Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Scheduling SIG - Wei Huang, IBM & RaviSantosh Gudimetla, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro: Scheduling SIG - Wei Huang, IBM & RaviSantosh Gudimetla, Red Hat

Kubernetes Scheduler is a critical component of Kubernetes that finds the best nodes for running pods based the configured scheduling requirements. In this talk, we will cover the scheduling workflow and the life of a Pod during a scheduling cycle. Then we will talk about recent SIG Scheduling projects and features, including the development of the scheduling framework, batch scheduling, descheduler, even pods spreading, and scalability improvements. We will also dedicate some time of the presentation to audience questions and users' feedback.

https://sched.co/Uajy

A

Schedule on time so welcome to this session, and we will give some introduction about how the schedule works in the juban, alias I'm wave one and from IBM as a freaking year. And this is hey.

B

Everyone, my name, is Ravi I work at software engineer at Red, Hat I, focus on scheduling aspects of Kona is an open shirt.

A

So so, first we had talked about how scheduler works in the whole cabin Elliott picture and there's a ticker very typical, yes of flow, so user create a deployment and the Department I request the girls to a guess. Over API server does some basic authentication of a registration check and if that's good, it will position the pay object. The deployment, if you have two to the backend storage 83, then the Java API server is down the after that controller manager, as well as many other components watch the API objects. They are interesting.

A

For example, catering manager are interesting that diploma and Rebecca said so. It gets an event there than your diploma is created, but it doesn't have any associate the Rivka set, so it creates one, and once the Rivka cells gets created beside controller notice, the desired State, for example, for this department is record. Have that number equals three but there's no pal? Yet so rep except controller pretty pass for them.

A

Then the surpass doesn't have one field set, which is the no name, because this not job of container manager, then it just creates three paths with the no suspect s. Blank has their that by this death, a tremendous jobs down. Then there were the three paths in a system we now to the name, no-name cell.

A

Then it's the scheduler going onto the stage it watched some the path as well as many other related resources, because it's neat only needs to know the overall view of the whole clusters so that it can make the good decision to put the power on to which now so, if watch the part which doesn't have the no name set, then does some internal magic about I will goes detail into that piece.

A

So it has some internal magic across the castle check the cluster, which now is the best fit for the incoming note sorry for the incoming pod, then it will create a API request to set the known name. To that perfect note, and this face, we call the point then Podesta. The double of schedule is done then, after that couplet we also watch on the past, which has the known name set to exactly is own know, so it only cares about which knows, which part will then turn itself right.

A

Then it will invoke the underlying contain a runtime, either Creole, LED or darker to spin up the real containers dang it. We also do some internal. We can say internal group to watch the part to report the past that has regulated API server. So this is the basically the whole picture of what happens underneath one user create a deployment then because we are scheduled introductions, so we focus on this piece. Let's take a look what's inside this black box, so internally basic, there are four big components inside schedule: the first wise Informer.

A

Well, if you are reading CDE or co2 controllers, you may be familiar with this. So this is the component that watch o all by page objects. You are interested and the ones API object. Your users interested get updated a little aquarium. You can vacation. Thank you do your thing so here in schedule with these two things. Why is to update its internal cache? So that means we don't query the APS over every time we need to decision.

A

Instead, we rely on this internal cache because some because you cannot query the 80s over so many times, because scheduling needs high volumes in your. Do you need to do a lot of scheduling by it's running. The second thing it does is build some specific data structure.

A

If you are writing C audio controller, you may be just use the tools like what cue that the kind girl provides right, but schedule works a little different because you need to make the scheduling decision very efficiently, so it has to rely on some specific data structure so make efficient decision. So this is the first part informers. Second policy. That is a cue. So basically you can think of Li. It's just one cue by internally. It has some sub kills, like I, could kill back off cue and schedule kill.

A

So, though, and we had a feature called preemption or priority JD 114 and before that, we're just making a very simple queue, which is the FIFO queue for seniors first, but after we introduced the priority we had to manage, which part has the higher priority right. So internal, you have the priority queue as well as we need to ensure the fairness is not the case that the higher priority public always can be reach high.

A

We need to sort of back off them to make them sleep a little bit so that the lower part has the chances to be scheduled as well so internally, the kills job is maintaining the own schedule and scheduling path properly sorted in the in the in the internal data structure, then it also works as the pub dispatcher every times. Pops up odd fries in internal priority queue and the consumer is the main scheduling girl routine.

A

So once this gets apart, the main scheduler thing will do its internal algorithm and as well as logic, to decide whether there is some room for the path to be laying down or there's no room then so to decision. There is the a and B here either put head back to the queue or post a pending request to the API server, which is a set, the no name to the exactly no name, and the first component here is called bending here.

A

It's a very simple just bill routine, but we do it in another goal, which means ISM another thread we did. This is because we don't want the bonding API requests blocking the main process, because sometimes the bending takes time. We don't want to that. Http communication pixel brought the main authority and by this step the bending phase, we have some internally called optimistic concurrency.

A

That means, by this step in ApS over the bond, maybe hasn't finished yet, but internally we assume the path has been assigned to the node, so we update the internal cash and but don't panic. If it's felt the band pair failed, we can get notification. We then invalidate the cache. So this is the basic internal flow of the schedule. So next we are focused on the internal algorithm and the logic of how the main schedule team gets apart and how it decides which the best of nerve is.

A

So basically, there are two phase in in that box. One is called fear, predicates also known as filtering, so they say you had to step. It goes through some Devon predicates, the predicates you're sure it comes from the pot, a pot API spec like how many resources you are and what's the pot affinity is etc, etc. So, internally we map those 8's back to a different practice so eat. So we go through every note and get a filter result that passes all the predicates.

A

But if Nano is qualified, then we will go to a second try which is called preemption two for the higher priority part. It has a chance to prevent, lower, pre-empting priority pass to make room for it.

A

So the little difference here is that if it enters the preemption phase, the result is that it only sets a speck of nominating note to the target node we want to eat for landau, and the preempt path then goes to next scheduled cycle because we don't want to eat to be stay full without we wanted to stay less so that it can better be managed in our internal logic. So this is the phase one and phase two is called priority. That means we do have some nails which can fit for the nail.

A

So the first tool called priority, also known as scoring, is that for each filter. No, we have internally have some priorities and we give score plus its weight to each node on each priority. Then we sum up the score and finally, the node with the highest score will be chosen. Then we go to the next phase, which the bending phase can give some example. For example, there is a priority called lists require, says requested, which means we once the part comes in, and there are three nails available right now.

A

We want to choose the know that we les you're cute ization, so we want to balance the workers, but of course you can choose the most requested to make the resource more bin packed onto the nails. You are right. Those different scenarios can use different strategy.

A

Okay, this is basic. The scheduling flow in the very high level, so funding I, don't need to mention. Penny penny just send a request to ApS over okay and the next wheel. If you do some recent developments in the fall schedule, so why is that the priority and preemption has been g18 140 so that it's a default behavior that high priority car high priority part can priam they're low priority low priority pass. A second wine is called.

A

We saw Skoda using scope selector, so we sort coated a price to the namespace and you can specify that the resource coder is apply to some specific priority class. This is what scopes like to come soon and along with the GA. We also remove the restriction that the system critical project cars can only be applied to the cube system. Right now, you can apply to any namespace. You want the other going features I just list some major features here. Why is the scheduling framework? So, as I know, extensibility is a very important party.

A

The whole kubernetes community and schedule is also no exceptional and in the before we do have some experiments mechanism, but this time we want to move a step further. So that's why we bring in the schedule for work. So basically we does some code refactoring to basically expose every face inside schedule outside, so that you can build your own plugins and the combined, which the deferred schedule to make it your own customized schedule. Job needs so I, so Abdullah will give more detail introduction to tomorrow's dip-dye.

A

So right now the this feature has been offered in 115 and 116. We completed it's like we are eating our own doctor. We completed the code migration from the all-star tour, this framework plugging step. The second one is called even past breath in yesterday's keynote. The wiki also mentioned that this is one of the features the community Oscar for a long time.

A

They want to sort of control how the path is spread across your cluster on different topology domains and after that also will give some detail examples and third wise called its extended priority on the on the requested to capita capacity ratio. The motivation is that for extended resources, we don't provided the option, see whether you want to be impact behavior or the even part, even spread behavior, for example, for some GPU. If there are two GPUs, whether you want the behaviors to use one as much as possible or just use them as average as possible.

A

So this feature gives you the option. You can provide your inputs, okay, so this is specially the recent development of a handle to Ravi.

B

Thanks we so I think this slide should have been the starting slide, but I want to set the tone to next part of the presentation where I will give some updates on on the the projects that fall under six scheduling purview so like to talk about the design rationale of scheduler like when we have started designing. What are the goals that we had in our mind?

B

The first one foremost thing is scheduler is not responsible for managing the entire lifecycle of a pod. So at a high level, when the pod has been bound to I know, the binding stage is done. Schedulers job is is done. It does not care what happens when the pod is in running state. So this is something that we have made as a choice consciously, and there are some side effects of that. We'll talk about that in a bit and how the projects that we that we are going to see are going to handle that.

B

So the second important thing that we have thought about is the minimum scheduling unit is always going to be a part. It's not going to be a group of pods or job or some other entities, so it's always the part that we are interested in in scheduling and we scheduled one part at a time. We also make sure that it's best effort than we are choosing a node for a particular part. We could have chosen a first fit approach like greedily.

B

In the predicate stage itself, we could have exited in the scheduler saying that we found a node that satisfies the resource requests for the part, but we did not do it. We do not. We wanted the scheduler to provide a best fit for the node. That's why the pod is actually going through the second stage, which is the prioritization of the nodes from the list of nodes that were filtered in the first stage.

B

So that's that's something that we have made as a as a design choice consciously, and we also wanted to make it configurable like the end user should be or the cluster admin should be able to pick and choose whatever the priorities or the predicates that he or she wants for the cluster. This is something that we that Kuban it is as a whole, as we explained, is interested in in going towards, and scheduler is no exception to that. The other thing that is of different for scheduler is making scheduler pluggable.

B

So previously we used to have, or even now we have scheduler extender, which runs at various stages like you can choose scheduled, scheduler extender to be run at the predicates level or after the priorities level or after binding. So we have gone ahead and taken it to the next level where we have broken it down. Broken has broken the scheduling cycle down to multiple stages like pre-filter, post, filter, filter and then prioritizing stage where we are ranking and and all that so for every phase.

B

We are going to have extensions going forward, and this is what Wei was explaining when he was mentioning that we are eating our own dog food, converting the existing predicates and priorities to this new scheduling framework Abdullah is going to talk about that in in the deep dive tomorrow, and the other important feature is multiple schedulers like like, as of now in Cuban. It is you can specify within the pod spec which scheduler you would like to.

B

You would like your pot to be scheduled by so that's something that we have thought through, and we thought that it would be good if we can make sure this extensibility is available to the user. So as at all, if you go back to the previous slide, the first three points and the design rationale like scheduler is responsible for managing lifecycle. The minimum scheduling unit is pod and scheduled. One part at a time. There are some side effects of that. The first thing is since scheduling is at the Java schedule.

B

It has done once a pod has been bound to a node at runtime. Like say when the pod is running, there is a very good chance that a new label has been applied to the part or attaint has been applied to a node or a Toleration has been removed or applied to the pod.

B

So in those cases the scheduler won't reschedule the parts, then they are in running state. So we are need of an entity which can do that. This is where d scheduler comes into picture.

B

The other use case, 4d scheduler, is ensuring that a cluster is evenly balanced, so you can configure D scheduler with some limits or the threshold saying that this is the lower threshold, and this is the target threshold like, for example, you can tell that I would like 30% and 70% as the range of the utilization for every node in the cluster and when the scheduler is run with that particular strategy, enable it ensures that all the nodes in the cluster are falling within that range. So this helps in solving the fragmentation issues in the cluster.

B

So this is another use case for D scheduler.

B

How does this work so D scheduler at a high level is, is an a victor, meaning it just evicts the parts that are not conforming to the scheduling decisions anymore and when the pod gets evicted. As you know, there are higher level controllers that are responsible for spinning up or recreating those spots and when out a new pod gets recreated.

B

It goes through the scheduling cycle again, but no, where D scheduler influences the scheduling cycle by saying, by adding a label to the part that is evicting or by adding the taint to a node or or something like that. So this scheduler is just an effector.

B

Some of the strategies that D scheduler has include remove duplicate parts. This is this kind of helpful.

B

If you have multiple replicas of a replica set or deployment running on the same node, it makes sure that there is only one copy of that collection running on the node and evicts the rest of them, and the lone odd utilization is the same example that I was talking about earlier, where you can configure the lower thresholds and the target thresholds and ensure that all the nodes are falling within the range within a cluster remove pods, while eating inter pod, anti affinity and node affinity. Both of them are similar to what I was explaining earlier.

B

Where we have, you have applied the pods, sorry, the labels to the pods or remote panes from the nodes or added panes from the node. So these are the strategies the D scheduler supports. We are also working on other strategies.

B

Like when we are evicting, we will ensure that certain types of pots are not evicted. For example, critical ports, static ports, the diamond set parts and parts with local storage.

B

We also ensure that the ture steers are respected when we are evicting the pots like, for example, when we are considering a node, we would ensure that first, the best effort pots are evicted, then burstable and then guaranteed, and if you notice that a parts pdb power disruption budget would get violated when we are awaiting, we would not go and if those kind of parts, so that's that's what the scheduler does now I will talk a bit about Q batch. So if you remember, I was mentioning about three different aspects of scheduler 1sv schedule.

B

One part at a time the basic scheduling entity is spot, so we schedule one part at a time, but there are scenarios where you would like to schedule five parts or ten parts at a time, so the default scheduler cannot handle those scenarios. Currently, that's where Q batch comes into picture.

B

It ensures that say if you, if you tell that I, would like all these five parts to be scheduled. You can create a custom resource definition which tells that this is a pod group and we need ten pots to be running and all of them would be scheduled at the same time. This is kind of helpful for machine learning jobs and there are like projects like tensorflow, apache spark and all those projects that are using cube batch. This also falls under six scheduling purview.

B

Some of the features of cube batch are like Co scheduling, fair scheduling, predicates tasks, priam preemption and then task priority. So the thing that is kind of important for most of the machine learning jobs is Co scheduling. It previously used to be called gang scheduling. We have changed, it changed its name to Co scheduling, and this is where Q batch helps in the next project. That also falls under six scheduling. Purview is Poseidon, Poseidon is a network flow based scheduler. The main advantage of using Poseidon when compared to default.

B

Scheduler is again, it comes back to back scheduling. It inherently is a batch scheduler, and this takes the basket leading to a step further like instead of taking ten parts or five parts at a time it continuously reevaluate, s-- and shuffles, the pods or all the parts within the cluster to ensure that all of them are optimally placed. So one of the things that this helps in is again coming back or going back to what I explained earlier batch processing.

B

We are also in need of an entity that watches the cluster and then say if a label has been applied or if a label has been removed or retained has been applied. All those kind of things are not taken into account by de schedule. Sorry scheduler, and we need external entity like D scheduler, to help us with those scenarios now with Poseidon, since it does a continuous reevaluation and rescheduling. We would not need that entity like D scheduler if we are using Poseidon.

B

Some of the features that Poseidon has are very much similar to default. Scheduler, like node affinity, anti affinity, teens and toleration and Co scheduling, but it's not feature-rich enough at this point like it's not on par with default, scheduler to replace it. So this is all of the projects that are falling in scheduling per view. There are like other projects like cluster capacity, analysis and other projects.

B

They are also under six scheduling per view. So one of the things that people frequently ask us about is how do those schedule? Let's work, and the answer is we do not know yet. This is something that we are working on like. How can we ensure that we can bring in batch processing capabilities from cube, batch or Poseidon, using the new scheduling framework that we have written like?

B

Can we use those extinction points to call cube batch or get the code from cube batch to ensure that both batch processing and default scheduler, the schedulers that are capable of providing bad scheduling and the default scheduler can coexist, there's something that we are working on. So if you have any questions or suggestions, please feel free to contact us.

B

That is the end of our presentation. If you have any questions, please feel free to ask sure.

B

They would elect answer so.

A

It's not only a scheduling order, also the wrong time order, even if scheduling is schedule schedule them on today, so I think.

A

It's sort of more my personal making it's more good fit for your specified order into a path rather than different paths, because in scheduling is perspective, it's just schedule, Papapa Papapa and once the three paths you mentioned cascade road Dan schedules, job is down by in run. Time is also the case. Is a cat and the red and random ha to be executed right so yeah.

B

Just to add to what we way was feeling like it's it's mostly at the cubelet level, eggs. If you look at the cubelet preemption logic that we have currently, it takes into account the priority of the pots, but when it is once the pods are in running state or one support lands on to the north during admission, we do not have this concept of taking priority into consideration and to be said.

B

That does not fall under schedulers realm as such right, because we have told that this part needs to be scheduled first when compared to other part, and then it's the cubelets job to identify like if there are five parts that needs to be admitted, it needs to find out which part is of higher priority, usually even if there is another part that has like come up first and there is no resource for higher priority part cubelet, thus preemption or it it's called qubit preemption I, believe.

B

But there is a feature which ensures that eventually we may reach a stage where this pod would would come into running when there are no resources. Does that answer your question? Okay,.

C

B

Yeah, so when we started out, we always wanted to schedule one part at a time, but it may not be the case like, for example, you may need to have- or you may have a scenario where you you have a very big data set and you would like to process it. Parallely like, for example, if you have a training data set, you would like to process it parallely, and you want to ensure that all the parts have come to running straight.

B

While the processing is happening before we move on to the actual job execution, you would like to ensure that the processing has happened and all the parts that do the processing are coming to running State, like as a group instead of one part and then second part, and then the third part. So this is where batch processing would help in, like you can take ten parts, all of them would come to running state or none of them would come to running state.

B

That is one example, which is called gang scheduling, but batch is like taking a group of parts and then scheduling all of them together.

A

To add something is that the four scheduler is not a controller or resource manager, so it doesn't reset possible for spinner past or destroy a pass so comparing some other scheduler. So if we want to bring the best capacity into schedule, one one one option is to call implement equivalence cache so that we know once we did the scheduling calculation. For the first part, we can do reuse the result of father for other part, so that can make it efficient.

A

This is one option, but we try kervins cache, but it does work as performant as we supposed to be. A second option is that the schedule also work has the resource manager, but it's not a case for the for schedule, so we want to see if other ways we can do this so that the scheduler can schedule all them in one time, rather than scheduling one by one. Yeah.

C

For sure, for sure of you express its personal purpose,.

B

A

So some some problems in the badge area, a knob, proper model in the different schedule, so some some some problem- I, actually not only a scheduling problem, for example, fair share or some resource sharing. There might be more need to control that the admission time right, for example, my need some additional control prism if you want to make his church so that you need to make sure u n stand different, either namespace or different, what kind of pub and part the relationship right.

A

But that is why we need some impose some extra capabilities instead of only focus on the power pod cycle. So we are trying to see whether the scheduling for work can happen us on that, and this is otherwise. You have to discuss stuff from scratch to Bill to another schedule only for their specific concert of what you need.

A

So that's what patches a complex problem live Jay.

D

B

Yeah, the current throughput that is documented is 100 pots per second, that's the throughput or the SLA that we support.

B

We are trying to provide or work on the improvements like in 114 or 115 I believe, like we have improved the performance of POD anti affinity by 20x. As of now, we don't see that scheduler is the bottleneck in the entire lifecycle of pot, but if we think or if it reaches stage where scheduler is actually causing those kind of problems, we would consider working on that.

A

The basic answer is: we are continuously improving the performance so very for very basic workflow. We have a which we have already reached 100 per second of 5000, real GCE clusters and in terms of some complex or close, especially the pot affinity, part anti affinity. Now it's invertible to to do some heavy calculation, because you, for example, for para affinity, which is check where the past can be coexist right, so that it's not only check in the past on the same on this node, but also check across the cluster. So it's invertible slower.

A

So you must to be assured that, but we are continually improving the data structure using proper benchmark to to improve on that.

A

Using multiple schedule, you me isn't.

B

Using multiple schedulers or multiple parts in a single scheduling cycle.

A

No, that has no authority. That's.

B

That's not in, but we are trying to have.

B

Like we are trying to see if we can fit in other schedulers into default, scheduler like using the extinction mechanism that we are talking about, but we haven't decided like what the approach will be. Yet it's something that we are going to work on and 118.

B

Yeah I think we just have like last question. Perhaps No.

E

B

Not going to change at all the apex rendered would be my.

A

Guess I can give a more detailed explanation.

B

A

Discounts answer is we, we will keep them in paranoia.

D

B

So we want with.

A

Their support, D, sophisticated.

A

So basically does accept. Our extender works is that you have this data external process, which is the ATP server, so that we expose several stage for you for the first schedule to interact with your extender. The extent is more webhook, but that means the communication and the Michoud. The measuring well takes a lot of resources, especially if you have a fat, 5,000 nose that has a lot of you know. They'd have to transfer and maturity mushroom, so.

B

There is the limitation.

A

Of the scheduling but for the framework we're doing another approach. Well, we are still exposing this point this different face, but is that we want you to implement the plugins and we provide a very simple interface for your vendor, so your vendor, given at his slash packages, schedules baba and you can now compile your binary. So all the communication works in the process, so no need to communicate with external things.

A

Not that different thing that differently yeah schedule.

A

For the extender no extender, we are still.

B

Yeah HTTP extended. We are not going to touch upon that, but if people are interested I mean we have like some proposals around or people have created, github issues around like adding gr, PC support and all that well.

A

You can here it is interest rate by even using G, obviously the cost us they exist. So it's I was a stupid before some small Custer and the you will be a very quick implantation onto your requirement before very large cluster. You have complicated every comment then try with us.

E

A

A far yeah we our plans in next race. We will use some real examples instead of color world, to give standard the presentation to know how to use that, because it's not very easy, comparing two extender. We can just you start to expand the framework we we must use. Something good compelling example to describe is okay. Oh maybe we are wrong. Auditory.

B

And I will believe in the corner. We can still answer decision mysteries, alright,.

A

Thank you. Thank you.