Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Scheduling Features or How Can I Make the System Do What I Want? [I] - Marek Grabowski

Description

Kubernetes Scheduling Features or How Can I Make the System Do What I Want? [I] - Marek Grabowski & Wojciech Tyczynski, Google

Each user has her own set of requirements and constraints on where their Pods should be placed in a cluster. Some want to increase utilization, thus they want to pack Pods as densely as possible. Other want to maximize reliability, so they spread the Pods as thinly as they can.

Another one may have Pods that interfere with each other, e.g. by writing a lot to a local disk and don't want to put those on a single Node.

During this talk I'll tell what features are present in the default Kubernetes scheduler that can help you to accomplish all those things and more.

Marek Grabowski
Marek is a Software Engineer working on Google’s Technical Infrastructure since 2013. He works on Kubernetes since February 2015, focusing mainly on scalability and machine management. Before that he was working on Omega project, doubling as an SRE for some clusters. Before his life at Google he attended University of Warsaw majoring in both Mathematics and Computer Science, and for a few years he was teaching at the Faculty of Mathematics, Informatics and Mechanics in Warsaw.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Welcome everyone: my name is Amara Grabovski, and this is what I teach in ski we're from Google Warsaw office and wearing the hoodie that Aparna actually advertised in in her keynote, and we are the part of kubernetes team for over two years now and part of our job was in Hopewell, was related to the scheduling and the scheduler itself. So we we're here today to tell you about all the kubernetes features that sorry scheduler features that allow you to take greater control over how pods help what's run in your cluster.

A

So what can you do to make the system put the pods on the nodes you want it to?

A

The main focus will be on features that went better in 1.6 released yesterday, which is mostly the ones that Aparna mentioned during the keynote, and we will do it by showing you some use cases and there and and explain how those features can help. In the situation. Scenarios like like that I want to know that scheduling in kubernetes a is a huge subject like pretty much every release from the beginning. There was some new features added to the scheduler. Some new functions some new, something which makes it impossible to just tell everything in this area.

A

In this short talk and in the time that we have, we also want tell anything about implementation, details so like, if you're interested in like how we deal with conflicts or pretty much any like caching and other internal staff in scheduler. Then this is not the talk for you, but you can catch us later. After the talk we can answer. Probably most of your questions.

A

Let's start with telling a bit how the scheduler works and actually what it's supposed to do, as probably most of you know, the goal, like the main goal of the scheduler is to assign a node to a port so like when the user creates a port and by user I mean, of course, like user or a reputed cassette or something that creates a part. That's not already bound to an out the the scheduler is.

A

That is the component that takes this part and figures out, which node is the best for this port to run and how it does it. The codec now I will quickly show you what is happening inside the scheduler when it's happening cause it's important to understand the back. What kind? What can you do to actually influence the output?

A

There are two: there are two phases of the scheduling. So when we try to schedule a pod, the scheduler takes all the nodes in the cluster and passes into passing it through two sets of functions. First of the functions is called predicates and second phase of the second phase is called priorities.

A

Predicates are responsible for figuring out which nodes are feasible at also like, on which nodes the pod can run at also like which ones we want to just discard and do not care about in the future processing.

A

While the priority face is just scoring the nodes with some like just assign some scores to nodes, after which the scheduler can just pick the note with the highest score or like on randomly one of the the nodes of the highest course and add the the port and then put the porter.

A

So when, like both of these displaces, this priority predicates base and priority phase, consists of a number of function. So there's a number of predicate functions, each of them computed independently and if any of those predicate functions fail forgiven now this note will not be processed any anymore.

A

On the other hand, like there's also the list of priority functions, which is also computed for every node from the output of integrate functions, predicates that we just assign this course and after all, of those functions, put their score for this node this course are combined in a way that voltic will be telling later to figure out the final score of a node which is kind of in her, like infer from all the scores from the single functions and priority functions from like how you can influence the output first thing. First way you can do.

A

It is to actually specify what kind of predicate and priority functions your scheduler should use. You can do it when, during the scheduled start up, you can just specify which functions which player predicate functions. The the scheduler will compute in the predicate face and which functions. It will use the Orient and other and the function that it will use during the priorities files, but this is what you do during the cluster setup and schedules can start up other way you can influence.

A

It is to use the features that were moved to do better in 1.6, which will which we will just go through right now. I will speak about predicates and vertical speed, speak about priorities.

A

Let's start with with the predicates functions and I want to begin with the very, very simple and basic kind of predicates that you actually do not need to specify anything to use them, which kind of protect your cluster from doing like bad things, for example, over commit notes when the predicate is called pod feeds resources and it does exactly as it what it promises it checks if resource requirements of the part can be met on a given note and by this I mean that, like as most of you, hopefully now resource requirements for the part, I split into two to five to two classes right.

A

First, it's like pots, sorry resource requests and the second one, since resource limits, scheduler do not care about resource limits at all like this is something for cubelet to use. Schedule occurs only about resource requests and it pays what what it does is basically subtracts all the requested resources of the run, the pot running on the nodes from the node capacity and checks. If the what's left is enough to fill full request of the pot, that's being scheduled by resources.

A

I currently mean CPU memory and GPUs that are like predefined kubernetes resources, but there are more to come in the future, probably so that was the very simple and very basic predicate that will always be computed unless you disable in for some reason, but you shouldn't now I'm going to tell you about those features that we added in in 1.6.

A

The first use case I want to talk about, is Kevin prevention of CO scheduling. So imagine you have two kinds of pots like the dog pots and card pots right like you'd, probably don't want to schedule card pot and a dog pot on a single note like because we all know what happens when you put a dark cat and a dog in a small container right like this, usually ends badly.

A

Even without this feature, you can kind of deal with it. For example, you can specify resource requests for cat and dog pots so that they won't fit on a single note. But then you will. The utilization of your cluster will job because, like you just need just like to just prevent them to be scheduled. On the same note,.

A

So, instead, you can use pot ante affinity that a partner spoke in your keynote, but I will explain it in a bit more more detail when you define patanti affinity, you specify two things topologically and the label selector at the pelagic is something that relates to note and label state or something to today's two pots and it works in a following way.

A

Imagine that you have a cluster with nine nine notes and on those note, those notes have labels like corresponds to colors. For example, the left-hand side notes have label color and value red middle ones have like value. It was yellow and right, one self-evaluations green, the the way the topologically works. Is you define like it and if you use the topological or it will cause the grouping of the nodes into this color group, so I could have a red group, the yellow group and the green group of nodes.

A

This is what topologically does to check if the pot kind of needs, the proton affinity. Predicate you do the following. For for that group, you take all the pots running on all the notes on this group. So, for example, you take all the pots running on the red nodes and check if any of those pots like match the label selector, which is the second part of the pot on TNT definition right. There is no card pod running on any of those nodes, which means like their notes.

A

All of those nodes are fine right, like you can schedule the dog pot on on any of those. Those nodes, on the other hand like there, is just one, but still there is a pot the carpet running on one of the yellow nodes, which means mmm you can't schedule the dog pot of any on the yellow, node, like the same goes for for for for the green ones right. So this is the the semantics of the pot, anti affinity and also affinity which also seconds.

A

But like, as you probably imagine, is pretty computed computer usually expensive to compute all of those, so we actually in 1.6, we limited the topological to be used to hostname, which basically makes a group of a single node, the groups of single nodes zone and the region, which also should be self-explanatory.

A

So you can prevent scheduling of the segments of the pot on the in the same region as some other pots are running, for example, and if you just put the this kind of put on the affinity in your dog pot, then we won't be able to schedule of any on the on the nodes when card pot is running.

A

There's also the opposite feature that it's kind of probably easy to figure out by now, which is pot affinity which forces call scheduling of pots. So, like imagine, you have a sapling pot and a watering can pot and you probably want them to run together. Like the examples, maybe there's like a just, you need a lot of network throughput between those two, the pot you probably want to put them as closest today, I must to itself as as possible. Hopefully, in the same note, so you could all, of course you can also hack around it.

A

Without this feature, like whatever put in a single pot, which you shouldn't do cuz I did break the whole of microservice idea, but the was able to do it before that right now, it's pretty simple. What you need to do is just define the pot affinity, which is exactly the same as put entire Finiti, except that this there's no untoward in the definition with the semantics. That is pretty easy to figure out.

A

Basically just reverses the logic like we discard the notes from the groups that do not have any of the post matching the selector running, and we keep the notes from the groups that do have pots or the given sector running all right. So this put root should be pretty pretty straightforward.

A

A

Another example, like we use case I, wanted to show you is right. The situation in which you want to create a dedicated set of notes for some users to use and prevent all other users from using it like what you was able to do before that before one point six, you could just force the users, you don't want to use those knows to use some labels. Note selector that won't match, did some set of office, some set of notes and will be running in everywhere else.

A

But in one point six we are the detained switch upon, also told about. We tell you to do it in a way better, but way better way. The example I have here is that you have a user, the black user, which ones serving pots that that is a blue pot and you have a red user, which runs like a huge bad job. I did you just wants to start like infinite number of some small pots just to compute something?

A

What can happen, of course, is that, like the red user will just schedule all his pots everywhere and just saturate the cluster, so there will be no space for the black user to schedule his serving pot, which is usually bad, because we care about serving pots right.

A

When you use paint, which is basically this slide, you specify, like you, can add attained to a note and the Tate says: scheduler do not schedule anything on this note unless it has a matching toleration. So, like this note, will be unavailable for the users who can't add distillation to to their pots.

A

What does it mean? The matching toleration for the paints have like ten techno schedules should be self-explanatory about his key key and value which either identifies the K. The taint and toleration has like four things: the key value and operator which defines which stains they match and the effect which should match these things as well. So the operator is either equal or exist.

A

If the operators equal, which is default, then both key and value of the thing needs to match what is written in toleration and when it exists only the key matter so like it means that we only care about existing of at another, given key so like. We do not check that the value of the value apart.

A

Also, if you, if I recall correctly, if you do not put taint there, it is treated as a wild-card, so with just much all the time. So it would be a toleration that matches everything.

A

So once you do that, like you kind of take out the blue node from from the cluster from the retinal red users perspective, so we won't be able to schedule those two pods that are an example, but the thing so that the black user will have always have space to schedule his serving job and will be happy. The red user won't won't be happy, on the other hand, but like if there's not really enough resources in a cluster to satisfy everyone. Someone will end up being unhappy right, like you can't do anything about it.

A

Okay, so that's the futures, the predicates I wanted to talk you about to quickly wrap up. It was like how the how predicates help you to prevent overcome it of the nodes, which is like the basic ones that is always applied, how you can use plaintiff. We need to prevent call scheduling how you can use pot affinity to force cross scheduling of thoughts and how you can use taint scintillations to create a dedicated nodes for some set of your users and right now, I want to leave you with Wojtek who'll. Tell you more about priorities.

A

B

Okay, so with the predator we've predicates already applied, we already filter out all the notes that we actually can't schedule our part, but still the group of remaining notes can be pretty large, so how we can actually choose this one note on which our pods should be scheduled, um and this is basically where priorities come into play. So I will start with like describing the most important ones.

B

I thought we don't have time to describe all of them, but I think that it should be fine, so the first one is actually connected to the amount of resources available on the notes. So after applying predicates, we know that there is enough space on on the note but which one is actually the best one. So the policy that we are that we chose here is basically that we would like to spread the amount of resources requested by pots as evenly as possible across notes. So technically, actually we are.

B

The priority function works on ratio of requested resources by my resources, requested by paul potts running on a note to denote capacity to make it work if there are notes with different sizes in the cluster, but for simplicity. Let's assume that all notes are equal and then like. This function basically means that the more available resources are on a note that better, the note is but actually like. Is it really the best Felisa? So so it seems roughly reasonable if the cluster is like has predefined size and like and stuff like that.

B

But if we are running person, for example in the cloud- and you have cloud autoscaler, then having an empty note means that we can basically remove it and and like reduce our the cost of the underlying infrastructure. So so that sounds like something that we would like to have and because of that, like we created this opposite priority function, which which basically prefers nodes that are rather like that Kenz that can handle our pot like have enough spare resources.

B

But like are more fooled and others, and with that we buy with that function, we are basically trying to make the notes that have some pots already running on them, full before scheduling anything to some empty pots. Empty notes, sorry and obviously like those two priority functions, conflict with each other, so we can't use both of them at the same time.

B

By default, we are using the previous one, which is like worst fit but like an important feature of scheduler, which mark briefly mentioned before, is that scheduler is configurable, so you can configure it so that, like it, will use this one function when starting a scheduler. You can configure it like, so that it will use this one instead of the previous one.

B

Okay, so another priority function is is about spreading. So imagine that you have a deployment or replication controller running and instances of a pod. So obviously you are running those and instances for some reason and you most probably don't want a bunch of them to be running. On the same note, because, if the note fails, then you will lose all of those replicas at the same time, but like fault, tolerance is just one reason of not wanting to have them.

B

On the same note, so the other reason can be that user spikes of all those pots can can be correlated and like we do, they would be spiking at the same time, so selector spreading priority function is supposed to solve this problem, so basically the more pods coming from it from the same deployment, service, replication, controller or something are running on. On a given note, the lowers the lower score this. This this note will get okay, so those two 5.

B

So now we are going to a slightly different group of priorities like the ones that I briefly described like before are just working out of the box for every part, but starting from now we have, we will have some functions, that a pod can define some preference where it would like to run, and this did, its preference will affect scores of the nodes. So the first example of such function is know.

B

The affinity where pod can specify a selector for nodes in on which it would prefer to run if it doesn't specify any selector, then all the nodes will be scored exactly with exactly the same score according to this priority function. So in the example from the slide, the pod specifies that it would like to run on green on it on a node with a green color and as a result like both green nodes, are getting the highest possible score and and all other nodes are getting like the lowest for this particular priority function.

B

So the next one is pod affinity, which is basically a sibling prayer. Priority function to the predicate that predicate that Marc described earlier. So what we would like to achieve is that we would like to recall.

B

To ensure to ensure that our pod is co-located, with as many pots selected specified by the preference selector as possible and like. Similarly as with predicate, the collocation doesn't necessarily mean collocation within the same note it this collocation within some group of nodes, which can be node itself, but it can be like rock or zone or region, or something like that and the example.

B

From this slide, we, our pod, would like to be co-located with with green, with green, with as many green pots as possible within the same within the notes of the same color and as a result like the yellow nodes, are getting the highest score, because there are like two green pods all together there and the red node, for example, is getting the lowest possible score because there are no green pods there.

B

Yes, so also. Similarly, as we switch predicates, we have untie affinity here, which were where we would like to our pot to be collocated with as few pots specified by it's like ante preference selector as possible within the same group of nodes. So maybe to give you an excel more Morial or real-life. Ish example of this, so imagine that you have you are you would like to schedule an end instances of a pod running a database and on one case you'd like have them to be spread across no drugs or there for fault tolerance.

B

But on the other hand, you would like to have a database as close to any to it to any application back-end. You are running as possible for to reduce latency for communication with it and to achieve it, you can, for example, specify the ante preference selector.

B

What I call it too much exactly those posts running this database and set for the instagrams, for example the grouping Iraq, and then what you will get is that those pots do those database pots will be spread across racks and why we can't use predicates, for that is basically because if the number of instances of the pot are is pretty big, then we still want to allow scheduling all of them and potentially running more of them within the same rack.

B

Whereas with predicates, you are pretty much allowed to run one instances within the same group of nodes, so so that so that's the example where we can't actually use use. Predicates.

B

Also getting back to example, from the slide like. Similarly as we've predicates here like we, the red node is getting the highest possible score, because there are no blue pots and like, for example, the green nodes are getting the lowest possible score, because there are two to the pods and we are not. Similarly, as with predicates, we are not differentiating groups from the SE notes from the same group.

B

Okay. So now now we we have a few more priority functions, but unfortunately we don't have enough time to go over all of them, but for now like we already have some number of scores for each note, and we somehow would like to combine them to get the final score of the of the of the note and to to clarify like the technique.

B

The technical detail of it is that priority function is supposed to return an integer from zero to ten, which I will be kind of using later, and the question is like how to combine those those those numbers for a single note to get one number. The final score of a note, and what we are doing here is basically we are just taking some linear combination of those scores with appropriate waves so where those waves are actually coming from.

B

So when, when you are defining when you are running a scheduler and we are defining a priority function, you are also supposed to assign a wave to this priority function and it can be like arbitrary integer and the the reasoning behind this integer is to reflect how important this priority function actually is, and those waves are exactly does that we are using here when by computing. This linear combination of of those scores from individual priority functions.

B

So once we have those the the final score for each node, we are simply just choosing the node with the highest score and if there, if there are Morgan, then one with the same highest score, then we are semi randomly choosing one of them.

B

Okay, so, as you may probably expect, there are some dangers in using priorities and this scheme, so, first of all, like you can't really rely on those priorities. So if even if you specify, for example, note affinity- and there is a space, there are notes where you are allowed to run this- there are enough space on dimmers and stuff like that.

B

There still can can potentially be scheduled somewhere else because, like, for example, scores for both affinity or or spreading, or something are significantly lower for those notes than for for other notes that are not matching the the note of in note affinity, selector and ii, like think that is also important here, is that if you are, if you, for example, slightly change the waves of your pretty you have your priority functions or just change the other new priority functions, you will basically, we might get a completely different, different binding, and yes, so maybe some of you know we.

B

This is like a scheme that different that is different than what we are using internally in park in google, and what we are doing in embark is that basically, we are ordering our priorities in some linear order and we are using the next prior desk next function, just to break the ties from coming from the previous one and so to clarify. What we are doing is that we are computing. The scores from the first priority function, leaving notes on you have the highest score from that function and like for those computing.

B

The second discourse for second priority function again like living on the notes with highest score and so on, and this is much more predictable in terms of like what which note will be finally chosen, so why we actually didn't choose the same skin here in kubernetes, and the answer is that basically, this kind of decision tree like ish behavior we can also achieve from from this from the skin we have in kubernetes by just like specifying appropriate waves of priority functions.

B

So if you specify the wave of one to the least important priority function, 10 for the next one and hundred the third one and so on, you will get exactly this behavior, so it's possible to simulate.

B

So to summarize about priorities like this is very powerful mekinese, but to avoid surprises, you need to really understand it and and use them very carefully.

B

Yes, so the feature that the the state of kubernetes as of 1.6 like as Aparna and Mark, already mentioned, like we have just moved to beta in 1.6 a bunch of features, including node affinity, both affinity and ante affinity, tines and tolerance. One more feature that we also moved to just move to beta. That was worth this worth mentioning ass, multi scheduler support, which, basically, if you have multiple different schedulers in your kubernetes cluster, allows your pot to specify which, which exact scheduler is responsible for schedule in this pot. A note about our future plans.

B

I think that, like the the most important thing from the scheduling perspective and the thing that a bunch of people are asking about and waiting for, is the mechanism for priority and preemption that will basically enable you to specify which pots are more important and which parts are less important, and it's in even case if the clusters, for example, too small- and there is enough not enough space for all of them- enable those more important pots to preempt the less important ones and another feature that is slightly related to it.

B

It's like reschedule err that will be responsible for improving the cluster state that is degrading over time, so to clarify like, even though the scheduling decision might be optimal, and at the moment when it was done like because of like the churn in the cluster and pods new pots are coming and some pots are being deleted.

B

It can no longer be optimal after like five minutes, or something like that, so the scheduler will be able to, for example, will be responsible, for example, to to improve the state of the cluster over time and, for example, improve spreading of pots that will be possible later and the last one and those both I. We hope that we will be able to at least have some initial versions in 2017.

B

There is one more thing which is resource estimation, which which, which will basically means that it will be estimating how many resources your pots, will be using and I'm, not sure if it's like, we will be able to deliver it when one in 2017, but that's Atlee, that that's definitely something that we would like to do at some point.

B

Okay, so that's mostly what we had for today, so we encourage anyone who is interested in in in scheduling to join the community scheduling community. There is like speche the scheduling special interest group that is like meeting meeting bi-weekly. There is like slack channel for strictly scheduling, related questions. There is the mailing group, so if you have any questions you can use those or you can also. If you have the closed questions now, you can ask and catch any of our any of us. We will try to like answer them. Okay, thank you.