Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Scheduling SIG - Babak Salamat, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Scheduling SIG - Babak Salamat, Google

Kubernetes Scheduler is a critical component of Kubernetes that finds the best nodes for running pods based the configured scheduling requirements. The scheduler provides features to run clusters with desired configuration. Users can choose to run their clusters with high resource efficiency, high reliability, or various other custom policies. This talk will provide information on recent SIG Scheduling projects and features, including the development of the scheduling framework, batch scheduling, pod scheduling policies for multi-tenant clusters, and scalability improvements. We will dedicate about half of the time of the presentation to audience questions and users' feedback.

https://sched.co/MPkg

A

Alright, let's get started hello and welcome to scheduling deep-dive. This is the last talk of the conference and I'm surprised that you're still here, but thank you for attending. My name is Bobby salamat I am one of the sig leads of sig scheduling. The other sig is Klaus. Who is here with us so.

A

Today, I'm going to talk about the scheduling and schedule and recent developments in the city in the scheduler, but before starting. How many of you by show of hands, think that the scheduler is boring.

A

Nobody, even you guys you guys, are at Google. You should believe that scheduler is boring. In that case, it means that we haven't done our job well, so so I Google. We believe that infrastructure should be boring. Basically, you should not think about it. You should not see it and if it's really the case that you think that scheduler is not boring, then means that we haven't done our job well, but all right. How many of you think the schedule is good kubernetes the scheduler is good. Does whatever you want to do. Alright, so you're.

B

A self-contradicting.

A

Bunch but ok, alright, so disclaimer we are going to go fast because sick, deep dives are meant to be more like Q&A, so I'm gonna go kind of fast over over what scheduler is and what it does, and recent development scheduler and then given ample time for Q&A, although or not that many people here in the room, so at least I will give you a little bit more time to enjoy viewing the city and seeing the things around here. So what is the scheduler kubernetes?

A

The scheduler is this component that usually runs on the master node it doesn't have to, but it what we normally. We run it on a master node and is responsible for placing pods on nodes. It does not concern itself with the lifecycle management of parts. What it means is that it doesn't create parts for you. It does not care about terminating parts. Basically, there's normally terminate parts only a particular scenario where for scheduling it doesn't find enough resources in a cluster or nodes in a cluster. In that case, it performs preemption.

A

That's the only case that it terminates what, but otherwise it does not concern itself with lifecycle of pods. Another thing is that the scheduler does not do any temporal decisions basically does not tell any nodes how many seconds or minutes it should run a pod. It just places pods on those nodes, so for for that reason the scheduler might be a little bit of a confusing name, but it's probably better than Placer. So we go, we do scheduler, it supports a whole slew of features.

A

Some of them are here, for example, it supports. Checking my slides are online. Is I need to take pictures. You can just download the PDF from schedule that makes your life a little easier, so it, for example, checks node resources. It does someone do stuff like spelling, pods among nodes or checks, affinity, anti affinity for nodes and also sorry for pods and affinity towards notes, like notes, selector and stuff, like that, supports the Saints and toleration these are. This is just a list.

A

You can take a look there or lots of documentation on kubernetes that IO about each one of these features. You might already be familiar with some of those I'm.

B

A

Go into any of those in depth, but if feel free, to ask questions about them towards the end. So what are recent developments in the scheduler one of the important things that we've done is improving performance of the scheduler, and this is a big effort to make the scheduler more scalable for larger clusters, so we have been able to achieve a hundred parts per second now in 4,000 node clusters, 5,000.

A

No, it was like a goal that we set for ourselves to reach to and get to a reasonable performance for some of these clusters with higher churn. So if your cluster is a smaller, it's very likely that kubernetes, the scheduler, can achieve a lot more than 100 parts per second.

A

So this is a good news for those who run large clusters and this improvement. Most of these improvements are already released. They are available in 114 and even older versions. Maybe they are not like all 3x. Maybe 2x is available in like 113 and 112, if you're using those and if you are using like 110 and older, we recommend to upgrade to a little bit newer version of kubernetes. If you need performance, another important performance, optimisation that we've made is to interpret affinity, an anti affinity with Ritchie.

A

We have received a lot of improvement here so, inter pod affinity Anand, it was super slow, like a thousand times slower than many other features of the scheduler. So if you used affinity and anti affinity, you couldn't probably in fact we had written this explicitly on kubernetes- that IO, that if you have any cluster larger than a couple hundred nodes or so do.

C

A

An affinity on or anti affinity because your pods are not going to get a scheduled, so a single part was taking like 600 milliseconds, close to a second to get scheduled. If you use this in slightly larger clusters, but now we have had like close, like 200 X performance improvement, so we are within like 4 or 5 X or maybe like 3x, slower than other no other features of kubernetes.

A

So this has been a huge improvement. Most of these are available I believe in 114, and there is a one, more improvement, which is another like 1/2 X ish in 1 115, which is the upcoming release. So if you use 114, you will enjoy most of these improvements, and another important thing is that we have graduated pod priority and preemption to staple in 114. So now you can use this feature with ease of mind. We are not going to advocate it soon or change it in an backward incompatible way. Now. What are the plan features?

A

So an important plan feature is even par spreading, but why do we need this feature? We have already had anti affinity right, so anti affinity was a way to say: I, don't want my part or multiple of my parts to be scheduled together on the same failure domain. A failure domain, for example, is a zone or in or a node, or could be a any group of part that you can arbitrarily create by placing the same label on those parts.

A

So you could say that I do not want my parts to be placed together on the same failure domain. But what was the problem with anti affinity? The problem was that hard part anti affinity, basically anti-fan. It has a hard and a soft, basically rule hard means that do not schedule if you cannot satisfy this and soft means try to try to basically honor this request. But if you cannot go ahead and schedule, the problem with soft is that if you use it with pod, autoscaler potolsky does not care about these preferences.

A

So when it wants to, you guys can correct me if I'm wrong, but then it very basically wants to scale down clusters, and it sees that. Oh, you only have preference. It goes ahead and scales down right. So what happens in those cases is that you have preference, so your paws will be scheduled on the same notes, for example, if you want it to spread, it'll just put schedule. Remove some of those parts from the nose, then remove the note from the cluster. This is done by the autoscaler and then schedule goes and picks.

A

That part puts it on one of the existing notes. So some people didn't want to use soft.

A

For that reason, so they wanted to go with hard pattern, tie affinity, but, as you can see here, if, for example, you have four pods, these blue circle, things are our pods and if you wanted to place them on two zones, let's say two of them could not get them scheduled, because you know you said an anti affinity in a zone towards other pods or other similar parts at least so two of them would get a schedule and the other two wouldn't so part of schedule, even part, spreading, basically allows you to say that I don't care about how many parts you want to put on the same zone as long as or an F fitness in the same failure domain I just want you to spread them evenly, among whatever failure domain that I have, for example, I have four four pods and I have two zones schedule automatically knows that it has to put like two pods per zone.

A

We also support as a part of this. We also support specifying imbalance between these failure domains. For example, you say I want to tolerate two three, four imbalance of pots. So, basically, what it means so, for example, two imbalance means that if a zone has two parts, the other one has four parts: okay, what I don't want zone to have two parts if the other zone has like five pots only.

C

A

Of imbalance is fine, so it supports all of that. This is this is planned for 115, but there is a chance that it may not make it to 115, because we are very close to the code freeze and some of the PRS are not married. Yet the.

D

A

And that we are, we are actually working on his pod schedule, scheduling framework, so the scheduling framework is an attempt to make this scheduling architecture a pluggable one. Currently, the scheduler is a monolithic piece of code. Any changes that we make basically goes to the guts of the schedule to the core of the scheduler, mostly and with the addition of new, more complex features we felt like the scheduler is becoming more and more complex and at times it's hard to maintain.

A

So we are introducing this new architecture where all these features will become plugins for this scheduler. It has two benefits for us. It makes maintaining the scheduler easier for you guys or some of you guys. It makes it easier to customize the scheduler, so you can build plug-ins for this scheduler and makes these plugins can be merged with upstream easily and then also supports a sophisticated conflict that you can basically pick and choose all these plugins that you want to enable or disable and configure them.

A

Another effort is gang scheduling. Gang scheduling is basically the idea that you have a series of pods or a group of pods. You want to say either you schedule all of these parts or do not escape do any of them. This is actually a feature that is oftentimes required with HPC workloads or batch processing. Many times you have like a master or masters that need a bunch of workers. If these masters are scheduled and no worker can be scheduled, those masters will just sit there, wasting your resources and are not making any progress.

A

Similarly, if, for example, workers are only scheduled you, they won't do any job, so basically, you can specify a group of parts that they should all be scheduled together and if some of them cannot be scheduled, then don't schedule any of these parts, so this is also being worked on it's available in head incubator project called Hugh patch. If you have been in in our contributor summit on on Monday, you might have already heard of it.

A

There are also some of these longer-term projects that we have in mind. We know that kubernetes, the scheduler is not great for running batch workloads. There are a lot of other batch schedulers like new yarn or slur more similar schedulers that have been around in the industry. They do a lot better job for, like lifecycle, management of patch workloads or like yank scheduling stuff like this. We we know that our kubernetes scheduler is not really great for those kind of workers. Kubernetes from the beginning has been mostly designed for running.

A

You know micro services, as opposed to batch jobs, but we're working on this. There are a number of these projects that we would like to add to make kubernetes scheduler better for batch workloads and with the help of gangue, with the help of the scheduling framework, we hope that we can deliver those soon.

A

Another incubator project that we have is the scheduler. So the idea here is that, basically, when this scheduler kubernetes, the scheduler only cares about the state of cluster at the time that it schedules a new part. Afterwards it doesn't care. So, for example, if you set anti affinity or affinity affinity is a better example. Actually, so let's say you put affinity on some of your parts to put two parts.

A

For example, together on it on the same note, but suddenly one of the parts, for example, terminates and disappears from that note, kubernetes, the scheduler does not care at all that this part, this affinity rule, is no longer satisfied. One of the part is gone. This one of them has remained so it doesn't care. These scheduler is an attempt to reapply some of these scheduling constraints at runtime and for the parts that are already running.

A

It supports a number of features already, for example, it's supposed to be balancing node resources distributes pods of collections when the number of nodes in the cluster change applies into a pod affinity. An anti affinity applies. No, definitely correct me if I'm missing anything wrong and it's as I said it's enough. It's available in an incubator project.

A

That's all okay. I did a relatively good job by finishing everything in almost 15 minutes. So if you have questions or comments, we are here to hopefully answer that some of those- and if you want to find us, you can go to our home page for kubernetes, scheduler or cg scheduling or our slack channel. You can subscribe to our mailing list or just send us messages or mention us on github any questions. Yes, let me give you the microphone.

A

I'm not sure if this is on but yeah.

E

Thanks it was a question about the even pod spreading, so how common well, the term was where you say you know the set number out with imbalance.

A

For example, yeah yeah, so let's.

E

Say you want that to be zero, so you want exactly the same number on each node zone. What's the behavior and if say no fails, and so that imbalance comes, does this scheduler then automatically reschedule pod to be created, or so.

A

So, basically again, our kubernetes scheduler does not care about the run time. It only cares at the time of the scheduling so right.

A

So one of your, for example, you have you, have two zones which are perfectly balanced, but suddenly one of the pods from one zone dies and goes away right, so schedule is not gonna do anything unless at the same part, for example, what not the same exact, but if the parties, for example, belong, belongs to a collection, let's say a replica said, then the replica said controller is likely to create a rep replacement for that part. So when that replacement comes up, then the scheduler puts it on that zone, which is missing a pod.

B

With with so many conditions that the scheduler needs to take into account when scheduling a pod, can you maybe say a few words? How do you test and assert that the pod is all things considered is being scheduled in the right place like what is the tech testing framework right.

A

So, generally, scheduler or many other components of kubernetes have three main types of tests. One is unit test which tests, like actual functions of you, know smaller units of code, which are the usual functions of the code. That's one one thing that we do, but it's really not enough for testing everything right so most of the tests of schedule, integration tests as we call them in integration tests, kubernetes generations, as we have a framework for integration where it brings up an API server and scheduler and all in a single process.

A

Basically it runs a net CD on the side. It brings up all of those, and then we have written a number of tests, for example, check a lot of these features. Each one of these features have usually at least one, sometimes many more tests to check various scenarios, and also we have end-to-end test which runs in a real cluster. These are usually the tests that we really want to make sure that end-to-end admission and scheduling and everything works and nodes actually run these workloads.

A

So we have these all these three tests to make sure that scheduler is reasonably reliable.

A

Have you noticed any particular.

A

They were not too bad actually, because we've built a library that you know and of they're all functions that we can pretty much lower. You can kind of like create some a lot of these scenarios relatively easily in our test environments. Yeah.

E

On the dish, Angela I know there's a future sort of thing, but the the previous talk was about the cluster autoscaler and the concept of scaling in so removing a node from the cluster see.

C

E

The scheduler needing to kind of bin pack their pods. How often does that will that they said you know to run? Is it something that gets triggered or is it kind of a try.

A

To go in tasks you so I think it's up to you, but Ravi do you? Do you know the answer to this question? How frequently the scheduler.

F

Cron job and then set the schedule to whatever you want it to be, so it's technically just a power and then create a job around it with appropriate service account that should work. Fine, thank.

A

G

Do you have any plans to introduce some kind of dependencies between pots don't schedule a before B yeah.

A

So actually, if I don't know, if you noticed or not as a part of supporting batch workloads, this is actually a very common thing for batch processing right. So a lot of batch processing pipelines are really like pipelines, so they have stages. One stage should finish before the next stage. Can you start? Yes, we are aware of that problem and we are working on supporting dependencies clause. Could probably you know I as far as I know, we have. We have this plan to add like dependency, management and dependency to basically batches right.

A

So we have that in mind. I, don't I, don't know if we have any particular a silly deadline to deliver this feature. We don't try it. You cannot really make any promises at this point, but it's sink.

G

Away you still focus on some. Some policy has meant to the batch workload and we do have some pipeline for the job with dependency and the task with it dependency for this. For this account, especially for the genomic workload, I'm, not sure what kind of kind of you know for genomic, they need to do some day to clarify DNA DNA test or something like that yeah, but I. We don't have generate with how timeline for this part, but they do. You know our pipeline.

G

Our first day is to try to some basic concept, such as draw queue. We don't have fun and finalize such kind of concept right now, right.

A

G

This in your pipeline, if there are some special Department we can, we can have some easy one have discussed in our meeting yeah.

A

One of them, one of the things, is that we really try to listen to feedback from our users and prioritize based on the number of feedback that we receive from our users and customers. If you have a real need, please say something: you know create an issue as as loud as possible. You know, usually we listen to the louder customers. More, so don't be shy. Yeah create an issue. Go on, it's like. Oh, my workers are not running. I have like a cluster of odd thousand, no isn't.

A

Yeah I would bring a million dollars to DK team and we immediately schedule your work.

A

Of course I was joking, but it could happen in practice too. Sometimes I.

H

Have a question about failure modes for even pot spreading, so let's assume that I have free failure, domains and I, don't know 100 pots, so they will be spread like 30 something on each and one of the failure domains fails. So a zone is gone. So now, probably my deployment or replica set or whatever control the pots will add to 30 pots and what would scheduler do? Would it schedule it 50/50 on the remaining two or will they become pending or is it in my control?

H

A

A good question so I mean I, don't know what you mean by in your control, but basically scheduler will try to spread all of those among the remaining failure domains, for example, remaining nodes or remaining zones whatever it is, and try tries to rebalance all those existing ones so that, if you have, for example, let's say you have thirteen, let's say you have sixty nodes and you have one pot on each and suddenly 30 of them disappear.

A

For example right so then you have like 30 pending pots after, like controllers, create another 30 pots for you. So the scheduler looks at looks at these nodes like 30 nodes are available after the planning, part and I need to spread evenly, we'll put one pot or no and basically for an extra pot or not any other questions.

D

You mentioned at the start that scheduling should be boring and that, because no one.

B

C

D

One raised their hand, so my question would be is the reason it's not boring for us, it is the default kind of spread the load, and we see a lot of workloads where we have small pots. We buy a small amount of resources and, what's requiring the full note and.

C

D

We very quickly see that we can can't schedule the large workloads with.

C

Scheduling yeah so.

D

We have to dig it into the predicate and understand what they have to put there. I'm, not sure if this is has a solution so that people don't have to deal with this. It.

A

Is actually a problem, you know. Basically, your problem oftentimes is referred to as fragmentation. You know in like academic terms, so basically it means that you have a bunch of nodes. Each one of them has a small amount of free resources, but each one of them has the three sources on each node is is smaller than your largest parts right. So this is something that in many cases for example. My background is in like cluster management, I was working on borg borg is like the cluster management at Google uses.

A

So in bored we have something similar, 2d scheduler, that we were talking about in work terminology, it's called three scheduler, but anyway, it's the same thing. So in those cases usually reschedule or D schedule kicks in, you know finds out that, for example, there are pending parts and all I see fragmentation and kicks in remove some of the parts. So that's one thing that can happen by D schedule: another possibility in with kubernetes that we have today if this may not apply to everybody.

A

But if you have different priorities for these parts- and you know your larger parts happen to have higher priority, then then preemption will take care of that for you. So it will basically remove some of those smaller parts and chances are that those smaller parts fit on other nodes which have like smaller amount of resources available, and it will schedule your larger pods on the freedoms.

A

Alright, any other questions.

A

Can you pass this.

C

Is scale a problem or like? Are they in the working like improving with truth, but since it? Oh so.

A

Say it's an again: it's.

C

A scale a problem for Java are we working on like increasing the performance for it, I.

A

Don't know if you were here earlier in the talk, but we said that we've improved performance of the scheduler by like three X, so this is. This is relatively recent base if it has happened in the past six months, so it's available in the more recent versions of this scheduler, but actually I was just before this talk. I was talking to one of our contributors who works at Alibaba Alibaba claims that they have been able to run clusters kubernetes clusters with tens of thousands of parts.

A

Sometimes they apparently run like clusters with 30,000 calls or so, and what they have found is that the scheduler is oftentimes actually is not. The bottleneck. Bottleneck is mostly on an API server and HCD, so they have had some patches to remove some of those bottlenecks and actually that's how they achieve to reach to these, like tens of thousands of pots per cluster. But scalability is a complex problem right. So it's if you've been to any of these scalability talks.

A

Folks from security and those who work on the scalability can tell you more information, but it's a complex problem. It's not just about scheduler by itself or about like the number of nodes in the cluster by it's, it's a very multi-dimensional problem. If you stretch it from one side, you have to compromise on some other sides.

A

Basically, for example, if you want to run clusters of many nodes, probably you cannot also have many parts per node, so you have to basically tone down on some of those or maybe you cannot have like many attached volumes for all of those nodes.

A

But overall, what we have seen so far is the scheduled at this at this point of time that we have all these improvements is not a major bottleneck in most of the clusters, unless you have like particular needs that, for example, I have a 12,000 north cluster and I wanna be able to schedule a thousand parts per second.

A

Yes, then schedule is a problem, but even if the scheduler was able to do a thousand parts per second I would say still you run into another problem on the API server side, because API server has to update these parts at like a rate of a thousand parts per second, an API server wouldn't be able to do it so yeah. There is more than just one problem to solve here.

A

Thank you any other questions. Yes yeah. So many questions.

E

Security, really, even at the start, that the scheduler normally runs on the masters and obviously that's the default. When what use cases do you see where that's not? Okay,.

A

I mean scheduler schedules not have to run on the master node. In fact, some of our customers and users run custom schedulers on their regular nodes in the cluster. As long as it can communicate with the API server, it can run on any node in the cluster.

A

No, no there's no specific use case, and if you have some concerns about like availability, for example, you know one like multiple schedules. For some reason, you can put some of these schedules on other nodes in the cluster.

A

So any other questions, I guess this is officially the end of the conference, for you guys, I hope you enjoyed the conference and hopefully see some of you folks in the future. Thank you very much.