Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Scheduling Deep Dive - Wei Huang, IBM & Abdullah Gharaibeh, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

SIG Scheduling Deep Dive - Wei Huang, IBM & Abdullah Gharaibeh, Google

Kube-Scheduler is the component of Kubernetes that assigns pods to nodes based on the configured scheduling requirements. These requirements can be high availability, resource efficiency and other policies and heuristics. This talk will provide an overview of the new and upcoming features in the scheduler, including the scheduler framework, and the new component config API that enables scheduling profiles. In addition, we'll present updates on other sig-scheduling projects like the descheduler, scheduler-plugins.

https://sched.co/ekHV

A

Hi everyone uh welcome to the north america virtual cube car. Today, I'm going to give you a deep dive on the sixth scheduling and uh a little bit on myself. My name is weinhuang. I work at ibm, I'm also the co-chair of the sikh scheduling.

A

So today's content is separated into three parts and at each part we want. The content is delivered to the exact audience. So we separate them on to different personas from the uh kubernetes beginner to some scheduling expert. So what we want to that the content to be delivered so that the specific audience can absorb easily okay, the first kind of audience. We are targeting those regular users who just write and deploy their applications onto the kubernetes platform.

A

So for them they just want to know some scheduled basics so that they can use the scheduler feature, features efficiently and correctly and also they will maybe want to do some basic troubleshooting to identify whether the issue is belongs to scheduler or other components and for sure they don't want to go too deep into their scheduled internals.

A

So fourth, um further thing is to answer them. What keyboard stack schedule does so in one sentence: keep scheduler just find all the pending paths and assign the part to the most best, the fit node. So that's it and in term of uh api perspective, so depending means the node that doesn't have spec download name field set it's empty, so you can see from the picture and then scheduler will go through some internal logic and we'll find it's the best node for the incoming part, and I assign the path to the node.

A

In this case, then spectrum name was sent to uh canned worker 2.. So this is basically schedule it does, and with that being said, there are some things I need to mention that which cuba schedule doesn't do which uh often missed understood by the users like quarter enforcement. This is the thing that the admission plugins does at the pod creation time. Instead of scheduling, time and also spinning up and sculling down, the replicas of deployment is the job of consumer manager because they are responsible for manage.

A

The number of replicas meets the desired number and also it's the cubelet responsible for effect. The running path, maybe because the working node has running out of memory running out of desk, etc, etc. So I list a few of the common misunderstanding that people would think that this thing belongs to schedule, but it doesn't so understanding. This will help you better identify the issues, whether it belongs to a schedule or not, and the next we will give a very high level introduction about the scheduling flow, the most the basics.

A

So once the scheduler gets a part, it goes through internal scheduling cycle. The sketch screen cycle will schedule the part one by one and one scanning cycle can be divided into several phases. The first one is called filtering, so filtering tries to answer the paths, hard constraints. It's like answer your question like as a user. I need my path to be scheduled to a node which has two gigabyte two gigabyte memory, or I need my part to be scheduled to another witch so that to code the existence with some kind of path.

A

So this kind of requirements I call them hardcore, they must be satisfied and all the hard constraints are ended, not all and in terms of the hub constraints they are. Usually. I think 99 of the cases are from the prospect which I specified directly by the user.

A

For example, the request and the path affinity so notice there here the primitive starts with required. So if you read out, some term starts with required is usually the hard requirement because it's required and some other primitive use, other terms like the project spread feature use a constant string to and identify. This is a hard constraint that, if the constraints cannot be satisfied, then do not schedule this part, so you can usually tell, from the literal meanings to tell whether it's a hard constraint, yeah or a soft constraint of a later mention.

A

This is important because only heart constraint only on unsatisfying hard constraint will make your part into pending status and next phase. So suppose you have 100 nails and only 10 nails has passed the filter phase. Then we go to the next phase called scoring so scoring.

A

In contrast of the filtering best efforts, so that try to answer some questions like I prefer my pile to be scheduled to know which has ssd, but if that note doesn't have it doesn't matter, I just want scheduler to try its best to do this kind of thing so, based on this software constraint, each filtered node filter means that node has passed the filter, filtering constraint, those filter, nails will get scored and we aggregate those score and finally pick up the higher the score and assign the path to the node, which has the highest score.

A

So I want to highlight that, in terms of soft constraint, it doesn't block scheduling your path. It's just describe a relation of preference. I prefer my path to be scheduled onto some some note, and maybe there are some soft constraints conflict with each other, so it happens so that and the software constraints has have two sources. The first one is from the prospect you can identify some stuff. The constraint in the prospect like for the node affinity and the primitive starts with preferred blah blah blah blah.

A

So this is the soft constraint and usually it comes with additional parameter like weight, so that can give the user a customized weight so that I favor this constraint more than other constraints of the construct and also some features are using other primitive, which are usually self-descriptive like when the pile cannot be satisfied.

A

I will still schedule scheduled anyway, so this is soft construct and there is a another kind of software constraint which are not specified by the user instead usually specified by the cluster admin upon the scheduled startup. So they are called. I call them implicit schedule config.

A

So there are sort of scheduling policies like the default policy tries to use the node, which has the least allocated resource as possible so that it will try to make the cluster the utilization balance rather than beam pack or the pause to one single note. So this is for softer constraint and you may think about think of a question that what if no node can satisfy all the cause, hard constraint in the filtering phase? What should we do?

A

So we have a special phase called preemption, so preemption is only triggered when there's no node can satisfy the hard constraint and it will see whether there are low priority candidates which can be sacrificed, can be preempted to make room for the high priority part. If this is possible, a schedule will go preempt the low priority path to make room so that the high priority part is schedulable.

A

So in terms of api, you have to define the priority class and specify their value the higher the value, the more important the priority is and in terms of the prospect, you'll define the spec dot priority class name and correspond to their priority priority test, which the scheduler will own in the long term.

A

All right. uh This is pretty much. I think in day. One a regular scheduling user needs to know about the basics of the schedule, and you just need to learn that scheduling has basically three phases: filter scoring and preemption, and if you see a party's pending status, first ensure that the path spectrum name is not set. So that is in the scope of scheduling which are trying to schedule.

A

The pattern to the note and also use cube control describe that part to check which constraint how the constraint is not satisfied. Okay, so in the impending status, you check those kind of go through those simply job shooting steps, but if a path can be scheduled, but it's not the desired node that it is scheduled onto.

A

There is usually something wrong in the scoring plugin scoring phase. Maybe some default scoring policies that given by the cluster and mean, doesn't fit your class. You need to check with them, or you do some debug to see why one score policy is way higher than the other. So this is basically for regular users all right uh for day two, you we are targeting the audience as customy or devops. So the goal here is a little different, so you are not only satisfied of trading schedule as a black box.

A

Instead, you want to know more about schedule internet and to get to know some configuration back best practices and also understand how the scheduling behavior correlated to internal. We call the scheduling plugins so that you can make the most of schedule and expose them to the end users. And of course you don't want to write additional code. No matter, it's a scheduler, extender or plugin. You just make customize the scheduler with the official scheduled image.

A

So the first thing you can look into is configuration so configuration as time goes changes a lot in the recent kubernetes releases, and the first thing I want to mention is that don't use command line arguments anymore because we have a better design and organized object, called cube, scheduler configurations so that you can pass on all the individual command line. Arguments through that config object through the dash dash config parameter.

A

But if you are still at like humanity's 1 7, you have to still adapt to the to the old style kind of cube schedule configuration and the second thing is even inside the cube scheduler configuration there are different versions, vary from v1, alpha 1 to v1, beta 1 and the difference. This is that before we won alpha 2, we are still using the old execution path, all terms like prior decays, corresponding to filtering and priority corresponding to scoring, and they are a little bit different in terms of their naming and the execution path.

A

So if you are still using the old config, you may be using some thing called the policy file, so the policy file is something like this. You define a schedule config with one f1 version and specify one policy as the policy file and inside the policy file. If you want to customize some behavior, you have to list all the predicates or the priorities, even if you just want to change one. So that is not good. You search user experience right and also here you can see that you have to provide two files.

A

One is the cube schedule. Config one is the policy file itself, it's also not user friendly. So, as time goes, we want to duplicate this kind of policy-based config and instead we are transitioning to the plugin-based config.

A

So plugin is simply a functional unit that is written to satisfy one specific constraint, either from the user's input or from some implicit general policy. So right now, if you are using v1 alpha, 2 or v1 beta 1, you can use the latest cube schedule configurations under the profiles.

A

Subfield, there is a plugin field, so there you can spotify, you can disable or enable or disable all the plugins there by writing the minimum yaml snippet as possible, so it's very user friendly and the more importantly so right now you can see that we, we have a new profiles field. That means so right now, starting with 118.

A

Your scheduler is not a single flavor schedule. You can build a multiple multi-flavored scheduler. I will give example so using the weaver beta wine configurations here I define four profiles and default scheduler image. First, the beam pack skip score and they can map to different behaviors in runtime.

A

Their different scheduling profiles like the skip score is something that I totally just disabled the scoring phase, because I don't care about which node is highest the highest score. I just want the part to be scheduled as fast as possible and the beam pack, on the other hand, try to favor the node, which has the most allocated resource as possible. So it's quite fit the the requirements of auto scale, because auto scada wants to save the cost of running machines.

A

They want to pack all the workloads and minimum machines as possible, etc, and in runtime in your workload, you just specify the spec dot, scheduler name and correspond to the profile. So in the example that we have four.

A

Workload, demos, and with different scheduler name and in runtime schedule, nails which specific flavor you want your workload to be scheduled. So that's the. I think the very key change in recent kubernetes releases so back to this picture.

A

So as a custard admin, you should know exactly what each plugin is doing, not only by understanding the logic and also knows their internal stuff like their arguments and how to conflict them and how that arguments reflect in terms of behavior, and here we have a page to list all the default plugins and which extension point they belongs to, for example, 10 toleration.

A

It implements several extension points in field phase, prescribed phase and score phase, so as a custom admin. If you want enable or disable do some customization, I do recommend to check out this.

A

The official website, and also there are some plugins- was compiled into the schedule image but not enabled by default, so you can choose to enable them. But again you should to understand the real semantics of them and some of them uh have conflict with default ones. You if you want to enable some of them. Maybe you want to disable some of these default plugins, so that is the basic.

A

uh The custom main need to understand.

A

In addition to that, there are some other global settings in the cube schedule, config like percentage of nodes to score, which are very useful if you're running on 1000 or 5000 now properly, you don't need to score all the filter, nails you just need to score, maybe 10 or 20 percent of the nails, because scoring is more to favor one node to to the other. So some of the workloads just care about whether this hot constraint is, has been satisfied and don't quite care about the soft constraints.

A

So that's a very useful features and as a crust I mean you may want to catch up with the latest uh progress in the upstreams.

A

I listed a few of them here and the eventual goal is that you, I suppose, just it's like you are running a burger shop like right in the before you just provide a one flavor burger, which has meat, onion and tomatoes.

A

That's it you use it just can take it or leave it right now you can have you have multiple ingredients right, you can compose them and into different flavors like it's a veggie one, it's uh with beef, it's with chicken and with beef. You can also divide it into with green, green onion or with wrong onion or with uh tomatoes. Without the middle east address edges, so you can.

A

The image is still the same image, but you can provide more flavors so that to adapt to your user's workload. So this is basically the. I think the trust admin wants to learn. Okay, let's go to day three days: three, we are talking to the audience which uh usually skating expert or innovators.

A

It's like you're still running a burger shop, but you are not satisfied with existing ingredients. You want to make some innovative ingredients. Our site, for example, then provide more flavors than the default schedule. Image provides.

A

So, in this case, they don't care about writing some sort sort of code and the ultimate goal is to fit diverse workflows such as batching workloads, which that the current scheduler is not supporting well, but they are they're. Not they don't want to start from scratch to write a secondary schedule because running a multiple scheduler will invictably cause some conflicts on the path scheduling.

A

So, for this kind of users still go back to this picture, you should not be satisfied with only knowing that filtering has some plugin xyz and the scoring has plugging xyz. You need to understand, for example, why we need pre-filter and how to use pre-filter right pre-filter, for example, is a phase that, especially for the requirement that you need to consider cross node, cross node status, so that you need to do some pre-calculations and put that result into sort of a cache and that result can be used later in filter phase.

A

That's for pre-filter, and also you need to understand what pre permit is and what reserve it right reserve is the face that you can before you part guest schedule. You can make a optimized uh assumption that this part has been scheduled so that you assume this part of resource in the scheduling cache and the permit is a very useful extension point that you can wait until a certain criteria, like a group of paths come inside. All of them are satisfied and you approve all this sort of pause in badge.

A

So this permit is a useful extension point in the bachelor clause and also poster filter is a new extension point we introduced in wine 19. It replaced the old, hard coded preemption logic, so that it's the preemption logic is more extensible like in the before we just in the default preemption.

A

We just consider the strategy to preempt the node. Sorry preempting, the path on a single node, but sometimes it's not not the case, and you want to extend this behavior like you want to preempt one group of paths and that kind of one group path may be scheduled onto different nails, etc, and also you need to follow up with the latest change in the scheduling framework and just read through the release.

A

Note, if you have any questions, go to the flag, go to a million less that to raise your concern and uh also we we are going to uh ga the scheduling framework in 120.

A

We have started to build this framework since 115, but now we are able to say that it's stable enough. It can be go to the next stage.

A

uh All right, it's still the same topic that if you want to implement what specific features you should know which specific extension point you want to extend and write the corresponding logic there. For example, if you want to uh get scheduling requirement, you may need to implement this kind of external point, and if you want to do some scoring plugins, you need to implement the preschool and score extension point so that so with the default scheduler on the base, you build something.

A

On top of that and finally, you get you compare them together and then you get a unified, scheduler binary. It's a new new binary which has 100 percent of the vanina schedule. Functionality, as well as the additional functionality you built into it.

A

And because uh building a schedule plugin has, I won't say it has a high bar, but it has a bar.

A

So we initiated a sub project called schedule, plugins, which uh writing some examples there and some guidance there for you to start with so right now we already have some caps on gas scheduling um capacity scheduling um because I say scheduling is about elastic quarter and also some ongoing care, like which favors to use some real-time metrics of the cluster, to do scheduling, decisions, etc, and also we do some ca optimizations to do automatic view, then, which so that you can just download the image from kubernetes.gcr.

A

All right and this this is pretty much for the day three, so that still the analogy that you are running a burger shop so right now you can use the default ingredients from the upstream and also you can buy some other ingredients from the local market right. You can buy some special cheese. You can buy some special ingredients hazelnut or what who knows and combine them together to provide additional flavors to for your user to use and feel the last contact us and you can find all the information here.

A

We have weekly meetings, raise your topic there and help us build this community better. Alright, thank you. I think this is pretty much for today's session. Thank you.