Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Enhancing the Kubernetes Scheduler for Diverse Workloads in Large Clusters - Yuan Chen & Yan Xu

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Enhancing the Kubernetes Scheduler for Diverse Workloads in Large Clusters - Yuan Chen & Yan Xu, Apple

As a wide diversity of workloads are being deployed in Kubernetes, the default scheduler has become insufficient in the light of scheduling performance and functionality. In this talk, Yuan Chen and Yan Xu will present their experience and results of leveraging the Kubernetes scheduling framework and developing new plugins to create a custom scheduler. The scheduler can meet different scheduling needs of diverse workloads in large scale clusters, from stateless to stateful services, big data jobs and machine learning applications. They will deep dive into (1) the design and implementation of the scheduling plugins for performance optimization, custom pod placement and group scheduling, and (2) the use of plugins and scheduling profiles to achieve a better balance between scheduling performance and quality. New features and enhancements of the scheduling framework will also be discussed.

https://sched.co/ekBk

A

Good morning, good afternoon, good evening, I'm yuan cheng from apple cloud infrastructure team. Today, I'm glad to present our work on enhancing kubernetes scheduler for diverse workloads in large clusters. It's a joint work with my colleague yanshi and many collaborators from upstream community.

A

This is the journey of my talk. I'd like to start with an introduction why we need to extend and enhance the existing cognitive schedulers. Now I'm going to discuss and present a few use cases how to use the scheduling framework to develop new features to support stateful applications, batch jobs and improve scheduling performance in large clusters.

A

Then conclude my talks with a summary. Why is native kubernetes scheduler is not good enough. Let's look at the history. Kubernetes scheduler was designed to many support stateless applications.

A

It used a port-by-port scheduling strategy and apply some simple scheduling logic. Also, its scheduling decision is based on some optimal strategy. It means it choose the nodes, scoring the nodes and choose the best nodes to be assigned to a port.

A

But if we look at today's workloads, there are many more other workloads like stateful applications, spread jobs, machine learning, deep learning and hpc applications start to run in kubernetes clusters.

A

They to some extent require advanced schedulings from gang scheduling. Topology awareness advanced beam packing also the kubernetes cluster size are increasing very fast. Today we have seen thousands those are clusters, ten thousands of those clusters. The performance is very important also as the we run more diverse workloads in kubernetes clusters in a multi-tenant environment, a single scheduler policy or strategy cannot meet the different requirements.

A

One size does not fit more. We need to support more customizable policy and algorithms for different workload.

A

So here, just a partial list of the new scheduling features that we see very important to support today's workload in kubernetes clusters.

A

Firstly, I'd like to so how do we and develop the new scheduling features? I'd like to give a quick overview introduction of the scheduling framework which provide a very powerful and uniform mechanism that enable us to write custom logic to extend and enhance the existing schedulers scheduling, logic and workflow.

A

So if we look at the flow here from a port submission to a port finally placed or assigned on a node, it has to go through of many steps or stages.

A

So the scheduling framework provide a api for each of this and extension point before and after each of the stage, for example, before the filter stage, which basically is find the feasible nodes that can run a node or the scoring stage and give a score to a node and rank all these nodes to choose the best nodes.

A

So the scheduler framework provided this api. We can leverage this and use this in the mechanism to create and many different kind of the scheduling, new features and custom scheduling logic compared to existing approaches to extend and develop and the new scheduling policy algorithm. The schedule framework is highly extensible and customizable because it, the plugin, is part of the scheduling free, the native and the schedulers. So we have only a single schedule and can use this and the cache and arrow handling. So it's provide a much better performance and better handle and errors.

A

Finally, because it's run as a single scheduler, so no conflicts or risk condition that is inherent in multi-schedulers.

A

Here is a summary of compared to existing or old way and to extend the cognitive schedules use just to change the code. It's definitely not recommended because it's not compatible cannot turn very difficult to to incorporate the new features or new versions kubernetes or use the scheduler extenders.

A

But the key problems here are one: it has very limited extension point basically can only and run extended after the filter stage and also after the scoring stage. Also it's run as a separate service or web hook. The communication overhead deceleration civilization very high.

A

The third approach is just the wrong and a separate customer scheduler alongside the native scheduler. The problem with this approach is very hard or difficult to coordinate the schedule. Schedulization conflicts, so it will introduce the risk so far and it's still an open question how to solve it in a larger scale and production systems.

A

By contrast, the scheduling framework can provide a uniform and very lightweight and fine granularity extension mechanism. You can customize extend the scheduling flow, logical algorithm before and after each stage of a scheduling cycle, also because it's wrong and built into the schedule binary run as a single scheduler.

A

It has eliminated the serialization deceleration overhead and can offer much bad performance.

A

So now, let's move to look at some of the use cases. We never use the scheduling framework to develop a new feature to support stateful ports batch job and improve the scheduling performance in large clusters.

A

So first use cases is about new features to support some stateful ports which require a static or fixed ip address.

A

This means whenever a stateful port is assigned an ipo address, it should and have this ip address during its entire life cycle. This is a not an uncommon requirement we have seen, and in many and end user environments, have this requirement either for the legacy and maintain the compatibility with nexus system or simplify some application. The logic this will require some changes to the existing scheduling logic.

A

So, firstly, we have to check this ip information, and this thing you have to know and which and state of reports already assigned on which ip address, also depending on how you manage your ip address. If it's a static, wii ipad and assign a fixed number life address to each node, you have to make sure you check this map from the ip address to the node or node to ipo chess.

A

Finally, during this and the filter stage of the scheduling and the decision find the feasible node, we have to check make sure and we assign the support to the right and the node if it's already have ipods or if there are still free or available iphones on a node.

A

So if we look at how we implement this and using the schedule framework, the simplest and the design implementation here is, we have to introduce under two plugins or extension point. One is for the pre-filter during the field free field. Basically, we have to sync up and make sure the scheduler have the right and correct information about the ip state, as well as the ip reservation information.

A

The second one is, it have to check and classify a port into three different categories, and one is a stateful port, new, stable ports without allocated ip or existing state proposal with a ip and a regular port just state this port, then the next plugin new plugin is during the filter stage.

A

The static ip scheduler has to do some additional things so for regular ports or standpoint.

A

It needs to check whether or not and this node or a rack have still available free ips then reject the nodes if it doesn't have any available ips. But if it's a stateful port, then it have to make sure right. The allocated ip and the node actually match means if there are some static ip already and assigned to a nodes, and the allocated ip should be available on this node.

A

So on the right side, this is the code segment just to show you how we could implement this static ip stock uh schedule plugin. This is the example of the filter plugin. So, as you can see, it's quite straightforward, simple, so compare with the existing approach using the scheduler extenders as a web hook.

A

The scheduling framework plugin provides a much simplified implementations, also because it's running as a lightweight plug-in without managing the the the event cache itself and the much easier handle errors. So it's much more robust, stable and also easier to maintain and manage.

A

Finally, without running as a separate component without the machining demanding overhead, the performance can be improved significantly. So here is some a benchmark. Performance result on top is the webhook implementation.

A

The percentage of the time spent on the predicated logic extender is measured as the percentage and over the entire scheduling algorithm and the time it's almost up to 50. By contrast, the field plug-in only take up to four percent of the scheduling algorithm durations. So it's a very and significant performance improvement.

A

Next and I'm going to discuss or share a new scheduling, features to support the best again scheduling for batch jobs, so many and the best jobs and like big data, workloads, spark machine learning. Deep learning require gain scheduling, which means give a group of the ports scheduling all of them, or none of them so far, because the existing native scheduler only supported this port by port group.

A

It's missing these features, so one of the solution we already have in the community is a night with co-scheduling plug-in. It was original proposed by alibaba, ching, chong wang and it's cornice.

A

During the last few months, we have been working very closely with the communities to improve and enhance this co-scheduling plug-in to better support against schedule for batch jobs.

A

We have the collaborators from alibaba apple, ibm, tencent and many others. So the idea is, it introduces two labels, the first enable and just define a group. You name it and, like my here's, my batch jobs, the second labels describe and specifies how many what's the minimum number of the ports should be scheduled as a group.

A

So, in order to support the scan scheduling, we have to introduce multiple extensions at a different stage. So first thing is for sorting the ports where port submitted to the kubernetes. It has to be sorted a scheduling queue instead of doing the port by port and the sorting the plugin make sure that the ports from the same groove are sorted together and avoiding and it interleaved and across the different port groups.

A

The second and the uh addition is to check and create a product group managed by the code, scheduling plugin so make sure we have. We know which port belong to, of which port group also is verify whether or not and the minimum number of reports in the class can meet the port group and minimum requirements that can reject the reports earlier to save the resources.

A

The third plugin is because now we'd like to support the port group scheduling and we cannot bind a port finding a node for it. We have to just reserve that port, so the reserve plugin also should have a timeout mechanism make sure it won't occupy resources forever.

A

The fourth plugin is this: permit plugin, it's check whether or not the ports belonging to a same group already reached its minimum number of reports to be wrong. If it's meet and post, then we allow all these schedule. Reports move to next stage to the binding stage to basically to start and creating the container supports to run the systems.

A

Also, we have to manage these port groups and make sure the deletion or product group clean up the product group. Also, if there are time out, we have to clean up the port group as well.

A

All these parameters can be customizable based on some parameters.

A

So, in a summary, this and the co-scheduling plugin provide a simple mechanism to run bachelors to as a group support this and gang scheduling requirement by many batch jobs. It also supports to define a product group across the jobs and deployment, and there are some mechanism to make sure the port group can be cleaned off and when deleted, or timed out, also have a nice and error check mechanisms.

A

So here is just example and policy fire define a scheduling profile to support the gang scheduling or co-scheduling.

A

So here, as you can see, is we just need and add this code scheduling plugin for the different stage, the sort stage, pre-filter stage permit stage and reserve stage? Also we can customize the parameters how to manage the port group for more information. Please visit the scheduler plugin in the ripple and also the users will welcome users and also contributors so moving forward.

A

They are not of the interesting or more advanced features associated related to the co-scheduling, the community and is working very hard to address or look into these issues and a new proposal or pr just to try to improve this and the label-based code, scheduling to a port group cid based implementation.

A

Also, we are looking at some more advanced customer preemption. For example, when you preamp to evict the product from product group, should we evict the entire all ports from the same group or just a single group, then it may violate violating the minimum number of the ports requirement defined in the gang uh the portal group definition.

A

Also, can we improve the utilization if we don't want a large port group, reserved resources and block some small port group from running? So what about introducing the back field strategy and a more advanced rescheduling strategy? Also? Finally, as we can see, there are a lot of new requirements on this and the port group based management.

A

So can we introduce a more general generic and sorting plug-in tree and even a single port, a port group consisting just a port, so we can manage in port or court group in a uniform way so for more information, and you can look at some of the discussion and the the issues and the cap. The third use cases is gainable scheduling how to improve the performance schedule performance at a scale.

A

We have seen many very large skill and clusters with thousands of ten thousands of nodes. Also, there are a lot of large jobs or services running in such clusters. Each have thousands of or ten thousands of ports for some the real-time interactive workloads. The auto scaling is important to quickly scale.

A

The number of the ports for service to handle increasing workloads so always require very good performance scheduler, but unfortunately, today, kubernetes native schedule performance can be limited by its port by port scheduling, algorithm and also the optimal placement strategy in very large scale clusters.

A

To address that, we have made two proposals. The first one is: can we customize some key scheduling, parameters for different workloads or different tenants to better perform balance, the scheduling, quantity and performance?

A

The second one is, instead of scheduling a port port by one by one we can assign the nodes to a group of port at a time.

A

So, firstly, let's look at customized scheduled parameters. So one of the key parameters that have a big impact on scheduling performance is called percentage of notes to score which determine how many of the nodes that the scheduler should check and score and rank them. So, for example, here is a benchmark results experiment.

A

So if we run this experiment in a 2000 node cluster, five percent of this parameter value means we'll score up to 100 nodes. So if we find 100 feasible nodes, we just stop search and rank this 100 nodes and choose the best one 100 means we look at up to the entire cluster up to 2000 nodes.

A

If we look at the results on the left, the graph shoe and for very simple strategy, the five percent and 100 percent- they don't have much difference- can reach 120 per second, that's the maximum throughput we have seen so far, but for some ports that require advanced placement constraint, like a affinity, anti-affinity five percent and the tempest one hundred percent show a huge difference.

A

Almost the 3x difference in terms of scheduling, throughput.

A

To address that, we propose a per profile parameter idea, so the current this percentage of nodes to score is a global parameters.

A

What we proposed is to have a per profile parameters so that we we could assign different percentage of the nodes to score to different scheduling profile, that different workloads or tenants can assign or use different scheduling profiles to better meet their scheduling, performance and scheduling quality.

A

The second idea: the proposal is to score and assign the nodes to port as a group, the key idea is, we can concede a group of homogeneous ports with the same resource requirement as a group, then we just need to score the nodes once assign the top key scoring nodes to the ports at a time to implement that we need, firstly, add a port group sort plugin, which is similar to the code, scheduling, plugin, sort, the product group together.

A

The second plugin is the group score plugin, which will apply this and top key scoring nodes to the k ports, but this is still in a design phase.

A

The challenge is: how can we support this port group as scheduling port, because the current scheduling framework, we can customize the scheduling logical before and after each scheduling stage, but it's still impossible to customize the scheduling flow logic itself. For example, can we skip a port for certain stage or customize some processing logical there, but this is a very hard problem, because we have to make a good balance and have good trade-off between the simplicity, robustness and the performance and advanced features in the scheduling, schedule, design and implementation.

A

Okay, so we have and discuss the three and use cases and how to develop a new scheduling feature to support the different use cases. Finally, here is example: how can we put together create a scheduling profile to use all these scheduling features? So here we show a example policy with the four different profiles. Each profile is associated with different plugins.

A

They have different names, different plugins, the only constraint is the queue sort. Plugin should be the same, because they're only a single scheduling plugin that for different profile. We can define different plugin associated with this and the scheduler profile, so sticky, ip scheduling, code, scheduling default scheduling or an integrated scheduler. This one we can use any combine all these three different type of the plugins together.

A

Also, we will see- and we can set different percentage of those to score for each of the profile that different workloads, different tenant. Different users can specify different profiles for their workload to better meet their workload, performance and scalability and scheduling quality requirements.

A

To summarize, we have seen and increasing needs for new features in kubernetes scheduler. The schedule scheduling framework provide a very powerful and generic mechanism to enable us to develop new scheduling features.

A

Many of them are already available and are under development last, but definitely not released. The community collaboration is the key. The work we have presented today is the outcome of close collaboration with the upstream community, especially the sikh scheduling. Community, more users and contributors are highly welcome to join the community for more information.

A

Please visit the scheduler plug-in repo.

A

With mike ready thanks the contributors in the upstream community, our collaborators we huang, abdulnah gereha, both co-chairs of sikh scheduling and ching chong wong kaijung from alibaba on a code, scheduling and gymnastic scheduling contribution. We don't sigh from 10 cents on a cid based code scheduling.

A

Finally, we are part of the apple cloud service team for more information. Please visit the apple virtual booth and we are hiring kubernetes engineers from core kubernetes infrastructure to platform to application layer. Thank you very much.