Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Capacity Scheduling for Elastic Resource Sharing in Kubernetes - Qingcan Wang & Yuan Chen

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Capacity Scheduling for Elastic Resource Sharing in Kubernetes - Qingcan Wang, Alibaba & Yuan Chen, Apple

Kubernetes manages resources capacity across multi-tenants/users/namespaces by allocating a fixed amount of resource quotas to each namespace. It lacks sufficient support of dynamic resource sharing within and across teams and organizations and can result in low cluster utilization. It has become a roadblock to migrating applications from other cluster management platforms (e.g., YARN) to Kubernetes. Qingcan Wang from Alibaba and Yuan Chen from Apple will present their collaborative work on a Kubernetes enhancement to address the issue. Capacity scheduling offers a similar feature to YARN’s capacity scheduler and enables elastic resource sharing to improve cluster utilization in Kubernetes. It supports hierarchical resource groups with guaranteed and maximum resources for dynamic sharing of resources, from CPU, memory, disk to extended resources like GPU. It is seamlessly integrated into Kubernetes as plugins and has been used in large scale production clusters such as Alibaba Cloud.

A

All right well, good morning, almost afternoon, everybody welcome. My name is randy, I'm a cloud native ambassador with the cncf, and this is the capacity scheduling for elastic resource sharing and kubernetes.

B

A

And I would like you all online and here in person to welcome juan chen from apple.

C

Thank you already.

C

Good morning, good afternoon, good evening, we are so excited to be here. It's the first on-site event, almost in two years.

C

It's my sixth and cubecon, was my sixth talk and now on-site and hope can get and make new friends get connected with the community for those people who are virtual and look forward to seeing you next time. So today, I'm co-presenting our work on capacity scheduling in kubernetes with ching chan.

C

Unfortunately, chinchon is not able to attend in person. They will present a demo in a pre-recorded video later so a little bit about ourselves. King tut is a software engineer from oni baba cloud. He has been a very and active contributor in kubernetes, especially the sikh scheduling group. He made a lot of the code contribution, design contribution to the kubernetes schedule.

C

I'm yan chen from apple cloud services, I'm a kubernetes contributor too, and my work is mainly focused on the kubernetes scheduling performance, scalability.

C

Okay, I'd like to start with some background context and motivations about the work. As all of all of us know, kubernetes has becoming a de facto cluster and cloud resource management platform. So more and more workloads and different type of workloads are running in kubernetes.

C

In addition to the traditional service workloads and now there are more batch workloads from the machine learning, deep learning to big data workloads or running kubernetes.

C

A key issue we have realized after talking to our customers. There are some look at today's kubernetes. How does kubernetes manage resources between different users or name space?

C

So the quarter is a basic concept, but quarter mainly used just for the admission, control or capacity planning, not involve the scheduling and in a more dynamic way, the secondary the quarter. So far we can define just a single value, for example, for each namespace. That's the request. The total request a namespace can ask for the third one is during the wrong time. The resource coordination is purely based on priority and the default preemption. We based on priority to do this and eviction to achieve this kind of the fairness or the resource education.

C

So, as a result, we can see and internet some of these more flexible or dynamic resource sharing. Traditional cluster management system, like the hadoop young, can offer as a also this can cause no resource and class resource utilization.

C

These two and the key of this under short time disadvantage, has become a roadblock for many customers to migrate from traditional cluster management system to kubernetes.

C

To address this.

C

We have been working and in upstream community and collaborating with a lot of contributors on two of the key and concept. One is the required terms, elastic quarter.

C

The idea is, can introduce a define and provide a more flexible. The building block can support more flexible, dynamic resource sharing across the name space and users. Also, it will provide some kind of resource guarantee and fairness.

C

Finally, we're extending the flat kind of the quarter structure to a tree or hierarchy level to better manage the resource across the different user organization.

C

That is what also other like hadoop young and offer capacity planning or scheduling or fair scheduling.

C

So now, let me and talk a little bit the detail about what the inestic court is so inestio quarter is a crd.

C

The difference from the traditional or existing built-in resource quarter is we introduce basically two fields or values for each elastic quarter. One is minimum that is guaranteed quarter for name space. The second one is maximum. Is the limit of the name space be aware here? When we talk about the quarter, we basically refer to the request request, not the limit, because this is more and what the schedule make decision and we look at the request only so it supports multiple resource type from the building, cpu memory disk to extended resources like gpu or other resources.

C

So I already mentioned so it's a different concept from the building or native and resource quarters. So this resource code is more for admission control, but the more industrial quarter is more used by the scheduler and coordinated resource sharing at wrong time. So the right side is a simple example.

C

Here is a elastic quarter named test, it's a social river named space test. So, as you can see, it's quite straightforward, so it's defined or specified maximum and minimum resources in terms of cpu memory and gpu by the way throughout my presentation, so I'm using the gpu as an example, but it can apply to any type of resources like cpu memory.

C

So how can we guarantee so the basic idea for each namespace with the elastic cortex means the minimum quarter will be guaranteed and you can get that much of the quarter. So maximum is optimistically. If there are additional resources available in a cluster, you can borrow the quarter from other namespace or user to run it. But what happened if there are contentions, so this is how the resource guarantee fair needs is very important so to implement this.

C

Basically, we implemented a customer preemption so at wrong time, if a namespace submits ports, but it cannot be, cannot be scheduled because it's quarter and has been used or borrowed by other namespace. So the preemption we try to identify those over used name space so means they already use more quarter than its guarantees.

C

Then, among all this and the candidate namespace, I should take off my end mask sorry about that. So after it identified those, then it will identify these of the candidates or victim ports victims. So so far we implemented a single policy based on priority and also try to minimize the number of the evicted ports. Of course, one of the extensions probably have more of the fairness and the key balance across different name space that can be added in the future versions.

C

So this and elastic quarter is a crd. The customer preemption is implemented as a scheduler plug-in. So for those who are not familiar with the scheduled framework, it's basically provide a uniform platform that allows the developer to customize optimize the schedule. So any of this plugin will be built into the main scheduler.

C

So we are not introducing a second scheduler. I think this is a key advantage means this is a kubernetes native solution. You still just run a single schedule with additional plugin. There are no risk condition problem. You can share the cache other thing also you can disable or enable the scheduled plugin.

C

So a lot of detail here, but the key thing is: there are three plugins the pre-filter we check the quarter, make sure we schedule a port right this port and have to make sure it's and meet the cannot exceed the maximum quarter.

C

Post filter basically is preemption. If a port, a name space is under, you use its guaranteed quarter. Then have to perform preemption. The reserve stage is, when you schedule a node with your scheduled port. The schedule has reserved some resources to avoid the risk condition.

C

So here is example of elastic or quarter. So there are two name: space name space, one have the minimum or guarantee and a quarter of four fourth gpu six maximum six gpu and another one have the minimum of six and a maximum of eight gpu respectively.

C

So at the beginning, each of them only use two gpu and the three gpu. So it's fine, it's even a less than their minimum one, but later the namespace, one or user one requested additional resources, then the scheduler can schedule more ports up to its maximum. One is six gpu, the next step, but the namespace two submit more ports or workloads that requested more resources.

C

Now the scheduler realized right. This name space, one over utilize used its guaranteed quarter, so the customer parameter will be kicked in and evicted ports from the namespace one. So now the name space, one use four gpu named space, two use six gpu this they are guaranteed or minimum one. So that's the fairness we want to achieve.

C

Next, I'm briefly describing a extension of the work so far. What we're describing here is a flat quarter and structure, but more and more users want to map the quarter management resource management to the organizational teams, so the hierarchy or tree kind of structure would be a very desirable feature.

C

So let's show you an example, so we can create this and tree or hierarchy quarter. Group from the root and aggregate the entire cluster resources also have a minimum or mean so it's children right. The sum of its children's minimum should be equal than its parent minimum. Then its maximum should be less than its parent's maximum. Then, at the leaf level the name space will be associated with the leaf levels and quarter group. So this week we can manage the resource in a more dynamic and flexible and manner.

C

I hope it's under readable and I apologize if the fonts too small, so here the example so just extend the previous example. So now, in order to define a quarter group, a user can specify the minimum and the maximum, but also their additional field called children. So it's a nested structure. So each quarter group can specify the the children the quarter group. Then the children called group define the minimum and the maximum and finally also can have its own children.

C

Next, I'm talking a little about it's related to the batch workload batch workload, support in kubernetes, but not directly and related to the scheduling. It's the job queue as we know the workloads and the batch workloads come and running and gone.

C

So typically, we need to manage or buffer the job when the resource is not available. So that's how we need a dual cue management.

C

So to address this and a job cue management is introduced to the system as well. So it's managing the workloads, not reports. The ports is the schedule and the concept so far. The policy is based on the priorities of the job and the creation, time of the job or submission time of the job, and also the quarter constraint then decided and which job or should be scheduled next.

C

So a new feature and we're still working on is concrete. Multiple queue right, for example, each user or each tenant can submit their jobs to different queue, then, based on certain policies and either fair, sharing or other more advanced policy, the management system can achieve or provide more flexible or fairness across the user and tenants.

C

So here is a high level and architecture about how the job queue is designed implemented. So the idea is where job is submitted.

C

Annotation like suspend annotation, is added so then later and the job will be added to the queue so based on the policy. When a job is dq from the job queue then annotation will be removed. Then different operators then got the notification will submit or create reports. So one thing I want to mention is because they are all types of different application level: specific workloads right from tensorflow, your pi torch spark and pi. They have different description or schema for their jobs.

C

So now and we are working on the extension- different extensions support, different workloads, so the key function of the extension is to convert or adapt the application, specific job definition to a unified and generic descriptions. In the queue we call, we terms is q unit. So that way you can support a different type of jobs and all clothes okay.

C

So next I'm going to show you a demo created by jintan so very unfortunate if we are not able to make the the audio and the work and otherwise qingtang would be the best present about the work. So I I'm trying to explain.

C

B

I can just creep it.

C

No, it's not working.

C

I have the video here right: okay,.

C

So in this demo it's really it's running a gpu cluster, so we create elastic quarter and namespace, then from different namespace submit to the jobs and you see and how the jobs and the different namespace users share the resources in a more dynamic way, also how the job queue to manage the different jobs.

C

Okay, let me quickly move through here, because we cannot hear the qin chang's voice and so then that's see and the command okay. So now the first step we just created and to name space name space, one.

C

Name: space two.

C

The next step is create the investigator with the minimum or guaranteed quarter and maximum or the limit of the quarter. Okay, so you can see here now we create a elastic quarter for each name. Space and the maximum is 4 gpu and the minimum is 2 gpu by the way, and so far we assume and the total capacity of this tiny and dummy cluster have total of four cpu cards.

C

So each namespace means it will get two gpu as a guaranteed resource, but it can use up to four gpu if another user is not using the resources.

C

So, as you can see here being at max, so next step is. We are going to create some jobs in each name. Space see how these jobs or ports are running and in cluster or schedule running in a cluster.

C

Let me see and okay, that's it. That's called so for the jobs now, so the name space one will submit three jobs. Each job have two workers because tencent flow work, so each job have to work. Each work is using one gpu, so it means each job is using two gpu.

C

That also means a total of four gpu means, because there are only four gpu in a cluster. Now, only two jobs can run at the same time. So you can see here is right. There are two workers, the replay cost is two and each one. The resource request, one gpu.

C

So, let's see if the user, one in the name space one submitter three, this type of the jobs. So what will happen.

C

Teen time probably have some the more insight here. So, okay, let's see it's still.

C

Okay, that's the first job created again. The each job basically is requesting two gpu, so user one launched three jobs.

C

So let's see the status okay as we just described so two jobs are running, each is using two gpu and then up to four gpu. The job3 is qe in the job queue because there are no resource available to run job3.

C

Next, let's see what's happening, if user 2 in name space 2, is submitting a job 2 to the system.

C

So name, space 2 is submitting a job, the same type and the request to gpu. So because there are two workers.

C

So what we are expecting is then, because so far there are no gpu free gpu available, so the preemption will be kicking, then evict some jobs or ports from name space 1.

C

Then the job user, 2 or name space 2 can get the resources from its job.

C

Let's see the status okay so far, and because it takes some time and the job too so far, it's not getting resources, it's queuing, but then, during this queuing and the stage the scheduler will identify the victims and the candidates.

C

So, as you can see now the job 2 in the name, space 1, is being evicted and the ports will be evicted, then the job now. Okay. So it's interesting because the job warning space one completes so that your three got a chance to run because it's requested to cpu then job two or job one of the name space two as we described and after the preemption eviction, succeeded it started running as well, then the job 2 of the namespace 1 was evicted and preempted because it overused the guaranteed resources.

C

So, as you can see this demo in the show how this resources dynamic, sharing between namespace1 and the namespace 2., so when namespace 1 didn't use the resources guaranteed resources named smith, one can run two jobs. Then, but later when name is race, two needed resources and the name space one has to yeah.

C

The scheduler will use preemption to reclaim the resources to run the workloads so now, and the implementation is simple and after evicting uh jobs and it will stay in the field feeling the stage but definitely and we can extend it either by application levels and your management resubmitted we're also discussing if it's because of this and inestic quarter and the contention that make a ports and terminate it early, then maybe we can automatically resubmit to the queue right. Then it will run later with resources available.

C

So this is the demo to show how the industrial quarter work and also the job queue hope you find it and useful and have a bad idea about how this elastic quarter working.

C

Okay, so now, and uh the current status of the index code of capacity planning is, we have two open source projects. The first one is implemented, is inestio quarter and capacity scheduling the core component is the customer. Preemption is part of the sig scheduling out of three plugin, so you can check out the code and play it with this and in the seek scheduler and plug-in repo.

C

So the second one is independent of separate and project repo is how to manage the workload and provide this and your queue management functionality. We just make it public and before the conference, it's still under active development. So now it supports uh tensorflow and the pi torch extension, so spark npr other thing and still working on. Also, we look forward to contributions from all of you, so.

D

The hierarchy quarter will.

C

Be released next so in terms of the early adopters, so alibaba already and apply this elastic cord capacity planning in production, the alibaba cloud, so at apple we are actively investigating how to provide this job queue and elastic scheduling, functions to support running the smart workload in our kubernetes infrastructure.

C

Baidu is looking at use this to run their self-driving and always ai and based on simulation workloads.

C

So here is a list of reference and you may find useful and the repo and some documents.

C

So this is a joint effort in upstream community without the collaboration, it would not be possible to even have to reach this stage, so we really want to thank all these contributors who has contributed to the ideas and the coding the designs.

C

We also look forward to working with everyone, and so you can try out the projects, provide the feedback or contribute the code, design or submitted issues or suggesting recommendations. Everything would be highly appreciated and welcome.

C

Okay, I think that's all I have and.

C

So I I'm happy to take some questions after the talk yeah I'll be around. So if you have any questions, yeah.

A

C

A

Just note for the online folks um we'll be taking questions online too, so feel free to drop those in if you're online and don't forget to review the session on sketch too.

D

Thank you for this. This is amazing work. uh One question I had basically you're taking all the boxes of what we need for scientific, like workloads, one that you didn't mention is gang scheduling. Is this also supported by the scheduler you described.

C

A cue speaker network.

D

Sorry, sorry, I was just saying that this is amazing, you're ticking all the boxes of the stuff we need, uh one that you didn't mention is gang scheduling uh when you're uh doing the workload allocation. Is this something that the scheduler also supports like.

B

D

Multiple jobs, at the same time, waiting for the slot things like this.

C

Oh yeah, so so by scheduling, and so I I should have a clarify right, as you can see now. Basically, there are two levels: one is at your q level. This is the workload or job as a unit, so the the queue management system will determine who which your is the next job to schedule. Right, as I mentioned, there are some policy there. So this is separate. You can think it's outside of the kubernetes and the schedule independent schedule, then the second one is the wheezing kubernetes, the key component and the cube schedule.

C

This is an important level right when the job is picked from the job queue, then all the ports within that job will be submitted to api server, then in the scheduled queue. Then, at that time it's just the default scheduled behavior and to do, but we have a plug-in, as I showed a pre-filled plug-in. Also, we checked make sure if the scheduler schedule a port, make sure that resources total amount of resources won't exceed right. This name space and the maximum unesco quarter resources did I answer your question.

C

A

Great, thank you. An online question, that's got a bunch of votes here is: are the elastic quota and job queue components um going to end up in the kate's scheduling? Sig? You know mix at any point in time.

C

Yeah so so far, we we believe. As I mentioned, this elastic capacity planning is already part of the sig scheduling and out of treat repo and it's been there well and we tested extensively and it's just the crd plus the plug-in of the schedule, which is basically just kubernetes native solution, alibaba already applied in production. So it definitely will come and more testing and try it and so so far, and uh we just hope and we can get more and yeah adoption and people test it and then yeah.

C

We can improve it, and so that's the goal and yeah make it turn the production, ready, yeah, widening and broadly.

A

Excellent. Thank you.

B

uh Just I may be completely misunderstanding or missing a problem, but could this be used for sort of like nomad style scheduling where, instead of having an hpa, that has to sort of respond to a metric not being in the right threshold and then scheduling more pods?

B

Make sure that you have pods scheduled maximally across all your nodes so that if there is a certain traffic on any workload like a web workload, it is possible that it already has more.

C

B

Needs and if not, then you can evict something that does have more than it needs and then do the scale up.

C

Yeah so so I would see like I said, there are multiple in the decision and the points and in the entire and the life cycle workload. So we talked about that and when the port, the key function of a cube scheduler, is basically assigned right place the ports on the nodes. So if you are familiar with the kubernetes scheduler and the functionality, there are different algorithms plug-in there and the bin packing algorithm, neither spreading the ports right or more best defeat, consolidation with other constraints.

C

All this can be under configurable by default, for example, the coupe schedule just try to spreading and the ports across all the different nodes, but you can configure it a different way. I think it's a little bit of circular to whatever and we want to do it, but the more interesting idea, I think you mentioned, and the comment will make a good point is, I think, his customer, probably care more, is about slo sla right. Can we use some high level metrics to drive this workload or report scheduling right?

C

I don't care what strategy you apply right. I want my ports can be scheduled around within one second, two seconds: yeah, that's, uh I think very interesting thing and yeah the community and we are still brainstorming, but I have to admit it's still a little bit too early stage how to have a sla, slo driven.

E

C

Yeah scheduling.

E

A

I was wondering.

E

If this was compatible with clusters with the cluster, auto scaler or auto scaling groups enabled um technically, in that scenario, shouldn't a cluster have like.

A

E

Resources, this seems more applicable for clusters with like a fixed set of resources.

C

So in terms of the compatibility again so the elastic quarter and the capacity sketch and pieces like I said, yeah it's purely and compatible with any upstream and kubernetes code and just met. He rebuilt the the schedule with the plugin and defined the cid the job queue and is a separate concept.

C

We try to make it more extensible like and you just provide, or the project will provide different extensions control, different workload so overall and I would see- and we try to make it and independent or agnostic to write what the application levels workloads or what and your workload management system. So by providing this standard api for job q by building this preemption and elastic quarter into the native in a native kubernetes way, but again yeah.

C

If there are any suggestions or concerns or some features, and you feel and we should have or desirable, and please yeah check out code or submit the issues and, of course, more and yeah. Hopefully you can help us or contribute to the project and code.

A

All right well, thank you very much man. We were about time here, so wonderful presentation.

C

Thank you very much. I wish you have a wonderful rest of events.