KubeVirt KubeVirt Summit, 24 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CPU Pinning with custom policies

Description

KubeVirt supports CPU pinning via the Kubernetes CPU Manager. However there are a few gaps with achieving CPU pinning only via CPU Manager: It supports only static policy and doesn’t allow for custom pinning. It supports only Guaranteed QoS class.

The insistence by CPU Manager to keep a shared pool means that it is impossible to overcommit in a way that allows all CPUs to be bound to guest CPUs. It provides a best-effort allocation of CPUs belonging to a socket and physical core. In such cases it is susceptible to corner cases and might lead to fragmentation. That is, Kubernetes keeps us from deploying VMs as densely as we can without Kubernetes.

An important requirement for us is to do away with the shared pool and let kubelet and containers that do not require dedicated placement to use any CPU, just as system processes do. Moreover, system services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs. The exclusivity offered by the CPU Manager only extends to other pods.

In this session we’d like to discuss the workarounds we use for supporting a custom CPU pinning using a dedicated CPU device plugin and integrating it with KubeVirt and discuss use cases.

Presenters:
- Sowmya Seetharaman, NVIDIA, github: sseetharaman6
- Dhanya Bhat, NVIDIA, github: dbbhat

A

With uh sonia dania all yours, thank you thanks beth hi, I'm samya and I'm from nvidia.

B

uh Hi everyone- this is dhanya and I'm an engineer with nvidia 2.

A

And in this session we'd like to talk about how we are implementing custom, cpu painting and integrating it with kubert, so um firstly, what is cpu pinning and what are really the benefits that you see.

A

um So if you launch a codeword vm, the word launcher pod, as well as the kvm vm, that's running inside the board are both are floating across all the cpus that are available on the underlying physical node. So let's say my node has 16 logical cores or cpus. That means my word launcher pod, as well as the vm are going to be floating across.

A

Cpu sets 0 to 15.. This means at any point of time um the workload that's running in the vm could be switched to a different cpu core, so it's not bound to a specific cpu on the node.

A

Many workloads are not really sensitive to this type of switching and you know not being bound to a specific core, but there are a lot of workloads like hpc workloads, for instance, that are sensitive to this they're, very sensitive to context, switches or you know, sensitive to cross-socket memory traffic, and they may also have requirements for optimizing performance, uh like you know, allocate having both I hyper threads from the same physical core allocated to them.

A

So uh the main uh purpose of cpu pinning is to boost performance, wherein you pin uh the vcpus in the vm to specific uh cpus on the underlying that way, they are always bound to the cpu and they're. Not you know, switching to different cores through the life cycle of the vm um kubert does actually support pinning um as long as you're using cpu manager framework but um cpu manager, framework, the be all and end. All of you know pinning there are certain gaps that are not addressed, at least in the current version of it.

A

uh Firstly, just supports two policies: one is a static policy and one is a non none policy. None is basically a new op so effectively or you have just one policy and it's not easily extensible. So let's say I want to implement a pinging based on my specific hardware, and you know my specific hardware requirements. It's not just plug and play. I can't just add a new policy and integrate it. So the framework is pretty. You know uh it's it's with a tablet.

A

So if you want to introduce any custom changes, then you maintain your own fork of cubelet and that's not easy to do. Secondly, one of the requirements for cpu pinning with cpu managers that should always be of guaranteed us class. What that means is your requests. Cpu requests must strictly match with your cpu limits and you should always request only whole number of cpus.

A

uh This means I can never use pinning for say pods that are of best effort, qs class or burstable qos class, um the, but this is probably the most uh problematic uh or the biggest gap. In my opinion, the way cpu manager functions is that um of all the cpus that are available on a node there's some reserved for cubelet and there's some uh reserved system processes, and things like that. So if I have a node with 16 cpus, a 100 millicode could be reserved for cube.

A

Led could make other reservations, so the rest of the cpu, which might be something like 15 900, would be put in a shared pool. So if you have a incoming request that is requesting, for uh you know, that's of qos uh guaranteed qs class and that's requesting for pinning, then the those cpu sets are removed from the shared pool and put in an exclusive pool and that pod is guaranteed these exclusive cpus for the entire lifetime of that port and there's no other system process or no other port.

A

That can, you know, run on these cpus. So any pod um that doesn't have this exclusive cpu request would be running from the shared port. So this means it's theoretically never possible. For me to you, know, bind all the underlying cpus that are available on my mode to my vms. So if I say 16 cpus, I can never create vms that are able to use all the 16 cpus because of this shared pool and because of cubelet reservation, and things like that.

A

So um I'm not able to densely pack vms uh with cpu manager, as I can without it, so that that's the biggest bottleneck that we were experiencing um and it's also best effort based. So it's going to always try to allocate from the same socket from the same pneuma uh as much as possible. But if it can't it's just going to do a best effort allocation and that's going to lead to fragmentation.

A

So that's where we started experimenting with custom, pinning and uh really our requirements where you know being able to pin all the available cpus on the node uh to the guest vms. So um that means allowing all the other pods um cubelet uh system processes cri to float across all the cpus on the node.

A

So the guest can be pinned to exclusive cp, specific cpus on the node, but other processes can float across all the cpus and one requirement that we have is that we don't really need exclusive access to the cpus that they are bound to. So, if my vm is say pinned to cpu, 0 and 1, it would continue to remain pinned to these cpus for the lifetime of the vm.

A

But that does not stop other processes from stealing time from cpu, zero and one so that that's one of the that's what we're trying to achieve, and secondly, we want to be able to uh you know, support a lot of policies. You know if I have a new hardware- and I want to have a very specific uh optimization for it, then I can implement it as my own custom pinning policy.

A

uh Thirdly, I want to be able to set vcpu pins uh in in my vm. That means I want to pin the uh my vcpus and the vm to specific pcpus on the node as well as pin all the threads that are associated with the vm, so that would be like chemo kvm threads. I o threads worker threads so on and so forth and yeah. I also want to be able to do pinning with any qos class.

A

So if I'm going to spin up a word launch report of best effort class, I want to be able to use spinning in that part and not just for guaranteed qs class.

A

So um how are we going about implementing this? So it is actually pretty straightforward. um uh Kubernetes has a device plug-in framework. uh In fact, cooper uses device plug-in framework too, um and this allows you to you know: advertise your resources as extended resources on the node. um The main advantages of using device plug-in framework is that it integrates very well with topology manager and also other device plugins.

A

So if you want to like allocate devices based on affinity- or you know- want to be able to support different topology manager policies, then it's very easy to do so um so device plugin basically runs as a daemon set on every node in the cluster, and it's going to discover the underlying cpu topology of the node and it's going to advertise the available physical course on a node as extended resources.

A

So one of the nice things about the device plug-in framework is um starting with kubernetes version, 1.19 and upwards. It has this preferred allocation method that support it. This means you can use this preferred allocation method to implement your own uh custom algorithm for allocation.

A

uh Without this, the devices would just be allocated at random without any algorithm deciding which device should be allocated, but because of this preferred allocation I can you know, create a number of allocation policies and implement a unique algorithm of which device needs to be picked based on that policy, and uh so this device plugin is mutually exclusive from cpu manager.

A

So we don't it is you know, managing pinning on the same node, so this device plugin will not run on a node where cpu manager is running and the best part is it's very easy to integrate with codeword cooper, it's already integrated with things like gpu device, plugin, sriv device plugin, and it's very straightforward to you know, hook it up with covert, um I think now, dania will walk over. um You know how the api looks like and things like that.

B

Yeah thanks samya, so uh in order to in order for custom cpu opening to work, there are some uh dependencies and assumptions, so one of them is that the kubernetes version should be 1.19 or higher, uh which is required for the preferred allocation like some I just spoke about so without this you can't uh implement your own custom policies and so on, and the next requirement is that the covert version should be 0.24 or greater and hyper threading should be enabled on the node.

B

So this is done by default, uh but there is a check to ensure that it's always in place. uh Cpu manager should be disabled or the policy should be set to none which is essentially a lower. This is because cpu manager and this custom cpu binning is mutually exclusive and we can't use both of them. At the same time,.

B

Yeah so uh yeah, so there were some changes that were uh needed to be made on the cooper side uh to support this custom.

B

Cpu penny um one of them is that uh there is the new feature, gate called custom, cpu placement, which needs to be enabled on the cooper's side um and then in the vmi spec that the user requests there is a boolean value called custom, cpu placement, which needs to be set to true in order to support requesting for cpus, and then there is also an option to specify the cache level or mode in the vmi spec, and this entire thing can be abstracted using vmi presets, as we see on the next slide.

B

uh Next slide piece yeah. So in this uh this is an example p set where you can see that in the under the spec domain, uh the custom cpu placement boolean is set to true and then the cache mode is set to pass through and then- and there are number of cores and threads and sockets being requested.

B

B

Okay, so there's also an admission web book on the world api side and cooper uh which validates the incoming bmi for request correctness. So um it rejects the certain bmi requests that don't follow certain rules, like suppose uh the cpu manager feature gate is enabled. Then it rejects the request, because cpu manager and custom cpu pinning cannot be used at the same time.

B

And then, if the custom, cpu placement feature, gate is not enabled or the dedicated cpu placement value is set to true or- and it also checks if the number of cores and threads in the request is even so, it should be even in order to prevent uh fragmentation of hyper threads.

B

So it rejects the request, if it's odd and if the number of cores and sockets and threads are not explicitly specified, it's assumed to be one and the admission by book also validates that the user is not requesting cpus under resources, requests or limits and is only requesting them under the spec domain. Cpu part of the spec.

B

Regarding when to use what so like samya mentioned, there's a concept of share pool with the cpu manager uh which in which means that you can't use all cpus because of this because of the shared pool and uh since there is a requirement of spinning all cpus to the guest vm and the guest vm will be pinned to specific cpus and then other other processes can float across all of the cpus and there's no need for exclusive access to cpus with the custom, cpu placement.

B

So in such use cases uh we will make use of the custom cpu placement and not the cpu manager uh yeah. I think now someone will go over the flow and then do a quick demo.

A

So this is the control flow of um how what happens when there's a vmi request with a custom cpu placement set to true. So the request first hits the api server and then it's. uh It goes to the word api, which is going to validate the request and choose whether to admit it or reject it. So if it passed all the checks that we were talking about earlier, the request is admitted and what controller then will translate the vmi spec into a pod spec.

A

So what it's essentially doing is um it's converting the boolean that you set in the vmi custom cpu placement through into the corresponding, uh extended resources that are exposed by the device plugin, and that would be added as requests and limits to your word launcher pod spec, once that's done, the scheduler picks a node where this launcher port could run, and it's scheduled and um cubelet on that node then talks to the device plugin using the device manager framework and it's requesting for allocation of these extended resources so inside the queue the cpu device.

A

Plugin is basically running one or more allocation policies and, depending on what what allocation request was made, it's going to send an allocate response with the appropriate cpus returned and cubelet then sets these cpus as an environment variable on the compute container in the word launcher board. So once the word launcher pods starts to run.

A

You can query these environment variable variables and set it as part of, and you can even run some validations to make sure that you got the appropriate number of cpus, and uh you know you can. If, if you have some special requirements, say they have to be threat, siblings or they're from the same uh code? Things like that you can add whatever checks you need to validate that the response that was returned by uh the device plugin was indeed right and matches your request.

A

Once you do so, you can um set these uh cpu cpus, as uh the vcpu pin in your cpu tune element in the dom xml. Once you do so, then you can start running the domain and when your vm comes up, it would have the pins set on each of the vcpus.

A

So I'll show you a quick demo so that this is all a bit clearer and I've just talked about this in detail in the next slide, but I think a demo would make a lot of sense. Are you able to see my terminal.

B

uh I can't see it yet somehow.

A

Okay, uh I think I have to go back and change the sharing settings. Sorry.

A

A

Window, how about now.

B

Yes, we can see it now.

A

Okay, okay, so, um firstly let me show you uh what's running in my cluster, so I have three nodes and I have deployed my cpu device plugin um as a daemon set, so it's running on all three nodes. um So if I do a describe node and query, what are the allocatable resources and allocated resources, um this would. This is the resource that is exposed by my device plugin.

A

So it's called nvidia.com course and socket, um uh and an interesting thing to note is these two: the first two nodes are single socket systems, so the device plug-in is exposing just one extended resource, but my third node is a dual socket system, so it's exposing two device uh resources. So it's running two device plugins for each of the sockets, so it's exposing it this way. So let me first show you an example of what preset I'm using and what vm I am trying to launch.

A

So I have created this preset, um that's requesting for eight vcpus and all split, as uh uh you know, from four cores and two threads and um I've set my custom cpu placement to be true and um just memories and other devices that I need and I've set a label. And now I have a vmi.

A

That's just going to you know call on this preset. So it's it's just using this preset that I showed earlier, but otherwise it's a pretty simple vmi spec, just using a default bridge interface and a pvc and a cloud in a volume.

A

So um I'm going to launch a vm so, like I said it's using this preset, I'm creating the persistent volume and then I'm spinning up the vmi, just waiting for the vm to get the running state and then we're going to see a bunch. I mean query a bunch of things on the word launcher board, as well as the.

A

A

3Ms got into running state.

A

So this is my vm um it's running on one of my nodes that has a single socket system, because I requested for single socket and if I describe my word launcher pod, you can see that in the limits, requests and limits, uh this extended resource has been uh requested. For so I'm asking my vm uh preset wants eight vcpus in total, but I'm requesting only for four course, because I'm requesting for both the hyper threads to be allocated from the same code. So there are two hyper threads per course.

A

So uh you know four cores actually translates to eight uh logical codes, so this is the physical code. If you look at the request, it's very interesting that the cpu request is just 100 millicourse. This is the default cpu request. If I were to be using dedicated cpu placement, this would look like you know. Eight limits and um request would be cpu eight, because uh it's going to have a request for exclusive access for eight cpus on that, so that this is one important thing to note.

A

Instead of setting it as cpus, we are just setting it as a different resource, which is what is exposed by my device plugin. um And if I query my word launcher pod. uh This is the environment variable that has been set by my device plug-in. So it's returning eight uh cpus in a very specific order. um The order is that uh you know uh cpu. The allocation is always in thread sibling pairs, so zero and eight are threats.

A

One and nine are threats, siblings, two and ten are three siblings. Three and eleven are dead siblings. So this can vary upon depending upon your use case, and you know whatever and policy you require. But for me I want my uh course physical course to be adjacent and I want uh hyper threads from the same physical code to be allocated together. So this is the you know, desired allocation that I need and that's what's being set as an environment variable by the device plugin.

A

um So if I query the cpu set running uh associated with my word launch report, you would see that it's actually floating across all the cpus on the nodes, so my node has 16 cpu, so it's floating from 0 to 15. um a dump, the dom xml of the domain vm.

A

You can see that there's been a cpu tune. Element that's been added, and this is what sets the pinning. So it's uh setting pin on vcpu0 to be on pcpu, 0, cpu, 1, on pcp, 8 and so on. So this order of pinning is what was you know dictated by the output from the device plugin? You can also see that I'm setting a cpu set um over here. What this means is all the threads uh like the camera, kvm thread, io thread worker thread associated with the vm, are all bound to these cpus.

A

It can float across these cpus. They are not bound to specific cpus. It can float across these cpu sets, but they can't run on any other cpus on the host and um I'm also you know, implicitly setting modes to host pass through and adding cash level implicitly.

A

um So if I now go and query the vm, uh you can see that the vm has eight cores and eight cpus, and you can see that they are from four cores so the order um you can see that they're from four course and on the node. If I query the allocated status and the allocatable status, you can see that it's allocated for of the extended resource for this particular vm and uh the best part is how do I validate.

A

You know that the pinging actually took uh took effect so for that I'm going to just p, grab the kmu kvm process on the node and see what are the you know allowed cpu list. So if you see over here, you can see that the these are the vcpus of the vm and they have been pinned to very specific pcpus on the node again.

A

This is the order that was provided by the device plugin and the order we set in the dom xml, and you can see that all the other threads associated with this process are floating across these cpu sets again. This is what we set as this element over here. So that's how we achieve cpu custom cpu, pinning for our use case.

A

Let's stop sharing now.

A

Does anybody have any questions or.

B

I think there's a question on the chat. Oh okay,.

A

um Do you support vm migration or okay, um so we haven't played around too much with vm migration to be honest because we've just been using bridge network, so that doesn't support migration, but um I guess it's possible to do. um I I'm not entirely sure, because we haven't experimented with it, um but it should be possible to do because you you can control the pin um because it's just another resource that is exposed on the node right.

A

So when you migrate, you would pick a node that satisfies the requirements, so it should be possible to do. I think. um Could you comment on what are the main use cases for cpu pinning without isolation?

A

So um essentially um in my use cases I have a set of nodes on which there are going to be very uh lightweight pods or processes. If any running- and my main use case is to pack those nodes with vms that can run my sensitive workloads. So I'm not going to have like elaborate demon sets or anything running on those nodes, and my exclusive purpose for those nodes is to you know, run these very highly sensitive, vm workloads.

A

So since I don't have too many other containers to care about, I don't really um it's so it's okay! If there are other processes or system demons that are stealing some cpu time, because uh again, everything is cfs quota because of how uh cpu limits work with q cubelet right. So uh at best it's going to be throttled and I I can control that by setting appropriate qos class for the other pods that are running on my node.

A

My I mainly care about optimizing all the available cpus on the node and being able to um uh you know, bind them to my uh vms guest vms. So that's my main use case to be able to use to be able to achieve pinning as well as use as much cpus available on the node as possible.

A

B

So that's just.

A

Made it uh on time for the end of the session. So thank you very, very much sonia and tanya. For this.