Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Optimized Resource Allocation in Kubernetes? Topology Manager is Here - Conor Nolan & Victor Pickard

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Optimized Resource Allocation in Kubernetes? Topology Manager is Here - Conor Nolan, Intel & Victor Pickard, Red Hat

To satisfy the resource demands of workloads, Kubernetes must be responsible for resources on the platform and the intricacies associated with them. The introduction of CPU Manager and Device Manager are two examples of K8s taking involved decisions for resource assignment. However, these two components act independently, which can lead to undesirable resource allocations most notable on a systems with multiple CPU Socket or NUMA Regions. Under the guidance of SIG node, a new component is being introduced to Kubelet called Topology Manager, which will resolve this undesirable situation by enabling optimal resource allocation. This talk will introduce Topology Manager, how it works to solve these problems as well as the reasons behind introducing such a feature.

https://sched.co/b6KQ

A

Hello, thank you for joining our talk on kubernetes topology manager. My name is victor pickard. I work at red hat. My colleague, connor nolan from intel will be, will be presenting with me today. Please type your questions in the q, a box, and we will address these at the end of the presentation.

A

Here's an outline of our discussion today we will introduce topology manager why it is desirable and useful for certain scenarios.

A

Now we can't talk about topology manager without first giving a brief overview of two essential components: cpu manager and device manager. So we will talk about these components a bit. Then we can dive into topology manager and show how this kubernetes component works with both cpu manager and device manager and health topology manager can be leveraged to maximize performance.

A

Why use topology manager? What benefit does it provide? Conor will share some testing results from our friends and colleagues at intel to help answer these questions. Are we done, is topology manager finished? Does it complete, there's tremendous interest in topology manager, with several enhancements being actively worked and investigated we'll go over and give some highlights of these activities?

A

Are you interested in learning more want to contribute? We'll have a few words on this topic as well.

A

Why numa a growing number of applications and workloads, especially in telco, 5g, machine learning, artificial intelligence and other type of workloads, often say we must be numa aligned.

A

A common example of such a workload would be a dvd based application here the application will desire to have a certain number of dedicated cpus, some chunk of huge page memory, a high-speed network interface such as sr iov virtual functions with perhaps two or more of these one for english one for ingress. All of these resources would need to be on the same numa node in order to get the best performance from a broader context. There's a desire to get the best performance possible out of the hardware.

A

How do we achieve optimal for performance with the lowest latency and high throughput? Well, we have to align the resources on the same pneuma node. These resources include cpus devices such as network gpu devices memory and possibly more to meet these objectives.

A

What is pneuma so we've mentioned numa several times. Some of you may already be familiar with. Pneuma know what it is and why we need it for those that aren't familiar with pneuma. Let's briefly, review pneuma is non-uniform memory, access. Looking at the diagram on the right, we see a two pneuma node system.

A

This system has two cpu sockets. Each cpu has local access to the directly connected memory and devices, and each cpu can certainly access all of the memory on the system. But, as you can see, for example, for cpu 0 on socket 0 to access memory on numa, node 1, it must go over the inter socket connection.

A

Accessing memory or devices over the interconnect adds delays so to get the best performance. Pod resources such as cpus and devices, should be on the same pneuma node and avoid the inter-socket connections.

A

How do we get exclusive cpus assigned to a container? We use cpu manager, cpu.

B

A

Is a kubernetes component that runs in the kubelet it allocates and assigns exclusive cpus to a container that specify a guaranteed quality of service, meaning that both requests and limits are entered your values, as shown in the sample pod spec now for cpu manager to allocate guaranteed exclusive cpus to a container.

A

The cpu manager policy must be set to static for more information about the cpu manager. We've included a link to the blog post below at the bottom of this slide.

A

So we have guaranteed exclusive cpus assigned to a container. How are devices assigned and allocated?

A

Well kubernetes has a device plug-in framework devices can use the device plug-in framework to implement plug-ins for gpus, nics, fpgas and other device resources that require vendor-specific setup and config.

A

With the device plug-in you can advertise system hardware resources to the cubelet that will be used to assign resources to the container, so we've introduced cpu manager, device plugins and how they can be utilized to allocate resources to a pod and or container.

A

But these allocations are done independently, no coordination, meaning the allocations could come from different pneuma nodes. How do we get these resources? Aligned? We use topology manager, topology manager is a component that runs inside the cubelet on each node. Topology manager was promoted to beta as of kubernetes 1.18.

A

It provides an interface to allow cpus and devices to coordinate resource assignment to pods and containers at the node level, both cpu manager and device manager support the topology manager interface with topology manager. We now have the ability to assign and allocate resources such as cpus gpus network interfaces like sriv virtual functions to a container from the same pneuma node.

A

Let's take a look at an example on the left is a simple pod spec with one container the container in this pod spec is requesting two cpus some memory and one instance of device a on the right. You see. We have a two pneuma node system, four cpus, each with two devices on each pneuma node now using topology manager in coordination with cpu management device manager, the cubelet will assign and allocate the resource for this container from the same pneuma node.

A

In this example using topology manager, you can see that we have an aligned pod with two cpus assigned from numa node zero and one device a also from the pneuma node zero. The resources for the pod are numa aligned, eliminating the overhead of the socket interconnect. Here we have optimized resource allocation.

A

Now, conor will take you through some of the details of topology manager, including policies in a workings, along with some detailed examples. Connor over to you.

C

Thanks victor, so um we've seen what topology manager is and why it's needed, so I'm going to go into some more detail on how it actually works, starting with the topology manager policies. So the chosen policy is set at node level as a cubelet flag, and there are a few different options with varying degrees of strictness.

C

The first is none which is set by default. It doesn't do any kind of resource alignment. The next is the best effort policy.

C

um This will attempt to perform some resource alignment, but the pod will always be admitted, regardless of whether that can be achieved or not, then the restricted policy is the same as the best effort policy in terms of how it actually works under the hood, but the difference being that pod admission can be failed if an optimal alignment cannot be achieved, and we've got some examples coming up to show to show you how that works, then.

C

Finally, the single luminol policy, as the name would suggest, will attempt alignment of resources for a container on a single luma node and will fail pot admission if that cannot be achieved, so how topology manager actually works under the hood. So at pod admission time the topology manager pod admit handler will loop over all containers in the pod, then for each container it will call out to the individual hint providers, which are currently the cpu manager and the device manager, as victor has spoken about and from those gather.

C

What we've termed as topology hints for each topology aware resource type. These topology hints are essentially all the possible allocations for a given resource and again these are covered in more detail in the next slide.

C

So then, once all of these hints have been accumulated, the topology manager using the selected policy will merge the gathered, hints and then select a best hint that will align all all requested resources.

C

So then we loop back over those hint providers instructing them to allocate their respective re respective resources uh using that best hint as a guide for which pneuma, node or pneuma nodes. They should allocate from then.

C

Finally, as we mentioned, this loop runs at pot admission time, so if any of these steps fail or the alignment cannot be satisfied according to the policy for any of the containers in the pod, the pot admission will fail with a topology affinity error and any of those allocations that occurred prior to the failure are cleaned up accordingly, so mentioned in the previous slide. A topology hint, uh a tv hint is a construct.

C

That's used that we use to describe how a resource request can be satisfied, so it currently consists of two fields: the pneuma node affinity. This is a bitmask of pneuma nodes, where a resource request can be satisfied, I.e which pneuma node are pneuma nodes. Then the preferred field contains a boolean that encodes, whether that given hint is preferred or not- and there are some examples we have to explain what we mean by preferred and how that can influence pot admission with respect to the given policy.

C

So if we take this example, similar to the diagram picture, showed you earlier. So if this is the underlying hardware of our of our node, uh we see over on the left, we've got numenol zero and connected to that. We have socket zero with four cpus and also connected to numeral.

C

Zero are three devices, so one gpu and two nics then over on the right across that intersocket connection, we've got socket one with the gain four cpus and three three connected devices, so two gpus and one nic- and these are all connected to new node one. So all of these resources are available for allocation and then the following part is scheduled to the node, and this pilot is requesting two exclusive cpus, one nic and one gpu.

C

So now we can actually take a look at the topology hints that are returned from the individual hemp providers, both the cpu manager and the device manager. So for the cpu resource type, we can see from the first cpu hint the one with the zero one.

C

True semantics: that a request for two cpus can be satisfied entirely on numerous zero, and this is indicated by the bitmask, which is encoded with numeral zero bit zero, and we also see that this allocation is preferred because it's the the narrowest possible placement and narrowness in this instance refers to the the least number of bit set in that bitmask.

C

So then the second cpu into the one zero true. This tells us that the request for two cpus can also be satisfied uh on pneuma node one, and this allocation is also preferred because again, it is the narrowest possible placement or equal to the narrowest placement.

C

The final third cpu hint tells us that the request can also be satisfied across both numa note, 0 and numa node 1. However, this allocation would not be preferred, as there are narrower possible placements, as we can see from the the first two hints.

C

So the same hints are also returned from the other resource types. In this particular scenario and for example, we can see that a request for one nic can be satisfied on either numero node alone, which would be preferred, are with a bit mask for both pneuma nodes, which would be unpreferred, and the same applies for the request of one gpu.

C

So now the topology manager has gathered all hints from all providers. The next step is to merge these hints, so this is ultimately done by taking a cross product of hints across all resource types and performing a bitwise and on the numerator affinities of those hints, and this gives us a new bitmask for a merged hinge, so for the preferred field in any of the in any of the hints in the cross product. If any of the hints in that cross product contain a non-preferred hint, then the merge end is also number otherwise.

C

It's set to true. um Also if the affinity of the merged hint is all zeros, we set the preferred field to false. So then, it's just a matter of iterating over all the possible permutations in those hints, and this continues right through until we have compiled a full list of our merged hints.

C

So, finally, once we have all those merchants, it's up to the policy to then choose a best hint, so for the uh for the best effort policy. In this scenario, the best hint chosen will be uh the one with the zero one. True semantics- and this is true for all: okay, all three policies- uh in other words, the best hint, is to allocate from pneumonia zero, and this is a preferred allocation.

C

So this means that the best effort, restricted and single numeral policies are all satisfied by this placement and the pod will be admitted successfully under all three policies, with the allocations shown. So all cpus and all devices are allocated from the one luma node numeral, zero, okay. So a slightly more complex example, this time again, our node has the same underlying hardware topology. However, on this occasion, gpu one on pneuma node one has already been allocated and is no longer available to us.

C

So now the following part is scheduled, and this part is one container requesting two exclusive cpus one nic and two gpus. So again, we look at what kind of hints are returned by the hint providers so for the cpu and nick resource types we see the same result. Each request can be satisfied on either pneuma node alone or across both pneuma nodes for the gpu resource type. We see a change so now. The request for two gpus cannot be satisfied on numer, node, zero or pneuma node, one alone.

C

The only hint we see tells us that the request can only be satisfied across both numerals and this allocation is not preferred.

C

The reason it's not preferred is that, although it is the narrowest possible placement for this resource request at this time, it's still possible to satisfy this request for two gpus on a single luma node, but just not right now, because, as we've seen, gpu one has already been allocated.

C

So then, if we look at the results per policy for the best effort policy, although the final best change is not preferred, this policy is all this policy always allows pod admission regardless and then uses the mer, the best hint to allocate resources on the best effort basis, as shown here, the restricted policy will always produce the same best hint as the best effort policy.

C

However, it will fail pot admission when the merged hint is not preferred, as we see here, the logic being that it's better to fail and try to reschedule than to schedule with a non-preferred alignment, and this exact scenario we have here is a good example of how the restricted policy differs from the best effort policy. Then. Finally, the single luminal policy's best hint tells us that the requested resources cannot be satisfied on a single luma node. So therefore, it will also fail pot admission.

C

So our final example again to know is the same underlying hardware. This time all resources are available to us and if we look at the pods this time the podge gets scheduled, has one container requesting two exclusive cpus, uh three nicks and three gpus, so the hints that are returned for the cpu resource type. We see the same result again. The request can be satisfied in either numeral alone are across both newer nodes for the gpu and nick resource types.

C

We now see that they both return a single hint with a bitmask of both pneuma nodes, and this allocation is preferred now. The reason this is now preferred is because it is actually the narrowest possible placement that could ever be achieved for that request.

C

For, for instance, a request for three gpus, as you can see, can only ever be satisfied across both pneuma nodes and likewise the request for three nicks. So then, for each policy we can see the result again.

C

The best effort policy will always allow part admission and use that best hint to allocate on a best effort basis, as shown here, the restricted policy, um the best hint is now preferred, so the preferred flag is set to true, and for that reason the pod is admitted with what what is now the narrowest alignment that could possibly be achieved for that request and again. Finally, we see the single luminol policy again will fail part admission, because its best hint tells us that the requested resources cannot be satisfied on a single luminor.

C

So what does all this mean for performance? Well at intel, we released some updated collateral for our container experience kits earlier this year and included in that was a technology guide on topology manager, with some information on performance, benchmarking and the link to that is included if you'd like to check it out in a bit more detail. But the key takeaway here is that workloads that have a new new alignment uh of their resources by topology manager can achieve a performance increase of over 2x versus workloads.

C

With that sub optimal allocation that victor spoke about earlier and this particular test. We did see a flat line on the performance improvements for the larger packet size once you went over 1024 bytes so, but this was just down to the the test setup and the the line rate essentially was maxed out at around 200 gigs. So we didn't get to see the performance beyond that. But again, as I said, the link to the technology guidance is included in the slides.

C

If you'd like to check that out and see what was actually done in detail for anyone who's interested in that.

C

So what the future looks like for topology managers so first thing their support for device specific constraints. uh This will be part of the 1.19 release, so this is an enhancement to the device plug-in api that allows allows device plugins to indicate a list of preferred allocations for its device.

C

So the plugin can take into account any internal topology constraints on the device. When returning this list, then this information can be incorporated into the ultimate allocation decision, so it extends it beyond uh simply just pneuma node affinity, support for pod level, resource alignment, so the topology manager currently attempts to align all resources for a single container.

C

This enhancement would extend the scope to allow alignment of all resources for all containers in a single pod and there's been a cap approved and merged, and that's linked here in the in the slides pneuma alignment for huge pages.

C

uh As mentioned earlier, the current hint providers for topology manager are the cpu manager and the device manager which provide hints for cpu and device resources naturally, but this proposal is for a new component called the memory manager which would provide hints and ultimately, new alignment for uh memory and huge pages and there's a cap under review on this uh topology aware scheduling, uh there's been lots of discussion and talk on this topic.

C

um You know, should the scheduler be pneuma aware, should this be an entry solution or an out of tree solution, etc, etc and there's a.

A

C

Of ongoing work in this area across red hat, intel, samsung, huawei and others, and there's a couple of kept included in the slides, if you again, if you'd like to learn more on that uh per pod alignment policy.

C

So, as mentioned in the on the topology manager policies, the policy is statically configured at node level at cubelet startup time, and so this enhancement would allow the user to specify a policy in the pod spec for a particular part. Now this change is something that would require an api change and that that in itself is challenging.

C

But it's something that's been discussed briefly. As of now there's no official plans are kept, but it's something that we've spoken about briefly.

C

So, finally, if you would like to find out more about topology manager, you can visit kubernetes.io to read up on it or you can read through the blog post that was released earlier this year to go inside with the 1.18 release and the graduation of the feature to beta and again, all of this work has been done under the stewardship of signature.

C

I just like to give a special mention to uh derek carr of red hat for all his help, and also a huge thanks to kevin kluse of nvidia who's done a huge amount of work in this and has been instrumental in taking topology manager to where it is today. So, if you'd like to get involved or hear more about this or any related projects, the link for the weekly signal meeting is also included.

C

So I hope you found this informative and thanks for listening and myself and victor are happy to take any questions.

B

All right, I see we have some questions on the q and a board we're working to answer those as we can are there any questions that folks would like to ask.

B

B

Okay, so we've got a list of questions. uh We'll start. um Let's start with, uh there was a question: are there memory allocation hits, and so the answer to that is currently, there are no memory allocation hints, but, as connor mentioned in the future enhancements, there is a memory manager cap that will allow for hints for both conventional memory and a huge page of memory.

C

Yeah, I see another one victor there um can topology manager reschedule already running pods to achieve best alignment with the request of new pod placement, uh so topology manager is a part of the cubeless. It's not part of the scheduling decision making um or you know if a pod can in fact be scheduled, but then fail. And so, if you have a deployment, you know if you're, if you don't get your optimal placement, it can be failed and then rescheduled.

C

um But as regards to this, I think from what I can understand from this specific question and no that's not something. That's doable in the current quality manner.

B

Yes, I I agree with that: connor can't uh currently no way to reschedule currently running, pods they'll either be admitted and be numered, or they will fail to be admitted based on the topology management policy.

B

um Next question is: uh there is a question which version of openshift will have topology manager at ga, and so the answer to that is topology manager is ga and open shift. 4.6.

B

Another question was: uh is it possible to have the slots for this presentation? So if you go to the schedule uh on the agenda, the slots are there, so you should be able to find them.

B

Okay, uh connor. You want to take the question about uh how to check the let's see. What's the question, how do you check the uh um to see if the?

B

How can we check in a running cluster if the requested resources for each pod are on the same number node? So there there are two answers.

B

One is you can, for example, on your system to check for the cpus, you can do ls cpu and you can on the multi-socket system, you'll see the cpus and the pneuma node assigned, and then you can, for example, ssh or connect to the pod and look at the cpus that are signed and connor. You also had another way to check this.

C

Yeah, so you can also check through the um the kubrick checkpoint file, um so that will list any exclusive cpu allocations to containers or devices, and it won't tell you the pneuma node, but you can you, it will tell you which cpu id it is and then you can cross-reference that with something like lscpu and there's some other things popped into my head there. Interestingly, I've seen um a kef uh in the last week or two to uh extend the pod resources, uh cube endpoints to be able to show you information such as.

A

C

Information on cpus allocated to containers and pods, and I think it's very early days, obviously, if there's only kept involved, but um that would be something that would be very useful. I think going forward.

B

Okay, next question: would it be possible for you guys to develop a scanning feature to check how much we could improve performance if we use topology manager?

B

I'm not quite sure what you mean by scanning feature, but you know what what has been done is connor and his team at intel have have done some testing and the results are uh posted and you know shared in some of the improvements that were that were there, I'm not quite sure what the what the scanning feature would be.

B

um So I think uh I don't know if you can rephrase the question. Maybe that would help connor. Do you have any input on that? One.

C

No just that and again there's a link to the white paper on that included in the slides and you'll want to check it out in a bit more detail, probably all for a bit more than what was mentioned in the in the presentation. um I'm just looking. There's one really long question in there um describe the setup. uh Take.

B

C

Few minutes to get through it and whoever asked that, maybe you could sync with us afterwards, um I I will be heading over to the the intel booth. I think there's a zoom room set up for any q and a's people have afterwards and also the slack channel will be open. If you want to carry on that discussion.

B

Okay next question: if I am using host network, if I am using host network with node with multiple interfaces, can I use topology manager to allocate interfaces to a pod? So um you know again: topology manager requires a hint provider for things like cpus and devices. So right now to use multiple interfaces in kubernetes we have, it requires um multis and so with multus. It does support having multiple srov interfaces and we can align sr, for example, srv virtual functions.

B

So if you have, if you want to use multiple interfaces, one way that is supported is to use the srav device plugin, and you can get high speed interfaces with that. Now the question about foundation host network- uh I'm not quite sure about that one um I'd- have to follow up with you offline, maybe or in slack, to get more info on that um tom. Do you have any additional input on that? One.

C

uh No sorry, I was just reading through another one. um This one there is the support for private and public clouds, um so financial manager was alpha for kubernetes 1.16 and base as of 1.18, um and it's it's integrated into the cubelet.

B

Okay, are there any questions that we haven't covered? I'm looking to see.

B

I think have we addressed them all: okay,.

B

I think that's all the questions that um I'm just going through reading these okay. So if you want to, uh you know, continue the conversation, you have more questions.

B

Take a look at the um channel number two kubecon custom extend k8 on your slack workspace after the sessions ends, we'll be there uh to answer any questions, and you know chat with you and thank you, for you know, attending feel free to uh you know reach us over there and as connor mentioned uh connor, I think you said there was a intel room. You want to share some more on that again.

C

Yeah I'd be heading over to the intel booth now to join. um I think there's a zoom room set up for any q. A people might have after the talk, so if anyone has any follow-ups feel free to head over there and yeah thanks for joining hope, you found it informative.