Cloud Native Computing Foundation Kubernetes Batch + HPC Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Get More Computing Power by Helping the OS Scheduler - Antti Kervinen, Intel & Alexander Kanevskiy

Description

Get More Computing Power by Helping the OS Scheduler - Antti Kervinen, Intel & Alexander Kanevskiy, Intel

When Linux schedules a thread on a CPU core, there is no guarantee which memories the thread will access. If the workload is lucky, the thread will use data that is already in CPU caches or in a memory that is close to the CPU core. But if not, millions of memory operations need to travel a longer way to reach the physical memory. Yet this may sound too low-level to be controllable and make a difference, you can easily help the scheduler running Kubernetes workloads, and make a big difference! Antti and Sasha show how to get a lot more computing power out of your CPUs by adding CRI Resource Manager (CRI-RM) on your Kubernetes nodes. CRI-RM affects process scheduling and memory locality by dynamically managing CPU and memory pinning of all Kubernetes containers on the node. In case studies CRI-RM has given major improvements in database and AI training performances without any workload-specific configurations or changes to upstream Kubernetes components.

A

So we are here to tell you about how to get more computing power from your hardware just by helping the operating system scheduler on your nodes, my name is antigeravid and I'm a cloud orchestration software engineer working at intel and now I'm going to let sasha continue.

B

From this and I'm alexander kaniewski, but less formally is sasha, I'm principal engineer of intel. We are working in the same team focusing on resource management, a lot of activities across the kubernetes, but today we are going to talk about like web performance and how it affects an app application and, first of all, let me be a bit of a bad cop, so you heard already like swati and francesca today, so I'm going to put a bit more darker colors to to what to what you already heard.

B

I know you seen this picture like that about like normal problem. Several times you have data dispersed, you have container dispersed across your whole machine, uh not working properly, so everything what we've seen so far today mentions: let's do the pneuma scheduling normal affinity, that's a usual scenario: let's pin the containers, let's allocate the data close to the cpus.

B

We will get something better true, but it's just a tip of iceberg right now in kubernetes it works for guaranteed containers for burstable for best support. It doesn't work. If you want to improve it, we need to do some better solution.

B

Same thing goes to containers, which is communicating to each other right now. Topology manager and the rest of herd of managers were focused on one single pot. Yes, we can handle several containers with input, but what happened? If you have multipod application, let's say front end plus your memcached and where communicated may see each other, you need to have affinity, you need to say somehow to kubernetes what those two will be working together.

B

This allows to hardware to do a bit more efficient scattering and data transfers. You get a common l3 caches, you get optimizations on the memory channels and so on. Kubernetes right now doesn't have it. We have it on the cluster level. On the scheduler side, we don't have it on the runtime site when they are running with things.

B

Another thing again how we from other users we can help with hardware and os to give you a bit more performance right now, most of the time when you run application, it creates a lot of threads. It consumes all web available, cpu cores and so what it means in reality, in the hardware it means what those scores are. Not sleeping those scores run like certain period of time when sleep, if we can somehow to instruct the kernel what those containers doesn't really actually need the whole set.

B

Of course, you don't need to run everything on every single available, cpu core group. It's grouped into a smaller pieces.

B

It will allow your kernel and your hardware to bump the turbo frequencies, so your workload, most probably will finish earlier, will benefit with the higher frequencies and, as a consequence, you will get a better like power savings, because not all the cores are running, and all of this is just like a keep of iceberg.

B

If you are starting to think how these pieces of hardware can be exposed to a end user and as we saw from uh francesca and zwaike our usual suspects kubernetes how to implement all of this scheduler how to get all of this information, and then last but not least, is user experience how to end user to express what he actually needs for his workload, and there are a bunch of problems with that.

B

So like in restaurant, you don't tell the chef which knife to use to cut the meat you just order with your steak, and you say I want rare medium or something same here. We need to have a way to express what is important for you and how to do it, a problem how we implement that. So, if we go to kubernetes side, we cannot implement all possible hardware. So the pictures, what you've seen it's for hardware, which was uh actual like five six ten years ago. The current hardware is not like that.

B

The upcoming hardware will be even more different in terms of what we considered as hardware resource zones.

B

Couplet should work should be simple, should be working everywhere, the same for the scheduler. Yes, we can try to expose with information to a scheduler, but we need to understand the cost. So the more algorithms we are exposing the bigger delays. We will have invested our side. You have a heterogeneous nodes.

B

You have more algorithms, your scheduler needs to be aware of. You want to optimize not for the hardware but wants to optimize, for, for example, for the few groups or pool groups of of your workloads or badge jobs. As we've seen in volcano presentation, you need to have another policies, so all of this in current kubernetes, it's simplified.

B

We are trying to find the way how to do it in extensive way and more user-friendly.

A

Thanks so as social emphasized that there are problems and we, what we need really is the flexibility to be able to manage the jobs.

A

We add to the whole framework, this kind of flexibility and there there are now a couple of options, so so that if we do that on the cubelet level there there is this herd of managers, the topology manager, cpu manager, memory manager, so those would meet need to be like pluggable, so that resource management algorithms could uh make their way into the kubernetes itself.

A

But there is also another option. So if we go one layer below so, if we go to the container runtime level there, we can do something something smarter. So we can forget that maybe we don't need to go through that door. Maybe we can jump out of that box in some other way, and this is what the cri resource manager is actually about.

A

So, regarding the features of this cri resource manager, we provide zero configuration, cpu and memory, pinning meaning that, if you just add the cri resource manager to your nodes in your cluster, you don't need to do actually anything else. And this year resource manager is starting to manage your jobs and like placing them to the cpus and memories so that they really give you the performance benefits. And here on the slides. We have the links to some material.

A

This is which is actually now reporting that what kind of performance benefits you we have observed and, on the other hand, as you saw these kind of affinities that were needed in the to solve these problems, that if two containers are uh tightly bound to each other, are communicating with each other a lot. So it makes them sense to run them very close to each other, even inside the node, so not only like, as we saw earlier nodes communicating with each other.

A

But now we are talking in the processor level, so put them to the same numa node so that they are using the same memory. Themes that are very close to that uh processors.

A

So for that purpose there is a way for users to add this kind of content, affinity, information and in testing to that uh cri resource manager also allows defining quite custom resource management policies, so there are pretty easy apis for creating your own resource alignment and management- algorithms, for instance, just to mention a couple of those so there's a port boost algorithm that allows you to say that I want like sets of three pots to run on wide cpus, say like four cpus and they share the cpus among each other.

A

So this was something that one of our customers needed, and that was that is implementable in a in a matter of hours or in a matter of days using cri resource manager, and you don't have to tell anything to kubernetes about that.

A

So another example is the balloons resource management policy, where we can pin containers to to cpus into the pools that are actually like growing inflating and deflating, depending on the needs resource needs of that those pods. So this is dynamic and you can also dynamically adjust cpu frequencies of the cpus inside the pod.

A

So how this all works, where it really goes, the cries host manager. So this is the how how the things are now, typically in kubernetes node. So there is a cubelet and it is communicating with a content. Runtime often container d can be also a cryo, and then uh what this means today that, if you want to add cri source manager into your node, then the cubelet starts communicating to it.

A

Instead of the content, runtime and here, resource manager is doing the cpu, pinning memory pinning and running these algorithms and then telling under the content or runtime, which is still your favorite content around time, but how to how to really pin those cpus and manage the resources.

A

But this is the where we are today, and this is where we really would like to be safe this year still, so that cubelet would be again communicating directly to the content around time and content around times would have a in their eye, which is like node resource interface, and that, through that, you could use cri resource manager.

A

What's the benefit on this compared to this, where we are today is that you can actually apply, and you can deploy cri resource manager to your comparing this node directly and you don't have to go to the node level and configure this kind of cubelet container runtime stack.

A

B

So, just as I mentioned, if we start to look at the hardware problems and how to expose it, it will be just a tip of iceberg, and actually, where is just our project, it started as a demo vehicle, but right now it's in a shape what it can be used in production.

B

But what was the thing to showcase? What how different policy, how house, with different variants of resources, can be exposed in user-friendly manner to applications beside that we have in sierra irm, implementation of last level, cache control memory, bandwidth, block, io and so on. What we are trying to do is all of those small pieces. What we have is going to appropriate places in the upstream so for block io for cache controls and memory bandwidth controls those patches just recently got merged into cryo and container d.

B

Already today, you can start using web functionality using annotations. We have a cap which is talking about the class-based resources, so cache block io is one example of those class-based resources. Another example can be like different memory types, so you can have a workload which says I want high bandwidth memory. I want combination of drum plus, slower type, tier of memory and so on, which is currently not possible, but hopefully will be done at some future.

B

We have uh ways how to use accelerator more properly, so our team, well, in addition to nvidia practically like, was for only two publicly available device plugins for for kubernetes. So we know how what is the problems uh in to expose the accelerators to kubernetes?

B

uh Together with nvidia, we implemented a cdi interface, its container device interface on well on the runtime level, how to expose the devices and now in kubernetes we have this dynamic resource allocation cap. This is about how you can fully control your accelerator devices, so you are not anymore requesting. I want just one gpu, you can say I want gpu of particular class. I want gpu with particular interconnect.

B

I want gpu with particular amount of memory and so on and so forth, and this is like iterative changes to existing kubernetes further down the line. We must. Probably, we will need to think about more um bigger changes to away how kubernetes working.

B

So, as we mentioned, the cubelet is already complex, hard to maintain uh and not really universal, to make it more simpler, but to have this good separation of concern between what and how we will definitely need to involve the cri protocol, we will need to figure out how information about product and containers communicated to to runtimes.

B

We need to understand how the resources are discovered and reported to the nodes. So we think what was presented as uh like separate topology exporter. It can be a part of a future cri protocol with things like the errors of admission of reports and rescheduling them or avoiding scheduling errors. It's again, it's something what we need to investigate, how we will feed it in the future and well, what's with what we had for today. Thank you.

B

I think we are right on the time uh questions. I don't think we have time right, so we probably.

C

Break so maybe one minute.

B

C

um Do you think that the pod spec? Is it good enough to express these? Do we have any problems at that level? Like the you know, do we need anything extra in the prospect express? I want these two gpus and same or you think that is not where the problem is.

B

So in pot specs we have several problems which is not solved and which, which is actually some of those caps are trying to to solve. So if we are saying I want two gpus, and I want this particular parameters with the current resource model, where gpus are represented as extended resources, it will not fit completely so we learn it from storage to have claim of those gpus.

B

So you get separate objects which describes. I I want to claim. I want to allocate me this amount of gpus with web properties and when the pod spec will be referencing. This object, saying I will be using that that's one side. So if we're talking about the classes, uh we don't have anything right now.

B

So what we're proposing is to have on the pod level and on a container level field resources, classes and when you have key value pairs, what you have and the current proof of concept implementation includes practically everything like the whole chain from discovery, runtimes, scheduler, resource quarters and so on. But where linked cap is simple, part is just like the first part like what is in report spec. What is in the cri protocol.

B

Regarding affinities, so right now in our pod spec we have node affinity constraints and you have required during scheduling, ignored in run time preferred, and so we need the same. But for runtime side. So I don't know. Will we start with exposing that on the scheduler side, but on one runtime side we need to have this similar fields required on runtime, preferred on runtime.

B

A bit complication compared to what we have with node affinity, is what like node affinity constraints. We usually work on the um scope of pod, so we are saying I want this spot if you needed a wet part or anti-affinity here, it's a bit more complicated. You want to say I want my front end with memcache d together, but you might have also more complex affinities. Let's say I have my database and I have my logger or backup container.

B

I want to be running this back low priority container in some every area in some our cpu pool. So you need to have way to express also with input container affinity and dfinity, so the syntax will be most probably a bit more complex.

B

Thank you so much thanks.