Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Maximizing Workload's Performance With Smarter Runtimes - Krisztian Litkey & Alexander Kanevskiy

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Maximizing Workload's Performance With Smarter Runtimes - Krisztian Litkey & Alexander Kanevskiy, Intel

Mitigating noisy neighbours in the world of containers is not an easy task. There are several solutions exists and many of those have own limitations. This presentation will be focusing on exploring new ways of innovations for container runtimes that helps get maximum performance and resource utilisation without modifications of the workloads. In this talk we are planning to briefly cover existing extension points for containerd and CRI-O, talk about new ideas from NRI proposal, as well as covering evolution of dynamic resource usage optimisations in our project CRI-Resource-Manager. We want to share our experience on dealing with heterogenous CPU resources, multi-tiered Memory, Caches, Memory Bandwidth and Block I/O usage. We want to demonstrate how using various metrics and hints provided by Linux kernel can lead to improvements of workload performance and dynamic hardware resource utilisation optimisations.

A

Hello kubecon welcome to our session, I'm alexander kaninsky, I'm cloud software architect working for intel and today with my colleague, we are going to talk about something interesting in the run times track.

B

Hello cubecon, my name is christian litke. I work for intel finland. As a linux and cloud software engineer, maximizing workloads, performance with smarter runtimes.

B

We will take a look at performance, optimization how kubernetes models, hardware, detour into hardware domains and then talk about smart runtimes.

A

So let me first start with achieving my performance. Optimization comes down to a very known problem of noisy neighbors. We all have dozens of hundreds of workloads running now and and all of those workloads. They have a different characteristics, different life cycles and so on. So one thing which I want to say in the very beginning, the silver moon doesn't exist, even though, if you have some good algorithms to optimize one set of workloads, most probably for ourselves, it will have differences, and the reason for that is what like.

A

We all have differences in the cpus caches memory and our hardware resources when usage of patterns of those resources also differs.

A

What is important is what you need to understand how those resources are used, how you can measure them and how you can react in those events of the life cycle of those workloads today, in our scope, is just a container runtimes dryer container b and some interfaces waters connecting roads, pieces together and even though here I'm talking about theory of things, we we have a project which is called cri resource manager where we validated most of our materials theories, and we have practical examples of what can be achieved by extending the runtimes.

A

But let's talk about how the resources are recognized, how we can look at them in our kubernetes world, where pristine and vanilla installation of kubernetes you have a very simplified model of resources. Everything is one big share pool. Every cpu is record. Every memory region is again one big shared set of resources, and obviously, in this setup you are not able to provide too much of your controls.

A

You have some priorities, but that's it to fully embrace our control. We need to start dividing those resources into a smaller challenge, so we can, when you can measure and control one and when people are thinking about the divisions of standard computer resources, where you immediately came to the idea of no one and this current implementation of aluminum kubernetes, you still have one big set of shared resources for cpus. You still have one big shared set of memory uh regions.

A

You have possibility to do some exclusive allocations, but it's still limited, and even if we look at the newer development in white area, you have some improvements. You can select multiple memory regions, but still this memory regions will be shared resource between the exclusive and shared allocations of workloads.

A

So when we are talking about new one, it might be not enough uh for a boundary of division of resources on the system and for reason for that we need to look more deeper in the history of human.

A

Two decades ago we had a setup of a system where you have multiple cpus single memory controller. You had a memory bus and when all of those cpus were accessing globally, that was called unifor. Uniform memory, access or symmetric multi-processing.

A

However, this setup was not really scalable and when the architecture evolved to non-uniform memory, access what it means, it means what you have groups. Of course you have independent memory controllers, you have independent set of memories and you have interconnect between those groups.

A

So each of this group is becoming uh norma domain and you can have multiple of those domains and you can have multiple different links between them, not necessarily like a full mesh interconnect but uh different sizes, different connections, and that is really model or system dependent.

A

That's why this idea of non-new format is coming from and when you're optimizing, we need to understand.

A

However, access between the memory regions is actually affecting your performance, but even if we double click on your memory nowadays, when memory became also heterogeneous resource, we all know about a standard memory era, but when we have already other types of memory, for example, obtain persistent memory, it's still connected to the same memory circuits, it's still dim interface.

A

However, even though it's seen as a normal system memory, it has a bit different performance characteristics and to fully utilize it for applications. It would be good to understand those differences, but that's not our end of story.

A

Very soon we will have another bus, which is called compute express, link and one of the profiles for this bus is a memory extension and by memory. It means what we're going to have a possibility to attach additional memory uh to the controllers, not to the two nodes, and that's not going to be like new phone type of memory. You can have different types of memory, give us different performance characteristics.

A

This memory can be hot plugged and so on. So all of these different memory regions will be seen in linux as a separate, newer memory, nodes.

A

So if we double click to cpu box, we can see what, like modern cpus, have multiple physical cores, why it might be different by the frequencies, but it can be also our differences, for example, with hyperturrets.

A

So you can have multiple hyperterrors per physical core and those hyper threads are sharing the same resource on the low level, meaning one until two caches.

A

So one of the division boundaries can be also physical, core versus uh one one. Hypertron.

A

We have alpha cash or last level cash before you reach for memory. This already becomes a shared resource between multiple cores within the same socket or die, and architecture is not something. What is studied.

A

uh The architecture of cpus are evolving and in new versions for new generations, you might have an absolutely different split of resources like water score, like what is the l3 cash domain and so on, and that actually also goes to even more deeper separation of things. So you can have the means splitted, physically or logically. So, for example, you can have multiple dice per one physical package or you might have a logical, pseudonumber clustering within one physical package, so based on red.

A

We need to understand what linux and actual hardware looks like in terms of hierarchy, so you have physical components like sockets or packages in linux terms, you have dies for multi-ship cpus, and then you have inside the cpu cores, and those cpu cores is practically can be grouped into same some pool or some some zone of resources which contains cpu caches memory, io and so on.

A

You have possibility to have inside this small group shared set of resources and exclusively allocated set of resources, and when you can mount your workloads into these small groups, my next problem or next question will be, of course, like what. If I have a workload which is bigger one with small group. Well, you still have ability to group on the higher level, so you can have a parent nodes or parent virtual groups uh which spawns across like multiple dice into one socket and one.

A

You can have a proper accounting for all of those which is practically like some of the resources of all the child nodes.

A

So to reiterate that when we are talking about resource zones where workloads about it's good to have a good understanding of what your physical layout or system looks like, and that might be a lot different based on generations based on the vendors and so on, because in different generations you have differences in how memory series are being organized and how what caches are divided, how io buses are connected?

A

You can have different types of memory, even though we will be connected to the same memory. Controller for linux, it will be visible as separate numbers, but what we actually want to understand as well. How will separate reported by the linux controller by the linux kernel? Numa numbers can be grouped into one logical and usable resource zone.

A

We have possibility at a moment to say which zone can be used, but what we don't have- and this is something what I hope will be improved in the next years of the linux- is the ability to control how much memory or how much of our resources can be used per zone for application. And when we have even more harder tasks to understand and fix, is how to understand.

A

Where lower details of a workload. For example, the working set size to have a good split of understanding.

A

What is where what size of a memory used by application versions were called area of for pages to use by application.

A

So going first, we have caches. We can at one moment split one we can assign workloads to the caches. However, we have a problem where first of all, the amount of cash classes is limited, so you can't assign each of your container to separate plus. Second, the configuration of the caches is something what is really hardwired.

A

We need to come up with a good abstraction and we think what class is one for good abstractions, what we can use in kubernetes or in cncf world, meaning we have a classes example like gold, silver, bronze and when we have a node level, mapping between those classes to the actual parameters of a hardware, so simplified ui for containers keep knowledge on one or level, and even though, if it's done for the caches, we have another resource, which is also can benefit from the same approach, and I'm talking here about the blockade.

A

This is virtual resource known by linux, kernel where you can define the priority. Sound limits for block storage, but again, like multiple nodes in the future. Originals cluster might have different configurations of storage devices.

A

You don't need to expose that to a end user, how to control it. But what you want to do is again probably with class-based abstraction for consuming of those resources, and that allows you to again properly split and control those resources on paranormal basis.

A

So all of this, I, the reason why I talked is what we have possibility to divide the resources. We have present possibility to measure one those are vendor-specific, but now when we have understanding how to divide how to measure now, we need to have a control for it.

B

So smarter run times ever since its introduction kubernetes has enjoyed an increasingly wide scale, adoption across public and private clouds. As a result, both the spectrum of workloads and the diversity of the cluster hardware is rapidly growing. You have ai databases, content streaming network function, virtualization automotive edge running anywhere from a virtualized generic public cloud, something like google cloud to a bare metal custom cloud with accelerators, for instance, 5gh with fpgas kubernetes is literally taking over the world. While this is a positive problem, it is also causing some headache for us.

B

Kubernetes node resource management is implemented in cubelet. The node agent cubelet has a simplistic view and limited understanding of the hardware. Moreover, it comes with a single one-size-fits-all resource, algorithm or resource allocation algorithm.

B

Now, while this algorithm is good for many common cases, critical workloads on bare metal require more optimal hardware, aware resource assignment for acceptable performance. Similarly, workloads in domain specific clusters often require custom resource allocation logic.

B

The one-size-fits-all algorithm cannot scale to satisfy all these requirements. We need multiple resource allocation and assignment algorithms and support for custom logic. That sounds a little bit like. We need support for pluggable algorithms, but where to plug these, we have very few candidates either the cubelet or the runtime itself.

B

Cubelet already hosts one policy. So should we extend it with pluggable policies? Well, that would be problematic for many reasons.

B

First of all, we would like to keep the orchestration hardware agnostic and, as it is critical infrastructure, we would like to keep it as simple as possible. A cubelet is already complex. Adding pluggable policies would definitely increase its complexity.

B

Luckily, another option suggests itself the run time, let's plug custom algorithms to the runtime and make it smarter. The kubernetes apis are predominantly declarative. The user describes what a workload needs and orchestration decides how this will be achieved. This is true for the resource api 2.

B

container resources are requested in abstract terms, for instance, 3, 000, millicourse and not course, 1 3 and 5..

B

But the api between the cubelet and the runtime, the container runtime interface, is fully imperative. It carries concrete resources and related parameters to bridge this gap. The cubelet turns abstract resource requirements into concrete resource assignments using its allocation algorithm.

B

But what would we need instead? So let's separate the what from the how and leave resource requirements unresolved in the cubelet, let's update the cri api to allow passing abstract resource requirements to the runtime itself.

B

Let's extend the runtime by pluggable custom, resource allocation and assignment algorithms and an api that these algorithms can use to resolve what spec requirements to concrete resources. In other words, let's make the runtime smarter.

B

There already exists an extension point for plugging resource allocation algorithms into runtimes. This is nri the node resource interface container, the subproject in draft status.

B

It is described as a new interface for managing resources on a node for pots and containers. It was largely inspired by the container network interface, which is used by runtimes to handle multiple implementations of the container network stack.

B

We share common interest with nri. It aims improving node resource management with a structured api and plug-in design for containers. We would like to do pretty much the same. It wants additional interfaces to customize and con a container's runtime environment. We also would like the same nris. Current focus, however, is on better control and more flexibility for injecting devices into containers.

B

This we would like to extend a bit so, ideally, we would like to make nri the primary integration point for extending the runtime and allow any vendor to plug in their any resource, allocation and optimization algorithm. So how do we want to extend nri well nri tabs into container lifecycle events and invokes corresponding plugin api functions?

B

The plugin receives information about the event, and then it selects and configures resources and responds back with data about how to set up the container. The current nri events are creation and removal of bots and containers to enable more generic support for resource optimization. Algorithms nri needs to step into a few more events to enable smart algorithms. Nri needs to pass all the necessary information to plugins to allow proper container setup. Nri plugins needs to be able to pass back enough data for correct container configuration.

B

So finally, let's look at the plugin invocation mechanism. The current implementation is a hook like one shot api, with a separate exact for every request for stateful plugins. We need another menus with minimal overhead, probably a proto booth based api over grpc or ttrpc.

B

So eventually we would like nri to become the common resource integration point for all oci compatible runtimes. Therefore, the nri core and data structures need to be run time. Agnostic adaptation to any corresponding runtime specific bits should happen in the runtime itself.

B

Container is already integrated with nri, but we need to extend on that a bit.

B

We have to hook into a few more additional lifecycle events and make sure that all the necessary data is passed back and forth to the plugins. Also, the current code ignores plugin replies altogether. That has to be updated to modify containers according to the plugin decisions.

B

Once we have ironed out all these details with container d, we can take cryo and integrate it to nri as well.

B

Cri needs to evolve also to provide better support for smarter, smarter runtimes. Smart algorithms need declarative resource requirements and the separation of what in the cubelet from the how in the runtime to do this cri needs to pass through both spec resource requirements, where button smartel algorithms, with better efficiency, also need a few group operations.

B

Vm-Based runtimes need full information about all container resource requirements for sandbox creation, also to optimize the collocation of related containers. It is better to get a single request for creating all the related containers at once.

B

We also need the ability to update multiple containers with a single request, and, finally, we would like a dedicated api for container state monitoring. We want to allow clients to subscribe for content as they change events then trigger and deliver notifications from the run time when such an event occurs.

A

So, based on the things what we just provided to you, there are three things: what we want you to take home out of this presentation. First of all, is a about the hardware.

A

A hard drive is evolving and the hardware is changing all the assumptions about how hardware works might be changing next day, so it means no one single algorithm which will satisfy all we needs. We need to have a flexibility.

A

Second thing about the kubernetes. Coblet holds the information of what and what should be run. What is the priority of things, but we need to split what and how?

A

What is about runtimes runtimes now contains some information about how, like linux container window, os windows is containers or vm, vm-based, containment and so on. So we want to have runtimes to control before how part of how containers are run, and we want to have a plugins in this interface. So what more custom logic and vendor specific projects or special installation logic can be extended.

A

On web, we have reached the end of our presentation me and preston available on the github or kubernetes slug, and we are happy to talk more and more details about all the things what we presented today and what we are doing for our hardware performance actualizations.

A

We will be available today after the sessions for live q, a so welcome and thank you for participating in our session today. Thank you. Thank you.