Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Do Not Disturb Mode: Fine-Tuning Resources for Latency Sensitive Workloads - A Kervinen & P Hunt

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Do Not Disturb Mode: Fine-Tuning Resources for Latency Sensitive Workloads - Antti Kervinen, Intel & Peter Hunt, Red Hat

Do you get your best work done without interruptions? Many people do, and Kubernetes workloads are no different. While there exist ways to run workloads in Kubernetes on dedicated CPUs, they still share resources like CPU cache, memory bandwidth and storage I/O. Recently, CRI-O and containerd have been extended to allow admins to control how these resources are shared. With the correct hardware and node configuration, users can put their applications in different classes that configure the amount of storage and memory bandwidth, and CPU cache that their applications need.

In this talk, Antti Kervinen and Peter Hunt will talk about required hardware, the extensions, and describe ways that admins can configure their workloads to finally get some peace and quiet.

A

Hello and welcome to do not disturb mode. That is fine tuning resources for latency sensitive, sensitive workloads. My name is antigeraviden. I work for intel and I've been working on resource management, especially on the cri and oci runtimes, hello,.

B

My name is peter hunt, I'm a software engineer at red hat, working primarily on cryo, but also on things in sigmod and other computer computer-related technologies. And I we also wanted to send a special shout out. This uh presentation and work was partially brought to you by marcus letzman, who also works at intel. I get the integration and design of the rdp work that we're going to describe a little bit so to start off, we're going to describe talk a little bit about qos or quality of service and kubernetes.

B

Since the beginning, kubernetes has provided knobs to configure how different workloads are prioritized and how they're configured to run on the node. So the qos classes were used to specify memory limits and cpu limits, as well as boom, killing behavior for different pods, and this behavior was able to be further customized by cpu management policies to be able to configure which cpus workloads ended up on which allows for users to configure cpu affinity so that there is a noisy neighbor problem among different workloads.

B

But this uh leaves a lot of other resources that exist on the node, but aren't quite as possible to configure, and this uh causes workloads to be disturbed when it would be a little bit better if they were less disturbed and were given more access to those resources and were less latent, so cryo has recently and there's a pr and continuity as well, but in cryo1.22 we have uh the capability of configuring uh two new resources, um this the first one is intel's resource director, technology or rdt.

B

It is a way to configure cache and memory bandwidth, so you can think of it as finer grading control over the way that memory and the memory bandwidth in the cpu cache are configured.

B

And next we have the capability of configuring block. I o classes in a similar way as the qos class. This gets class based.

B

uh You know, on a c group, granularity block io uh controller, so you can configure how given a c group, you can figure how that c group is able to the scheduler is going to prioritize it. You can give it extra weight or you can throttle that workload so that it can only access a certain amount of the block I o.

B

So, let's describe you, know the situation before any of this qos stuff. So we have our three workloads running on the same cluster.

B

It's very realistic scenario of having 911 our emergency telephone service and a flower delivery service, which is an important uh you know site that it needs to stay up because it's you know, people want to be able to order their flowers at all the time, but if it goes down or if it, you know, lags a little bit, uh no one's gonna get hurt and then finally, we have a scan file system which will look all over the file system and you know to see what's out there.

B

um This is something that we want to be doing fairly regularly, but we don't want to be. You know it's. It's not really mission critical, there's, no apps or you know things that will go down if it doesn't run. So it is our lowest priority, so by default, if we just run on a kubernetes cluster, no configurations of limits or memory, or anything like that.

B

This is the what it's going to look like. Each of these three workloads are going to end up on each of these uh three cpus, the two cpu cores and they're, going to be split up among they're, going to argue with each other in the cache causing uh you know, things to get evicted back and forth. They're gonna contend for memory bandwidth as well as uh for storage block, I o.

B

So this is not great if we want our 911 process to, you know be undisturbed, so the first thing we can do is- and this is available as like 112 beta, so it's fairly, uh you know well tested now- um is setting a static, cpu manager policy, so this allows the user to configure which cpu affinity each of the processes are scheduled onto this, allows, for you know the 911 process to have its own dedicated to you, and no other process would get in the way of it.

B

That's a lot like allowing it to you know, be less late and then the other two processes can go on the other core and they can share and they can content with each other and it's not as big a deal, but we still have all of these other resources that they're contending with so something we can do now with cryo1.22 is configure the uh the cpu cache um behavior so that um the the 911 process has a uh you know is given the a specific uh you know weight to how uh its cache is being configured, and it can run a little bit nicer with less contention around it uh further.

B

We can configure how our scan file system process, which is a lot less important, um is configured with memory bandwidth. So we can throttle the memory bandwidth allowing for the scan file system process to not take up too many of uh too much memory, bandwidth that would contend with the 911 or the flower delivery processes, which are much more important.

B

Next up. We have the capability of uh throttle, uh giving higher block. I o priority, so by default you know any the block. I o priority weight is 100, so if we give the 911 process a weight of 400, that makes it uh have higher priority and it's given access to the block, io scheduling more frequently, and it's allowed to do its things faster, which means it can run in. uh You know a more in a way that it gives it more access to the node and it can run less hysteric.

B

Finally, we can also throttle block io, and this allows us to specify a specific amount of you know read and write rate that a certain c group is given access to.

B

In this case, our scan file system process is only going to give access to a certain amount, a certain rate, and this will also free up room for the flower, delivery and 911 services, in other words, with all of these uh all these capabilities combined we're giving this 911 access to all of these resources in a way that it is, uh you know, relatively undisturbed, um much like this cute puppy here, so 911 can do all of its normal activities and not worry about resource contention with other processes.

B

So next up we're going to unti. Do a bit of a demo of the block, I o throttling.

A

Thanks peter so I'll start sharing my console and now that we know that what can be done on slides, let's see that how it works on a virtual machine.

A

So in my virtual machine setup I have a configuration file I did for cryo, which tells that there is a separate block. I o config file where block I o classes are defined and the file is in the etc containers.

A

And if we check the contents of that file, there is a single class definition.

A

So there's a slow reader class introduced in this file and the introduction goes so that for all the block devices that match this regular expression, this wild card actually uh for all those block devices throttle reading to down to the five megabytes per second, and now we are going to start our file system scanner here to go to this class. So, let's see how this file system scanner put and actually demonstrate. Yaml looks like so uh in the annotations.

A

You can find block io resources, beta kubernetes, io annotation and I'm giving it the scope of port, meaning that I want this annotation to apply to all containers inside this pot. I could also give here a container name to uh restrict to this. This uh block io class only to a container okay and the block I o class name is the slow reader and how this file system scanner then works. There's one container running busy box and inside the busy box it is running a while loop and inside the while loop.

A

It is looking for files under the scan directory and it is passing those files as command line arguments to md5 sum, and then that would be sorted and saved to a file and then the files would be differed compared to previous md5 sims, and that way we could track that.

A

What is actually what is changed in the file system and what are the checksums for all this all these files, so the most important thing here is actually to know that what is actually being done and who is actually reading the block devices, is the md5 sum process.

A

So it is reading the files and maybe for completeness I'll, show quickly that what is under this con so in the scan directory, I have mounted user bin and user lib from the host file system as read only mounts.

A

Okay, I think that we are ready to launch this file system scanner there. It is and if we now check out what this c groups look like, so I have a helper script digging out z groups details.

A

I can see that in the default namespace there is a file system scanner port inside it there is a busy box container and for that container in in its c groups there is a read bps device file for clock, io controller defined, and it has found two devices from this virtual machine. So two devices are actually matching this wild card here and the read is now limited down to five megabytes per second for both of those devices.

A

Let's see now the mp5 process really should be running and if I'm checking out what's in the proc and pit of uh mv5 and io file, so there is something really happening. Numbers are changing.

A

So if I take this nice little uh pulling loop here and compare the numbers all the time to the previous ones that we have read and make that human readable, we can see that it's always the five megabytes per second at most that the md5some process is reading.

A

Okay. So now we have seen that this, this actually works. So we can continue and take a step back and think that what have we actually introduced here in a bigger scale, so I'll start sharing.

B

A

A

And, let's see so, we have been defining here the first steps for this kind of class based qos, so we have enabled workloads to be annotated with qos related class names.

A

There is block, I o classes, and there are rdd classes both for uh that can like for high priority workloads and also sort of limiting classes for low priority workloads, which we want to make sure that won't interfere, that much with other higher priority workloads.

A

And here the name of the class in the workload configuration it is kind of node independent. So we at this point where workloads are annotated. We don't actually have to care too much about the node hardware details. So we do not give any absolute numbers here, for instance, but those are hidden inside the per node configuration files that are read by container runtimes cryo in this example, and here why why we are doing this so that we have this kind of node configurations separately.

A

Is that if you think of the block coil, for instance, and compare spinning hard drive to ssd or compare that to non-volatile memory like obtain and the characteristics of those block devices? In performances are completely different, so the value is how you would like to throttle this kind of file system scanner on real hard, like physical, hard drive or non-volatile memory.

A

Those values could be totally different, but anyway, you might want to have this kind of heterogeneous cluster, where you really have different kind of nodes, and it would be very nice to be able to schedule this kind of chops on any node. So it's kind of foreign file system on any node so that it won't disturb the more prior higher priority jobs on those nodes.

A

And now these node specific configurations that depends on the hardware characteristics enable you to do that so keep it simple. In the workload and in the user perspective.

A

Then a couple of words about the qos in future, so this is what we have been getting so far up and running, and what we are doing is using the annotations as usual, and this is good and only good for working getting this working now. But obviously this is not a long-term solution, so there needs to be some cleaner, cleaner way to do the class-based qos in the future.

A

On the other hand, uh in kubernetes there the resources, the numbers for cpu and memory they currently imply the whole qos class for the container and for the port, and that qos class then implies things like how to be how to configure out of memory killer and whether or not this port should have exclusive cpus.

A

And if you think about the future, where the hardware is going, there is going to be different memory types you might have like high band with memory. You might have low latency memory and not that low latency memory and there might be different cpu core speeds that you would like to somehow state for your workloads that this workload would prefer like a slow low power, cpu or another workload, wouldn't really need like high performance cores these kind of things.

A

I do not believe that we can any more or any longer encode to these numbers of cpu and memory. What is requested and limited so one way just a food for thought for the qs future I'd like to like sketch, somehow that how this could be made more explicit.

A

So, of course we could have these numbers for limits and requests, as we have done for now and have the default behavior unchanged, so that everything would be backwards compatible.

A

But in order to make this more future proof, uh one option could be to introduce this kind of qos section or container resources where we could be giving them this kind of rdt and block. I o classes how we want this workload to be run and why not, at the same time, enable overriding this kind of implicit assumptions that we have had for out of memory killer or cpu exclusiveness, or maybe at some point for memory, speed and latency.

A

These kind of things could be then stated explicitly, and even if you have this kind of qos section, also in the port level, and not only on container level, then we could also affect the assumptions that we have on what level qos like eviction. So when cubelet is noticing that the node resources are running low, it starts to evict parts that are not high high priority.

A

So why not make that also explicit so that users can configure it as they want and not only imply through the numbers of cpu and memory. But anyway that was just a food for thought for the future and let's get back and wrap up what we have actually been doing so we've been tuning workloads. We have seen that okay, there is already a good way to say that this workload needs explicit, cpu cores.

A

But now, with these new extensions, we can also give workloads exclusive, cpu cache and we can throttle memory band with if we do not want other workloads to take those and from from high priority workloads, and similarly, we can uh configure pro block. I o priority for high priority jobs and on there again we can throttle plug. I o bandwidth of low priority jobs and, as a result we can be. We can see our most important workloads, relax a bit and don't be disturbed by other workloads.

A

So thank you very much here are the links to trio, project and rdt and plug io, and now I think it's time for questions.