Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Mitigating Noisy Neighbours: Advanced Container Resource Management - Alexander Kanevskiy, Intel

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Mitigating Noisy Neighbours: Advanced Container Resource Management - Alexander Kanevskiy, Intel

In large clusters, some applications attempt to consume a majority of shared resources. These "noisy neighbours" cause performance degradation for other workloads in the cluster. At this time, Kubernetes has mechanisms to mitigate this behaviour for CPU and memory only. This talk discusses methods for extending fine-grained resource control on other shared resources, such as block and PCIe I/O, shared CPU caches, and others. It demonstrates how to utilize extensibility points of CRI-O and containerd runtimes to achieve fine-grained resource control. The talk also presents an approach for evolving this method into an extensive and fully dynamic resource management solution for Kubernetes. Agenda - Problem Statement: different types of "noisy neighbours" - Resource management on kernel, OCI, and Kubernetes levels - Stitching the pieces together: dynamic container resource management

https://sched.co/UabM

A

I think it's time to start hello, welcome everybody I'm Alexander! Can you ski I started to work in ciencia for kubernetes related projects about three years ago, initially, I started to work on things related to sequester. Lifecycle like how to install a kubernetes on bare metal and bit later, I started to look at different kind of device begins again like how to use accelerators and over time, I got interested in the topics related to resource management. So one thing what we are going to talk today, it's in this area so noisy neighbors.

A

So don't we know what noisy neighbor skis like like it's, it's your neighbor with a hammer drill trying to do some renovation in the morning like drunken company in the restaurant and so on, even right now for us like when oil Abel is about our time. So, for example like not yet folks at the same time as my slot we're doing talk related to 5g and for time-critical application, so very close to what either they're talking about and.

A

What I wanted to say is what these kind of problems were unique to all of us, I mean so well. Really, we silver bullet doesn't exist, so you have in your class through your own problems. What are some sitting on a chair next to you, you have a bit different problem, so what I'm going to talk today? What is out of scope is, if you can mitigate it somehow by adding modules spreading more quotes around your cluster. So it's not a problem. It's expensive like how much you can afford to buy in your hardware.

A

However, what is in the scope of this talk today is about one single node, so you get work load scheduled. You have more hardware resources on you know like its CPU memory, yada yada and now those different workloads where misbehaving busy child. So practically we have like playing with Tetris on the same note like, but we figures what we are using is like: it's not simple two-dimensional figures like every resource, everything what you are using it, let's put in a figure two different place, and you also like what perfect line doesn't disappear.

A

Your your figures in overtime, changing the shape during the execution and now I think what is also in the scope today, so we're in runtimes track.

A

So what I'm going today to talk it's about like cryo, container D, we see I spec, run see, unfortunately, I'm limited in my time, so I can't talk about like cottagey, wiser and so on, but we will try to keep in well like smaller scope, which is most common for everybody, one small node, so if you're on production still using a couplet directly talking to a doctor, so this docker shrimp I have a bad news for you like sorry, you cannot do much with it.

A

So probably you want to switch to something but like cry or container B in your setup. Alright. So let's talk about hardware resources.

A

Just a standard picture, you know like most common server, like two circuits, you have a bunch. Of course you have some memory, you have interconnect, you have some accelerator devices. So raise your hand. What do you think, like it's real-world picture?

A

Okay, not what I'm good. So now. Let me scare you so indeed in real war to a situation, a lot more complex, so you have a sockets in inside your sockets. You have a physical course. It can be one chip. It can be multiple chimp within one socket.

A

You can have different sizes, of course, so, for example, like in your phones, must probably have a like big.little architecture.

A

When you can have a machines which have like one or more thread on the physical course, your physical cores can run on different speeds, so, like not all the course in reality, equal when interconnect between the CPUs again when people are usually same, like I have no problem.

A

Usually, what we mean is actually one of those interconnect is get saturated by like data transfers and amount of is interconnected will be really dependent on your, like the obvious, every texture of like model of CPUs you are using and so on, and if we are talking about where Numa problemas is so, the key part about Mumma is the letter M when memory access. So you have a multiple memory controllers on your CPUs, so you might have different setups of a memory.

A

You can have like normal memory across persistent memory, which is a bit slower or vice versa. So you can have a normal memory and high bandwidth memory and for kublai that will be seen as like normal memory, all like all together, but in reality for your workloads, it might have a difference. Another thing is what like, when people are thinking about new weights like well its memory plus CPU, it's a wrong assumption, so you might have completely legged new man old completely, without course.

A

So no my describes how your memory layout in the system is done. So when you have devices it connected to be shy, buses, when you have interconnect between those devices, you probably like goblet or your orchestration, doesn't really know those devices can be connected not only to PCI but our resources and when, of course, you have also like all your integrated things like Network, blocker, your storage and so on. So the reason why I wanted to show this picture is actually like to notice all those errors. So all those errors is your like toll bridges.

A

Every time you cross in when you are paying the cost, so mitigation of those things is actually trying to combine or workloads of how to museum minimize going through vowels or how to find welcome a route to do minimize with cost. So, let's look how our kubernetes thinks about where resources we have on what host we're all like. Well, negative resources like CPU and memory and as I say it like for the kubernetes CPU and memory are equal.

A

So if you want to try to express I want to a course with a bit faster speeds, oh I want one physical core or one physical thread or higher bandhas memory. You can't do it. Of course, we can do like some kind of leverage of extended resources or device begins, but usually for native resources. You end up with double bookkeeping, which usually it doesn't end that well you can use something would like to make decisions from a pod class like some artificial thing, which implicitly done by kubernetes looking at your native resources good.

A

But if you have some scenarios where, like sidecar injected and so on, it's easily get screwed up and I will break how to enforce resources is using the metadata of reports or like labels, annotations and I would say most of the time. It's not only use it, but sometimes it's even abused annotations model. So, like example, like well network guys are using it for second verb amber ribbon, so sorry network bandwidth.

A

Second guys, we say like well use this profile and actually, in the background we expanded like huge list of available and the problem is annotation is what it's visible to a user, and you need to think how you actually will be exposing like what is with user. Experience of this thing is going to be, and the most bigger problem is what, when you're trying to expose resources, which a bit more complex, when just like one single parameter example, would block I/o.

A

So imagine you have a workloads which are competing on a storage and we need some how to prioritize it. So in also aspect you can say like well, we have the weight and when, if you go down to the kernel level, well Colonel understands what you can have weight per device. You can set amount of I/o, ops or bandwidth again paired device. So if you try to expose all of these parameters to reuse or user, all say like all them. Thank you know, so you need some how to find with balance.

A

So, for example, I create a class that bloke at your bronze silver gold and when, in my background, try to convert it on the node level, like this class main to specific parameters to a specific device on whatnot, because again you can have on the node. Well, you can have feature genius cluster, where, like one nodes, have a bit differences and in with Heartland another complex resource which I wanted to bring into this example is where Linux kernel has file system, which is called a source control.

A

What it allows to do it allows you control where allocation and monitoring of layer, 3, caches and memory bandwidth and difference with C group is what, like in case of C group, you can easily like it's it's software, it you can do like as many groups as you want in case of this resource control. It's all Hardware dependent, so you have well eight sixteen different classes markets. You can't create moment wealth and another thing is to control our parameters inside. You really need to understand what what kind of hardware behind of this thing you have.

A

So again, you need to think not only how to treat where resources what hard web gives to you, but also how to present it. We use around how to do the mapping between all those two things, so how we can control those things. Just oh, let's look at our nice free, layered cake. So no bottom, we have we run times like run, see kata, divisor and so on. We have a layer between models like OCI speck.

A

On top, we have four run times like most commonly used like container diem crier and when we have one more layer of interfaces, Sierra interface and when we have a couplet, so each of those is responsible for own things, but one interesting thing here. So if you look precisely so the more you go down the more it knows about hardware, even more, it knows about the control, so even more you can do. The only problem is what, where information about? Actually what user wanted? It's all about.

A

The couplet and reason is those interfaces between those, not passing all of the information. So, for example like if we look at the Sierra interface, so Sierra interface says like wishes, were your cpu parameters? Visas were memory limit, misses way, CPU set, which you need to use this as my memory set, which you need to use and if you noticed a few things missing so one thing is so, for example, what CPU requests and limits when? Well, we don't know you can reverse calculated from.

A

We see a fast shares, but you don't have it memory request. You know what we limit is will be, but you already lost to information how much user actually want to have at minimum and when all the extended resources couplet doesn't it calculates it itself and keeps it by itself?

A

The huge pages gooblat says like oh well, run time. I know you can handle it, but thank you. I will do it myself and in case of cpu set memory.

A

We have it in the spec, but it's not used well, maybe in the future it like with introduction of topology management and maybe in soon we will have like something for memory. We will have usage of red, but right now it's not used and if we go down to OCA level, for example, if we take well run see like I know, we're going to continue ready. I want to update some of our resources like beside who thinks.

A

What's here, I already communicated to down to around zero I can do a few more things like I can serve a block. I await. I can tweak what kernel memory parameters, for example like for TCP stack I, can set all three caches and memory bandwidth schemer, but again what problem is l3 caches? It will be per container. So, like you using wet interface, you can say that only like for limited amount of containers and next thing. If, if you somehow able to intercept oshi, I spec like we config what runs you get before?

A

We contain your created you, you can see what you have a few more parameters. What you can control so like, like cache classes, can be done properly and everything which is super important in notion spec is actually you can control execution of the hooks and the hooks on OCR level. It's the first thing where you can plug your own custom policies to do something. So what is it it's, a simple like binary or script or whatever you can figure what on which stage of a container of time.

A

You need to execute that you know he defined arguments. What needs to be passed.

A

It's called by the runtime itself. It's called in the host name space, so you can tweak. For example, we see groups of a container, but what you cannot actually touch is actually we configure for container itself. So if you want to inject more mounds, if you want to inject more devices well, you are not completely out of luck. You can do it by yourself.

A

All the same cannot and see. Groups filtering well. Well, not easy. We have few hooks different right now in Ostia expect and where is a PR from NV year to actually add even more froze my bit user phone.

A

Maybe you can do some something with it, how to enable it so inquire, it's easy it's disabled by default, but you can easily comment out one one line and it will start to look for configuration options where to find the hook, so it it looks for like quite simple JSON file, which says like well on with stages execute with hook, and you can say, execute it always or you can execute it. Only particular annotation is passed, but but what's simple in case of crab in case of container dear, unfortunately, it's not what easier.

A

So this functionality does not exist, but soon I really hope what it will. The existent from thanks to my Brown, which actually helping to get it done. I thinking like it's, it's hooks, you can do certain amount, but you are limited so the next thing. What if you want to do a bit more complex thing? You need to go a bit level up, so you need to define your custom runtime and custom runtime things to kubernetes latest features.

A

It's quite easy, so I grant implies you define like well, I have a runtime, which is a wearable. Ok or it will be handled by hunger, cold, walk I or in my pot I can say: well, you run type last words for this port and on level duo in cryo. You say: well, this block I, your handler is actually with binary what we need to in container B. It's almost the same where differences like actually a couple of underwater stones, so first of all, continually supports two kind of interaction with the actual runtime.

A

So you have watch I, see exact based interface. Oh, you have G RPC. So in this scenario you can see what it's it's about. A previous like exact, based interface and I. Think like if you're trying to pass the parameters using annotations, you need to make sure what you instruct container D of actually which annotation needs to be passed down. Crime in what sense like it's passing, almost everything what it knows. Where are some exceptions?

A

How you can find particular annotation prepare container, but it passes container D require with additional configuration options and when I'm talking about like.

A

Special runtime I'm actually talking about like quite simple script or quite simple binary. So it's something approach close to what a media guys are doing so you're practically wrapping. We exist in runtime or shimming, whatever you prefer to code. You know in this particular example: what I have I just created a simple shell script like for all runs: ecommons I am just passing it, but before we container started variations, I want to enforce some of resources and what I do in start, a creation.

A

I know we container ID, so out of container ID I can request to a state of it from a state. I can get to a config from what config I can get a name of. We see group, and this is the place where couplet actually tells me about like what kind of course classes. It is so guaranteed doors the below best afford and in this particular scenario, so I'm saying like okay, well, guaranteed I, don't touch it for door stubble or best afford.

A

I am low a priority of bloke I/o operations using a standard runs the update command trivial works.

A

What are a few problems with it? One of the problems is, would like you need to maintain those routers. You need to like, for example, this code that has bugs about parsing the parameters. You need to be careful. This word. Another thing is you have one runtime class per port? So if you have multiple scenarios which needs to be handled, you do out of work all you need to have a your own special, let's say, like default runtime class, which will be handling all of those.

A

So we we thought about that, and we try to do different experiments on what like how to control where resources, what I showed you like like ashes, memory, loci or yadda yadda, and we realize that what.

A

Hooking into low levels it might help, but we not necessarily have enough information, so we know about containers. We not necessarily know about higher-level like pots, and we, some time ago we created a tool to actually open source at our internal to be colored cRIO resource manager. So what is it to put it simple? It's a proxy for Sierra, so it takes requests from corporate it processes to container beam so what it does beside procs in those things.

A

What a touch actually also can do it. It can enforce certain decisions of example, like second one memory, things or injecting more mods or like trying to ship something or actually even do some and what is not part of Russia I spec, but talking directly to the colonel, about changing those parameters.

A

So I yesterday heard a talk from uber. We did something similar actually to to migrate from way message set up to kubernetes gain to enforce, like local policies. What we need to have always were a logger interface, so again, why most people would say like well guys? Why are we doing a separate tool like why I'm not doing it upstream? I need to say history about waste. All this tool started as a internal debugging tool. We just wanted to understand like well. We have standard cluster set up kubernetes non-budget.

A

We have Cryer container gear again non-budget something goes wrong. We need to debug it, so we started with just what let's add tracing we found to ensure what we actually wanted in couplet and when we realize it, okay, well, if couplet will do something, XYZ can be simulated with red tool and which is how is to get a bit more developed.

A

We are using this tool as a proof-of-concept work, so we can try something and afterwards we can, together with the data we can go upstream and propose those changes to turbo couplet architecture. Wise, it's quite simple, so like web proxy part is, as I said, well get it pass it log trace and so on. The interesting part starts a bit later, so we have set of internal policy engines.

A

You can buy these policies. Actually you have all the information. What is past and CRI for both pod and containers? You can modify it directly, you can go to see groups or, for example, like rest control, as I mentioned. We are also created down Ward's API. So, for example, if we do some decisions, we want to inform where workload about with decisions, what we made and if we do something dynamically again workload can monitor with downwards API file and get information about it.

A

Another thing which is also most important, what we are trying to do is actually.

A

We want this tool not to be maintained it as a set of static policies, but also give you a possibility to try own thing so, for example, like Netflix, we have idea of Tytos project where you can say like we have a machine learning actually how workload needs to be placed within one old. We want to give you a tool which gives you possibility to experiment this similar kind of decision and with this kind of dynamic interface, also serves as aa purpose of trying to do like dynamic, dynamic reconfiguration of the nodes.

A

So what we can do now, we started with a simple policies, so we decided like okay. Well, let's take our static policy out of couplet and what, if a couplet will be supporting either CP goes, doesn't give workload proper, my performance gains, so we added support of aizen's abuse and our external policy. It worked when we start to think ok. Well what?

A

If workload is clever enough to ask not like two exclusive CPUs but work, 2.5 and two will be used exclusively for the attack threat and 0.5 from the shared pool can be used as controller threads of your application. So we added this kind of policy, and this is where is downwards?

A

Api came saying we are communicating like we also your exclusive ones, and those are your shared pool and the last thing what we have from a policy perspective is a topology where so I know we have a topology manager inside it with Google Earth right now.

A

It mostly works about aligning devices and aligning with CPUs, but we want to have few more steps forward, so we able to detect what kind of storage you are using and we are trying to move your workloads like closer to your storage, so that you can get from, for example, track from nvme disks. Well, the best of your performance we are aware of, like vo, we work load feeds into one memory controller.

A

If not, we can try to find like well a rack on top still to feed still to minimize the cost of your workload, to be a correctional and another thing which we also try to experiment and I hope to see it someday in the kubernetes. Actually, we have cross container affinity ante affinity. So, for example, if you have like two containers in your pod, which does a lot of IPC, you want to keep it close.

A

But if you have like short container logger, which shows low priority and shouldn't be too much affecting you'll, probably want to move it to two separate memory, node two separate course out of it and works not only visiting like one port scope, because we are in Serie level. We know about practically know everything. What couplet is nobody like all the ports, all the containers of what is running and the whole view of the system.

A

What we also have we have support for memory and caches, so you can define your own classes. You can map it to like with best afford and so on, QAS classes from couplet, and you can set the parameters like how much you want for each or for class, and what we also have right now is very dynamic API so like. If you have a character at the Redeemers cluster, you can define like well business with default for your notes wishes.

A

We can pick for group of your notes, and this is where individual config of this particular mod and you can dynamically change it. It will adopt. It will rebalance based on more settings what we don't yet have, but I really hope to do it quite soon is what we haven't finished web local implementation, but it will be similar to caches and memory, so you don't have a classes for each class. You will have parameters per device and when you have what we also want to experiment is about like we have Hardware guys right.

A

So we we have ability to measure things which, like in how much interconnect is used by certain processes, how much of vector flow it in unit used by certain processors. How much like this were called cows in thermal throttling of a CPU, and we can try to do dynamic, rebalancing like well, if, with scarves in with thermal throttling, can we move it somewhere, so the rest is not affecting so this coming next and I really hope very soon and last, but of course, not least, is external policy API?

A

So all the information, what we have we are ready to share and if you want to experiment in your data center with your own policies, machine learning, whatever you'd like go ahead.

A

I don't have time today to show with Emma's, but what slides are available links are available watch a screencast we prepared with a nice set of demos actually showcasing or what we can do right now unless to wrap up with key takeaways. What I would like you to remember from this presentation? So, first of all, I know. We include I, know what some of your machines might be virtualized, but still think about where underlying hardware, it's still somebody's computer.

A

You still can predict how it will behave and keep in mind what like architecture of computers are changing quite rapidly. Like assumptions of a hardware, what you've had like ten years back, it's not animal wallet even like five years back. It's already like many things changed.

A

Second thing: what I wanted to say to you is think about like how our castrations actually can mark a state in a container how you represent the resources right now, where kubernetes have a very simple model about like countable, larger objects, but if we want to control more fine human resources, we need something more so think about your experience.

A

Think what you want to control come to the kubernetes community, let's discuss and we'll last and the most important so I give you hints how you can do it on a hook swell runtime levels well, erm our to what we are using for experiments. But what is the most important is what you you can try your own ideas. You will have your own results from your real production, application and data, and this thing is the most abundant thing for us as a community to actually design how it will be in the future of kubernetes.

A

So that's what I wanted to say today. I now have probably few minutes for the questions.

A

Can we get a microphone.

B

It looks like you have most of the guts of what couplet is doing, with the exception of the external api's. Have you considered just adding the external API, so you can be a drop-in replacement for equivalent.

A

Yes, we actually, yes, it's a advance. Wait, wait, wait! It's our long-term goal to improve for CPU manager on in couplet sight to actually do this kind of dynamic policies, but to propose something like web to a couplet.

A

We need to come up with a well-designed API, so it's vendor independent and we need to have a well well qualified data to show like why it's useful.

A

So what we are right now in this stage, what we want to collect data and this Web data, we want to go to signal community and saying like well. This is how we want to put.

A

If you can put your microphone over here,.

C

So you're talking about Lois and Abel right so that resource manager or proxies or staff right so, but there is some kind of categorical device right like for example, if I have critical device, so whatever I want to in those c group or whatever run run, say controller may not can do those lloyd's label or radio can really name it whatever those kind of stuff right. Yes, so I'm not sure. How can we so that one without particularly property well.

A

What's one of the reasons witness go back to one slide it's over here so.

A

So it's exactly about like this part, so if your device actually has some of the kernel interfaces which allows you to control when terminal resources of what device you can plug your implementation, which we'll be doing this fine fine grained kernel tuning? Oh well, if, if it's enough on your case on OC I hook level, you can do it where it's all the question like where you do. You have enough information to perform whatever voice operation so one or, for example, like nvidia gpus with in McLouth.

A

It's mostly well exclusive usage devices on our case with Intel GPUs, its shareable resource and right now, Intel and AMD. We are having discussion in Linux kernel, community of actually how to come up with proper cgroups interface, how to manage the resources of GPU like time time shares memory and so on. So if you have so your specific, like crypto devices, probably you can join the same conversation and see if we can do something like that for your kind of device.

A

All right so we're on enough time! Thank you. So much.