Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Working your Cluster: Smarter Scheduling Decisions for Your Work... Madalina Lazar & Denisio Togashi

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Working your Cluster: Smarter Scheduling Decisions for Your Workloads - Madalina Lazar & Denisio Togashi, Intel

When deciding where to schedule your workloads, you have to consider more than just CPU and memory. Whether you are in 5G, AI/ML, HPC, or NFV, you have many more considerations to optimize your workloads. You may care about how busy the node is, how many GPU cards are attached, whether a minimal throughput is available, or whether the node is cooler than the temperature required for basic cooking. Fortunately, Kubernetes allows for extensions to its scheduling paradigm, which allows for new creative solutions going forward. Using these capabilities, we have created a way to use knowledge of your resources to impact your scheduling decisions. Telemetry Aware Scheduling and GPU Aware Scheduling, both open-source projects, enable you to use a variety of metrics in intelligent scheduling. In this talk, we will explain how to deploy and configure your system to handle your varied use cases.

A

Hi everyone. Thank you very much for joining us today, uh especially in person. I know it's been uh somewhere like the weird.

A

The last years have been a bit weird, so at least from my point, not sure if you can relate this, this is like I haven't seen this many people in like the last two or three years. So it's a bit different. uh So thank you very much for being here. Thank you very much for the people that are uh virtually attending.

A

um My name is madalina I'll, be one of your co-hosts. Today I am a software engineer in intel's resource management space in ireland. I have a web service and distributed systems background, but I recently moved into the cloud native and kubernetes space. I won't be giving this presentation alone, I'm here with my teammate denisio and he's going to say a couple of words about himself.

B

Hello, my name is denis otogashes or ice mandolin. I am also a cloud native engineer at intel. My background is science.

B

Then, a couple years ago um I got uh opportunities to fast track and get a degree on the soft development and, and then so I end up on this huge incred, incredible company that intel, so we both are based in ireland.

B

We work on the team on the resource management team and in this team um we have so many different projects, but each one then, with your challenge: it's your the challenges and so on, but um we we always looking on different ways to make an improvement kind of different innovations related, but with the mindset what was talked on this morning uh relate to sustainability.

B

So, as I said, we we have a dual school projects and uh yeah. I hope you enjoy so madeline and I is a first time um community uh cubicon. So um it's really great to be here and yeah um today, because of you, and thank you, give you this opportunity we're going to talk about one of those projects that is related on the um you know, this math scheduling and and then we follow up the presentation that so I stopped to talk and head back to mata. Thank you.

A

So I um so the name of our talk is working. Your cluster, smarter scheduling decisions for your workload. The aim of it is to show you how you can leverage telemetry from your own cluster to make better scheduling decisions.

A

We're going to start off with why resource dates and scheduling matter. How would they, how would you go around combining them and what's the, what are the benefits of doing so we're going to touch on one of our projects, which is called telemetry, aware scheduling it's an open source project?

A

Then you spoke a bit about it: we're going to look at a high-level intro system design and then the basic building blocks we're going to see telemetry aware scheduling or tas it's a bit of a mouthful and in a quick demo and we're going to conclude the session with a bit of q, a so I'm guessing.

A

Most of you here today run your workloads on clusters, whether they are on premise or via or or hosted by a cloud provider, um especially if you work with big clusters, you have a large number of nodes, and with that comes the problem, how do you deal with uh failures because it's the more hosts you actually have in a cluster the harder it gets for you to pinpoint with like accurate precision when a host or a node can become unhealthy?

A

So when you design a system or even think of a system, you look at. How will I deal with these fail failure scenarios? Some ideas that will come to mind is: whenever you want to schedule a workload you can avoid scheduling to an unhealthy node or once you know that a node becomes unhealthy. You migrate your workloads away.

A

One level going above this that you can think about is, I know my workload. So I know what I want to do and then I know my cluster. I know my hardware configurations what if I take the metrics of interest like temperature or power or load, and I make a ranking of my nodes and I pick the best outcome, especially if you're in kubernetes.

A

You don't want to also reinvent the wheel, so you're going to think of how can I make my system work, but out of the box tools like like the kubernetes, the scheduler or the gpu, aware scheduler.

A

So the next topic that I want to talk about is at least in how we thought about handling telemetry and smarter scheduling decisions, and this is where I'm going to talk a bit about one of our projects. It's called telemetry, aware scheduler, as I said before, it's an open source project and, as the name says- and I guess you already thought about it- it uses telemetry to help make scheduling and scheduling decisions, it's an extender of the kubernetes scheduler, and that comes with a couple of perks.

A

The first thing you think of is because it's an extender, I have the capability of filtering and uh scoring nodes, and then the spice that I think comes a bit above is: I can also utilize node affinity rules via fixed and also custom labels, and we're going to look at this in a bit when you're talking about scheduling, at least in kubernetes and with a default scheduler. One way you can actually touch upon a recommended. Pods, like a pause recommended schedule, is via policies.

A

That is pretty much how you would tell the scheduler how to react, what actions are taken and when, in our specific case, we work with something called telemetry, where uh telemetry we're scheduling policies and we're gonna look at examples, and these this policy is structurally based on rules which in turns, are based on metrics that actually come from your own cluster.

A

If you have more complex scenarios, we also have a capability in telemetry or scheduler or tis, that we support multimetric rules. So you can build rules that contain multiple metrics and you can link them together with operators such as any off or all of.

A

So, to set telemetry we're scheduling up, you don't need much. The first component that you need is a metrics pipeline because we're talking about telemetry- and I I'm going to talk about the first part I'm going to talk about- is in the top left corner, so the metrics pipeline. You need that because you need to expose, collect, store and then make metrics available to the kubernetes custom metrics api, in our specific case, we're using the prometheus node exporter to make telemetry available to us.

A

We use prometheus for collection and storage, and we use the adapter to make our metrics ready, making them ready for the custom metrics api. The second part that I want to talk about is the actual telemetry aware scheduler. It works together with the default scheduler. So every time the default scheduler actually wants to make a decision it reaches out to tas.

A

If the workload has a policy that ts recognizes it just returns like a suggested outcome of that pod placement to the default scheduler. The last piece of the puzzle is the actual telemetry aware policy and, as I said this part, this is harshly how you control the actions. It's a custom resource known by the telemetry aware scheduler, and um you the way you actually work with it is in your in a high level workload type like a deployment.

A

You would just add a label saying which policy you want to link it to, but don't be afraid we're going to show you examples.

A

As I said before, structurally, this policy has a couple of rules, and tas itself supports uh four types, and I'm going to ask denise to come along and show you what are these four types and how they behave and how do they actually look like.

B

Thank you madalina. So thank you for a nice intro on that. So I'm going to talk about the a little bit money structure, the other another piece of the building block, that is, the js pulse, so the ts policy composed of four strategies.

B

Okay, um the two fourth strategies- are directly related about the the the schedule, the native schedule, the full scatter, uh which telemetry was getting working like a standard. The two less the two less strategies are more uh communicating director of the the difference via the northern finnish rules. So you can go yeah. Thank you. So we got to take a look on the first strategy. Don't schedule each one of those strategies are composed for the action so and the metric name and the target val and the operator.

B

So we say that the the strategy is broken or uh in case of down schedule when the the metric rules are broken. So what that means. So in this case we have the health metric uh metric, like that example. So if you know closely you define you have that metric and if that method is at that current state have that value. That is equal to 1 in this case, so that node particularly going to be pulled out from the scattering process.

B

So that's the equivalent of the filtering process on the the default scheduling and now the second. The second strategy is categorically started, so it's the one that actually gives a kind of priority or those nodes that have been uh available with information to the cluster. So once that is, you only have those up suitable nodes which one that you think that will be uh more adequate, so they can't get a kind of the prior priority uh in front of the other.

B

So that's you can define only these strategies using another metric method, in this case there's a temperature. That is a good opportunity to know to save. If you consider temperatures are power related. So so, in these examples that we have, that strategy is just saying that okay nodes with that metric temperature, the lowest, really going to have the high priority so now uh move on for them. Now we are looking on the disk schedule strategy.

B

The schedule strategy is from that on the telemeter, where scheduling use the node affinity rules, so it doesn't communicate directly with the default schedule. In this case, you have any other parameter files. You have no way to link it using the node affinity uh rules.

B

So if you have an apart that eventually break those rules, the pod going to be uh sorry, the node that have the method is being uh broken going to be labeled. So uh it is, in this case the heart code labels. It just won't receive a, let me say violating so once that label is present on the node. We can use the other components- uh external components. In this case we use a cabinet scheduler that identifies that the node.

B

There is one nodes that broken the node definitive rules and proceeding that way you can go uh to infect the party risk and so on. So the last strategies that one recent future added to the to our application. That is a labeling.

B

Basically, uh it's worked like a discarded strategy, but the main difference here is that you can customize the user can customize the type of the label that you want.

B

A node receive if you break that rules, so uh you can think like that that you are transferring information uh some metric to the north, so why this is important, this kind of flexibility um that that would come from one, the collaboration across our our company, so we developed together with the attenuation that where they are interesting use case relatable gpu, so which they they have also gpos scheduling that another open source project, that is, we work together with telemeter west scan.

B

So let's, let's imagine uh sorry, there's that kind of fault here now the last image we have across and this cluster have a node and this node have attached supposed to gpus. Now.

B

Imagine that we have you're able to write a telemeter with the policy that tell you that if your metric is related, like might say, power, consumption break the rules. So if you have one gpu that break the hose- and you have a workload that tells you that one part per request- one gpu, you have one people that have a problematic concern, a lot of power, but the other gpu is working. Fine.

B

If you use this schedule that node that have those distributes going to mark as a violator and all the parts going to be discovered and evicted, so that's not a a suitable situation so and we we create this new feature that are customized in the label. That way we can transfer the information just tell and label the node that have those two gpus which one gpu will have a high power consumer.

B

They say you can write a kind of the label, let's say disable card gpu one or something like that, and then once that is you have that information on that node gpu, always kind of kick off and do the business there and make it ready for for the kubernetes to discuss just that part that use that gpu. That has high power consumption that that's uh yeah. I think that's enough. We can go for the demo, but we we have to make a kind of deployment.

B

Those are the three files that we use normally used when we work with the discarder community schedule, so you can see on the board there all the sections that you are related. So you you! You have the uh the deployment file in this case we have an application, simple nginx and uh we're going to deploy five cops or that fight parts, and you have that the link to the telemetry, so you identify that each one of those parts are going to be associated to a policy related gs and the other section.

B

It says where they know the fins who's coming out, because we have a uh in all gis policy discovery.

B

uh Sorry, the labeling strategy strategy and the final one, the simple pulse for the combination of the scheduler that just tell you to invict the party that violating all the finished rules so um so apologize if some demo effect happen here, even if it's a recorder but okay, let's start here um this example. Here we have a closer that composes, the three workers under one control brand. They normally stand so the only the work on those received parts.

B

So we have the the right side. Is you have the metrics associated to each one of the workers and on the further right? You have the excuse me, the labels, the labels associate for each one of those notes.

B

Now, on the top panel is where the you're going to observe the all the resources being called uh during the demo, so like the pods that have been deploying and all all the all the operations that happen and that the middle is where you have the information, they show the events which you want those steps on the bottom and eventually you don't see part of the logs.

B

So we already have deployed the telemetry we're scheduling under the policy. The police again has been shown on the left side, so it composed for this number three uh strategy again down schedule, standalone and labeling. So each one again have your own metric rules.

B

So um again, don't schedule you're going to tell the chairs that to future not evaluate their result, we can see that working on trees that have that help northern health matter, the health metric name that is equal to one. That's the method that that current state is showing. So you look around the on the telematics policy. You have that that that exactly that value should not be that in that case, that node will be put out on the in the schedule process. So, um therefore, we'll have three results: filter out.

B

um It's okay, so the next. The next one is.

B

Excuse me, uh they schedule a method strategy, so, as I mentioned before, this is based on the priority and the metric used here that simply temperatures and gone selected between those nodes, the the works that have lowest temperature. They want to be with the going to receive majority of the pods. In this case, uh we only have five, so they all then gone to land and the node that have the highest priority: the lowest temperature that the work they would choose.

B

So they deployed our demo and we can see that the container created and so on and that you have all the file being up running on the work too here which, in example, all the the logs so the logs on the chairs, which are the steps each step did that happen.

B

um I think it's a pretty clear debt so in that fine, what the police, the strategy police, you know, the filtering part and then go to uh select what nodes are available that doesn't break the don't scan the rule and then go to the prioritization part. That is basically going to relate to the scheduling method strategy and give the rank and select based on this rank, sent back to the full schedule. Okay, what should they have the hyper arts? And that way we go okay.

B

Now um we have now five parts running on the walk too. So there is reason to to think that. Okay, so we have one method, that's related on the memory use, so what we expected that the value matrix for them, the the the nodes that have their map should increase in case on the on the walker two, because we uh we spent yeah, we have more positive than that. So we do a little bit of the push that, in order just to go above the threshold uh related the labeling strategy.

B

So we saw just saw that the labeling happening there. Oh sorry, so the labeling happened in that and then we take a look on the logs and it's just saying that. Okay, the metric provider on the walker tube just crossed over the threshold and for the label strategy break the dead. And then uh here's going to write off the label that have been described on the policy to the to the node that broken that rules.

B

So just continue that and that's that's what is actually just describing on the log.

B

Okay, so that that's what is at the moment, so um we have what we have now two nodes that are not able to receive so one because the breaker they don't scatter us and uh the other one. That's because then they are finished rules and you know able to receive a lot.

B

So what we expected just one left and if we increase the number of the parts to be deploying on the uh in the closets or just not upstairs, and they should end up as just show that on the walker on the or on the first one so and yeah so that again, so, let's just show the process how this one and you have the label on the work and we get what respect than that. um We still have parts on the work truth, so they most likely are not happening.

B

That's a little dependent situation and depend on your policy, but we we would like to move as that migrate from the more suitable nodes. So gs, as I mentioned we mentioned before- that- is not capable to evict the parts of in this case.

B

We we have to call the help and help it on the kubernetes schedule, and uh once that we have the pulse the config configuration deployed, we just deployed the application and you see immediately that this community schedule going to look at and see there are what's uh parts associated on the nodes that break their finished rules and then immediately evicted that so the cabinet schedule work on the cycle.

B

So once that you probably immediately do those checks and then yeah so going to put the other parts on the queue in the comments on the cycle to defer scatter and find a better, better node available. That is work.

B

Now we don't have any more uh parts running on the orchestra again, so no pods, so memory should go down the memory using something else so that matters associated should at least go to back on the same levels.

B

And when that happen, okay, uh we don't have a delivering strategy rules and a broken anymore. So we expected the label to be removed and that's what happened once that happened. uh Work to working on. You is again background alive, so right to receive the parts again, but the work now have a 10 points of the same story: memory use going to increase so as inspected break the rules, because above the threshold on the level stretch, uh yes write the labels on that nodes and once the label is there remember.

B

The kubernetes scattering now is up running on the cluster they're going to look on the node list, and you see there is one node that breaking the affinity rules and the one is started in vector all the parts on that and that node and we scatter to the work that are available and in the class at the moment.

B

Okay, so yeah, that's it so.

A

Okay, so before we conclude this session, just wanted to give give out a bit more details. So, as I said, telemetry aware, scheduler is an open source project and, above you can find the link to our public repo. If you're interested, you can look, look at the repo push pr's or issues. If you see anything that you would want to work on or fix, if you're also, if you're interested about features you can reach out to us via the kubernetes slack room or emails, we have our emails in the bottom right.

A

You can also find us in our booth, so feel free to stop by and talk to us if you're um interested about learning a bit more like architecture wise, a bit more details, you have links to 12 hour white papers so feel free to check them out. If you want more details architecturally this is it in terms of presentation. Thank you very much for for your attention and I'm going to open it up for questions. If you have any.

A

C

Hello, I'm here over here, I'm.

A

C

No problem what you show us scheduler has now a dependency with the metrics pipeline. If the matrix pipeline fails, will scheduling be blocked or is it able to still scale ignoring the telemetry of our scheduling.

C

I mean we now have a dependency of the metrics pipeline. Is it a hard dependency? Is it optional? I don't know.

A

I I would say it might not able to react as fast, so you might need to wait a bit until the matrix come comes back online, but I think I need to test a bit more. The scenario I think this. This is my opinion, so I think you need to wait until it comes back.

A

Oh okay, and do you want to add more details to this? Okay? Sorry, can you come actually? Can you come to us once we finish, and we can talk about this work because I can't hear you okay. I need to know where to look okay or us to look.

D

Yeah I was going to ask what happens when the telemetry scheduling you have configured for for a deployment or whatever doesn't find any suitable node, but you still have available nodes. What happens in this scenario.

A

So I think it will just look continuously for nodes. So if something happens- and maybe your node comes back online- it might just like continuously look for nodes. If, if it wasn't, maybe you might need to go back and alter your policies, at least that's that's what I think it would do.

A

Tasc yeah, so with tas, you would get a suggestion of a placement, so maybe when that's not running, you would just default to whatever the default scheduler wants to do and your policies might not. Actually the ts policies might not work. Okay, any.

E

High, so so the metrics uh pipeline, it generates new metrics or you also work with default. Metrics and the telemetry policy can also be applied on the default metrics from the nodes or any custom metrics that are generated like via script or manually.

A

Can you, okay, yeah.

B

So you hear me so basically, you can use uh any metrics because it depends on your exporter. So in this example, we use node export for the primitives you can. You can use. Also qb metrics api as well so is, is up to you to select what type. What type of the method you are interested so collected also can be used as also okay. Thank.

F

You so I'm going to read out a question from the online users. Does tis work well with topology spread constraints to ensure an even distribution of application? Pods.

B

So they even distribute distribution on the deployment. I assume.

F

I guess they're asking yeah deployment or stateful set or.

B

um Yeah, I can't I can't yeah. We have to think about that. This is a good question.

F

um Yeah, maybe maybe follow up in slack afterwards, yeah.

A

With you again, whoever wrote can ping us on slack yeah.

F

A

The question yeah sure we'll try our best answer: okay,.

F

Thank you. We got a couple more minutes in the room.

F

uh I think he was first.

G

uh Hello, I have a small question, so what will happen after our port is scheduled? So when it's uh during uh scheduling the port is placed one of the nodes and metric changes, will it port will be rescheduled during execution or it's ignored during execution?.

B

So your question is to know what happened on the pod after the discovery.

G

Yes, true, so actually default scheduler usually check this rules. Once you decide.

B

G

To schedule port and it ignores changes further, so in this case a metric is always changing, and during execution it changed what will happen? Okay,.

B

I think there is a time related on that, so it depends on the how how how the process it takes and if eventually yeah, if you get that time where the deposit has been discharged- and this guy definitely started getting on that, it might have that kind of conflict yeah, it's something that we haven't fought, because we basically use this schedule just from the distance in terms of to show the defectiveness of the of the application. So but yeah we is. Thank you we'll take a look at that as well.

F

Okay, I think I think we have time for one more and.

H

Hello, so my question is: is this scheduler going to respect the pdb constraints that we have for a report for the deployment.

B

Can you repeat that, can you.

A

Repeat the question yeah.

H

Is this a scheduler going to respect the restrictions set by a pdb by a disruption budget? How is that going? How I mean? How is this going to work together with a pdb set for a for about for a deployment or a pod.

A

I don't think we tested that approach. I don't know what the pdp is done. I think we need to look a bit more into this because I don't think I understand what are at least I'm I'm not able to understand what you're asking me. Sorry.

H

Sorry I mean a pub disruption budget, it's a configuration or a policy that you define in kubernetes in order to for uh to restrict how kubernetes a big spot bots from a node or the number of or the minimum number of pots that you have to run for a certain application.

H

So I'm asking that because if the scheduler is going to move bots around, it could actually violate the restrictions set by the pdb or the constraints.

H

Does that make sense.

A

Not quite I think I need to follow up sorry, please.

F

Thank you, okay. Well, thank you very much and that's we're at time. Thank you. Thank you.