Kubernetes SIG Scheduling, 21 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Scheduling Weekly Meeting for 20220421

Description

Demonstrated fake-kubelet

A

All right, hi, everyone uh today is april 21 and welcome to this week's six schedule meeting uh this meeting is being recorded so be respectful to each other. Let me share my screen.

A

So basically, there is only one item in the agenda because this side, this item involves some demo, so we'll spend maybe 10 to 15 minutes. So if you have some other items, you can just put it on the agenda. I can get this discussed so for this item.

A

The background is that in terms of the scheduling, evaluation or scheduling testing, as long as with some other control points testing, we actually do not need a real physical cubelet running out there. So this is why, in some of our skeleton scalability testing, we use the hollow node other than this to simulate the accumulator's behavior. So holo node works well and has been developed for a couple of years and also you should use some uh scalability testing tool like cube mock.

A

I also use it in the daily performance testing, but the issue with holoneau is that holoness actually represent a single, fake, cubelet right and it marks almost every behavior of the cubelet, but just make the importation as a hollowing stuff, just just have no operation out there. So, basically, if you want to simulate a 1000 cluster, you have to run the holo node 1000 times and given holo is not that high performance comparing to the fake calculator implementation and also you have to spin up one thousand times of the column note.

A

So it's almost like almost impossible to run that in your local laptop. So so, and this project, which called fake cubelet, I think more. Accurate name, is a fake kubrick manager because its primary function is to watch the node object, submits to the cluster and provide some very slim actions to make the note be consistent, simulating all the the cube latest functionality like once the part comes in. We just do nothing but make it ready and also for uh for no, it will maintain the hot bits. So basically, it has a very slim invitation.

A

So in my local desktop I can even start a 5000 mils without any memory pressure, just local, desktop and yeah. If you are interested, you can check out the the the dark and the implementation here. So one interesting implementation here is that it doesn't use the traditional node, informer or something alike. They just use a very raw and events channel to watch the no events, events and then that I think that's.

A

This is why it achieves a better performance than some other implementation to simulate the keyboard cable. Okay. This is basically the how the fake cubelet looks like I'll give you a short demo. So I suppose you can see my screen right.

A

Can you? Yes? Let me let me know if you can so basically to use this.

A

It only runs as a controller to watch the nails you add into the cluster and also it has a mod to start a couple of default nails right for you. So to make it works, you have to have a fully functional control plane. So right here, I just use the latest no latest kubernetes oscar and run the control plane without the cubelet yeah. There's a script called the local cluster, so I can step barely a control plane without any kill blade.

A

Just give me a few seconds. It should be quick because I'm running out more max.

B

So if I understand correctly, the main.

C

B

Well, there are two differences with cube mark right. uh One is that in cube mark there is one part for each node right there, one one container very.

A

B

There and here there's a single container simulating.

A

B

C

B

From scratch, as opposed to cubemark, which uses a cubelet code,.

A

Yeah, I think cubemark in particular in particular holland node. I think leverages some cubelet. You know kiloplate implementation and uh unless, in my local laptop it doesn't scale well the hollow node, so yeah. So you can see here. I have a control plane.

A

Is more focused on simulating a particular provider like you can simulate a vm? You can provide a whatever entity that can join the cluster, but underneath it can be anything so for for us, the requirement is not that complex. We just want to like simulate the testing performance testing functionality, testing on the control, plane, components.

A

We are not trying to simulate the particular entity as a as a cube rate and join the cube cluster so different purposes, but it's just the name is a little confusing, as I planned. As I explained here, it's just a controller and take over the the node join the cluster and make them behave like a cubelet. So that's the case, and uh so basically I have uh just combined.

B

Here, given that it doesn't actually start containers yeah, I guess there is not a lot of risk in terms of drift or or how do you handle drift like uh if there are new features in cubelet, you need no.

A

It's not it's not aimed to to test any cubelet feature. It's just testing the scheduling, feature or control manager or api server like the scalability stuff.

B

I guess more generally.

B

What does cubelet do huh so if, if.

A

Cubed is more associated with the containers. Spin ups containers, runtime behavior, and but if you think about scheduling, we don't care too much about the runtime compared behavior right. So definitely it has the limitation, but it's the clear boundary that we want to raise this to. It doesn't touch anything about the cubit.

A

So here I want to stop this uh battery and uh let me do something crazy. Is that uh mr 1000 notes I'll give you to take a look to see how it works. So, basically you you can just ignore this option. If you're ignoring this option, it just means okay, it just starts as a controller, and then you can watch anything else gets added into this character later, so the takeover parameter shows that you will take over any else add into the cluster.

A

So if this parameter is just to ensure you maybe have a hybrid mod, you want the real node to join the cursor. But in our case just all the notes are fake. Let me show you it's just basically less than 13 seconds.

A

You can see.

A

Basically, all the nodes gets managed by the fake keyboard manager.

A

Okay, 1000 that should be ready, and so why spin up a lot culture matters is because we want to simulate some, not only the scalability.

A

Concerns as well as some, if you know that inside schedule there is some parameter called percentage of notice to score, and if you test a particular feature in a small cluster versus logical, the behavior may not the same.

A

So you can see. Okay, you just take uh 15 seconds to start 1000, fake nails, and uh if now you only have some default.

A

So here you can see, this thing is scheduled to a very fake note, 12, and if you, if I look, I usually use it to simulate or verify some like the weight, as well as I'm testing some new plugin, especially the scoring plugins behavior, because I want all the scoring plugins to work together against the particular workload to see how it works like this. Jumbo is a experiment that some customer says that they want to evaluate interpreter affinity's, plugin wave along with the workload level weight.

A

So this is why I need this kind of to to help. You help me.

A

So you can see it's spinning up to a different uh different, the note.

A

Okay, this is basically the how the cuban manager works and what's the scenario I want it to be applied to another thing I can show you is that so basically it has some default node, which is spin up by the parameter 1000. I just show you right.

A

It has another cool feature is to take over any node object that gets introduced to this cluster, so I have 1 000 node, already right the log because it starts with index zero. So I want to introduce the 1001 node to the cluster.

A

So with this feature, you.

A

A

You can see the new node has been added to the cursor. So in this with this feature, you can very flexibly uh compose your cluster with whatever spec you want, you can you kind of have some hybrids spec for the nails, so you can easily compose a cluster with the fake specs.

A

Okay, any questions so far.

B

um I was wondering if this would.

A

B

A

Scalability scalability, I think I I that's my impression, but I need to talk with six capability about these. Maybe there's a potential uh consideration out there yeah, but to my understanding yeah, it can be a better physical for that.

B

And also yeah, I mean they, they might uh have some insight of what you could be missing yeah, because I I don't really know what um what cubelet does besides starting containers, I don't.

D

B

Like there's, some behavior you're missing out and yeah.

A

Goodbye is also responsible for maintaining, like some no less heartbeat, as well as some other functionality. Those part has been has been covered in this tool.

B

C

Doesn't report heartbeats what.

B

Happens, the the body is deleted or something.

A

Yeah, if you, if, if a kubelet, doesn't report her bit, the node manager will think this node has been unhealthy or unreachable, so it will add a change to them to another. So the hotbed thing is required, so you can see that it will submit the hobbit to the apso regularly.

A

um So yeah go ahead.

B

Yeah, my last question is: uh can you repeat that what you said about the watcher versus informers.

A

Yeah, this is more.

A

Detailed a little bit.

A

A

Call the refactor.

C

A

So basically yeah it's yours- are raw channel go channel instead of using the power informer to do, because you feel think about informal is sort of overkill in some scenarios so for performances sake, the implication is intentionally use a raw channel to watch this kind of thing and yeah. This is the. I think this sort of magic run here to make it perform it.

D

Yeah, I think you know using this channel um yeah. You will improve this raw. China will improve performance, um but that's it, but it doesn't simulate on the situation that you know. If you use um in the real situation, the couplet will watch the all those uh object resource um through the informal mechanism which may be a bottleneck there right with the api server you know, could be about make there. But if we, if this change so that part will not be simulated, if there's a problem there.

A

Again, any cubelet specific feature functionality won't be tested or covered right. Here we just here is a fake. It's just maintaining the minimum requirement to claim this. Node is ready and also the minimum function, like a part, comes into a now and make that part as a ready step. So basically it's just yeah. It's just has some limitation. It doesn't fully simulate the cable behavior. I think this is the I'm not 100 sure, but it's almost the case for hollow node, which you use in our scalability test.

A

So the scope is just right here we don't the the behavior okay, I see thanks okay, so this is one mod one particular persona is that as a regular kubernetes developer, so this is the most uh feasible workflow. I want to use to testing some scheduling, changes like in place and start the control plane and then started effectively to simulate uh to, to finish the whole loop, to evaluate a particular uh uh verification work. So, but there's a to a larger scope is there is some other personas like sies, like application developers?

A

They can also be a to be an audience of these tools, so basically how they run the kubernetes cluster is either they run the real cluster or they run a kind cluster. So so this tool can also work with the whatever control plane easily. So let me create a cluster, so basically, this command means I create a onenote cluster, and that know the worksites, the both the control plane and the cubelet.

A

But we don't use that node for scaling purposes, because the byte default has a as a tint.

A

And just to ensure there's no other items that can used all the time.

A

All right, so it's ready it's basically a medina 127 cluster and next I'm going to deploy the fake cubelet manager to this component.

A

So it does nothing special other than doing the privileged thing and the row binding our back thing and.

C

A

Okay, the core case is here, is just a deployment and the specifier image so that the image is just nothing but the binary. I just assure you- and it has some environments like take over all the nodes that guess I did it into the cluster and by I just spin 100 nails here, so you can specify any other number.

A

Oops I've changed.

A

Let's look at this.

A

A

Parameter, there's nothing else, changes.

A

Anyway, uh let me just ignore the takeover parameter.

A

And then it gets created and for now there's only still one note and underneath the fake cube, it is starting itself and also spinning up. One thousand notes.

A

Now you can see just within seconds, you can just all the fake notes here. So this is a more typical use case for the sies and the application developers, because they don't want. They usually don't touch the source code of the kubernetes and what they charge is the usually the configuration of a particular component like scheduling they change the like change the default weight or arrange the plugins differently.

A

So in that case they just.

A

They should provide a configuration file to the current, so they can make customize the cluster and we saw the fake kubelet. They can just run their simulation test suite to to do the verifications. Yes, so that is why I think this is useful for both the kubernetes developers as as well as the regular users srbs.

E

We actually posted a question in the chat: uh what's the difference between schedule and perf, and this like the only thing that I can notice, is uh separation of configuration like technically. We have the node object and the pod diamond that we want to test in the code there, whereas I think this particular utility has separated that configuration out which makes it easier. Is that the only difference or am I missing something so basically.

A

Different personas and different mechanisms so for the integration test, it's basically oriented for the programmers and also it's spin up the cluster and do the testing very programmatically so which is not easy and not intuitive for the regular user like the ieee or application developers to use, and also they may all, they may also don't have the make chain. They have the golem uh the environment.

A

To do this kind of thing and also they need to specify now the what's the each test case out there under the scheduling perth and if they want to add something. This will be very difficult for them because they may lack of the knowledge out there so yeah. That is the main difference.

E

I see so from a developer standpoint is going to be beneficial. Is what you think, because we are separating out uh like the api server starting or uh technically, we are creating our own control plane, using kind or whatever speed or whatever the tool that we want to use, and we can add this fake, no uh to get it moving. So I think what you're saying is it's the separation which would help the developers?

E

Yes got it. Okay, that makes sense. Thank.

A

You basically for for the kubernetes developer. You can change the source code as as in any way as you want and use this tool to test the changes you made for sres.

A

They just have the bernina control plane and just tweak some configuration parameters and then run their tester on top of that. So this can facilitate the the different persona studios there and but the benchmark testa still has is a place to be there, because we care more about like the go, long embed the benchmark testing, as well as some uh prometheus metrics running along with the code. So we in the latest benchmark.

A

We extract the metrics and give you the final results, but for for these two, I think uh for now it's just the first step. It's like a it's like a very convenient tool for you to set up your cluster with with any number of nodes you want, and based on that, you can connect this because it's almost a real cluster, you can connect it to the regular tools. Regular monitoring stack, you use to do some further stuff. So it's more yes, it's more, not that programmatically to do changes to do tests against the cluster.

A

It's more have all this wrapped in a very convenient way and test against the cluster.

C

Yeah, I have a question yeah yeah. If I want to deploy uh two types nodes like one hundred for uh four cpus and another hundred four turn cpus, and do I need to create two free copy to manager here? No, you don't have to.

A

You don't have to so you just provide two node templates so for here you can see.

A

So you know the cpu.

A

And change the cpu stack here and then another ammo, you change. The memory is back and then you can just you like for, like uh you, want to start this cpu 1000 times and then to kill supply, cpu, etc and for another one, do another 1000 iteration to apply another spec. You don't need to start the manager points when manager is just a single turn around in there yeah yeah. Thank you.

A

Yeah, I do like this too pretty much so this tool is uh authored by bashimi and he turned to me I think, by the end of last year, and by that time I thought is almost duplicated, with holo note, because, usually by running some ad hoc casting spin up the hollow node, but I didn't spin up hollow node 1000 scale that when I tried to spin up the whole node with even with 50, it doesn't scale well on my local single laptop.

A

So that is why I think this this tool is super powerful to do the things for scheduling. I think it's also beneficial for other components as well. So next step is that I I may also demonstrate this to see scalability and also maybe seek node, but signal node is pretty overloaded. I don't think they have.

A

They maybe can afford to be the sponsors so next step. I will mainly demo this to see scalability to see how it interests them. And finally, I will try to raise it as a kubernetes, 6 sub project, so to draw more attention uh to contribute, contribute to it and also facilitate your internal companies.

A

uh Developer experience to shorten, for example, when your developer schedule plugin or tweak some weights for a particular customer, it can short shorten the deciding decision cycles.

B

Wait and my other piece of feedback would be to rename the problem.

C

Exactly I would even stream the word cubelet.

B

And just say, maybe fake node manager, yes or.

A

Or even column, something.

B

Like that, at the beginning,.

A

At the beginning, I was also yeah used by by the fake cube plate. It seems like the same thing as the hollow node right. It's spinning up the fake cube rate a thousand times, and each fake keyboard connects to the api server. But it's not the case actually.

B

Sounds good yeah. I think I think um it's better to ask see, see scalability first right there and then then we decide because it doesn't fit. The charter of 6 kelvin yeah.

A

It's just a yeah, just a scheduling can be a primary consumer of this tool.

B

Yes, I agree that we could be primary consumers, but yeah. That's why we can reconsider after talking with scalability. I think.

A

A

All right, uh this is all I have for the cure plate.

A

Yeah any other things you want to discuss so one other thing is that I think starting the previous meeting. We have extended the time from 30 minutes to 45 minutes so yeah you can add more items you can accomplish.

A

And if you don't have any other things to discuss, I will give eight minutes back to you and thank you everyone for joining today's meeting. We will see you. We will see you in two weeks, yep bye. Thank you. Bye.