Kubernetes SIG Cluster Lifecycle, 6 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cluster Lifecycle - Cluster API Provider Nested office hours 2021-04-06

Description

#sig-cluster-lifecycle
#capn
#capi

A

All right, good morning, everybody, uh this is the april wow we're in april uh april sixth uh cap and office hours. uh We have a pretty short agenda, but phase I'll hand it over to you uh to demo the multi-cluster scheduler that you've been working on.

B

A

I need to make you a host. Give me one. Second.

C

Yeah before that, how about just go through the status I mean obviously.

A

uh I'm still working on the on the the the. Why am I blanking on names right now? The control, the um nested control plane uh controller, so the types got merged, uh ciao who's, not on the call. I basically just put a I put a note into slack, I'm inclined to merge the the api server vr that he has open. um I don't know if anybody else wants to give that a a quick look before we before I give it an uh looks good to me because it'll just merge once we do that. So, if.

B

You've already yeah, so you you haven't shipped it right.

A

uh Yeah, it looks good to me, I just don't wanna. I don't wanna comment that yet in case uh in case anybody else has feedback before we go through and I will.

C

Take a look after meeting uh if I think it's okay, I'm going to merge it perfect. So now next thing is uh uh nested uh controller.

B

C

This child is online, so um maybe child can give a quick update on that. Is that ready or do you need some time.

D

um Yeah, I'm I'm still waiting for the nasty api server controller to be merged, so after that one is merged our rebase and submit the pr for the nasdaq controller manager, it's quite similar to nested api server. So I think it should be ready in one or two days it can be ready in this week, so yeah. If everything's, okay, we can merge these three nasty component, all to the mark, uh ultimate master, yeah.

A

Nice and I'll probably have the I'll, probably have the ncp uh controller tomorrow uh by end of day tomorrow, uh at least a pull request up there. So we can take a look at that, so we can start to converge everything together.

D

Right right, yeah sounds good mm-hmm cool.

C

Cool, then, okay, if there are no more agents, I can do a kind of brief discuss description about the multi-console thing.

A

Let me make you host real, quick. Sorry give me one. Second, all right, you're good to go.

C

So yeah, let me see you I I'm kind of what should I do should I click share screen uh yeah, okay, yeah.

B

B

Let me try to share this one.

B

uh What did you guys see, we can see terminal there, we go. Okay. uh Can you see this one? Oh, can you see.

A

This one we can see one that has uh your root to context in there.

C

No, can you see the chrome, the chrome web now just like this? We know okay, so I I think I I'd better start with some uh high level distribution. First, yeah that'd.

B

C

Sharing screen and this one, how about now.

A

Chrome is showing and you've got the uh cap and office hour stock.

B

B

C

Yeah, let me actually just come with this. This figure. Can you guys see this okay yeah, so um some backgrounds for the people, who may not very closely know what I'm trying to do and the problem was that um we have incoming requests that people ask for saying. If this system, the vc architecture can be extended to support multiple super clusters, because our previous design is kind of unmapped mapq one can means number of the tenants.

C

So it's mapped to one model uh yeah the sinker was designed to you know, have a marty mc controller to sync all the multiple tenant controllers request into one super clusters. So uh there are quite some use cases for this in terms of supporting multiple super clusters, one is, for example, for the edge computing. They have multiple uh clusters spread across different geographic regions. So um there are some technical issues or whatever management issues they don't want to combine with a huge genetic.

C

You know how much just one giant clusters they'll spread multiple small clusters, so if we have a way to utilize the resource of that many clusters, that's always a good question. uh Another thing is: um we cannot so in a cloud company. Probably we can.

C

We can somehow assume our supercluster has auto scaler enabled or that capability, but for other companies like on-prem data, centers or other use cases other than the cloud vendor. You probably cannot make assumption that the supercluster can be auto-scaled, then in that problem. uh In that case, uh you always has a has a problem such that, because we don't expose the detailed supercluster details to the tenants, so tenants always assume they have an infinite capacity, but if, in fact, that's not the case, so how can we scale?

C

The superclass is also another questions. So another way of scaling is just give another super cluster, so that is another way of scaling intel capacity from a capacity perspective. So due to these a few reasons, uh I'm thinking of so um so in the past a few months, I'm trying to see if we can extend the model export multiple super clusters, um the the the idea is uh there there are. There are a few problems right. The first problem is: how can we um change the thinker, somehow support kind of m2n mapping?

C

It's kind of m is number of tenants and its number of superclusters so um because you, you need to have a way to do the synchronization between this kind of a multiple entities uh we, I have come think of through the um possible. You know changes on the sinker um in in. In the end, we have two ways. One way is you know it's kind of we let the sinker to be uh a kind of two-way mc controller, so ymc controller for 10, another mc controller for for for super, that's that's doable kind of doable.

C

Another way is kind of um like the the way that I did in the figure, so we kind of uh horizontal scale the number of sinkers. So we still use one sinker to take off one silver clusters, um but uh there is a one-to-one mapping between sync and supercluster, so I tend to go this route due to a few reasons. First is so architecture wise? This is simpler and the changes to synchro to support the to making housing work is minimum second thing.

C

The second reason is that uh from performance perspective, this has per least performance. You know impact of. Otherwise, if one sinker is connecting multiple super cluster, you always have contention. You will have another level of contention on the super cluster uh accesses because you have a centralized hub in the sinker. uh uh um I don't want you bring too much. You know uh performance concern on this design, so instead I'm go. uh I tend to go with the multiple sinker approach like each one.

C

Supercluster has one sinker um so another, so so another problem is: um uh if you look at this way, so you always have a problem uh in a sense in the past. It is very clear if you have a tenant part, it's clear which supercross it is it is. It has to has to only have one single cluster. So there's no doubt that there's no scheduling decision here, um but now you have a scheduling decision in the sense that you have multiple super clusters, you need, you basically decide which super classical the part should go.

C

This is a kind of I would say you know in all the multiple multi multi cluster design. This problem is always kind of kind of ignored or or or overlooked, for whatever reason, probably, why is this hard problem? uh Secondly, is uh but without this capability you so every existing, you know multi-tenancy multi-cluster workload management, you, the the user, has to somehow understand the the usage of the super cluster or the multiple kind of clusters so decide um the for example.

C

They they need to decide how many revocable to use super has one replica super class 2. This is the way that you know kobe fed does, um but I want to do this in a kind of more transparent way, in the sense that I want to. You know abstract the entire super clusters, multiple super classes as a one. So from the user perspective, they shouldn't be worried or care about how to distribute the the paths instances across the civil cluster, which I think is a kind of manual tedious work.

C

uh Making a lot of you know, automation, framework broken. So in a sense, I want to do uh more, intelligent in the sense that if there is a tenant apart, we need to you know the system needs to automatically decide which superclass it goes um so for for this guy, I want to so pretty much. We need to introduce another level of scheduling in our framework.

C

That's um basically you you need to have one schedule schedule decide um which supercluster, a canon part should go and uh another level scheduler is in the silver class itself, so you decide which node the part should go. So this is a pretty much the the the high level uh uh from the party. You know scaling perspective. This is the way that I think uh it goes and and then this is the model I'm trying to achieve.

C

um Another reason I want to, if you look at the figure, I call it a namespace scheduler, which is not a part scheduler.

C

So the reason of doing that is I'm thinking of another routing sense that if you people really want to use the system again, you still need to some users still need to tell us that the quota, the the the resource, the how many resources they probably need in the first place again, because we hide all the super cluster details without doing that, it is possible that you know one user, because this is a shared multi-tenant platform. It is possible that one user- without you know this quarter, to enforce kind of capacity planning.

C

uh It is possible that the user just keeps running the the job and use of the entire supercluster resources where others just cannot use it. So, from the you know, capacity you still have need to have one level have a level of capacity planning, um but it's already on capacity parenting. There is no scheduling decision, it's just a capacity planning. I still, I think this capacity this you know this kind of capacitive planet is still needed.

C

um I'm here I'm kind of leveraging the existing quarter system and because the coda system is, you know, has a tight. You know uh relationship with the name space, so I call it this kind of namespace scheduler, but in reality what it does is trying to schedule a quarter in within the namespace across multiple superclusters. This is what the namespace nemesis space scheduler does um so overall, as this figure shows, um what do we? What what the essentially people the general workflow is something like this so tenants create.

C

So we have a per supercluster sinker, and this thinker need can synchronize the tenant object to the corresponding supercluster and the talent object. You need to have you know, label or whatever tell the owner so which supercluster this. This object should go right and this labeling work is done by the scheduler uh any, and so once the people create a courthouse within attendant namespace, the name space scheduler, trying to break this quota into small slices and spread the quota and do the scaling based on the available capacity of the super cluster.

C

For in this at this showing these figures in this figure just show. Example, if you look at the red placement, the placement is the scheduling decision. um uh You know down by the name space scheduler saying that okay, this name, this namespace, has a coda, cpus memory y and the schedule tell you this code. This name space should be placed in class one class two. So once we have this label, you know attached to this namespace the what the following the following things.

C

We do is the synchro notice that oh, this name in space can be placed in c2 and c1. The c1 sinker we are created namespace in c in cluster one, and this class two single will create another, the same name space in the superclass2. So this is the first step. Every time you have you create a quota and the scheduler carry you. The placement cluster, the namespace should go. The corresponding synchro will create the namespace in in the corresponding superclusters.

C

um This is the first step, then next level, when the part is actually create creating in the namespace. The namespace scheduler also need to decide which superclass should should go, but now it doesn't need to go through all the superclusters, because the namespace schedule tells you. This namespace resource can already go class, one class two. Now the the now oh, the name specific scheduler only need to decide okay within the c1 and c2 which cluster the part should go.

C

This is this is another steps of the for the part, scheduling but uh think about this way so because, in general the placement cluster number should be small and the overhead of doing this step should be small, because even you have, for example, you have thousands of super clusters, but for your name, space, probably only placing a few very few uh superclasses that can accommodate uh order in a quarter request.

C

Then then, this level scaling just from the three clusters peak one which can satisfy the pop pass request and that's it then, once the part have a uh annotation say which cluster superclass should go, then the sinker real copy that we are seeing that part to the corresponding supercluster, and this is kind of the general workflow. um Do you guys have questions up here.

A

All makes sense so far: okay,.

C

Cool then I'm going to switch to my.

D

D

A

Can you guys see it.

A

Yep uh as long as it's just.

C

One terminal yeah one terminal: yes, yep, oh um all right, so this is the one the current cookie config connect to the meta clusters. So basically the meta cluster has everything about the um the um the vccr and the class cr. So if you guys I haven't mentioned, but I'm using still using vccr to model the um the virtual clusters, I'm using the class capi cast cr to to monitor the to to represent our superclusters.

C

uh I think, if you so yeah forget about this tenant, so this is probably the wrong way. So if you look at the.

C

uh I think I have one here yeah. This is the. This is a secret in the super. Me silver means the name service, I'm I'm here. I I'm trying to model the super clusters, so this is the cluster cr. uh I have two. I have two uh super clusters with different names. That's probably okay, um and if you.

C

I have one tenant, so this is the model that uh this is the model that I show in the figure in the demo. This is the meta clusters. I generated a super cluster using using mini using mini cube. So if you look at.

B

C

So this is the two that could be coming to get these two um super clusters in the um to see the name space. So I'm going to clear everything so make sure. Then I can go from the start. Let's see, I have.

C

A um this is the way that I can access the this is a tenant master, so I have a one and one name name: space called default and I I'm using currently I'm using default namespace to do the experiment.

C

uh If you look at everything I have now, I'm going to go through something like how to clear everything. um If you can get the code, I think I have a deployment here. Yeah it has four ports. Let me try to delete everything.

B

C

C

A one corner try to delete this.

C

C

C

Yeah here on, uh as I can see in the vc manager, we I have besides the vm agent uh vc manager, I have a scheduler 20 and two sinkers one sinkers for superclusters.

B

The typical way that I see.

C

C

C

Too many logs, uh let me try to make these things that will be simpler.

B

C

Yeah give a few time for the schedule to restart. I want to start with scheduler.

C

Yeah, um so in the schedule, every minute I'm going to dunk intel on the schedule cache. um This is the description about the cache. The cache have three components like clusters and namespace and the parts so in this case this empty one, because I just deleted the code, so there's no namespace being scheduled because the code has gone, the path is empty, as no part is running. So in these thumb dumpsters, you only show the two root clusters: the status of these two root clusters.

C

um There are some important things like you have how many are allocated, what are allocated atoms? What is the capacity, so this is from the tenant perspective. Basically, uh from canada perspective, there is no part it's allocated, there's the entire capacity has not been used and the other rest of things pretty simple and there's positive. What are the parts in these clusters?

C

um This provision is another is the it is the capacity that allocated but reported from the supercluster so which means the you have a schedule. So, every time you schedule something the scheduler cache will update to the allocate and if the scheduler, if the scheduled decision has has been reached to the supercluster by the sinker, the silver clusters, then we we are, uh then the cluster, the the scheduler. We are updating the provision.

C

One saying that okay, uh this is th. This allocation is actually are provisioned to this. There there are, a bunch of allocations, are actually provisioned in the rule castle already. um I can't explain this a little bit later, but I think this is not affects the overall functionality. This is odin. I I make this design to your white watercase like for the for the fader recovery, because we will.

C

We will have a case that um the super cluster, the telomere cluster, is kind of offline for some reason, but the the parts still running in the superclusters.

C

So if, when we recreate when we recreate the entire uh the scheduler cache, if we don't get information from the super customer, what are actually running, then our cache will be wrong because we are we, we have this capacity, but we are not accounting for the actual usage. The scheduling will be will be wrong. So that's the reason I introduced this provision and the provision items um yeah, that's pretty much, it is we have. We have two superclass here and we uh they are completely empty.

C

uh Now, if you look at the key is here we play with the namespace and the quota as a user interface. uh If you, if you look at this default namespace, then we we have an annotation, so this annotation indicates so for every namespace.

C

If we don't specify we, we will use a default slides, but I would recommend people to introduce a slice. The slice indicating the smaller is the schedule entity for the scheduler. The reason why I introduced slices- um I want to avoid kind of fragmentation in the sense that if people specify a quarter of this big value, then then, if we cannot break down this color into small pieces, it is partly where it is highly possible in the supercar.

C

So you will introduce the fragmentation and the idea is is based on the quarter slice and the total quota. You are specified, I'm going to calculate how many slices you have. Then then, the scheduler just trying to schedule this slice across the cluster. So basically you can think about. The analog is like a memory page, so you have a matter of big memory but in the end operating system, the body system- just you know,.

C

C

Control the distribution, in the sense that, if so, if you want you your name space to be scattered to as many super classes as possible, you probably will make a smaller slice.

C

Then you, you have a chance to spread more slices to more clusters where, if you want to the uh the number space to be cons, concentrated to a smaller number of clusters, probably you can make the slice number a little bit bigger, so they're kind of a pro always this is kind of classical trade-off between the page size, allocation and fragmentation. So I gave you this so I introduced slides to make this flexibility in our scheduling.

C

Otherwise I is kind of very strict in the sense that you have to place a big chunk, which is the cortex. uh Sometimes it is hard to find a placement hard to find a sloth to feed this quota, and sometimes it will introduce fragmentation now. This is the one key concept that I call it slice um so now. Next, if you look at the quota, uh this is the code I'm going to specify uh the quota. Saying I'm just you using this chord saying that you have a two uh cpu 200 and the memory 200.

C

If you, if you look at my slice, is 100 pretty much. You know if you do this you're going to place two slices. So what I do is I'm going.

C

To okay, now, if you look at the name space again, so you will see this one, so this one is the scheduling results the scheduler for this scheduling without saying that, okay, you have two. Now you have two slices uh and I tell you replacements: is these two slices go to the root cluster? One.

C

Because root cause apparently has enough um capacity to to just hold the 200. uh 200 uh uh uh mini mini core. It's just me. I think it's meaningful yeah and a 200 megabyte.

C

C

If you now look at the dump again, you will see now the down show that I have one namespace being scheduled this. This namespace uh has a quota of 200, but the slicer has a 100 100 200 quota in the 100 slides. The scheduling result is to pull this to root class, one with two slices.

C

Now, if you look at the root cluster one, you will find that okay, now it has this many allocated and the allocated atoms. What are the items? The items of granite is the slice.

C

And there's actually provision proving in a sense that if you look at the root cluster, one.

C

You show you will see that this this this names before name specie, is synced to the root class one so, which means the actual allocation are provisioned, so because th, this provision item in indicate that, in the supercluster this this this name, space is actually synchronized to the supercluster um root cast root. Cluster zero, zero, zero one, okay.

C

Now my current designs, you can change the code so and we will do rescheduling. So this is a kind of something slightly different with pop scheduling is, like you know, the immutable results once part. Scheduler node cannot change that anyway, but I think in this model you have to provide this functionality. People should be able to change the quota if they want. So um in current design. I allow people to change the quota, for example, if I do this.

C

If I change the quota from 200 to.

C

C

See that now I have four slices and our schedule. We are going to change the scheduling with ours, because we have four slices. I'm going to spread these four slices to two super clusters. One supercross has has three and another surprise has one um once you do this. If you look at the superclass two, you should see that supercars 2 also synchronize. This have created a namespace using the namespace and if you look at the logs, I hope it will wait sometimes to print every minute.

C

Okay, now you can see that the super class y should have three has three allocated items each item: the unit is the slice so because they have three slices and the root class root. Two has one size allocated and the name space is kind of three one distribution.

C

Okay, so this is the from now uh yeah tiana. I show you how this namespace scheduling work, basically leveraging the slice and based on the quarter, that the user specified our scheduler will schedule the parts to the uh sorry schedule, the the name specializes to different clusters and update the scaling results in the placements, and currently the sinker has been changed. Enhanced, so sinker will every time they find the name space. The 10 names has a change.

C

They will watch for the change and check for these placements to see if the sinker's owner supercluster is in the placement list. If it is in the placement list, it will sync the route, the namespace to the supercluster, the corresponding supercluster yeah um till now. Do you guys have any questions.

A

This is super cool. uh I got one quick question um so with the slicing, what does that mean for uh the the network space? I know when you originally wrote the doc on this. You talked about this. This basically negating the use of things like cluster ip and um unless you have some some higher level custom network, that's that's able to operate that. um I assume this fits into that same model where we, where you end up still needing like every namespace. Now it has to navigate through internal load balancers or.

A

However you, however, you operate cross cluster. Is that still the case? Yes, it is.

C

So so the slice is kind of uh the the you can see. Think of slice as a tubal of the resource that you care about. So for now the cpu memory, uh for instance, it can be extends to gpu, because, for example, you can make saying that I want cpu memory and gpu is another number right, so you because uh assuming in the future, if you want to extend it, you need to extend the quarter to cpu memory and something else.

C

So this has a one to one correspond: the um the one to one relation between the quota, the everything that you can specify in the quota in the slice you can use the same. You know the tuple, but for the networking. Yes, I, uh the network is completely orthogonal problem. As of now, we should always assume you have a customized load balancer to do to to do the kind of the service to provide a service to provide the services.

C

um You also need to make assumption like say in a sense that the the entire super cluster you need to have a way to do the routing across the clusters. um This doesn't affect doesn't affect networking at all. It's purely like resource like cpu memory or can be standard.

A

Yeah that makes perfect sense awesome. This is super cool I'll. Let anybody else ask questions if they have them yeah.

E

uh To say it's a very good. The demo is a very exciting feature, so I think, there's still uh uh some uh complications.

E

How we deal with the vin agent, like a couplet uh reporting, the status where you have a lot of uh management to filter out different uh couplet, doing different parts, because your schedule is a one direction, but you need also all the time to getting this complex status.

C

From different clusters uh I haven't tried, but my I thought the problem is as long as you, so so here's a few things right, so my thinker is currently installed in the meta clusters. My scheduler is currently installing the meta clusters, but the agent need to install in as a demon setting every super clusters.

C

So if you think about this, if you, um if you, if you look, if you remember my previous figure- is like the pod, the um so think about that way. So as what I can say that that, if you assuming that you don't have any network problems, everything is a flat flat network.

C

uh If you have a via agent installed in the uh in the uh in the super clusters and the rest of the model is still the old, you know vc model the the sinker will copy the create a v node in the tenant right. We will give it a real agent ip in the tenant so tenant. Every kubernetes code will still be directed to the agent.

E

Okay, so still doing the same thing: it's not too overwhelmed too much no to taking care and got some.

C

Bottleneck, I haven't fully sort of thought about it, but if you look at the the model that I'm currently using, is we still using uh one agent pronoun when we agent pernode.

E

Yeah essentially,.

C

The load is how many parts it can possibly support right, how many parts, but the single, but but but that number of parts running, that node is kind of limited. It's capped by the node capacity, so it won't break too much security pressure. In my opinion, because, regardless how big the intel system is, the number of parts one reagent is taking takeoff is still limited, like 100 parts.

E

So you're going to have a 1bn agent for all the kind of uh uh yeah.

C

Okay, it's the old model, one rear agent. Now, I'm just thinking about the old model for reagent one reagent pronoun.

A

E

So you're just expecting.

A

That that every cubelet is still addressable by all of by any control plane, that's deployed in the meta cluster is what you're getting at, which makes perfect sense. I mean that that that fits the flat network model and then, if you wanted.

E

A

I know we've talked about how how this is less efficient when I was just responding to a github issue about this, but you could technically technically use the deployment route um for the vn agent that we that we added uh that would expose one vn agent load balance via an agent per super cluster that you could do different policies around to kind of um to abstract that more. If you didn't have a flat flat network, you could almost expose that vn agent again.

A

This would be kind of insecure and you'd have to set up the policies properly, but you could expose one via agent per per scheduling cluster per super cluster and then route through that using that that feature flag, that we added.

C

Yeah, the good thing is, I didn't break. The entire system is leveraging the vc model, so so, which means everything that you have been doing should be applicable in this model, because that's the key, I don't so everything that you are you have we have done so far because I didn't break the synchro. So I don't. I didn't, introduce a new api. Everything is just use native api uh and the mechanism is the same. I'm still using synchronization mechanism, I'm still going to expose the v node to talent master.

C

So then you can play all the trigger about which reagent so which proxy, which in a kubernetes proxy, the the request should go because you essentially just manipulate the v-note ip address or part number right.

E

Yeah uh another question for fay that, since we are getting to the scheduling domain, uh I think for the scheduling user may think of there are some affinities anti-affinities or this uh existing scheduling feature as well as how to handle link preemptions. So are we going to have some plugins uh in the future too? That uh you're not rewriting? All these uh scheduling features.

C

So here's the thing! So, if you look at my repo, the scheduling thing is, is pretty much just empty. It's just like I'm just implementing yeah first of fit algorithm, so that can be extended. I I think that is up to the users. I I because you have a good references which is a kubernetes scheduler. You can try to you, you can. You know, enhance everything that you want. Based on the how the kubernetes schedule does, um every every affinity we can.

C

We can achieve affinity because uh it's like, we can add all the labels in the namespace. If you look at my my scheduled cache, I already trying to save all the labels.

C

Although it's empty, I'm going to keep all the labels in the root clusters, these labels, these labels of root clusters- you can add it to the cluster cr, because in rcr you can add all the labels you want, then in schedule cache you have the labels for the root clusters and the in the name space. If you look at you even name space, if you look at the uh the, if you, if you add the labels in the name space, then it will show up here.

C

So basically you you can steal every entire labeling system in the kubernetes and do whatever things you want for them affinity, anything the rest of things, just algorithm implementation, okay! Well now, I'm just everything that I show for now is just a framework basically enable the scheduling just making people can do the schedule. Work. Just writing the algorithm and forget about the entire framework. So, uh for example, how they update the cache. How do you handle the you know the cost of line? How do you handle the you know the multi-cluster communication?

C

How do you handle the cluster cache? Everything is done by the framework, so it's a wild one achieved. Every users just need to focus on that arizona directory. Any implementation.

C

C

Okay, now I can show how the path scheduling is done.

C

I think I have one.

C

Let me take this test deployment as an example, so in this example, uh I'm trying to use 100 100 and you have four rabbit cars, so each rubber cars and have these two okay.

C

um If you look at this one.

C

See you have you have four running if you look at.

C

Let's see, there's one running route two and a three running with.

C

C

Yeah, this is the schedule for the part, basically um based on this, uh the slice distribution, the scheduler finder. Okay, the there is the root root, 2 the the root the cluster root. 2 has enough capacity to to satisfy this part. Then it go to root 2.. So this is the label annotation that scheduler place and a sinker will watch for this annotation and the root 2 sinker will sync this part to root 2 and exactly running.

C

Yeah, this is how the pod scheduling is done.

C

Okay, uh let me see we have what this classic actually look like yeah, the cache will have everything. Now it has all the parts.

C

Yeah and this part will show up with two: we have three parts shows in the root one. The customer.

C

Okay, so any questions so far.

F

D

Quick question.

F

On this is, can a pod cross the slice size, meaning if a pod had the 200 millicourse? Would it just take two slices.

C

F

C

F

uh That's the limit for a pod is the slice size.

C

Slides, like oh, the uh the the slice size is not the limit of the pot. So what I do is I'm going to sum, for example, uh if you look I'm going to for every cluster, I'm going to sum this uh the size of all slices and due to scheduling so, for example, if I have in this example, I have three one.

C

uh I have three one. Let me show this, so I have three one right. So in theory, you can run a part with three slices in brutal class one.

C

My current schedule will allow that.

F

Okay, so we're able to create, even though we have four slices, we can create a pod up to the maximum number of slices that are available in a specific super cluster. Exactly okay,.

C

So that again, that's that's the power. I want you. I allow you because, by placing with the slice size, you can somehow decide.

C

First, the distribution or the the max number, the max kind of part size you can you can you can do um uh again. This doesn't fit every usage model, but in general, in general uses the model in the sense that um you in reality you will. The quarter you caught up will be some some somehow more reasonable. It's bigger than this 100 100 things right and the slice is kind of moderate slices, and you can your your your part. The request should be also I'm assuming in modular case.

C

You are running a workload, so it's kind of just the parts within the workload templates um you I'm assuming you're wrong five, six, ten parts, um so uh you you can think of the the slide size and the um basically the quartus life size and the parts I I understand uh there probably have some trick things while you're saying about setting up the slide size. But again, if you don't want to set it, I have a default anyway.

C

This is the flexibility. For instance, uh you, the better one best practice. Is you the better? You know making the slice size and your quota size kind of proportional. You don't want to have a skewed in the configuration like you know, the the slice doesn't, the cpu memoration, the cpu, and the memory ratio in the quota is different than the cpu memory ratio in the slice, which is a bad example.

C

uh Is that clear.

F

Yeah makes sense.

C

Okay, uh next, I'm going to show, uh I think the last one is I'm going to show how to do the cleanup.

C

Yeah, the bad thing is uh the the current kubernetes. um If you in this case, if you reduce a quota, um the kubernete currently kubernetes doesn't clear the parts I have tried. um I don't know dozens uh which probably they just doesn't implement that in my opinion, they probably should kill the positive either they shouldn't allow the quota to be reduced um or they should be able to kill the parts but in, but for now what I observe.

C

That is that if you reduce the quota, for example, around four parts use up the entire quota, but you, if you reduce the quota by half um the positive, is not going to be killed. um So we have to take that.

C

The same same, I didn't change that that.

E

B

B

C

Okay, they need parts.

C

Repeat the quarter.

C

Yeah, so if once you delete the quota, you will see that I'm going to clear the placement results from the namespace this way. Next is this schedule, so whatever happens in this in that uh the the silver cluster sinker will do the gc uh to remove the namespace, because this namespace doesn't belongs to this class anymore.

C

Let me see, but it will take some time. Yep, it's already gone. This may take some time, so it relies on one minute: the gce, the one one, the the petroleum, the one minute petrol to give it cleanup.

C

The root wines are already.

C

Cleaned see it's terminating now.

C

It's gone all right. uh This is my end of demo today,.

A

This is fantastic thanks, so much for showing all this and and being able to do it just uh off the cuff real quick um cool. Well we're basically at the end, if there's any other questions pop them out real quick.

A

Otherwise we can call it and uh we'll see all next week and we'll look out for everybody can look out for the next two pull requests that should be coming up this week for the controller manager and the nested control plane and we'll get moving on cap end stuff as well cool thanks. Everybody have a great one, great and wonderful.

A