Kubernetes WG Multi Tenancy, 18 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG-Multitenancy BI-Weekly Meeting for 20221018

Description

WG-Multitenancy BI-Weekly Meeting for 20221018

A

Now it'll got it so now we are official online, kick things off yeah! Thank you, Jim and Tima for having us join this meeting uh so glad to be here. uh So my name is kaiishu. I lead the kubernetes team at by dance. Since January uh Charles delivered the first session to this group. We have made a lot of progress. So if you look at the cncf landscape right right now, Cooper Zoo already officially on the landscape.

A

So that's great news for everybody and we also have made some other updates like fixed a lot of issues from the community, and this time we want to present the update, mainly only cluster resource code implementation. We have even though it's right now still in Tech preview, but you know we soon will be publishing this to the open source. Community. That's why uh they prepare this discussion with the team and seek for any feedback before finalize the publication uh shortly.

A

So with that I just want to head over to Linton, you can start your presentation.

A

One quick question can I shared, so you mentioned that this is um in the cncf landscape. So kubzu is a. Is it a Sandbox project or incubating project or um right now, I think it's incubator uh project, yeah, fantastic.

B

All right, yeah, so is it right time for me to take over here.

A

Yes, please go ahead. Okay,.

B

Yeah thanks again uh uh uh yeah yeah uh thanks everyone for joining yeah. It's a great pleasure to meet you guys in this working group meeting so yeah. This is the Lincoln from by dance, so I'm here, together with uh kaishu and ishan from our team, to give an a status update for kubzu. So it's an incremental. uh We are going to present the incremental change since Charles presented early uh this January and uh okay. So here is the agenda. I will prepare for this uh session.

B

So first of all we are going to give a quick uh recap of uh kubzu, so I think it would be very helpful for folks who are new to Google and a second. However, uh there are a few news to share with you folks. So, uh firstly, uh kubzu was open source in literal, uh Late July and was just included in the cncf landscape I. Think in this month and yeah we improved the kubernetes compatibility of poop Zoom. So far, uh Google supports kubernetes up to 1.24, so yeah.

B

The thing was has been working very hard to uh run a lot of like performance testing and do a lot of bug fixing and uh the third one uh is uh What. uh There is a new feature: uh yeah we're going to share a new feature of the cluster level, resource code and management, uh even though it's still uh something like a tech view feature uh yeah.

B

That's that's why you folks still cannot have access to it from cubes to uh GitHub repo uh yeah it'll be generally available very soon, so please stay tuned and, in the last part of the session, we're going to have a quick demo of this new feature.

B

And yeah, let's first of all, let's take I, have a quick recap of kubuzu, so kubuzu is just uh lightweight gateway, gateway service for kubernetes, Mountain tenancy. uh It presents uh a new tenancy model for kubernetes So. Currently, there are three main tendency model in kubernetes, so namespace as a service control, plane, another service and the cluster as a service.

B

uh Kubernetes API, as a service presented by uh Google, is designed to sit between uh namespace as a service and the control plane as a service which enabled kubzu to be very lightweight and also very fast, and also to provide a view level. Isolation just described in this diagram uh simply speaking uh wrong. Its own API server, along with distributed uh key value, store on its metadata storage, which would serve for the server for the API request from all from inside all uh tenants.

B

Meanwhile, it also serves as API Gateway with protocol conversion, which submit uh API requested with proper conversion to the backend API server.

B

uh In terms of the tenant management, we introduced the tenant crd of the API and the Tenant controller within the kubzu server to manage all the tenant instance uh that kubzu or tenant controller will be responsible for silence certificate and generating codes of uh Cube config file. This is how like kubzu, give each tenant an isolated view of the cluster.

B

So, as we have already refresh our memory about Cooper zoo, let's take a look at uh the new feature, uh which is a cluster level uh resource code management. So. Currently, uh kubernetes building resource quota is just namespace scope, so it will be uh insufficient to support tendency model beyond the single namespace. So is that that being said, we expect the resource quota at the tenant level, which support resource code and management spending multiple namespaces uh we need class.

B

uh So that's why we need the cluster level resource code, time management, uh AKA class uh cluster resource code. So it is a cluster scope to crd uh and it supports cross namespace resource code and management, and the status of the cluster resource quota will be the place to persist. The aggregated resources usage in each of the namespace, which managed by the tenant.

B

So yeah here we are going to keep uh uh and I'll a little bit more information about the key components of the class resource quota feature so uh the same as a kubernetes building resource quota uh cluster resource code also has two uh main components, so the admission components and the controller so as we've already uh familiar with the resource code, animation will be responsible for both uh quota, admissible evaluation and usage updating, while it's controller reconcile a code update and recalculate the actual resource usage.

B

uh So basically it serves more uh server as a source of choose in terms of resource code. The classroom resource code, controller uh yeah, we'll just keep watching for the creation of the of its instances and then create them, create one managed resource uh quota instance for each of the namespace, with matching label as specified in spec in in the format of the namespace selector.

B

So it's basically it's just a label uh since the operation, the class resource code controller will continue watching uh all of this instance and keep thinking and aggregate quota usage from all managed resource code and persist. The aggregate usage in the status of the cluster resource code so uh but in the admission part so different from uh uh depending on the entry admission, uh the as in the building resource quota, the cluster resource code are just leveraged the admission weapon, uh also it it is only responsible for automating, compatible resource or resource requests.

B

Well, not updating any resource usage uh as a building resource code admission does so. The desired here here is just to rely on both the mission and the controller of the building resource quota to further uh evaluate the compatibility of the resource request and update the usage quantity.

B

So yeah here's a diagram of the architecture overview about uh kubernetes with cluster resource quota. So it also shows that the control flow of how cluster resource code works with the building resource quota when there is a part, is created. So here, let's assume that the class resource code instance has been created for one tenant and all the managed resource code instance has been created in each of the each of its namespace, so with a part is created by the tenant.

B

So the request will firstly go through the out of tree admission: web hook, a classroom resource code and the classroom source code. I will evaluate uh the resource, compatibility uh based on the code and the current usage, and it will either deny it or just allow it. So if the request is allowed by the animation weapon, uh the request will go through the entry uh resource code admission, which will admit an update the resource code usage so which it's just a building uh logic.

B

So after that, the resource code controller will reconcile the code and recalculate the usage to provide the source of juice. What uh this step is also uh just building logic and after a result, quota I have uh just completed the work. The class results. Quota controller will keep watching for all the managed resource quota and just thinking and aggregate the usage of this uh of each resource code under all the managed namespace.

B

So yeah there is some uh advantage on disadvantage in terms of this kind of uh design, so yeah. The advantage is like it like effectively avoid the contention in usage calculation and update from both a class resource quota and resource code, so yeah that uh that is compared to if the design of cluster resource quota just inherit, what the resource code animation does. So it so another there will if the classrooms quota also uh work uh work for it.

B

Like uh uh usage update, so yeah there will be contention in terms of usage, calculation, updating so yeah. That being said, the this kind of design, because those simplify the class of resource code admission part. So the yeah permission is easy enough. Just do the like the compatibility, evaluation and just either admit or uh reject so yeah uh in terms of the disadvantages, so overseeing resource is possible because they're uh the asynchronous model of the uh will at least a watch of the crd update.

B

So because we rely on the usage update, which is aggregated and sync synced uh from uh being to the classroom resource code, AS status. So since it's asynchronous, so if some uh uh ongoing request has not yet been uh like persist or has not been updated into the status of the class resource quota uh and you've got before that, there is some like other admissible uh request comes in, so there will be a resource code uh like over sailing.

B

So yeah, that's pretty uh that's kind of like no issue here, so we try very hard and we can have some alert or we can uh introduce the extra like allocated field to just uh like Pro aggressively to just uh prevent the overseeing but I think it's just a trade-off between either overselling or underselling.

B

So yeah uh so much for the classroom resource quota itself. So, let's see now, let's uh take a look at how like poop Zoo uh leverage, this class resource code feature so inside the tenant uh spec. So we add a new field for specifying the cluster resource code.

B

So whenever a tenant instance is created, uh technical controller will create create a cluster resource quota instance in the kubzu API server and cluster resource code controller will take over the rest and, as we mentioned in the previous slides uh yeah, something like it will create uh one of the one node resource code itself. It's managed namespace.

B

Under this tenant, so yeah similar thing and reconciling is reconciliation. uh Handling will apply to the update and delete the events uh for the uh cluster resource code spec uh in the tenant.

B

So here uh yeah so much for the like theory about this uh new feature and a status update so yeah. Now, let's uh we're going to have a quick demo about this new feature.

A

Do you want to pause the years if anybody has any questions.

B

uh Yeah, uh why not yeah? We can do that. Yeah.

C

This is the phrase: I got a question. So um can you go go back to your previous slide.

C

How do you distribute the the thing called The Limit? How do you distribute the limit to your organ spaces?.

B

uh So yeah, so basically, uh since we, uh the customers record controller, will create why resource code in each of the namespace so yeah, the namespace will be uh yeah way to identify the namespace by labels and uh do.

C

You have a crd definition in your slides, or maybe maybe that can explain so. Basically, you need to specify each names. That's how much quarter in the oh.

B

Yeah so yeah for the hard limits we just specify uh just inherit what whatever the limit in the cluster resource code, so yeah the I didn't, prepare the spec uh CRT spec, uh you know slice but I think the demo will show that. So, oh you can think so you you.

C

Just that's why?

C

uh What is the limit of our namespace and if you, if we have unnamed, especially it's going to to be the total called out, would be n times? The high limit is that.

B

Yeah, so basically the if uh yeah, let's take an example, so cluster is required. I have like two CPU and uh two uh two gig of memory right, so that means in each of the names is. It will have like two uh uh yeah the same exactly the same so the two core and to a gig of memory. Okay,.

C

B

Yeah, the total uh quota will not be the sum of that, so the cluster resource code, animation controller, will just prevent, but that it goes beyond more than the higher limit. Yeah I think we just use the result counter to collect the usage. Okay.

A

Didn't do any limitation.

C

Sorry sorry I just want to make it very clear because so, if people want to use cost results quota, they only have a let's say they want to put a hot limit for attendance. How do you? How can you do that if they want to put a high limit for tenant tenant? uh How can you do that? If you want to duplicate every namespace with the limit, um can you do that? Basically,.

B

A hard limit for yeah I think so, but it cannot be granular so uh so for each names. Basically we'll just see uh all the namespace under the same thing, and it will be the same, have the same yes, yeah.

C

But then you have a problem right. So so let's say you know, regardless how many demonstration you created if I want to this Talent only have a CPU. Why why memory even.

A

C

Whether you create one name space or a thousand namespace, they share the same code.

B

uh Yeah, so we have two level of uh check right, so resource quota, so cluster resource code I will check uh and prevent this kind of uh yeah. All the requests will not go beyond uh The Encore and Y uh gig of memory, uh um which it's it's kind of like shared pool, I'm all the namespace under this tenant. So but we don't uh prevent like if why the tenants just want to create all the resources within one namespace. It's also okay. So as long as those uh limit of the.

A

Culture results.

B

C

My control limit, which is the sum of the namespace. You also have a way to tell each individuals namespace. What's the limit and the individual numbers limited, we are going to be duplicated for every number. You create yeah, but you have a bigger cap: okay, yeah.

A

So yeah, let me let me tell you I think the demo might help people to understand this better, since we have some real specific data right to show how this works. Maybe we should go to the demo and then we can answer questions from failure.

B

Illustrate that uh yeah I believe Ryan has some question.

A

I was just gonna, I. Guess reiterate my understanding, because I think it's similar to how we handle it for HMC and so yeah the um they essentially they they put a resource code on each namespace and then um anytime that changes they add it to the global. So you have that Global status and there's an emission controller on that. So there's a delay but I think that that's my understanding of how it works right.

C

Okay, I understand I, understand yes,.

B

Oh yeah I think uh yeah, so any any other folks have any question so.

C

But that that means uh within the tenant namespace can contend for the quota right.

A

B

uh Content, you mean contention yeah, I,.

C

Mean content within talent, because you so so apparently each University have a kind of a little bit bigger quota right. You you want you want this. Classical quota is fully utilized. Then you.

B

C

Have fish namespace that kind of a little bit bigger color, then this tune-up space may concern because the a may get more be Nemesis. B making NS depends on the usage.

A

B

Yeah I got your point. So if the tenants like trying to create uh resources in two different namespace yeah.

C

They have contention because each one will have a bigger limit. Otherwise you know the cost of limited is going to be waste. Then you know so then one Nemesis may may get all resources to reach that class classical Memphis quarter and the other Nemesis cannot be created. Anything right.

B

Yeah I think I did uh like theoretically yeah it's possible in some like extreme cases, so yeah.

C

Okay, then, let's go ahead with the demo. So let's exactly this pretty much is a configuration issue, so something that you probably can mitigate with a proper configuration but yeah.

B

Okay yeah: let's take a look at the demo here, so I.

A

Oh sure so yeah. Let me just.

B

I prepare uh recording here so yeah just make sure everything goes smooth.

B

Yeah, let me just start the recording and so yeah uh after the thing is set up, so we can see that we have two uh text here. So one is for the backend API server and what the other one spoke.

B

B

In The Next Step we're going to create a tenant so yeah after we apply that we just share uh what is included in the tenant, so we just name it with six uh ones and with two CPU to GPU as a memory. So then we, let's check out check the status of the cluster resource code. We can see that yeah. We can see that uh we have let's go back to.

A

B

Yeah, we can see that we have uh yeah two CPU and two gig memory has been specified in the spec and in the status. The usage is just uh both of the usage is zero.

B

um In the next step, we are going to download the Cube config for the specific tenant.

B

Cool yeah we're going to take a look at the weather if there's anything in the cube, config file, uh yeah, so yeah. Let's just take a look at the resource quota uh from the tenant perspective uh in the default namespace, the resource code uh yeah, which is the same as what we specify in the cluster resource code.

B

And here we're going to create a pod steal from the tenant perspective.

B

Now, here the here's, a part with one CPU and uh one gig of the mirroring, which is under the yeah classroom resource code.specified, being respect so yeah. We can see that the names uh the pod has been created successfully and yeah. The party is running now and uh and then yeah we're going to check the quota usage.

B

Yeah, as you can see in this resource quota in the default namespace yeah, the resource code usage has been updated.

B

Yeah, we also take a look at the all zip code from the perspective of the backend API server, so yeah as you can see which my answer phase question. So each of the namespace will just get uh the same uh like harder limit with the same amounts of specified in the classroom source code aspect.

B

Yeah in the next step, we are going to create another uh part in a different namespace.

B

So yeah we are going to create a namespace called test and create a pod uh under the name name, space.

B

So yeah we can see that uh in this request it has two CPU and two two gig. uh So if we just take a uh just uh look at the resource code under test uh namespace, so it should be admissible, but it will be rejected by the class resource code because it code, if the part is uh created it will go. The total, like usage, will go beyond the hard limit of the cluster resource code. That's why it will be denied.

B

So yeah, you can see that the code has not yet been updated and from the global view right so for all the namespace, so yeah there's no change to the code, app of course the the pause. The second part request was rejected.

B

Yeah same uh view from the backend API server perspective.

B

So in the next step.

B

Yeah we're going to like uh show you guys how the resource quota cost, uh which is uh customized resource code uh collaborate with the cluster resource code. That's specified in the tenant spec.

B

So here we are going to create a customized resource code on one of the namespace under the tenant.

B

Yeah before we do that, we just delete the first part we created so make sure everything is empty and delete it. It's just a brand new tester.

B

So now the all the quota has been updated accordingly,.

B

Yeah we prepared uh yamo for create resource quota, so it has a hard limit for one CPU and one gig of memory, and then we, the tenant, will apply this resource quota so which will create the resource code under the default namespace.

B

So yeah we can see that underneath a default namespace from the tenant perspective uh yeah, we have two Originals quota. There.

B

Now we're going to create a pod request so which is which is beyond the cluster resource code, oh, which is beyond the customized resource code, but under the cluster resource code. So you can see what will happen so here if you look at the error message, so it's not! uh The error is not erased by cluster resource code, so that means clutter resource code. I just admit this request, but the resource code admission component just rejected.

B

Of course it uh the specific uh the customized resource quota or the request just goes beyond the customized resource code that we just created.

B

So yeah I think that concludes my demo and yeah I hope we, uh it just properly explained the question if he has.

B

Yeah, so that's it about the demo.

C

Yeah, uh okay, yeah I kind of understand so.

C

Because then you can get a usage, you don't have to account to use it yourself. You can continue stage for every namespace. So is this the case? So if you use so so, is it required every namespace has kind of color I think so. Space.

C

In the underlying Community, significant needs to have a question.

B

Yeah so yeah we are going to take a uh look at the like extreme case.

C

I'm just saying that okay is: is that possible for any of your Olympics, let's say to remove the product I, don't think so.

B

To remove the quota.

A

B

Mean you mean the yeah can.

C

You go back uh any of the demo, so each Canada at least they'll have a full name space default. It could be system um whatever. So can the cookie system.

C

That means it doesn't have the code it has to have.

B

One yeah I have to and the classroom resource quota controller will keep reconciling. So if someone just trying to delete it, it will be just recreated. Okay,.

C

Okay, so yeah, because you have to have a product. Otherwise you you don't have an effective way to track the usage, because the code, the building controller, will do the usage calculation, okay,.

A

So in terms of the roadmap, any other major features or things planned, uh we have we're actually working at like the marketing DNS and uh the offset control for multi. Tenants like we found some access to leaking issue and.

B

A

To build a.

B

A

Security uh capability and in.

B

Impact, in that uh you know social with us, we will.

A

We will support like the service accounts and the rbac rule-based access control to support the uh the we would be attained by being object to help.

B

Us improve the uh of the access control.

A

I think that's the uh two of two of the major feature we are working with: okay,.

A

So we'll hope, a lot more Community users right to use the coupon so can send us more input and feedback. Regarding our next steps so uh depends on your guide, Jim and Ryan. You know, hey, you guys have to uh evangelist. You know this. This is a cool thing right to the community better, so we can have a more audience yeah.

C

But uh yeah I think that the entire class to discover resources can be independent of cookie tool right, even yeah right. So right. If you look at their design, I mean, as you can see, you can use.

B

A

One of the use case I mean we implemented the exact same thing in our I'm, assuming mostly in hnc. So that's already there but yeah. um It seems like it's a pretty similar design.

C

Oh, did you do that, so every so every name service again, you haven't really got the before classical product individually decided to override a date for a second. If this is your design.

A

uh So Adrian uh uh and I forget: who else was on the team at that point, but they they rolled it into anthos for the so it's the Google hrq um and then they backwarded it to hnc not that long ago. So yeah.

C

Yeah they share the thing that stuff. It's.

A

Like pretty similar, I was actually looking through back in the code to try to figure out exactly what we were doing um and so yeah like. Essentially we you know there's it, but since we're hierarchical we know all the child tenants, and so we we manage it, and so um we do the calculation up front. um uh So we can make the emission web hook yeah.

C

I I think the funny part is the key funny part. Is they don't do the usage packaging in any of the internal memory scrub stuff, uh internal cash? They just leverage the building uh uh coal house stuff to do the usage calculations so which coming from ABS should be accurate and the synchronized for my API silver perspective, so yeah yeah, the trick is um you have to give individual namespace as a cluster scope limit by default? It has because one namespace can use it all.

C

You should allow that then, uh but if people wanted your final granity control, you can override the namespace is the final final gravity. One right I understand.

B

C

I understand correctly, that's the purpose of the dealer am I right.

A

C

Yeah yeah I think that's a that's. A thumb. uh I think that that does not make sense in the sense of uh in in a case from an event in perspective and the simplified Divine I mean that can be wrong.

A

Cool yeah, this is good. Thank you for the update and you know it's very looking forward to some of the other features uh they mentioned, and we will you know it's up, yeah, we'll post the recording on the channel and we can kind of publish it there.

A

Thank you, team and team.

B

Yeah thanks for having us again and thanks for the feedback.

A

Absolutely all right yeah, it doesn't look like we have any other topics to discuss. So if you know with that, we can just conclude the meeting so by the way you guys going to kubacom this way, Yes actually.

B

A

Am but I think most others I think Ryan I saw you shaking your head, so you're, not traveling, right or.

C

A

Okay, yes I'll, be there I'm there Sunday through the following week, um I think Saturday here so pretty much all week, yeah.

B

A

Tried the way we never made the decision yet people uh uh people are tired and sick, so we probably were steep this time, but uh good luck to your session Jimmy and the folks yeah. Thank you all right. Thanks.

C

A

Bye-Bye thank you. Yes,.

C

A