Kubernetes WG Multi Tenancy, 18 Jun 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Working Group Multitenancy 20190618

Description

Agenda/notes:
- Co-chair announcements: Sanjeev Rampal is new co-chair!
- Yushiro Furukawa is going to present his coredump feature. How do you make sure the correct users can access their logs in a multi-tenant cluster?
- Kural to discuss Tenant resource quota feature request/proposal in Tenant controller PoC https://wiki.onap.org/display/DW/ONAP+Cloud+Native+Multi+tenancy+proposal#ONAPCloudNativeMultitenancyproposal-ResourcequotaproposalforthetenantCRD

A

Cool so welcome everybody to today's multi-tenancy working group meeting. Thanks very sending our agenda today. First off, congratulations to Sanjeev Rumpel, who is our new co-chair of the multi-tenancy group along with me, Tasha drew Thank You spongey you've for stepping up and I definitely look forward to working with you, Thank.

B

You Tasha thanks. Everyone look forward to working with everyone and helping this continued to progress on this group.

A

And then the agenda for today is ushiro is going to talk about core dumps and multi-tenancy, and then carol is going to talk about a tenant resource quota feature request, slash proposal, so you shiro. Would you like to take it away? Okay,.

C

So hi nice to meet you I'm usual okay. Can you hear me, is it okay? Okay, so thank you so today, I are you sure and Bono? Is she and one channel will present age some explanation about our quartum feature with multi-channel kubernetes environment? So just women and one song, could you could you explain? Could you continue to explain or cordon feature? Yes.

D

E

C

So just a moment bond will shared our some document or for explanation just moments. Please.

C

E

Can you baby yeah.

C

I can see okay, but just women a little a little unstable just just women.

A

C

A

Presentation mode and just show the browser yeah.

C

It's waiver yeah: please stop the presentation mode and just to explain with some normal yeah I think it's.

E

A

Yeah I mean I, guess yeah. We can see most of it. Yeah.

C

E

Cause I'm, it is fun. You know.

C

How much it'll take me to the hospital already the mirin day.

E

We will explain automatic Odom cricketer feature. First, I will briefly introduce this feature.

E

First, motivation in much secret. Yes, much can provide just a monkey. Crested amis researcher access code, umpires but general meaning straighter work relative of operator channel because vampire is third on blockers plays no own work, I know so to access Kadam files created in by container a tenant user should ask crust administrator to fetch my and perfect client so to facilitate investigation of crust application. I turned to you that we are developing this feature and this automatic Adam create a picture and I'm.

E

Quite the emotive rage, not know cartridge and authenticate and authorize users access to identify, so Nick read wrong one. He explained how this feature was one. Please, ok,.

D

Hi, everyone page to choose the architecture of our design. Basically, there are three parts: the code on fires generate at the caught on fire download pod and the backend stores part the caught on fire generate part would launch an amine pod to those nodes with scrotum Labour's admin code or the dual following scenes. Firstly, it will copy an executable file. Executable file called hinder to the hosts file system.

D

Then it would modify the core pattern fire on hosta to make the to make sure the kernel or the in work the handle to copy an extender to the hosta file system. Then it would modify the copepod and firearm hosta to make sure the corner would invoke the handler we just copied when job crash happens, the dongle part, which is she resists, the aggregation api to master and could not access backing stores directly. They mustard on all the coal fires by our aggregation API.

D

In this way, we can implement access control with existing design like a BSA, the backhand storage is simple: it stores all the code on files generated by any pod inside constable.

D

D

D

E

D

E

I think the next two page about what happened when Kadam happened: okay,.

D

Good job inside hold crash Colonel, our worker node, would generate ten fires and pass it to our hinder. According to content of cotton hinders years with caught on fires and stores it in our back in stores.

D

And when tenant wants to download code on fire, they could access aggregation API, like any other original ap highs in kinetics, to get the code on fires they want.

D

That means administrator could control this by existing authority design, like obviously.

E

Nicholi yeah.

D

Next page and the last page shows the administrator could choose which no sport coat on Fisher aspire, Labor's and whether admin code is deployed on worker node, who has Kadampa capability, is determined by no selector faring on spec.

D

And this is all the current design of our column features in communities so.

B

This is, this is a thing your your Shiro is sorry today, I.

C

Yeah yeah I can't hear you, okay,.

B

Good good good it all so so great. Thank you. Thank you, for you know bringing this to the working group. Are you become so basic questions on sort of just the feature positioning? um Could it have been implemented by instead of pushing the code um file to remote storage and then providing access credentials?

B

Could it have been to simply provide a.

B

Limited access directly from the host storage, like a a privileged container that that has access to the host local storage mod but uses the credentials of that user or that namespace you know to allow retrieval of that code, am from the local storage directly into the queue API without needing to push it to a remote storage. I mean I'm just trying to understand whether this could have been done without needing to push it to a remote storage.

B

As an intermediate step is my question clear or I mean I'm just trying to understand the positioning and kind of architecture of the of the feature and wondering whether it could also be done by simply pulling it from local storage but controlling access. Mm-Hmm.

C

Yeah yeah I think it's it's your your right, so just we are tenant user can send on some API for after that yeah, okay, once huh yeah I think it's correct. So what do you think about that?.

C

In there, okay, by the way friend, oh yeah, yeah, it's okay,.

D

We design actually why we design like this, because.

D

If we put all the, if we put the code on fire, seeing cost.

D

When powered shifting between nodes, we think maybe there will be some problem about.

D

Running note a at first and it it which rescheduling to another node, and so we maybe how folks know they and not be the core files. So we think this is not she's, not she's, not good.

B

Okay, maybe dig into that a little bit more okay. Firstly, you could could you please add this document to the working group document page so that we can look at it without flickering and yeah have more time for the working group to think about it and provide further comments.

C

Okay, so citizenship, so the after finishing this meeting I will up. We will upload some. This tie into yeah some Google Google Drive or some somewhere, and we can. We can refer this slide with no ranking. So yes, and so one sound, could you could you paste or this your your URL for it have repository and then.

C

Yeah core dump detector and this issue: could you paste the ring something yeah? Thank you.

C

Yeah she saw I think it's okay to stop the sharing that your screen so because our little yes ranking so yeah, you know. Thank you.

B

So I think it needs a little bit more review by the working group. I mean gee sure is: is the remote storage the right option, or should it be just one of one of multiple options you know, so maybe we can all provide comments once you put the document in Google Docs and provide the link I. Think. Okay, sorry, have you already started prototyping the CRD for retrieving the code on files and so on?

B

B

I'm, sorry, a quick aside again, have you already prototype the controller in the C or D for retrieving the code.

C

Yeah we are just now trying to implement so they're, not not finished, yet maybe some 60% or so doesn't not finished yet, but and now so once on. How is your implementation right now.

D

C

So your are quadrant detectors. Implementations need us like.

D

C

Now now you are, you know, writing right, yeah.

D

C

So that sums so yeah I'll him as he explained, so that we we don't have any moving some cold right in right now. So now we are yeah just under in implementation.

F

G

Okay, so how? How does the administrator know what our back to put onto the core dumps, because in any given namespace like there could be lots, different people or groups or service accounts that have different levels of access? Has the administrator know who should be able to get to core dumps.

D

Okay, it means the access control. Yes,.

D

Here's the idea like if you, if a tenant, want to access any files from he must create about yet, firstly, and the object the object. The name of this object is.

D

It's the same to the name of the powder he he won't gets the column from, and the cluster would check that if this powder you see existing and the customer would lock the UID of this pod.

D

So if the pod was deleted, delete delete' and like a new pod with no say no, the same name was created. The user can not get that the column from that code, because the UID has changed.

D

F

G

F

Yeah, okay, um so.

G

Part of that, so is that there's a CID for that is this so and that would have the our back to applies to say who's allowed to get the core dump I'm. Sorry.

D

I didn't get you.

G

So for any given pod for any given called up for a pod, there's I back around your API call to control who can access that? So you don't want everybody to who has access to that names place typically to have access to all of the core dumps? Yes, so where is the are back defined for the core dumps for a particular pod.

D

Like, like other object or API in kubernetes, oh okay, yeah, so you can use it like any other object or IPs in communities. So abacus the setting of a bank is similar.

D

G

H

What is the object on which you do the are back.

D

H

D

H

Asking what is the object on which you do the are back for this feature is thatis ERD.

D

Uuid currently is we design it with the name code on the endpoint? We call it code him caught on in point. Okay,.

H

So the our bet would be defined on that.

D

Yeah administrator could control if user for the existing object.

H

Okay sounds good yeah I'm, looking forward to the document.

D

C

Okay, so maybe we are, we will share that some today document and yeah, and if possibly so, the next meeting I'd like to we like to explain again and yeah, let me think think up more in the. We are just updating some our implementation. Okay,.

A

Well, put you down before I follow up next next meeting, no.

C

Thank you so much yeah yeah.

A

Cool and then so the next item on our agenda, okay, is that we were gonna, have a presentation on the someone that has a update, so the tenancy CRD.

I

Hi anyway, this is cool, something doesn't yeah. Let me put my proposal thinking that I put the link actually Hyundai. We share my screen.

A

Do you know how people can get access to that wiki to provide feedback, because I asked a couple people and no one seems to know so.

I

The Wikipedia it's just Linux, Foundation, username and password you can just log in with that. So it's the same thing: what you guys uses for CN CF right. It's the same thing! Anyone can comment on that. One.

I

Let me do a little bit introduction of whatever team, so myself is Chrome the Ramakrishnan and I'm working as a senior software Intel, so I have my teammate it on also here in the meeting and we working on own app. Our three know project. So the one happy is basically like a service auto station, it's one layer above the kubernetes and then we have feature requests wenona for multi-tenancy.

I

We are looking at kubernetes, whether the kubernetes as a resource or destitution of the multi-tenancy future, and we were following this working group for a couple of weeks for like four weeks, I am following this working group and this working group is really very active and they also did some kind of multi-tenancy POC work. So we thought we should consume that work for the our project in one app and as well as in that Reno in links foundation. So we went through all the document and also we had run.

I

The initial p was if Walker fish is done by Sanjeev and his team, so this wiki page is basically little bit explaining about the own app recommend on multi-tenancy and also to show all the multi-tenancy works in kubernetes. So the goal of this scope or the proposal within the own app, is to work with kubernetes SiC groups or working groups, and we are not going to have any has a different solution other than from this working group and I.

I

Also following this working group, there are other solutions is for multi-tenancy, where you have a separate control frame for each tenant in your cluster. We find it's quite expensive and also it's easy. It's difficult to maintain in point of for our project in order for in at Reno in Linux Foundation, so we want to consume.

I

We thought this project, the multi 10 controller project, it's very simple, and we think this is where we will hit hug 8 to provide some kind of your future or your POS to this project, and the recommend we had is other use cases so sorry to reduce this one sorry. So our use case is basically from multiple users. We have telecom comb segments and that into user or basically like a tenant, and we want to track the usage of the resources of each tenant, and that is one of the use.

I

Cases in own admin acronym a stack, and we also make sure that what is the level of resource consumed to by application. Also, and even if it's a application is running in multiple tenant. We should get that resource allocation and how many resource is used so that when we place a workload from one app or something it should knows in which kubernetes cluster it has to place that workload. And it has to also understand how many resource Cotta is available for each tenant in each cluster.

I

So that is our use cases and we have other use cases which is not much related to multi, tenant or but it's related to scheduling but- and we also want to create a centralized v document- is to create a centralized credentials for all the tenants and also the applications which will be running in one app. And then that will be passed down to kubernetes by saying that I won't go in depth about the multi tenant proposal, because you guys are expertise on that.

I

So all the materials which we put here it's referred to from the reference documentation which we provided at the bottom. So we are not the sole author for this one. We said that the document and all the stuff is related to the authors who have written the document in the reference section. So this section just explained how to run that one.

I

We just doing some kind of financial people series on that, and this is what we want to discuss today and we want to have a future request or a future which here provides CRD for resource Kota, and this resource quota is basically like bookkeeping, tracking and discovering of the tenants actually. So it's just a wrapper for the resource quota, which is already in Canaries.

I

What we are doing is like we accumulating all the resource Kota of each namespaces and putting on top of the tenant, and we basically do using the working group parent controller, which we, which is our aim to reuse it again and- and this is just a structure, how the resource quota should look like and in the resource, spec Kota spec. We just reusing the kubernetes resource list and resource cope. So this is, as I said before.

I

Right is just a wrapper for the resource code, um and this is quite where we want to explain what is the tenant resource so from one app or from a Cree, no side. We have this resources so suppose you have a tenant or a department, or something like that. We will say this much CPU or memory or the pots or resource they can use. So we are basically using device.

I

Plugin called turmeric device plug-in which is open sourced by Red Hat City office, and this is very handy so once we able to be using it just for the experimenting purpose, so it this. Basically, this device plug-in just give a dummy resource, so you can instead of this one. You can introduce some other device plugins like every GAA, qat or chip use anything here, and so this is where you say like how many resources you are allocating for this tenant, so the CPUs memories and pots.

I

So instead of what you can also say that how many pots this tenant can run, you know nap or something like that, and then then we will say how many resources it can use like FPGAs or chip use, or anything like that, and then we will allocate the resource this.

D

I

We introducing like a wrapper and that wrapper will have the and to be using the same tenant namespace, what is already created by the tenant controller, II Wasi, and where we specify each tenant, CPU memory, ports and device plug-in resources. Actually, so what we are trying to do is, like you, be creating your one layer above that, where you create like give us some amount of resources for this tenant within the tenant.

I

You can divide it for each team or, if for each client, and we creating a pool like silver gold, and this is per team, and the next one is per application. As I said in one of our recommend, we want to track how many resources this particular application is using. So our use case is basically on Telecom and comm segments where we can true want to track the resources of the BNF applications. So I we just put like the firewall.

I

What is firewall cutoff, this particular tenant, and that we basically using the list, which is already there in kubernetes, and this list basically reused the resource Kota with the values, so you can within the tenant.

I

You can have multiple levels like v1, v2 and v3, and, having said that, I think the diagrams will explain more so you have a tenant and you have team like gold or the pool of customers, who is the gold customers and team of other customers who is like a silver customers, so the gold customers will able to use a lot of resources. The silver customers used to use less resources apart from these two tenant. Even you can draw any application we had. We want to track this application with the different versions right.

I

So it's not that we just concentrating on the client or the user, but if you want to also track the application, how much are this application can run? So we want to do that. So in the example we we are just putting in the same namespace, but the application can run in a different name space and it can share by a different tenant.

I

Actually, so do you guys have any questions so far.

I

Again, I think, let me good.

J

I

Yeah good, how.

J

Does the tenant spec up at the top for total resources spec that hard reflect the sum of all of the resource quotas to find in user resource quota list? So.

I

So I think that diagram little bit explain more. So then the admin creates the totals this one right. He creates this particular of the first one. The overall.

D

I

And traces quota and then within the resource quota, it will refers to this. This is code of the individual client or the tenants within that, and we proposing a kind of admission, controller or controller, which you will review whether this over all things is summing up or not. If it's not summing up, it will turn out to error, saying that whatever you put it here, it's not something to the resource quota you bring below this one. Is that answer your question? Yes,.

J

So basically, you want the sum of all your product. Our namespaces resource quota objects to some to be that spec that hard value yeah.

I

Exactly yeah, so it's basically a bookkeeping as I said right, so it's basically doing the bookkeeping. It's not doing any kind of storm isolation and that, though, one more thing, it's all so see whether this particular resource is available or not. Then when, when the user or the client tried to run your pod or your workload, it also see whether this tenant can run this workload or not, because that's what basically the recess quota do in coop in it is right.

I

So this one is basically like a wrapper to that resource quota and which reflect the resource Kota for ten and clever. So it's a very.

B

I

B

Does that mean that you have to provide the quota for each space within the ten? In addition to providing the overall tenant kohta? It's.

I

Up to us how we want to take that one.

A

I

Doesn't means that each namespace should have that one if that is not it, we'll try to pick so our example right, if you don't have this resource quota for this particular tenant, we can put your logic saying that this is a common resource, and this tenant will also use that resource. Actually so.

H

It's not sorry.

B

Go ahead, yeah just to kind of I, think there's a little bit more clarification on the resource manager model needed here. Okay, if, if we are looking so there can be a couple of ways, my initial expectation had been for this resource quota that there is a quota for the tenant as a whole and it doesn't matter whether individual namespaces within that code are within that tenant.

B

Rather, basically it's it's a quota act at the tenant level and which namespaces don't need to have individual quotas, because you know they can yeah here the quota but yeah, but in that case that's a slightly different model than in which you have the tenant recess quota and you have the namespace resource quota and the controller is simply checking that the sum of the namespaces is not greater than an encoder yeah forcement. In that case. In that second model, the actual enforcement is still happening at the namespace level. Yeah.

I

B

In the other model,.

I

B

So, which of these two models is closer to what you're thinking.

I

At this point so far as we are thinking at the bottom models actually and because the second model, we are very much inclined because that's where we can show this particular namespace as a resource Kota, the first model, which you talking without it as a resource quota, it's kind of second priority for us. Okay,.

B

But I think I think. Let's again we will review it more of fly.

I

B

Comments but I think what I would encourage us to think about is that if the controller is just doing the arithmetic of checking that the names faces are adding up no more than the tenant then start delivering that much more value right, because it's just saying you know just it's: it's really just a configuration checker really at this point it's.

I

A configuration checker why we are doing that because we want to showcase that it's a you know. Overall, it's summing up the value for the each namespace for each client or the customers, using the particular in resource or H cloud, and the first proposal you mentioned is without any resource, Kota I. Think that's also. We have the values for our use case actually right now, but we've want this first future to be a semester of future request for us actually, but what we can do we can able to provide the both the futures.

I

Actually I will put the both. The features in I include that one as well in this proposal. I can do that yeah right.

B

And then and then let's have more detail: let's, let's because I feel that will kind of add a little bit more value and then.

I

B

More comments so.

I

For that use, KS right, you saying that without any resource Kota, so in that case it you see it's a common pool and any tenant or the names within that pool will use that resource support. Okay, now.

B

So let's speak here in terminology right: it's not called within the tenant, those individual namespaces as let's not call those also tenants, because.

E

B

To be clear on terminology there's X number of namespaces inside a tenant.

E

B

Those X namespaces are themselves not canach right there, just namespaces namespaces.

I

Within the thing it's like, we can a bunch of clients or customers yeah.

B

I

B

The thought was that, could we have a tenant level quota which applies through the aggregation of the namespaces, the namespaces themselves, needing quotas? Okay,.

I

J

Follow the openshift model with cluster resource quotas, but having some better aggregation of knowing and assigning namespaces to given tenants is that what you're, requesting or suggesting so.

B

In the open chip model, it's one namespace, pertinent right. One is faithful project, so there is resource Kota equals of ACOTA.

J

Namespace first quota is something different, that is a multi project and in that situation, resource quota that allows you to kind of trade resources amongst multiple projects identified by a label selector for those projects, but our issue has always been ensuring that the right label is on those projects.

B

You're right plus the resource Kota, is different and I need to read up on that one. So what the bottom line is and let's be clear on precisely which model we are going after here in this.

I

So what I see is there so I put like only two use cases or the two models I had, so your model is like I, have an answer SS in s for namespace, which is doesn't have any resource Kota, and what will happen that scenario right right? In that case, okay,.

B

Now I'm not saying we have an answer, I'm just suggesting that we think about it right, maybe a.

E

B

H

So the way I understood it is maybe so are you was talking, but I was I am talking to the person who is presenting the slide right yeah. You assume that the resource code always is exists in each namespace. So for our use cases.

I

They exist because we have to do.

H

Oh sorry, I was going to say that I think that makes I mean yeah. So I got your understanding. But what makes more sense is that for us to be like able to define across namespaces and then and then maybe the implementation is that you actually go and create resource kotas per namespace, which is which divides the whole aggregate across these namespaces equally and then creates a resource code upper namespace. So that could be an implementation detail and one way yeah.

I

I think that will create multiple scenarios varieties. This.

H

Implementation but I think I think I agree that it would be good to have a mechanism which basically creates one tenant and then somehow it got implemented.

I

In this case, as you see it right, if they anus one and anus to have the recess Kota, then you will have the missus quota allocated for that and, if NS for doesn't have any resource Kota, we have to see the scenario like whether we have to create a resource Kota for the remaining resources matching to the ns4 or in future, if someone's, create the new regime space or in s5 right of the sharing should be so yeah I think there is multiple scenarios and the third scenario what we are targeting is application.

I

Actually, so the application could run in the same namespace or it could run in the multiple namespace with different priorities or resource Kota, so that also we are targeting, but I we will take the both three use. Cases and I will include that use case. You guys suggesting here and I will rewrite this one. So the one question I want to ask is that is this: you want this proposal to be be here or we can move it to the Google Docs and add it in no. No, your proposal actually.

B

Yeah Sakura yeah. We can definitely add it to one of the existing proposals, so on I just feel like we need to define, it will define it a little bit more precisely for now, okay, and so, let's keep it as it is right now, let's have a little bit more review by the over the next. You know couple of weeks: okay, and really the at least my individual thing and I'm sure others have their own thoughts as well.

B

Is that the the primary value of resource quota pertinent would be if individual namespaces don't need quotas, but you still have a fluid pool of quota for the tenant as a whole. Okay,.

I

B

About that- and you know there are implications in terms of the runtime- does that actually map into the underlying cgroups of the runtime or not right, so that there's a little bit more analysis, we should do in certainly merge it in as appropriate. This is obviously very much affiliated to the controller CRD, let's evaluate exactly the model here and then we can. Then we can merge it as needed. Yes, actually.

G

There's a there's a fourth level of grouping here more to the is it is it just exactly the same? You take the whole tenant if they've got five names of spaces, do you just split it in you know and you've got a tenant quota? Do you just put 20% of it in each main space there's a whole other level of grouping here, because in in our situation we have a tenant.

G

Doesn't it's not just one kind of you know when you have the the concept of environment, so you've got dev and test, and the integration test and staging and production and stuff like that. So we, the the controller web building, actually has there's a bunch of things. But it's you know.

G

I can say my my app when it's in sorry, my dev environment has a certain amount of resource quota and it maybe it's you know it's it's eight CPUs and there's a total disk total PVCs of you know 20 gigabyte in dev, but in prod it's actually a hundred C CPUs and can use four terabytes in prod that I'm still a tenant, but I have multiple environments.

G

Within my tenancy so that you know Minister, there's another level down of specification of it's not just this just and the homogenous set of namespaces that make up the tenant is. They can be grouped at another level of abstraction, yeah.

B

No I totally agree that we definitely should not assume that you know the cut has divided equally amongst all new spaces. I think that's overly simplistic. What we really want is statistical sharing of that resource across all namespaces of the tenant. The model here is, at least in my mind, is that the tenant is the one getting charged okay, so the charge back is based on the resources that the tenant has a whole uses. It doesn't matter whether the namespace is a uses, 20% of it or 30% of it, or it's totally fluid and dynamic.

B

What the chargeback model is that this is a tenant and he's going to be charged on the quota of the tenant, regardless of the individual, so it's like, and so that's at least the way I will have been thinking about it and suddenly you know we can look at all different options here and so all those other models of saying, ok, let's just assume it's divided, equally I- think those are probably not I, wouldn't sort of consider them to be useful.

B

So we need we need to have a couple of options here which we we can you know, and then we can provide comments on the dock offline and they can come back to it in the next column. Ok, so if you're, just thinking.

G

If this is a billing constructing a.

B

Billing is one of the implications of it. The reason you have resource coder is because you bill for it right, so you know tenant a has. You know, quota of you know X CPUs and Y memory, and maybe these are the pieces, can fit some profiles in or later and and so he's going to be, build on it, and so.

G

Thank You social should that we we kind of put requires and limits altogether. So it's it's also I.

H

Think that use kids is definitely one name space, not using up everything that was allocated to other namespaces right other than building they.

H

B

The point is that, if you want to have, you can additionally want to have in space quotas, but that's more because you want to have some fairness within your tenant across. You know: applications using namespace one versus applications using namespace too so I give it a quota, so that I'm sure some fairness, III want to get at least I want to get twenty percent of the CPU right or something like that.

B

But in terms of what is the resource consumption of this tenant, that's the tenant quota, the tenant limit by the way, that's another question like: are we, including both quotas and limits at the tenant level? I, would assume yes right. Yeah.

I

B

So that at the higher level that has implications for billing is one right. So but let's, let's you know, think about it: offline and kind of what.

I

I do right: I will create some scenarios for the in s4, which is without the recess quota and I will put like multiple scenarios and use cases like I. Think we discuss over like four models here. I will put all these four models. We only had two models. I will put all those four models and you guys can comment on this page. Actually, it's the same scenes here for Linux didn't.

H

Make sense to put it in a dog, wouldn't that be.

I

I can see it in Google, Docs and I can share it. That yeah I can do that. Yeah.

H

I

Sure I will do that both yeah.

I

H

I have a high-level question: what's the like, so we have tenancy already in this tenant controller. Like is this. What state is that in like is that has has that been reviewed and agreed upon and yeah? Is that part of the incubator or is is so.

B

That was a POC right. What we did so far was a POC actually that bill will have and that as an agenda item in the next call as well, where we will review the state of the tale in control and sort of what needs to be done to make a more official version of it, because right now, it's just a POC right, and there are some open questions there as well like the namespace naming and things like that, and then this resource quota thing.

B

We also come into play there so yeah, so that is a POC right now. It is open for comments. If somebody has any concerns showstoppers, you know just add comments on on that against the github POC against the github repo.

H

B

Doc as well, well so yeah you can look at the docs page in the links and you'll see the docs as well. Okay, and you can go back and refer to our talks from the skip corner on the deep dive. Every demo know that so the status of that is it's a POC. It is a working proposal. It is by no means final.

B

We need to have a plan for the final, a little bit more official version of that, so in any one of the upcoming calls, one of us may be myself, or one of the others will have an update on that in terms of the official direction for that controller.

B

So it's still a working progress so feel free to provide your comments, but we will bring it up in the next call in terms of what's the next steps for that sounds.

H

Good thanks, thank you.

A

Oh okay, I think that's it for today. So thanks everybody and chat soon.