Kubernetes WG Multi Tenancy, 25 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 20200825 Kubernetes Working Group for Multi-Tenancy

Description

Presentation on the Multitenancy Operator by Capsule

A

What's that I just hit record okay thanks yeah: uh do you wanna? Do the intros yeah go for it? Oh okay! uh Welcome everybody to the multi-tenancy working group meeting for uh this late summer day, if we're in the northern hemisphere.

A

um So I asked adriana to come and join us today, because he is the author or one of the authors of capsule, which is a multi-tenancy operator, um and I've noticed that there's uh there we're starting to see a couple of these pop up like that, so there's capsule, uh there's kiosk, which I don't know very well. I only found about that one about a month ago.

A

There is, of course, the tenant operator or the aka, the tenant crd, which is a project within this group, um and so those are both kind of like, let's say, user level constructs, and then you have hnc, which, like one level below that, I think on the stack, um and I think this is a good time to start understanding.

A

What's the same like what's common between these systems, what's different, can they work together? Should they work together, like there's, no need to consolidate things if they're truly solving different use cases, but uh but yeah? I wanted to hear about what was going on with capsule, uh and maybe we could have a discussion afterwards about how these pieces should fit together, even though we haven't heard from the kiosk team yet um so uh sorry adriana. Is that the right way to pronounce your name.

B

Yeah yeah, it's correct. It's like your name similar to it.

A

My name in english and french are very different uh and you're joining us joining us from.

C

Yeah uh so, okay, um first of all, thank you very much for uh inviting me to this uh to this group. That's my first time and uh let me just uh introduce capsule and myself, um so we developed a capsule when we developed the capsule. We were not aware about this working group. Really we discovered this later after we released the the first prototype.

C

So what's the story of a capsule capsule was designed because a requirement because the requirements coming from our customer, we are a small uh consultancy, firm basic in italy and we provide consultancy, support and on kubernetes and cognitive technologies. So uh we have. We have a customer that is uh trying to build a public uh container as a service platform uh using bernates, and so we soon discovered that uh straight kubernetes was not enough to support multi-tenancy, and so that's uh why we developed a capsule.

C

Capsule is a seal in early early days. We did not yet release the zero zero one and we are still on zero, zero, zero and so, but we are receiving uh very good feedbacks from customers, of course, and also from the community. uh So then we discovered uh this working group. The multi-tenancy working group then, and uh the uh a um hnc, h and c yeah uh that that's the uh the.

A

Hierarchical namespace controller, yeah, yeah, yeah.

C

Yeah the hierarchy, namespace controller, and uh so uh we are really really interested in this in your work we are still discussing if we can in some way adopt a part or totally your code base.

C

We are still uh trying to understand the the common part, uh the a and the the the approach that the common approach.

C

I don't know if this is a um a feasible uh thing, but we hope we hope to uh uh how to say to to adopt um your project uh as a foundation for uh our uh for our uh capsule. So uh please consider that capsule, the open source project is uh just the the building block for uh an enterprise platform that we are designing, and so a possible approach is to adopt your code base into this platform.

C

uh So we are just trying to understand it's.

D

C

Really uh we a lot of work we have to. We have to do uh so anyway, uh this uh this meeting for me, is it's just the first step uh in this in this direction, and so thank you very much for uh for uh uh for for inviting me to this uh to this.

A

Yeah thanks for for joining us, um I think maybe one of the I think the the best place to go now is maybe for you to tell us a little bit about what capsule does like what features. Does it add to chord kubernetes.

C

Yeah, so basically the idea behind capsule is uh to to uh is the the kiss approach try to. uh We adopted a minimalistic approach to design the the operator, so one of the requirement was avoid custom resources to give to give custom resources to the end user.

C

We we defined the only one crd, uh the tenant crd, but this resource is not uh is not for the end users. This resource is only for the cluster administrator.

C

So all the experience for the end users is just straight kubernetes, so the end user uh uh doesn't have done, don't need to uh to use custom resource, don't need to use uh uh cube, ctl, plugins.

C

He just needs to operate in a regular in a regular way. So all the job is done by the uh by the by the the operator and the operator takes care to group namespaces in uh in a tenant.

C

So it's just a grouping.

C

The job of the operator is just grouping the namespace in uh in a tenant, uh so there is also something that is uh in common with your project. That is the um the inheritance.

C

So when we, for example, we can assign airbag to the tenant, we can assign a resource quota, we can assign network policies, we can assign node selectors affinities and so on to the tenant when the user creates a namespace in a tenant. So this namespace uh inherits all these objects from the tenant.

C

So, uh for example, uh this namespace in reads the resource quota, the limit range storage classes and so on and so on. So that's excuse.

E

Me do we have any slides or it's just audio.

C

uh Yes, I have a slides. Let me let me try to find it. I used these slides for the presentation to the customer. Let me try to find them.

C

Where is this one? Okay, can I share my desktop share screen? Yes,.

A

Let me just want to check it stay on the line. Are you there yeah? Do you see my screen yeah? We see your screen uh fake wow. Are you? Are you online yeah? Okay, great is any. Is anybody else who works on the tenant operator uh online as well or can you represent any discussion about the the tenant operator.

C

Yeah, so this uh uh okay, these slides, are uh a bit old. uh We don't have updated slides, but just to give you the raw idea about the about the the project. uh Okay. So basically we want to group namespaces in um in.

C

So that's the the approach namespace aggregation. We want to aggregate namespace in a tenant abstraction, uh so that's the basically what we uh okay, we we present this in this way. So namespace is a flat uh structure. uh Any namespace, it's an isolated environment with constraints and these constraints cannot be grouped, cannot be shared between different namespaces.

C

So that's capsule capsule. Does this job to assign the resources to the tenant and all namespaces inside the tenant inherit the these resources that that's the the basic uh the basic idea behind the project? Can you just ask a.

A

Question, how did you, how did you implement that for the resource quotas, um how do you implement the sharing.

C

The resource quota you mean the resource quota yeah. We uh we avoided. You know, probably you know that uh in open shift. For example, you have the project, a resource quota that is a crd, a castle resource that is used to share resources between uh different uh namespaces different projects in openshift. They are called projects.

C

uh We have uh uh avoided to define uh uh custom resource for this and we uh just implemented uh just it's a uh a knack that we are trying to improve.

C

uh So uh basically, when we uh we assign the resource quota to the tenant, uh then the namespace inherits this resource quota, but we calculate the aggregate instead to we calculate the aggregate and the controller change changed the resource quota usage into the single uh namespace into the single namespace.

C

So at this point the uh standard, uh wrestler admission controller, the resource code, admission controller blocks, the creation of a new new pods or a new.

C

Or new resources.

C

So let me let me show you this. If I try uh so before to go in this uh in this detail, I wanted to just show you the main scenarios that we uh can cover with the capsule, and these scenarios come from the requirement, the requirement of our customer.

C

So we designed a capsule having in mind these requirements, uh so we, for example, we can enforce the quota at tenant level. We can enforce the node selector and the affinity for the tenant.

C

We can enforce the ingress selector for the tenant. In case you wanted to assign a dedicated ingress controller to the tenant.

A

Does that actually share a single ingress or just a single ingress controller.

C

Yeah yeah, it depends, it depends if you have a single ingress controller, for example, you can uh shard, you can partition the depending on the ingress controller, but in our case the requirement was that um we have a multiple ingress controllers, one for a different tenant, so each tenant has its own ingress controller, and so uh we use capsule to assign the ingress control the the ingress class to the uh to the tenant, and so only the tenant.

C

Only the ingress controller that you assigned to the tenant is able to uh to uh to expose the the service the interest, but.

A

If they have, if a tenant has five name spaces with services in each one, you still need to create five ingresses one for each namespace. Is that correct.

C

No, no, no, it depends inside the namespace you can, uh you can do uh you can work in a regular in a regular way. So let me show just to if you check the uh the github, you can see uh uh more. um These use cases with more detail.

A

Okay, maybe I'll have a I'll have a look later. In that case,.

C

Yeah yeah, but you see here this is the uh the tenant uh the tenant object, so you can assign the ingress classes to the tenant, the storage classes, uh the name.

A

I'm still looking I'm still looking at your slides. I can't see.

C

A

Okay, you're just playing at your side: okay, yeah.

C

Okay, uh so other use cases, the storage class. You can assign a storage class to the tenant so.

F

How about uh how about the pv uh persistent volume object? It's a non-namespace scope, yeah.

C

Yeah, exactly exactly exactly so all the persistent volumes created in the namespace entertainment in the namespaces that belong to the tenant uh will receive the um the uh the annotation telling you the uh the storage class to use and the. uh In addition to the controller, there are a set of admission control admission web book, uh validating a web book that enforce the usage of the specific uh storage class. For for this, so let me let me show, for example, you see you assign these this one to the tenant to the oil tenant. Okay.

C

So when the user creates a pv, a pvc, sorry, a pvc.

F

Sorry, are you showing a diagram or something we are still seeing your slides.

C

Yeah but I don't have this into the slide.

F

C

A diagram- I don't have a diagram, it's better for me to show you this. So let me show this just be patient.

C

C

Storage, glass.

A

I think that the problem is that we are looking at your slides, but I think you're not looking at your slides anymore.

F

Yeah, you are shooting something else, but we are still.

C

Okay. Okay, sorry, sorry, sorry, sorry, you are just seeing the slides new share desktop sure. Okay, probably you see this now you see yeah.

G

C

Okay, so you can assign storage class.

C

We probably don't have this multiple namespace permission quota.

C

Ingress selector storage class, for example, you see you assign this storage classes to the tenant. Then all the pvc created in these namespaces will receive this storage class, and so the end user is not allowed to change this, because this is. This is a requirement that comes from the cluster admin. So the cluster admin uh set, the storage class, the alloyed storage classes into the tenant, and so the regular user uh receive automatically this search class and cannot change.

F

I see, I guess my question is since pv objects are not in any namespace, it's cluster scope. How do you isolate this scope? Objects.

C

Yeah you, uh the pv the persistent volume, the persistent bond yeah. We we assume that the dynamic storage provisioning is in place. We don't deal, we don't consider it uh the the manual uh pvr assignment manual, pb creation.

C

So if you have different storage classes, so you can- uh and you force the end user to use uh yes only as designed storage classes automatically, you have the isolation. For example, you can assign a storage class to a tenant, and this is, for example, um a sf. You are using a sef storage class, and so this storage class, for example, can point to a storage pool in yourself uh environment, and so you have a storage pool assigned to the uh to the single tenant.

F

But I see I get how it's created, but how do you guarantee a talent can only uh like a list or get his own periods? Not other talents appear.

F

I don't get the questions like. Let's say you or you have several pvs belong to different storage class created by the provision right. They are all in the class level. um How do how do we guarantee one talent can only access his own, his own tv, instead of other people, there.

C

But user, usually the uh the the end user is not allowed to access the pv. He only sees the pvc.

F

Yes, that's, yes, that's the usual access pattern, but on the api level, what's the um what's the mechanism that we prevent one time and from accessing other talents.

C

The airbag the standard airbag are back.

A

C

User, the the the end user here allies, the end, our end user is not able to get pv to get storage classes to get uh nodes because it's uh it's you see. uh Let me show you: when the uh allies uh login to the uh to the to the to the platform allies can check. Can I get namespace? No. Can I get nodes? No can can I get persistent volume, no and so on.

F

C

It exactly exactly exactly from from the uh allies point of view. uh The uh hero experience is the is like a a regular, uh a regular kubernetes user. This this was uh intentional. uh This was one of the key uh design, key criteria for design capsule.

F

Get it okay! Thank you. Yeah.

C

Yeah that that's the uh and so uh you can see uh capsule as a combination of a controller uh for the crd and a combination of uh admission, controller and validating controller.

C

You see we have a set of the dynamic admission controller uh in addition to the uh to the basic one partner, selector limit range resource quota, mutating and validating.

C

We have these, for example. uh Let me check where are they should be here? Not here.

C

This one, the box.

C

You see we have a set of mutating and validating the book, one for, for example, for the ingress another one for the storage class. Another one.

D

For the namespace.

C

Quota, another one for the network policy and so on. Okay, so it's a combination of a custom of a single custom resource, a single controller and a a set of uh um uh validating and mutating uh web books.

A

Well, why would why was it important to you uh to only have one crd, especially if regular users can't see them um just to make it easier to manage.

C

Sorry, I don't get the question sorry, my my understanding of is not so good. So.

A

um No problem uh you've said a few times that you only have one crd was that important um or it was that just the way it happened to work.

C

No, no, we did the this was intentional. We designed it uh having in mind this requirement, we do well. You mentioned the kiosk, for example, uh because in koski you have a lot of crd, and so the end user has to learn a new crd, new object and so.

D

C

Wanted to avoid this it's by by design we only uh the experience at the end, the experience of the end user is uh it's like a regular user. You, don't you don't know the end user is not aware of. Capsule is not aware of. Multi-Tenancy is not aware of crd and so on. When.

D

C

User, when the end user login, I mean the end user- is the the so-called tenant owner, the the the people that can create namespace inside the tenant. uh So this user is not aware of the of the of the tenant and the crd.

A

And sorry, how does a user create a name source.

C

Namespaces, yes, it's able to create namespaces, because when we assign, um when you create a tenant, let me show we also uh create these two cluster role: uh namespace, provisional.

A

To create the namespace okay.

C

Yeah cluster rule, okay and the class rule binding okay, it's a namespace, provisional. This row, this class rule is assigned to the uh group of the user, so the only requirements, but we we are changing this. We are changing this behavior. The only requirement for uh to have a capsule work uh to have a working cap, a working setup with cars with capsule, is to have the user belonging to this group. Okay.

C

So that's the only requirement, so we assign to this group this cluster role, uh this cluster rule. So all users all end users are able to create namespaces, but but the user can delete namespaces because we assigned the permission to delete namespaces, but not at the cluster level, but only at namespace level. So when uh the user, when alice, creates a new namespace, she gets this role binding namespace later. But this this is assigned only to the namespace to only to the namespace that she created.

C

So she can. She is able to delete only her namespaces, not the other namespaces you see get world binding, create other namespace.

C

Create other namespace and then oh, I don't have here blah blah blah blah blah.

C

No, I don't have here, but uh so trust me. She is not able to uh to she's able to delete only hair, namespaces.

A

And that's enforced through our back or through something else.

C

No, no, our box regular, are back.

A

Okay, because you you create, so what is it? You automatically create a role that.

C

C

This was another uh requirement, only uh used at only the standard, kubernetes tools, uh nothing to invent other stuff, because that this requirement uh went from the the the customer went from the client.

E

So can a tenant user list their namespaces like? Can they see which namespaces they have created.

C

List namespace, so you are touching a very huge topic.

C

We have this opened.

C

Okay, ability for tenant owner to list namespaces in the tenant, as you can see, this topic was long discussed in the community, uh but there are a lot of.

E

C

Closed issues, because you know you know- probably you know this comment from these guys, jordan legit, so we don't have yeah we. This is, we have a don't, have the option. We are working.

D

C

This by implementing, um so I told you before that uh capsule will be the building block for a a more uh wide platform, uh so this platform will implement an api that that it's a open shift style api. That will give you the the option to list only your namespaces, but this will be just uh a customization, is not the the regular kubernetes is not allowing you. This.

E

So you mean using api aggregation, yeah.

C

Okay, that's the that's the the the the point for this.

A

Yeah, you need either either aggregation or some kind of crd to do that right.

C

Yeah yeah, uh so uh anything else. No, that's it's very! It's just a minimal tool.

C

So that's it so the the idea. uh Anyway. The idea is to try to to to find something in common with your uh project and try to try to use part part or all uh of your project to build something uh more complex. I don't know it's. We are just studying your uh your code base just to understand, and uh so, uh but we have, uh because we are in the early uh days.

C

uh We uh we want. We desire uh to.

C

B

C

uh A more uh a project that is uh supported by by the community.

C

This is just our uh opinionated way to implement uh multi-tenancy. I I know that these other people cannot agree with this approach, but you know I told you before uh this went from uh requirement from our customer, and so we we we had to implement this in this way because of the requirements.

A

Yeah, this is all really interesting, and thanks for joining to to tell us about it, um let's see I'm not sure what the right thing is to do. Next, um how does this compare to the tenant.

C

Crd, the cld to tenant crd.

G

Yeah, so we we, we have implemented a similar version of the tenant crd. The idea is um so you use every api server, but we do we use crd to encapsulate the namespace to work on the program that you list about and how to view their own namespace. Only so we use a kind of crd called tenant namespace, which has a kind of representative proxy for the actual name space.

G

The lieutenant owen can manage the tenant name. Space object, so we set right object, rules for tenant, namespace object. Then um I mean this has some you know learning curve. People should understand. Okay, one tenant name space is corresponding to one real name space. So that is the way that we we does- and I I checked, um but you you does actually does more than that. So we currently only do that.

G

So resolving the you know, namespace management like create delete list, and that's all about it, so each tenant can only look at its own, um but uh this is the talent crd. I mean that that that we have in in upstream I mean in a working group right now, um but yeah, but we do have some other ways of you know: grouping namespace, you probably already know we have. You know I chance or name space to have another way of grouping them space um yeah. We do.

G

We have a virtual cluster, which is uh a pretty different way of another, very different way to grouping them space um yeah that that that's what all I can see, but I think you guys so you got definitely does more than the currently what we do here. We already look at the name, space list power and this power.

A

How many people are using um the tenant operator right now other than virtual clusters? Do you know.

G

uh That I'm not I'm, I don't know. Actually I don't have. I don't have the right, I don't have the concrete number for now. um I don't know. Is that way to do the polling or actually this is always my question. How do you know so? How do you know how many people use your project.

A

um Yeah, so right so like I'm interacting now with some gcp customers, um I'm I'm getting some uh like somebody posted a blog post today from sainsbury's the the british grocery store. So I'm that basically I hear it because if people want to use it, they end up, they end up getting in touch with me.

A

So I can I'd say that there's maybe two or three large customers who are looking at it or using it seriously like we started accepting pull requests now from murakari in japan um and in internally at google, we're talking to some other customers that I can't talk about, obviously but um yeah.

A

So I I do see some people starting to use it and just to be clear, like with h c, we've been working on it for close to a year now, but we only recently like really tried to sort of push out um getting people to like asking people to use it, because it wasn't ready for a long time. We didn't want people to be using it until we reached a certain level of maturity.

A

So I think we are now at that level of maturity. And so um that's why we uh talked about it at qcon.

A

uh We we've released it at google as a supportive product um or sorry it's beta right now, so you don't get full support yet, um but and we wrote the blog on uh on kubernetes.io, so I am seeing an uptick in interest now, which is what we were, which is what we were aiming for now that we're at this kind of um approaching stability level.

A

um I think it would be good to look at how I haven't looked at kate spin.

C

Yeah yeah, that's uh sorry uh d. uh I see uh we see a lot of interest in multi-tenancy in these days and we also discovered that is another project, another open source project. It is called the ka k8 spin operator that is uh trying to implement the multi-tenant.

C

In the same way, the these guys are from spain, and uh so also they are trying to implement the multi-tenancy.

A

Yeah angel or angela, I'm not sure uh he got in touch with me- he's been trying out agency as well.

C

Yeah yeah yeah a lot of interest in the community uh about to market tennessee, because you uh I I followed your talk adrian in uh at cubicon and.

D

C

It very interesting- and so I liked the slide when, uh where you showed the cluster uh sprawl, probably yeah this one- I like it, I like it this yeah yeah. Exactly this, you see, I we see, we see a lot of a solution for multi-cluster management from any vendor. Any vendor wants to sell you cluster uh management solution with thousands of clusters, but we found this a bit pain.

C

It's a it's, a it's a pain when you have to deal with 100 of the clusters, so we are trying to uh to use a different approach instead to create a multiple clusters, a cluster for each department, a cluster for each user.

C

We want to try to.

C

Put the things uh back to the simplicity with a single cluster and multi-tenants: that's our that this was our uh our scope, our goal.

A

So how about this for the next step? um Okay,.

D

One quick question before you go there, so how are you dealing with uh the crd problem? Are you tenants not having crd collisions or things like that for multiple tenants? Let me try to ask it in a simple way, so you have multiple tenants that happen to be using similar components. They want to deploy they're, bringing but maybe they're different versions or or different expectations, and and they don't have the rights to deliver the crds to the cluster. So now you have.

D

How are you dealing with that? Are your? Are your if your tenants bring operators that are clashing with each other or competing with each other in terms of the need to deploy it, and you don't have permission to do that right.

C

So you mean from the cluster admin or from the end user from.

D

The end user right, the end user, you have tenant a tenant b, they each for whatever reason are using the same operator, but maybe they're using different versions. um So they each want to bring a c or d to the to their name, spaces, which you cannot create.

C

D

C

We honestly, we did not consider this this case for now, but probably it's something that we have to to to check to try.

A

Yeah, I once had a chat about it with um daniel smith, who's on the api machinery sig um and yeah. So there is no good way to do that right now um there are a bunch of hacks that you could build, um but they all break down in various entertaining ways.

A

um I've I've looked at a couple of ways of possibly doing it, but the one thing that I could never figure out how to do is how just uh the discovery docs would work. If anybody knows what discovery docs are. I I wanted to actually go ask a couple of people. What would happen if we just didn't do discovery? uh What would break um but yeah, so hnc took the approach that we're just adding features to namespaces.

A

uh We are not trying to create virtual clusters, so virtual clusters will solve this problem because, with virtual clusters each tenant gets their own api server, yeah, and uh so with you with your own api server. Obviously you get your own crds and so right now that is one of the I would say like if you want a multi-tenant cluster, but you also want crds uh pretending. Crds virtual clusters are the way to go. You don't really have any options and there's either um so yeah fake and can help you out there.

A

um I'd love to see namespace crds become a thing um to hack them in, like without major modifications to core kubernetes. As I said, I have to figure out the discovery problem um with that. um If they were to become popular, then we could probably get them into core kubernetes.

A

Does that answer your question.

D

Oh yeah, I was familiar with that. I was more interested in whether capsule had some creative way of dealing with it sounds like you're telling me. There are no creative ways at the moment right, essentially uh and, and the problem with something like virtual clusters is. If you have networking an example that is a little more sophisticated than than a flat net typical approach, it kind of breaks down.

A

D

A

um And then I do want to go back and talk a little bit more about capsule. But can I ask um what is the the use case? It's where you have different, basically um different tenants wanting to install possibly different versions of the same operator which might have incompatible crds.

D

I can tell you what we have right: we have our tenants, our vendors that come with their own products and they're all might bring their own telemetry or or their own, in some cases, they're trying to run istia, which we know we understand those problems, there's no real multi-tenancy in this example, but but things like that, right or or even they might bring their own cert managers built into this thing, um they're all kinds of basic stuff like that, really it's and so sure you can try to create a coalition where you get all these vendors to align in the same versions and but that's that's complex uh in this space that we're dealing with, which is a telecom world, and so that's why so what we landed now is I looked at vnc and based on our requirements at the moment.

D

I I built a solution that builds bare metal cluster. I I took phase 10 and operator and I expanded it to do something along the lines what capsule is doing, but but we always run into this crd problem. So at the moment, we roll back we're dealing with basically building building clusters on vms, on behalf of the vendors, to avoid this problem.

A

Yeah, I think that was it. I'm sure faye would want to know more about why vc isn't working for you uh virtual clusters, maybe there's some reduced version of virtual clusters that could work better.

D

D

Yeah. Thank you. Oh.

H

There's also just a call out a handful of other things. I know uh we're experimenting with virtual cluster and it seems pretty great for what we're trying to do, but uh there's also a handful of other projects to check out in this space. Like I know, darren shepard from rancher has an old project called k3v, which is meant to be like k3s virtual clusters, but that changes the way that everything is structured, where it ends up mutating all requests and putting them into a single name space.

H

So it's a little bit different than how virtual cluster works and might be more lightweight. If that's something that you're you're just looking for isolation of crds.

H

A3V I'll take a look yeah check out. uh I think I think it's github handles. I build the cloud uh and it's just I build slash. K3V should be the project.

D

I don't think it's.

H

Maintained at this point, but uh I know when I've when I talked with him in the past, uh there's always there's always the opportunity to respark projects.

G

Yeah there is another project I think called the loft loft, I think, which is uh uh involved the version of the k3v, but that is a past past platform. If it provides more services, you know seriously to handle the virtual cluster you can.

G

I would say that is a commercial version of the virtual cluster, but again the the their implementation is kind of still different from what exactly we do in the details, but high level picture the same thing: each talent has its own dedicated api server and they share one super master for the particle vision. um But we differ in the details. um I don't want yeah, I don't think uh yeah. I I want to think more.

G

I don't want to dig more in this meeting, but if you're interested, I can tell you in a slack channel, what's exactly the difference between the loft version of virtual class and the virtual class that we do in multi-tenancy working group.

G

C

Can I ask a question: can I ask a question to go yeah? Is there any any anybody from working on this multi-tenancy benchmark yeah? That's.

E

Something that I've been working on here- okay,.

C

So we uh uh we are, uh we, I told I told you before, that we want to approach this your work, and I think that the the first approach could be this. uh We want to try.

D

C

uh Something about uh about this benchmark and try to adopt these benchmarks in our solution, just as a first step.

E

C

The one one thing and.

E

No absolutely- and this is why we sort of initiated this track, because I think what we sort of foresaw- even even you know, as as before some of these newer projects came about. There was this growing interest in different ways of doing multi-tenancy, but there was no quick way of determining if a namespace was configured for multi-tenancy right. So that's that's what we're trying to accomplish here- and I believe I I think, maybe we you know- maybe we interacted on one of the pull requests you created.

E

If I recall um so, we have the scoop cuddle, mtb uh plug-in now, which you can run on a namespace. It will go through a set of benchmarks and give you a report. We demonstrated that I think a few weeks ago in one of our meetings, so the idea is, we need to go back and define now what the right you know profile level. One and two benchmarks would be for different levels of multi-tenancy right, one of the things that was important, though, and I think you've solved this um a bit.

E

You know with your project is allowing self-service and sticking to. You know just kubernetes concepts versus introducing uh new abstractions right, so that's those are some of the things we're trying to figure out and then also, I think, for this working group. What's interesting is you know, do we sort of accept that okay, there's gonna be several ways of configuring multi-tenancy and we want to encourage different ways of doing this, but maybe there's a few things. We want to standardize on like with the benchmarks and perhaps with some annotations and things we can define.

E

For example, one problem we're running with the into the benchmarks is: how do we know like, for example uh like if you create a in you know an ingress controller um for a particular tenant or let's say you create even a quota or something within the namespace.

E

Maybe quota is not a good example, because you can't edit that with the namespace admin role, but let's say you create, you know, network policy right and how do you prevent the tenant from deleting this? So I think what you've done is you've written a custom, mutating or validating webhook for that, um but if there's a general way to know that this resource needs to be protected, those are the sort of things we might need to standardize on, so that this particular resource is not something.

E

A tenant user should be able to delete or manage.

C

Okay: okay, but are you planning to implement these as a controller or it's a just.

E

Yeah, the benchmarks will be a cli tool and that cli we can also run as a job, and it produces a report so.

D

E

Is there's actually- and this is some work that we're doing in the policy working group is we're creating a common report for different policy engines, so the benchmark tool will also produce that common report, which can then be consumed programmatically or just as a cr. However, you wish okay so yeah. The idea would be. You can periodically scan your cluster if a new namespace is created, you can audit it for multi-tenancy and at least uh show compliance right. So, okay,.

C

Okay, so, okay, okay, so that's so no plan to implement this as a as a controller, something like this. You know probably polaris.

E

All right so not yeah, not so much from the benchmark point of view, but if you're looking for configuration, audits, uh polaris is one there's kiberno, there's oppa gatekeeper.

E

So all of those policy engines do configuration validation, uh for example, a lot of what you're doing with your custom web hooks uh kiverno can also implement as validating and mutating checks right. So there's two different ways of doing that as well: okay, okay, great so, but yeah on the benchmarks.

E

Definitely, let's, let's collaborate, and maybe we can discuss more on the slack channel um and you know it would be interesting to run so if you have a cluster set up or if you know we can try this out on one of our test clusters and also run the benchmarks and we're going through the same process with hnc as well uh and we'll do the same with virtual clusters to audit. For you know: compliance okay,.

C

And uh what about this tenant operator is some. I is something that is.

E

A

E

A

Yeah, that's the one, that's the most similar to what you've built in that, whereas agency is quite low level and you could probably build not all of capsule on it because we do have like for name space self service. We do have a special object. It is the it's only one that users have to deal with, but it is one um so yeah we do other than that.

A

I think you could probably build a capsule on top of hmc or on top of the features that we would like to add to hmc, whereas the tenant operator actually is a slightly alternate usage model they're similar but they're. Not.

A

I don't know that they're composable, exactly because you've got the concept of like um correct me if I'm wrong, but but like in the tenant operator, you've got sort of like the tenant name space, and then you put a bunch of stuff in there and then you have the the sort of lower level name spaces as well.

A

You don't have that you don't have any one object. That represents the sorry you don't have. You have exactly one object that represents the tenant, but there's no one namespace that represents the tenant. It is an aggregation of namespaces, whereas that's not true in the tenant operator, um I'm not sure which I don't know. If we we need both, it could be that it would be nice to sort of sit down. I'd love to do it in person, but that's not possible. um It would be nice to have a look sometime and see like.

A

Could the use cases be resolved the same way.

E

Okay, well, the other interesting question. There is also what sort of flexibility do cluster admins require right. So I think, with with some of these names, multi-tenancy or namespace based constructs like, for example, with ingresses right, allowing shared ingresses versus per namespace like who should make that decision? Is it like a cluster admin, or is it a a project that makes that decision right.

C

Do you mean in uh in capsule, you know, do you mean cups.

E

No, no just in general, right.

C

General in general,.

E

A

How does shared ingresses work just use a an external name, you import the service to an external name. Is that the online.

E

Or different url prefixes right, so different paths.

A

I just because an ingress can only reference a kubernetes service in the same name space. I'm just wondering how you get around that limitation, other than external names.

C

So by sp I can. uh I can speak from my experience from my field experience uh this. uh This is something that is uh uh the the clustered means uh tend to to keep for themselves, uh yeah that they don't uh don't leave these freedom to the end users so because, because uh they uh so you know the ingress, uh it's more um related to the to the networking uh to the networking part yeah. So you know uh we we can.

C

We everybody want to wants to to remove silos in I.t, but silos are still there and so sometimes the network guys they don't allow end users to decide which which uh how to expose the the services they want to to keep this under their control. So it's more something that is from the cluster admin duty.

A

Yeah with that, I'm afraid we're out of time, um but this has been a good discussion and I think that we should uh maybe jim fay isano, and I should get together, possibly on slack and talk about our next steps. Yeah. That sounds good.

E

Yeah sounds good yeah.

A