Kubernetes WG Multi Tenancy, 5 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG-Multi-Tenancy Bi-Weekly Meeting for 20220405

Description

WG-Multi-Tenancy Bi-Weekly Meeting for 20220405

A

Sasha, you don't want to do the intro, or would you like me to.

B

Yeah sure I can do it, hey everybody welcome to our regularly scheduled multi-tenancy working group for kubernetes meeting. uh Today we have a pretty packed agenda or with some uh co-chair proposal, changes um which everyone should have seen go out on the mailing list and also uh jim and team have some updates on the kubernetes multi-tenancy docs proposal.

B

um We also have an announcement so I'll just start with the announcements and then I'll hand it over to jim and team to talk about the docs. um So first announcement, uh the hierarchical namespace controller, hnc 1.0, is now out big milestone for the project super excited for the team, and you know whoever, if anyone's checking it out using it, has feedback.

B

Please let the team know uh we're all watching github issues and chat, so you know uh take it kick the tires. uh Let folks know what you think, but the 1.0 milestone has been achieved and really really big, big, congratulations to adrian ludwig and everyone. Who's worked on it so far, um and then the second update is that we are going to be nominating some new co-chairs for the multi-tenancy working group.

B

um Sanji vrompal is going to move to emeritus status um and we are going to be we've put together a proposal to move uh jim and uh adrian and faye and ryan um into co-chairs.

B

All four folks have been really engaged in the project for probably three years at this point, um leading various teams um and then ryan has been our most consistent sort of cross project contributor, which we really appreciate him for so I sent out sort of the announcement um on the mailing list for this proposal to add new co-chairs um and we will be moving uh forward with that using lazy consensus. So if anyone wants to reply to the chain um with a plus one or feedback, um please feel free to do so.

B

If anyone wants to give me any feedback offline, I'm definitely here. But if we don't hear any objections, then we'll be moving everyone into co-chair status at the end of next week on friday, which is the 15th of april. um So those are the two uh bookkeeping announcements. um I do see that someone's in our slack channel asking for the link for the zoom meeting, it looks like someone's, replied: okay, cool thanks.

B

What did I say? Always there to reply to the slack message or pick up the pr, um so thanks ryan and uh yeah cool. So I think that's all of our bookkeeping for today uh and now I'll hand it over to jim to talk about the docs.

A

All right, thanks, tasha, okay, let me share my screen and I'll also um add the docs, the doc link into chat for those who might not have it so yeah a few weeks ago. I think we started- um or maybe I guess it's been perhaps has been maybe longer than that a couple of months. We started this effort um to propose an update to the community stocks for multi-tenancy right, because one of the key questions we get is, it seems like we have reached like tasha mentioned, with hnc and also with other projects.

A

We have reached some level of maturity and knowledge on multi-tenancy, but there's not much or there's. Actually, no, you know guidance in the community stocks around multi-tenancy, so the idea was to propose an update there. So this is a working document which we will convert into a pr hopefully fairly soon. um I think we're almost there we've completed most of the tasks where you know.

A

I think we have a few pending items and then we can- and even while those are in discussion, we can go ahead and at least create a draft pr and start getting some wider reviews on it. um So the idea was for today was to just go through some of the main questions which I had listed in the agenda, and you know once we're good with those we can. I feel we can convert to a pr and they'll always be minor changes.

A

I'm sure we'll get, and hopefully we'll get plenty of other good feedback, and we can continue to iterate um before we dive into details. Does anybody else have any just general thoughts on how this has progressed so far or where we are? Does everyone kind of share the same feeling that we're in a good spot, to start converting to a pr? After a few um you know, kind of items have been resolved.

C

It looks good, I love it.

D

Yeah, it's looking good.

A

Okay, all right so yeah, maybe just going down through the list, and I I had listed a few things but feel feel free to bring up others, which you, you feel a worthy of discussion uh right now and, of course, about things. We would um those few minor things we can continue changing just offline and asynchronously.

A

um The first thing I had kind of listed is on the storage, isolation and- and I saw ranjit thank you for making the updates and um I'll go.

E

A

Also reread, but maybe just so that we're all kind of clear- and maybe if you want to kind of just describe what is the guidance here we that we want to provide and if somebody's implementing multi-tenancy either team base, or you know the customer facing multi-tenancy.

A

What do we want to say about storage or what should we say.

F

Yeah, um can you can you hear me? Yes, okay, cool, so I did make some updates based on your. um You know, feedback gym, so you know so. The way I was thinking about this uh is that you know we have this whole uh or kubernetes. Has this whole um concept of storage class and versus persistent volumes and uh pvcs right, so the claims that you could attach to a namespace?

F

um You know I I I thought so how about we use some of those constructs and uh because, because this whole section we are talking about control, plane, isolation and how we can use a name space per tenant strategy. I thought the whole pvc uh claim per tenant that you could attach to a namespace could be a good uh story to tell um you know because you're right, like you know, I was reading through your comment.

F

You could uh have an app that is inherently multi-tenant and you could talk about things like sharding and you know other isolation strategy when it comes to storage. But that's not a kubernetes story right. So we are you know. So the thing I mean. What we're doing here is to try uh try to tell that. Kubernetes story and from that angle, so I I thought maybe this would be a good um topic to talk about. I mean I love feedback, obviously from other than what others think, but that's essentially what I was thinking.

F

So you could take a claim or you could take a pvc and attach that to a namespace and if you're, following a name space for a pretend strategy, um you know so yeah, so you can assign some dedicated storage uh for that tenant right or for the workloads or the pods running within that name, space dedicated to a tenant right. So that's that's the whole idea again right.

A

Yeah, no, that makes a lot of sense, and I think it's good to you know outline some best practices and I recall there was there were some prior discussions and on this um so seems like one. One option could be that if you're running in a multi-tenant environment you have a storage class per tenant, which you know prevents any any chance of um somebody else claiming a pv reclaiming right. uh The other.

A

The other thing would be the reclaim policy right to make sure that it's configured as delete and pvs don't get go back into a pool for reuse and things like that. So those.

E

Two things.

A

Definitely seem like worth pointing out.

E

um Generally, we've recommended a dynamic with a deletion. Reclaim right is the preferred and then storage class for namespace is kind of the. If you can't do dynamic, um ppcs or pvs.

F

I was looking at the whole storage class for namespace, but I couldn't find good. um So are we able to associate associate a storage class with the namespace.

A

So there's no built-in construct for it. You would have to use something like oppa, gatekeeper or caverno to enforce those type of policies right, but you would need some some way of saying any any storage usage in this namespace will has to abide or, as is forced into a storage class.

A

A

But that seems like yeah I mean even so I guess the question is, if someone's using delete and if reinforce again that would have to be enforced somehow, but let's say if they use the reclaimed policy of delete. That seems like it's safe enough right I mean there's no right or anyone else. Do you see any other risks with but running with multi-tenancy and uh any chance of misuse of storage.

E

I mean obviously yeah um I mean under the covers right. It's it's a volume mounted to your a node at some point, so there's there's risk there.

E

So it's it's really going to depend on the other policies you have in place to prevent you know privileged escalations, host path um or host mount pass and all that other stuff right. So um so so really it's!

E

The storage is kind of only as good as the sum of all the other parts of the security.

A

So so the guidance we're saying is use dynamic storage do not use um like you know, any any host mounts or even like yeah. If you have temp and things like that, uh you want to make sure that you're not mapping to empty.

E

So you want to host try to like limit that, though right I mean and that's where it gets hard, because most security auditing tools that people use to leverage a lot of this like twist lock, um does all of that right, yeah, um and so you have to kind of be mindful that you do have some some risk exposure, regardless from that. From that perspective, right.

A

Well, or even pod security standards um with restricted. All of this is prevented right. So.

G

The thing is for storage isolation uh we using sandbox and not using sandbox, has difference so.

A

E

Yeah and in realistically, if you're, using a dynamic storage, it's going to provision it and attach a pvc immediately and when the pvc goes away, the rekindling policy will kick off and it will delete it so the risk of anyone. Stealing the data between you know, pod restarts is minimized, but but if you're, not using dynamic storage, then or dynamic provisioning you're exposed um and that's where the storage class come to play. But it's still files on a disk right somewhere.

A

Yep and of course, nfs and other things have their own challenges right, because.

E

Yeah wow yeah yeah.

G

I actually, I would suggest you know, move the story. Networking isolation, part in the sound in the wrong time, part basically with the sandbox, because they are so different. We then without sandbox and and- and I don't think in this document we are strongly. Are we encouraging sandbox? Or we say it is okay, if you don't have sandbox.

A

Yeah, I don't think we've taken a strong stance to say that yeah sandboxes, you know recommended or anything like that. We're saying I think the reality is- and you know maybe others have kind of seen use cases also, but a lot of folks, especially sas providers, are running without sandbox containers and they are using namespace based multi-tenancy.

A

So I think, of course, in team base. There's other challenges, because now the user has access to kubernetes apis, but at least in the in the sas multi-tenancy, just using namespaces seems fairly common.

H

Yeah, hey jim, this is jeremy, so uh sandboxes should really be used if you're going to run untrusted code code that you haven't had an opportunity to scan or review.

H

I feel comfortable making that stance.

A

Okay, so you would say that for storage isolation, sandboxing is not necessarily.

H

Oh, I specifically for storage, isolation, yeah.

G

Yeah so at least you know, storage, network isolation and narrow isolation shouldn't put into the control plane isolation, so it is data planning. So that's my point.

A

Okay, yeah, I see what you mean by so so I think yeah, but let's, let's go back and check the so here we're saying name space for so this section.

A

Yes, so the one one view is: okay, is this really? Is our storage and networking data plane or control plane? I I think here what we're talking about is, if you want a namespace per tenant, what kind of kubernetes control plane or kubernetes api constructs? Can you use to you know for configuring, some storage, isolation right so yeah? I guess it gets a little bit fuzzy and I think.

G

Yeah so because, if you are talking about, like you know, dynamic storages right, some house amounts it's all data plan things, but if you just talk about the pvc object and pv object, okay, there are control pane things, so either we make it clear. We just talk about any control, plane objects in the api server. How do you deal with that and forget about exactly how that api is implemented by the whatever pv provider? And that's also fine? So maybe we have made it clear.

G

So if we are talking about control, plane, installation, networking, isolation, we are talking about network control, player, object or network. This kind of thing so make it very specific. All right.

D

Hey jim fine, this is dead yeah. I agree with you. Basically, if we limit this discussion in that particular section to pvc and storage classes and then reclaim policy of delete, uh then it will still fall into what jim you were saying about.

D

uh If you want namespace list for 10 and then what all the things we need to ensure from storage network, our back point of view, but if we start going down the path of like the data plane isolation, then that starts blurring the boundary between what we have in sandboxing and uh here so we might. We might want to just restrict this part to pvcs and storage classes.

A

I see yeah that could work so maybe renaming this section to something like pvcs and storage classes or you know, uh and then adding a.

G

Network policy, I think just three yeah.

A

Yeah I like that, because that makes it very clear that we're talking about these kubernetes api resources- and you know, then we can revisit network isolation and storage isolation in the data plane, isolation, section.

D

Yeah something along those lines.

F

Basically, removing the word isolation from from here right.

A

Yeah, so just maybe we're saying rename to.

A

Storage classes or.

A

I don't know what the kubernetes stocks name it like pvcs pvs, but we can, you know, follow the same whatever this section is named, um like configuring storage, or something like that. Okay and.

E

A

Thing for network isolation, what we're saying is renamed to network policy.

A

All right, yeah adrian, had the same comment on you know. I had added this additional security considerations and I had the same comment on even pod security standards right.

A

I think the trade-off is yes, of course, pods you could say belong to the data plane, but really we want to make sure if somebody's configuring namespace for tenant they implement pod security standards and image scanning and cis benchmarks.

D

Yeah, although jim I mean, wouldn't those be uh whether it's namespace based or not, I mean these are.

D

So, actually, if we are slightly talking about those four sections and transitioning off from that.

E

D

Point then, uh maybe that point you had uh written in the comments. I thought it was good where all those four and potentially three or four whatever will end up being, but we move them uh in advanced isolation or advanced concepts, something like a complete separate section where I felt that those are cutting across the board right, whether it's a multi customer or multi-team.

D

They are some of these advanced things that cluster admins need to be aware of when they think about multidimension.

D

And, like the last section, uh additional considerations I had put in that for sas providers, but I mean that is the sort of the bucket that I was thinking is the place where we can uh yeah have all these uh additional things which are not, let's say so. Probably, quotas are still namespace specific, but api priorities and fairness like there is that one mention of.

D

um If you have uh logging and monitoring as services which are cutting across the board, then how do you give priorities to the parts so that those get handled with a higher priority right? So those actually are not specific to name space pertaining that they can apply to. uh I mean multi-team multi-customer, even in uh probably virtual control plane, but also those parts which are for super cluster or api requests which belong to the super cluster. They need to be handled with higher priority right.

D

uh So from that angle I felt that qos or- and then dns was one section that I felt and ranjit you had mentioned in the comments that you were okay to club, that with network isolation, and I wanted to hear from others as well.

D

If that makes sense like when we talk about now, we are going to change the network isolation to just network policies, but if we were to still look at dns from that particular lens that it is still applied to part of the networking right like it probably doesn't uh have to be a separate subsection on its own.

D

uh What do you all think.

F

Yeah so before we jump into dns, uh I mean to your earlier point: yeah uh you know, so I would like to keep the resource quotas and the limit ranges, though the whole quota section under the name, space yeah.

D

F

D

With cody, yes, uh limit ranges, quotas, those make okay, because.

F

D

Are associated with namespace api priority quality of service, additional security and dns, and I think jim was also talking about the last four in that list. Okay, you see on the screen there right the last four yeah. Those four yeah quota should remain with namespace, okay,.

F

That makes sense all right.

D

Yeah yeah yeah yeah, the last four is whatever.

A

Okay, so we're suggesting we, we move these four into additional considerations and we remove sas providers and just say additional considerations.

D

That was my sorry to interrupt jim, because I just wanted to make one additional point regarding the additional, like that last section where, rather than calling it additional considerations, which I agree with you and ranjit that uh they should not, people should not think that those are options. They're.

F

Optional yeah right that makes.

D

Perfect sense, so naming the title of that section will still be very important. uh Advanced uh isolation is a good title. I'm fine with that.

D

If we all agree with that, but then we can also talk about that last point and we will separately I'd like to talk about that uh gym uh the operators and how they actually is a big concern when we talk with teams like what operators to use and all, but that can also just fold into that, so it will be a list of like five things and maybe something else that doesn't fit in the uh neatly fits into these other sections.

D

We can still you use that additional advanced isolation section to discuss and outline these things, because we want to be comprehensive without, uh like you know, missing anything which all of us have seen in the field, and it will be good for us to just capture it in some place. Sure, okay, so let me stop and uh yeah we can continue with the earlier flow. I didn't mean to interrupt you.

A

No no worries. I think this is a good segue into that uh topic itself, so yeah. If we, if we move these down, we rename this into, like you know, and I even advance seems you know that it could be considered that if somebody just wants basic isolation, they can skip that. But then.

D

Oh okay, stuff.

A

Like we're saying that you know um in the security settings, we want folks to consider, so we could say like common considerations or things like that right, which or we'll we'll find some proper name for it. The other option is we move some of that into key concepts right. We could move security into key concepts so that it gets prominence there uh even before talking about control, plane and data plane, isolation.

D

So, actually, that's a good point, jim and now I'm the main problem with that. I think with that will be it's just too much of details before you know, uh building up the whole story right now, the way it is flowing. It's just like use case is very high level. Then key concepts. Okay, everyone.

E

Understands them, then: okay,.

D

Now you are driving into details and then these are like really additional details, so uh that can just go.

A

D

We don't want people to. I would.

A

Yeah, I agree for the other sections like qos, etc. I feel security may need that emphasis. Oh.

D

Oh you you're saying just that particular stuff: correct: okay, yeah yeah! Probably you are right because those image scanning and those things are fairly.

A

D

Yeah, those are concepts yeah, I think you can. We can move that particular part in the key concept section.

A

Okay, all right so that I think from the organization perspective should help right. So just going back and recapping what we just said.

A

So we'll move security up and we'll just call it security under key concepts in control, plane, isolation, we'll call this network policy we'll call this pvcs pv something related to the control plane. The api objects, then, under data plane, isolation will move um storage, isolation.

A

You know if there's yeah and there we. I guess the question is what what else do we want to say about storage? Then? um So again, we could revisit sandboxing, but we have covered sandboxing and node isolation here, yeah.

D

I think ryan, uh it was you right. uh You were mentioning about this talking about.

A

Right right, right, host volumes and things.

D

A

Yeah, so we can call that out.

D

I think ryan, you should just take a look at uh the part which is there currently in sandboxing and then what additional things you think should go as part of our data plane, studio solution.

A

Yeah take a look okay and network. I don't know if there's anything else to be said about isolating networks from the data plane perspective, but dns, I guess, could yeah. Where would dns go uh under yeah.

D

So what do you all think like should dns uh be its own subsection, or can we fold it as part of network isolation? I mean whatever we want to call like the network: segregation, not segregation. We are not using that word, but.

F

So yeah we are going to change that to network policy. So in that case I don't know if dns is adding dns makes sense.

D

Yeah exactly now, that's what I was thinking, because we decided to rename that it uh is probably a different uh issue. Now, okay, it can stand on its own. I.

F

Think it can stand on yeah standard zone, okay,.

A

All right sounds good, okay, so I'll make those changes. You know at least the structural changes to the dock and send out a note. I think, all of that's good.

F

uh So there's there's just one thing: I I sort of purposely stayed away from is uh bringing in service mesh when writing the network isolation part, uh because that can open up a can of worms right. So I don't know just wanted to get the thoughts from the group here. Is that worth bringing in or just give that.

E

I mean, I don't think it change fundamentally changes anything because you still have network policies right and the service national has their own flavor of a network quality, but.

D

No, but actually I'd like to hear ranjit what sort of specific problems have you encountered in the sense that what ryan is saying yeah from the technology point of view, that is true, service measures will have their own network policies and what not, but when it comes to configuring configuring, a cluster for multi-tenancy, uh what we're like there is something that you were thinking. That is a problem right. That's why you uh so! I would like I'll be happy to actually hear what you have in mind. There.

F

Well, yeah, I know I've heard some customers have spoken to you know. um You know, especially the one. The ones that are building a sas, multi-tenant environment, bring bring in a service mesh to control, restrict traffic more at the infrastructure layer right and it's it's something that always comes up, but I wasn't sure if that that was the focus of this document or you know, do you even bring.

D

That actually, that you- because you mentioned that I'm reminded of something which I wanted to bring up to the group and also specifically ask jim- is see. Opm keyword know that they are good uh sort of building blocks or tools that can also help with some of these restrictions, and there is a passing reference to policy engines in the dock so separately from service meshes.

D

uh Where what should we should we, when the first question is, should we not actually bring those at least passing references to those tools in here uh and how do they help with the multi-tenancy configuration like if there is some guidance, because, ultimately cluster admins? If they know that okay, I can uh use opr keyword? No, that's a good thing for them. Right like it could be comprehensive.

A

All right yeah, so, let's, let's answer the service mesh question first and then we can discuss policy engines as well. um Some, I think you know again, if you're creating like an other considerations or you know an other sort of section, it's it's fair.

E

A

Service mesh there and say that can be used for network policies. um I think they I don't know, then whether we have to go into details like okay. You know some service meshes operate at layer, seven versus layer, four.

F

A

That creates other.

F

You know complications.

A

To explain um and and if we're putting this in the kubernetes docks, I mean the more we stay close to the kubernetes api, the better right, because uh you know there could be perhaps other higher level docs, which explain you know um other things right so so I think it yeah and obviously even with service mesh. There's things like network service mesh and then celium operates with ebpf. So there's so many different implementations.

A

D

Won't be able to.

A

Do that justice.

A

But you know we could briefly mention it and say: service meshes can be used and different service meshes may operate at different layers of the network. The osi stack right and leave it. I.

F

Like that idea, so briefly.

A

Mention that in.

F

The other considerations.

D

Another considerations section.

F

A

Right and if you do that, then we could give the same treatment to policy engines right to say: hey policy engines can be used to restrict and manage configurations for namespace or per tenant, but different engines will have different ways of doing things so mileage.

E

A

E

I guess the only suggestion I'd say is: maybe we could even make it more generic and saying um you know admission and mutating web hooks.

G

E

Used to enforce- and there are various tools that you know make it easy right, because I mean we, I know quite a few people that just write their own admission web hooks right and skip it all together. So.

D

Yeah yeah, I mean that's fair, uh so in fact, uh looks like um uh adrian is not here today right because I had asked him to also uh write a blurb on hnc, which yeah, which can also go in that section. So basically that's our section where we have all these, I mean for lack of better word, I'm just calling those as advanced concepts, but they are in some way. They are advanced concept right. Not everyone will need those uh at the same time, they are important. So.

A

Right so so for tools, though, like projects and tools, we can have just a separate section and we can move this towards the end and we'll have to check. I know we've so some other documents. I have not seen this in the core kubernetes docs, but other docs do include like a section where you know any any project can kind of add themselves and there's a disclaimer about third party projects.

A

Things like that, so we can follow those guidelines and just have a section on projects with links to um you know cappy nested, hnc and then any third-party tools like we cluster capsule cloud arc. You know that want to be listed so, but we'll have to check on what the allowed guidelines are and how to do this. um You know where things can be listed, but it's not there's no sort of you know.

A

um The goal here is to educate and not just sort of advertise projects.

D

A

You know prefer one project over the other right, so.

D

I think I've seen that network uh see cnn.

A

Right so we can follow the same guidelines and have a separate section for that. Yeah.

D

Okay, uh just um I know we are uh sort of towards the end, but uh jim. I wanted to ask our discuss with the group and because we had the specific question the last point there.

A

D

Yeah, so let me uh share with the group what that point is about and where it's uh so see when all of us understand what operators are and controllers are and what we have when we talk with customer seen is. uh Invariably there is some operator, one or two in fact, uh sometimes more than two in their clusters and uh they are trying to give a sas. uh Their operator is just part of the story.

D

So, for example, let's say it's a wordpress as a service and the database might be mysql might be coming from a mysql operator.

D

That's a very typical example, and there are such containerized applications that we are seeing more and more every day. Now, uh where the storage is being managed by the operator and the application has its own, you know it may or may not have an operator. So from a cluster admins point of view, the trouble that they seem to face is okay. I have all these bicycle operators out there.

D

Some are open source, some are uh proprietary, but uh how do I know which one is best from my needs of creating a sas and from that when we started working with uh such teams, we realized that there are some basic things that an operator needs to support and those are the five things that I have uh listed out there uh and I won't go into the details of those and, as I have mentioned in the comment uh jim, I will shorten that, but I wanted to wait for this discussion to happen before I take a stab at that.

D

But the main point was: if we uh bring that out as well, then it just helps the cluster admin that when they are choosing operators- and I agree with your point where we ultimately the operators are for providing some service, and the question comes, I mean sas model is where this becomes more relevant.

D

What how should an operator be now? We are not going to guide them on operator development? That's not. This document is meant for, but as a cluster I mean when they have to choose existing operators uh from let's say a marketplace or even some operative repository.

D

They can use at this guidance to check which operator will satisfy their needs. So that's the background and then the team you all can read up on that, but that was what my thoughts were, that if we are having a separate section for like advanced concepts, advanced isolation, then we can put this discussion as part of it.

A

Right now, thank you that helps explain and frame. You know the. uh I guess what needs to or what's described here my I guess on my first reading, and maybe this was based on the prior versions which you're waiting to update, but this seemed very much focused on operator authors. So if we can flip that and say.

D

A

What should an operator admin care about.

D

A

It and how do they evaluate if that operator is fit for multi-tenancy, then yeah that makes yeah in fact yeah. It's not yeah. I see.

D

That that's the sentence which can mislead right written and I have written that, but it's in the like a while back so yeah, but uh I will shorten this if you are okay and I mean if the team feels that this is part of uh that kind of the document that we are creating, this will add value. Then definitely I will work on it.

A

So my related question, though, was, is this guidance for operators specifically, or should we generalize it for any shared service right, so is there or is there something you know, sort of unique or specific that we and, of course we can in in that section we can talk about. You know operators also, but.

F

A

D

A

Installed or managed by an operator or it's managed manually or through terraform or whatever.

E

A

E

I mean, I think it does, because crds are cluster-wide and a shared service. If it's not using custom crds, then it's kind of by default has to be namespace in some regard right and so then you have namespace isolation. If you're following everything else, but from an operator perspective, it largely has to be cluster-wide. So I guess you could just say: cluster scoped services.

A

E

That was my assumption.

A

Of what a shared service would be, but maybe that's not a valid term.

D

Let me uh ask you jim, I mean I roughly understand what you are implying, but I had a very specific guidance as part of this session. So if we uh to answer your question, if we see ultimately, we want to provide a concrete guidance and when it comes to an operator, uh is the vehicle or the mechanism through which a service uh gets created or is enabled in the cluster right.

D

You have a operator and from them based on that you can create wordpress services so, but, but when it comes to concrete guidance, it's not about the service, the guidance that we will be able to give, but it will be, the guidance- will need to be for the specific implementation.

D

So if you are using a mysql operator as part of lets, say a wordpress service, then that mysql operator needs to do one two three or need to have these things, one two, three four whatever so I I think when I rewrite this, I will uh give that context to the discussion that from where the operators are, what roles and operators play as part of a shared service and then the specific guidance that we recommend that kind of structure that we are liking right.

D

So I will rewrite this in that flavor, where uh the guidance is very specific, like if you are using a mysql operator, for example, make sure that that controller or the crd spec has certain things defined so that you are, you know like, for example, guaranteed scheduling. You need to make sure that there are uh the resource request and limits defined. Otherwise, kubernetes will bump out your uh board and that can just uh crash your wordpress service. I mean that particular instance. Obviously,.

A

Yes, so, but even in that example, I guess the question I have is you know so, of course, one best practice is that you have a operator which is automating the configuration and life cycle management of that service.

A

But what you just said about configuring, like you know whether it's resource limits or other best practices, even if in some case somebody is manually configuring a mysql instance, wouldn't they want to follow the same guidelines.

D

ah You're saying from that again I mean without even an operator.

A

D

A

These are just I'm running, I'm providing this as a.

D

So that's true, that's actually a very valid point and valid question, uh but because we have already sort of talked about those things as part of, let's say the name, space isolation uh where, if you have as a cluster admin, if you have control over uh the traffic, let's say through policy engine or something to apply those policies, no matter what, then, yes and that point has been covered, but I'm specifically now referring to the situation where a control plane is getting extended right.

D

All these controls, an operator, will add a new api and that api is not going to expose ports or anything. It will be a very high level, uh abstraction and behind the scene the control is going to create whatever resources that it wants, or as part of handling that crud operation on that new api. So now the question is: what should we look at? I mean as a cluster admin.

D

uh What should I look at when I see a two competing mysql operators which I want to install in my cluster, and the guidance will be specifically around that point where we will tell them that? Okay, look at some of these things that your controller is doing.

D

It might require them to you know: analyze the operator's code, just look at eyeball it if it's possible just look at spec and all of that, so this section will definitely shrink, but um that's where I felt that we might be able to handhold some of the decisions for them. You know.

E

Yeah- and I think it gets to jim's point- I think, all the same recommendations we would make for an operator probably apply to any shared service on the cluster or any any I'd say any service that has a cluster role right. um It's going to be roughly a lot of the same guidance.

A

Is it everything.

E

We put in control.

A

Plane, isolation or for workloads.

D

Not I don't think we have everything in there I mean.

D

See uh and ryan. I understand that my still the question is: um they are not the same thing right like when I add a new. uh Yes broadly, it is all controlled, plane, isolation, but uh now we are talking about control planes that are written by third party, whereas control planes and.

E

Yeah but I'm saying any any shared service right like that's, not necessarily an operator right, um because I guess I mean technically I would I don't know if I would consider kiverno an operator, but it's essentially implementing um you, know uh crds that have reconcilers.

G

That are cluster wide right.

E

So, like I mean I guess, it's a shared service versus.

D

Yeah, but actually that you make an absolutely great point, then, when they were working on the operator white paper that like what to call an operator, was a hotly debated.

E

D

Point so uh yes, these days and as we have heard, the team and everyone talk about everything will be a custom resource. Eventually, everything is moving in that direction, uh but and even though everyone has their own custom resources right.

D

What I'm specifically concerned about- and this is again coming from the discussions in the field- is there are certain uh certain operators which are more uh just for the lack of better term. We can call them a stateful service. Let's say like cassandra data.

E

Stacks of their operator- and we have worked with so many people.

D

uh Elastic and, uh of course, my sequence we have studied, uh they have very specific route right. They they uh define a custom resource whose own intention is to provide a high level abstraction to uh create instances of that underlying service, and it it's not a service which is going to be used across the board, like, let's say or across the cluster like given, but it will be an instance of mysql which will be used as part of a particular application stack. It could be whatever the application you are using.

D

So I'm talking about those kind of operators which system.

E

D

Have to choose no matter what, especially in my sas kind of setup, that becomes a key sort of time swing right. I guess.

E

I'm still not sure what guidance would be different from that sort of situation than a kiverno type situation, where it's it's just a a c or d with the cluster role. That hat can do a lot of things on the thing. It's a shared service right, it's shared across the cluster.

D

So I guess maybe some verbiage.

E

Around a shared service or an operator per name space right like that, that could be.

D

Let me just answer that particular point: ryan. The difference between keyword, no and a bicycle operator is killer. Law is not going to create new ports right.

E

It can it has. It has the ad option right. It has features to potentially do that right. So there's there's a lot of security concerns around kiberno and that, from that same regard, right.

D

I know what is like uh no or opa opa's main role is not that right as against that the main role of mi sql operator is to actually create behind the scene. The entire, uh like, whatever is needed to run a mysql uh as a instance in the kubernetes. So uh the the difference I feel uh between something like even a service mesh has their own crds right.

D

So if we take a broad brush and not call this out, I I feel that we will miss out on providing a good guidance there, because they are not the same things like just saying that yeah all the cluster wide services need to be uh need to have all these things anyways. So you know we we won't be able to finally uh tell the our readers that what specifically need to be looked at when they are choosing something like uh is operator for a shared storage service.

D

I mean not shared, but a storage uh kind of service.

G

Yeah, I think that the recommendation here is actually highlighted the thing that we have discussed in previous documents. So is that, okay to say, if we write an operator in a multi-tenancy scenario, how about we trying to automate the thing that we we have discussed? For example, we we we can that guaranteed scheduling. Is it possible for this operator to have people like saying travel like a web hook, so to automatically patching all the requests for the parts that are that other that customer?

G

That requests basically make everything that you list as a automation for the operator. So, for example, the operator can automatically set up the network policy for the tenants. I would say that kind of thing makes a little bit more. I would say just like a recommendation, not enforce everything that you have mentioned, because um the thing that I mentioned is kind of specific to a particular use case. That.

D

Yeah and that's what I had originally called out, that these are the recommendations uh coming from the field for the sas uh kind of use case, not the multi team we haven't seen so much uh probably they might arise in that scenario as well, uh so see, I'm not suggesting. First of all, I'll be rewriting this. The main point is, uh if I to answer your question uh yeah like some of these things, can be automated uh yes and to answer ryan's question. Those can apply across the board for every service out there.

D

The the main point is today in the field what's happening. Is uh there are operators which have uh which are doing certain things right.

E

D

For cluster admins, uh it's we are not recommending or telling them uh how to automate things. We are helping them choose one operator or another uh based on uh certain criteria that they can easily check uh to just maintain their multi-tenancy posture. Of course they can. All of these things can be done through opa. Probably keyword, no can do it. We have an admission controller, that's part of two plus, which uh does many of these things uh and yeah.

D

We don't actually uh mention those how to automate them here, because it is mostly about uh look. You are these third-party controllers that you are adding to the cluster and they need to follow certain things. That's the whole point.

G

Okay yeah, my work is for existing operators. So if we write this way, are we saying that all the existing operators they are not multi-tenancy ready? uh Maybe it is they are, but I just don't want to deliver that message. No.

D

No, I'm not saying no, no, no! So fine, that's a very good point. I'm not declaring any operator here to be multi-tenancy radio or not. These are just the criteria that cluster admins can keep in their back pocket as part of their evaluation.

D

uh Then they have to choose an operator for something I don't know and for this discussion this particular discussion. I am not going to call out keyword, no as a operator from that perspective, because it's more an admission controller which is doing the policy engine, uh but something like uh yeah, my secret, a lot so many other services, right, mango, etc.

D

So yeah, none of the operators are multi-tenancy ready or not ready. We cannot as a team, we cannot make that decision. I'm just saying that we can help cluster admins uh with some great criteria or guidance on what to look for when they have to choose certain operators as part of their control, plane and add new things from you know third party repositories into their cluster. That's all the main point is.

A

So why don't we do this? I know we're almost at time, so uh you know dev data. If you want to rewrite this section.

E

A

Cluster admin perspective we can give it another read. I do feel that if we're specifically talking about operators, we may need to add something just about other types of shared services which are not you know, name spaced or not, creating an instance per tenant.

A

So that way, we then it seem, may flow a little bit better to say that as a cluster admin you're going to be configuring, some required services like dns, there's optional, shared services, perhaps like oppa, gatekeeper or caberno, there's other operators. You might configure for per tenant services right, and we can cover all three and provide some guidance, some high-level guidance on how to go about selecting or evaluating these services and what they want to put into their cluster.

D

Yeah I mean I'll take a step at rewriting this from the cluster admins perspective. I'm not still 100 percent convinced that we need to club given no opa and third party services along with these operators.

A

I'm not suggesting we do I'm saying we we, if we're talking about operators, it seems like we should also mention something about other shared services which are non-operated not driven through operators right because there are several examples like valero right. So if I want to install valero, is it multi-tenant? Is it not? Does there might.

D

Not be an operator for it right.

A

D

No that's a great question and um yeah I mean uh that's the reason we are having a separate section where we can definitely uh again probably uh shared services uh can uh have its own subsection and try to the reason I'm resisting. That particular point is because what's going to happen, you know is um the guidance that we have learned. uh That is more specific to the operators that we have seen and work with in the field and something like uh the question that you raised are valid questions like valero.

D

Is it multi-tenancy like ready or not? uh Honestly, I.

E

Won't be able to tell you right now, because I haven't studied that or even.

D

How the guidance that I am going to give later on how that relates to keyword? No, I won't be able to be. uh You know confident about that. So, rather than diluting the whole message, maybe what we can do is we can have anyway, we are going to have a separate section of advanced isolation right, so we can have shared services like qrno valero and then maybe you can take a stab at writing about those, and this can be very specific section that let me first take a stab at that.

D

Writing it and I'll do that later today, and then we can go from there.

A

All right yeah, let's let's kind of review once you have that covered- and I am also agreeing with you that the shared services would be a separate section right, not combined with the operator section yeah but see.

D

I like that, you call.

A

D

Correct we want to be comprehensive, no matter what so I agree with you.

A

All right, I.

D

A

A little bit over time, but great discussion.

C

A

Again, I feel we're pretty close to getting this pr ready. um So it's exciting to see it come along.

D

uh Jim just one last point: uh the zoom link that I used from our uh main document. Is that not is this not that zoom link? I think, because nobody was there on that particular.

B

This is the link. Could you dm me on slack wherever you found that other zoom link and I'll try and update it.

D

Yeah, I will. It was in our main dock.

A

All right thanks.