Kubernetes WG Multi Tenancy, 15 Jan 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG Multitenancy 20190115

Description

Working Group Multitenancy Meeting Notes/Agenda
When: Biweekly on Tuesdays at 11am Pacific Time (opposite weeks of sig-auth on Wednesdays)

Working Group Multitenancy Info: https://github.com/kubernetes/community/blob/master/wg-multitenancy/README.md

Charter: https://docs.google.com/document/d/1SkVdOPR4jozYDT8ro51hU3yrf1sHS8Gez73xM3PCsVo/edit

Sanjeev Rampal goes over working group doc generated from in person meeting at Kubecon

Agenda:
Multitenancy Working Group Proposal:
- Motivation
- Models/profiles of multitenancy overview
- Commonly asked questions and proposed answers
- Detailed proposal for each model of multitenancy with sample configs and proposed work items
- Action Items and Next Steps
- Timelines

Presentation: https://docs.google.com/presentation/d/1dsAsVm8kCA9Dx9_gMEYeJL7pduAbnfnxT9lhbyCvHDg/edit#slide=id.p1

A

um So hey this is Asha. Today we're going to go over a presentation that is putting together a pencil for how we can all work together for the next couple months. It seeks to define some vocabulary that we found a lot of people coming into. The working group are a little confused by so soft wants a tendency, hard multi-tenancy. What are we looking at doing and then we have some profiles that were going over after we've defined those there's a lot of content and I want to make sure we get through it.

A

It is all up for negotiation. This is a rough draft. It's not! You know. This is in no way a final product and there's some straw, man and architectures that are also very open for conversation.

A

So what I'd like to propose, we do is, you know, stop definitely for comment but try to get through the whole proposal in this hour and then we're going to open up a presentation for everybody to make detailed comments on and then I'll reach out to people who have you know more detailed comments, so we can make sure that we're kind of getting the whole feedback and kind of be able to incorporate kind of more depth so feedback over time and then we'll put together sort of the next draft of this and then share with everybody.

A

Are there any questions before we get started any anything?

A

Okay, great, um so I'm going to hand it over to Sanji who's done a ton of work on this and he's going to take us through it.

B

Okay, Thank You Tasha. Can you all hear me.

B

Okay, yes, with okay I'm, going to share some slides I put together, which has them some of the content. That of describing.

B

B

Okay, so can you see this light? Junction? Yes, yes, okay! So a lot of this is content that many of us have discussed in various meetings and forums and Pat said so we're just trying to consolidate and create a framework going forward. Okay.

B

So what you're going to be covering is just to do a quick review of motivation of why we're doing this. Many people already know that, but since some of you may be coming from a slightly different background, we'll spend few minutes on that, and then we talk about.

B

You know from molds slash profiles that we, we believe, might make sense to pursue and then we'll go into a little bit more detail on to what kind of work items would be needed to fully flesh out and define those moves, and then we'll cover some action items and next steps and timelines.

B

Okay. So let me start so very briefly: we will review why we're even doing any of this right and pardon me if some of you this is redundant for some of you. So this will help of answer questions when we get it from outside of the working group right. Why are we even doing multi-tenancy? So the motivation here is that clusters will need to be shared by multiple application teams.

B

For example, it could be multiple development teams within an enterprise, and the benefits of shared clusters include reducing the number of clusters that are all around in an enterprise, so reducing cost of small and improving utilization of a smaller number of clusters, because you do not have to dedicate a cluster for every application team and correspondingly not have to dedicate nodes servers, separate control, stacks up at monitoring, stack and so on, reduce operations from the admin team because they do not have to maintain an lifecycle.

B

Hundreds of stacks of kubernetes plus committee s, plus e FK, + HT, o + x/y/z, everything that goes into building a production. You will have a fewer number of stacks to manage, because each of those clusters will be supporting multiple teams again. What we're doing here is trying to formally define an upstream kubernetes, something that is actually already in common use.

B

Projects like openshift and rancher. Already have the concepts of multiple projects which are essentially like tenants with an open ship convention or who can do a kind of informal DIY being space methylation.

B

So the goal here is that since there are all these similar things going on, but not quite standardized, why don't we take some similar elements motivated by these similar, similar work and formally define it in upstream kubernetes, so that there is a essentially a standard way of doing multi-tenancy, which people are already doing through some proprietary types or ad hoc kinds of solutions such as open, shaped, rancher or DIY solutions, and we believe this is going to be more significant.

B

As the industry moves to containers running directly on biomech, because there it will be even less likely that you would want to give each app team their own dedicated, bare metal, cluster right, I mean think of a minimum of three all in one bare metal servers or possibly, six bare metal servers for kubernetes cluster. Do you want to give six bare metal servers to have the team? You probably don't so as containers as this technology is becoming more towards running containers on bare metal, as opposed to running containers on a hypervisor.

B

The need for multi-tenancy will be even greater right when you have VM the viens provide a kind of 410 empty layers, so in some sense that's why people are able to use kubernetes today on top every virtualization platform like VMware or OpenStack, because that provides the currency, but we need to design for the future where containers on them. It will play a big fool.

B

Okay, so we use the term what a tendency to refer to any kind of kubernetes cluster sharing solution, and these solutions could have various levels of soft or hard boundaries between teams that we will call tenants.

B

We can readjust the terminology later right now be using this terminology just for the purpose of our discussion and in parallel, the current approach of using non shaped cluster will continue so I should today there are a lot of products which can create lots of coverage clusters and give each Kunitz each team their own cover and expresses, and that that approach will continue.

B

So we need to so what we are saying here, that we are trying to do something similar to what openshift and ranchoddas, but do it in a native extreme manner for the native upstream coconut is so any questions on motivation, and we all in agreement on the value of doing this.

B

Okay, so hopefully we can answer these questions, as you know, folks from other working groups or ask us why we're doing this. So here's a potential reference model right and we have what I'm calling modes you can think of it as profiles again terminology we can adjust as we go, so a kubernetes cluster will run in 1 out of n possible multi-tenancy moves, ok weather for now. These modes are for the purpose of documentation and testing in future, really in future phases. These will be. The mode itself could being explicitly configured attribute of the trusted.

B

But this attribute this cluster is running in mode 1 or mode 2, but we will do this gradually so that we want to get the cigars and other groups buy into our basic assumptions before we make too many assumptions about. New Covenant is object right, so we want to gradually get the buy-in from the community with simple solutions, and then your objects see straight to the goals and talk about team profiles, which are new series that tends to get resistance from the community or at least from the different things.

B

So multi-tenancy mode 0 is effectively current to mulleted right. There is effectively no infrastructure multi-tenancy. The entire cluster is used by a single team, but we may, as part of this work, add some security recommendations such as it be for security profile. Even for this case right for the non multi-tenant case.

B

Ok I know: I've got a lot of text on these slides, so I'm not going to walk through every sentence, but you can we'll have the slides later you'll be able to take it'll, take a look later or if you have questions on any particular point feel free to ask so so more zeros. Is that right, whatever we need to do to have baseline security without having actual but a tendency, but multi-tenancy starts with more s1 and you're, using it s1 to stand for what some people have called soft multi-tenancy right.

B

So the in the past was always being these discussions about. Well, are we doing soft multi-tenancy hard market and latinos? Let's put those down as effective as besides modes right so mode s1 is enterprise. Soft multi-tenancy using current kubernetes feature set right. We are talking about what can we do create the doubt creating any new theories without creating any new controllers, we want to have a solution that will support any tenants per cluster and it will rely on kubernetes namespace. One namespace for tenant, probably not defined any new controllers again.

B

If the team feels strongly, we could change that. But this is the current sort of draft thinking and applies to both on-premise kubernetes clusters, as well as kubernetes customs, hosted by public cloud or hosted kubernetes such as elastic community service from amazon and so on, but the primary reference model, which will be easier to think of, is an on-premises kubernetes cluster right. So we want to think of that case first, but eventually the solution needs to apply to hosted kubernetes cluster that one.

B

So that's more, that's one more tensile is stock. Yes, thank.

C

You yeah so for more the one animal s1.

C

Is there any specific difference, because if we don't run any additional controllers or we don't need any CRTs in the cluster, what's the difference between all one and more es? One.

B

Yeah, more one and more as one of the same things, but really more s, one has a curated reference configuration there. This group will document and publish as a proof of concept for getting the sanction from the fig arch for having the new controllers. So more zeros is no matter. I didn't see, okay, so we the difference between mostly or modaress. One is a curated reference configuration that we will curate and document as part of this teams, output, okay, and that might include some sample automated tents.

B

Now let me explain the point here we want to if we, whatever working for it from a few different people, is that if we directly go to let's just a couch and say you know we wanted to. Finally you can tourism and I. Think you see you've seen this in response to the camps that that you had introduced right.

B

It tends to take a long time to get approved and there's there's some amount of resistance to it. So what we are proposing here- and this is up for debate of course right so feel, free to agree and disagree.

B

So what we're proposing here is that, to make things easier to start with, we simply have a documented reference model without new controllers, but we clearly document and provide reference configuration for everything from pod security policies, network security policy, our back and so on, for somebody to develop to add this to their existing cluster, using that we give a demo to the cigar CH team and that then, hopefully, will give us a better blessing from them that okay, this makes sense, now put a controller on top of this right, but it can that work can continue in parallel to the work 4s2 right, we can continue.

B

We can start off doing the controller work in parallel if we feel that is appropriate, but we don't have to necessarily couple it because functionally these are two different pieces of work. Agreeing on the right duration model is a different task than implementing the controller for the curated curation model is.

D

It a requirement in your opinion, for the controller and the custom resource to exist in the upstream kubernetes project. Or would you see this being more of a community effort.

B

So that will have to better all these are great. Questions will be nothing played by ear based on the response we get from folks like the gaucho. Eventually we may walk in. It would be great if we can actually put the controller in upstream trunk and make it an official trunk feature.

B

But what we have heard from different folks is that when you try to do that, the various things tend to be very, very circumspect because they really need a lot of justifications. So what I'm saying here that let's use model mode s12 based confidence, essentially a proof of concept to agree on the parameters by which multi-tenancy is defined. Somebody can use it today and build their own DIY, but they can refer to our standard recommendation.

B

So if you can think of more s, one more like a best practices document, whereas more s two, we do that we develop a controller, but we feel that the controller has more chance of getting into extreme chunk. If we have already got buy-in with our work around s1 now, that's the thinking so far feel free to you know, I mean I.

D

Think modes I think I agree that material of s1 don't seem to have like enough significant differences that I can understand for them to be discrete but I. Think having s1, s2 and h1 makes sense insofar as h1 has a lot of features that are like hitherto unknown. You know we don't have the features required to build h1 so that by part of digging into that will have to be discovering like and digging into the deficiencies there, but for s2.

D

If you did want to build a custom resource definition and a controller I think one of the best things about CRD API is that it's really easy to build them as external projects and then, if and when it makes sense, integrate them or leave them as additional add-ons, depending on how will make sense once they're like completely nice.

B

Exactly exactly that's what thinking and if, for any reason, the sig net safety guard says this. This work, we don't think should have its own controllers. Then s1 is still useful by itself like we are not getting. The 2 s2 is not just new controllers, but s2 is also some new. We expect that you know there's new work happening with kubernetes runtimes, with things like cata containers and so on with s2.

B

We would potentially have explicit theories for the concept of a tenant exception right, so there would be concept of tenant as a new object, as well as tenancy profile as a new object. So there would be two categories of new objects. One is the tenants themselves and, secondly, is the tenancy profile right to get buying from the guards for all of that might take time, so we developed it yeah. So.

E

I just want to jump in Tim. Hawkins did a lightning talk, a kook on about how they actually want to get turn all of the upstream. All of the current trunk stop eventually into CR DS and so they're. Actually wanting to get less controllers in the trunk then, rather than add, more I, think it's their strategic goal, but he didn't really say what time frames. But anybody else see that one yeah.

F

He Amenia it seems clear that they wanted everything to be in that. Even you know core pieces.

G

Right, yeah hi. Another example- is the dynamic volume resizing stuff that you know. There was a lot of discussion in the sig arc, where you know it was initially proposed as something that is built-in, and then it moved into a separate controller and see release. So the pattern is already set, which is basically do your CRTs. Do your laterz and whatever else you need and then you know, make sure that things work and then it will be easier to get things into the main trunk.

G

But the idea here, I guess, is that we should make sure that you know we can work on this on our own quickly and only if we need some basic primitives that we need an s2, then we can have a smaller cap which might have a easier.

G

You know, time tool and say in signal or something like that. Does that make sense.

B

Yeah the way I was looking at it is that we would work on both s1 and s2. As we go along, we might move things we might actually merge, s1 and s2, because this is really the same thing or we might move, they might add the ability of controllers into s1, but still keep s2 for, let's say future cable kubernetes capability, which still needs off-mic a tendency right. So there is still more. Let's say that this is KP on namespace population.

B

You know that that's an example of self-service. If these are logically separate things, so they could come in later, so at the moment, I'm not assuming a specific boundary between what falls into s1 and s2. This is an initial thought, but we can move things around between and s1 and s2 as we see fit and that may include. Okay, we do need a controller for more x1 is 1 and, or we may say, these are really the same thing and we just the way of flip.

B

This is simply to get some working, POC first using s1, and but we yeah so the boundary between s1 and s2 is kind of fluid. At this time.

H

So question on regarding 0 s1 like today, without implementing any of that, can we assume that, like estos, the superset of s 1 and s, 1 is a superset of 0? Will we be building upon things or do you think they would be entirely this giant set of features yeah.

B

You can think of it that way you can think of s 2 as a superset of s 1. So really, fundamentally, it does come down to two categories: soft multi-tenancy and hard multi-tenancy.

B

Only the reason, I split up swapping to Arizona net two in x1 is soft, not a tendency using mostly today's constructs and s2 in soft method, anything using future constants, but the broad categories are still soft and hard. That's really it! We don't want to create five different categories, search.

H

And the work in making all that test work when these configurations are placed, why do you bucket them in yes, one I guess, but just one.

B

S1 is more towards the profile definition of kubernetes objects like hot security policies and so on sure s2 s2 has more of that right. Plus it has the controller, mechanics enforcing those policies. So but, like I, said, the control mechanics could end up in as well as well. So really the baseline way to look at this s1 is soft multi-tenancy using currently available functionality. Maybe we stop at kubernetes 113 right, basically using functionality up to an inclusive of 130 in Kouvola days. That's what is funded and s2 is all additional capabilities.

B

/ new controllers, possibly using all upcoming capabilities in kubernetes, 114 and beyond. So this basically puts a timeline on s1 is essentially really the next three months worth of functional.

H

Content but I think my question is: while we tell you an s-1, would we also make sure that tests that need to be parsing are passing with all that in place if those changes are applied as pxpr? Any of that, or do you think that would be morning is to thing yeah.

B

So, okay, so is there things that.

G

Didn't work out, you know that was not me.

I

B

I heard those charges, yeah John, so did James and I had a chat. Today. Jim was very much involved in the conformance testing and so on. So his suggestion was that for something like Morris one, we can create a third-party tape suite. It doesn't have to be part of the official kubernetes conformance tests.

B

Initially, we use that third-party test, which has an additional optionally available test, suite it's available as a best practices document as well as we use that to demo this when we demo this to s1 sorry cigar to get their blessing okay and then, when we do s2 is we do a formal test suite? So we will have test automation, both in s1 and s2, but an.

F

B

It will be more like a third-party tape suite just as validating.

F

For concepts that we've.

B

Got some tenancy separation and then x2 is where we have a formality right.

G

I can add a little bit more color, so in s1 the idea is that it is targeted towards existing users. They can just pick it up pick up what we publish and without adding more binaries more operators, see our DS.

G

They should be able to apply to their existing environment, and then the conformance tests should work as well as these additional tests that we write should also work, and the idea is to have a test suite which will publish to the test grid, runs against master and make sure that you know all the things that we say in our document is not broken. Essentially.

G

B

Yeah onion, as sasha said, this is just a drosphila thinking right. This is a straw man to help us all think about these things and we can adjust it as we go.

B

Okay and then finally mode h1 is where many of the folks have said. We need to be thinking of sort of the ultimate multi-tenancy solution that could include things like totally independent control.

B

Planes pertinent, you know true isolation across all dimensions, whether it's control pain management, monitoring, lodging all of that and and truly supporting untrusted current tenants within the same cluster, essentially like a Naiad level multi-tenancy that will require quite a bit more work and, potentially you know new machinery and code kubernetes council constructs like maybe a channel, so yes that also we can start in parallel, but we don't need to get the other one. So they're allowing this kind of categorization essentially allows us to work on s1, s2 and h1 in parallel.

B

They're closing gates would be different. S1 is something you know we're thinking about next few months. Right s2 is still you know within within within the year with in 2019. S1 could be three to six months. S2 could be you know, six to twelve months and then h1 could be potentially post twelve months.

B

So you could start on all of these infinite yeah go ahead. Question! Yes,.

F

So I guess with h1 I guess one of my questions was I've been looking a lot into like the Federation kind of capabilities being added, as h1 makes sense to maybe more so kind of go down that route of almost like a federation type feature where there's a master kubernetes cluster that you know the admins are using and they're basically giving some clusters for, for others to use or I guess. Have you thought of that or what's the how's that for yeah, so you know I had.

B

Not so physician and multi-cloud is an interesting point. You, yes, obviously a full full full! You know. Comprehensive solution would need to be part of each one, but because even for s1 and s2, we are saying it applies to not just on-premises kubernetes clusters, but also to kubernetes clusters hosted by public cloud providers like eks and jke, and all that.

B

So there is some amount of multi-cloud in s1 s2 as well, because we are not saying that in the long term, if somebody has h1, they will not need a hot market, and you see at all, there could be in trying to files use cases where they never need the complexity of hard multi-tenancy right, because this is all within an interface. So so right.

I

B

F

I guess my question is I: guess would what h1 be almost more I guess better suited towards like a federation type scenario? And then, if you need you know soft multi-tenancy within those Federation's, you could do that. But if I mean I, guess I guess I'm wondering like h1. If we, if we really have to basically isolate the entire control planning data playings, does it not make sense just to do basically hierarchical, kubernetes clusters with with different masters that are controlled by others? I, don't know if that's affect I guess is way out of bounds.

F

B

So we are very early in h1 thinking. I personally have been thinking mostly about s1 and s2. So yes, what you say could be valid, but we really need to think it through. This is a straw man, let's all think about it, and contribute your thoughts. So, yes, you could be right, but I have yeah.

F

B

F

B

F

If I was I, guess if that's been brought up or fleas kind of discuss, data wrong.

B

Yeah, so let's keep moving all that it starts in all good terms. I just want to give an aside just to illustrate a side example from OpenStack, not that I want to model this in OpenStack or anything like that, but just to give you a example of another platform that has multi-tenancy right in OpenStack. Some of you probably know there is a single API server, so a PS server is not for tenant, even in OpenStack, but it has strong internal data plane and management penetration across change. So there is a for example.

B

This is showing OpenStack lands, which is one of the OpenStack services. It's a single API server for multiple users of multiple tenants coming in through that API. But then internally, it's there's isolation, but the controller API server is not partition, so just something to keep in mind that you can build a pretty strong, hard multi-tenancy boundary, even without splitting up the API server. But in that case they typically put things like rate limits pertinent. So there is all of the the separation between all these categories is not black and white.

B

It it's slightly great and we'll have to take a stab at it and I'd rate it as we go. Ok, so that's just a side example of the stack, but we don't need to spend time.

B

Sometimes the question does soft multi-tenancy need to be standardized at all, and the working assumption is yes because you know people are doing this already. Today. People are doing this with oppressive projects and rancher projects. So if we do not standardize this in native kubernetes upstream, we don't have the analog, there's always lose discussion. Well, you can do namespace base isolation, but what exactly? Does that mean? That's what we are trying to standardize, ok and and same thing.

B

We will validate that using fair testing and conformance testing, eventually, ok and then how do we work with overlapping work in other working groups? Well, we have a well-defined reference architecture which we collectively build, and then we create dependencies on other working groups and we track those dependencies and we expect to be needing to deal with lots of other groups like the ones listed here. Ok, so we create our own reference architecture. We do proof of concepts with it.

B

We do controllers, as we see appropriate, but in many cases we'll have to just track dependencies on other teams. It could be, for example, defining new features in the way Network policies are done or in the Polish working group, okay, call in the API machinery.

B

So that's how we're going to do this, so so in order to for us to do that, we need to have a well defined reference architecture, which is what we are trying to lay the foundation for okay, so that we can then point to back here's our reference architecture when we go talk to signature, say, consul and so on.

B

Okay, so now little bit specifics on what the cluster multi-tenant cluster will look like so again in list again I'm going to focus mostly on s1, because just to keep things simple, but s1 slashes to right.

B

So in this model it's one namespace pertinent. So here we have three namespaces and intake. We're using existing are back like admin, a digital view, and we have created a role for tenancy admin. Okay, so because this is s1, we don't have the controller once you have a controller, the controller can do the job of the tenancy admin, but right now it's a manual process. This tendency admin is responsible for creating and deleting tenants. Okay, so and this tenancy admin has the privileges to create the isolation boundaries between tenants.

B

The tenants themselves do not have that, which is because they can step on each of these pieces had the privilege. So there is, the separation is via again because right now we are talking about s1 here. Existing constructs of our back rows and I have some samples in the coming slides and opinionated pod security policies and the Capone C is example, so the initial model could very well be that we do not allow any privileged containers in the in the tenant space.

B

Well, we'll come back to that, but that's an example of of a opening, a policy which the tenancy admin would around provision. So bottom line is that there is a separate author band process for creating projects right and only the sole. Only the sole can create delete, namespaces and label namespaces and more very likely. This would also be the only role that can create network policies, so equivalence cluster would look something like this.

B

You would have consistent namespaces like cube system and Q public, and we would need to support the notion of shared services, which we would in our reference. Architecture would say that those shared services need to run in the default namespace and each tenant gets their own namespace. The tenants are isolated from each other, but services in the default namespace can access one or more tenants because they're providing shared services to one or more of these tenants.

B

These shared services are things like the monitoring service registry service and English controller service, and so on so, and there could be multiple instances of these. So, for example, there are two instances of English controller English controller. One needs to be able to provide service to tenants a and B and English controller. Two may be providing service returns, C and D.

B

So in the curated reference model that we we would publish, we would need to define the.

B

Reference security policies to support this sulfa model.

B

Yeah and then the default namespace also has things like monitoring services and and which we would model as Prometheus, based monitoring.

B

This is very much modeled after efficient OpenShift kind of does something like this. There are shared services running the default namespace and individual projects run in their own namespaces.

B

They also have the concept of joint tenants and joint tenant is where tenant a a and B can actually talk to each other. They are not isolated.

B

Initially, we would go to the model that all tenants are isolates from each other, but we may want to support an additional capability of joint tenants and then the other thing is is having multiple do we want to have all services running in the same default namespace?

B

Is that, as part of our security analysis, we would need to? You know convince ourselves that ok running ingress controller for tenants, a and B and another thing that controller for C and D for joining them both in the same default namespace? Is it fine or do we need multiple services namespaces?

B

So the point the point here is that having a reference model for how we gonna lay out the names papers well,.

C

There are questions we have the question: what's whose ownership illness disease share a service who owns what? Whose owner of these shared service because seems like to me? No tenants the share service, no.

B

So this in this model, the shared services are owned either by the tenant admin of the cluster admin. These are centralized share service. If the tenant owns his own service, then it runs inside his own tenant anyway, in which case we don't care about. Just looks like a nap. The reason we have brought this out here is because the model needs to cover the concept of shared services which are not owned by the tenants just consumed by the tenants, but provided by the cluster admin of the tenancy admin.

B

C

Today, if the services provided by the crust enemy, most of them, we have city in Cuba system,.

B

Well, that's debatable. I mean we. There are cases where people feel that cube system should only have the actual kubernetes control plane, but things like ingress and prometheus should actually run in the default namespace which, again by the way, is what open shipped us. So a lot of this, what I'm, showing here is essentially modeled by you know doing what open shifters. So a lot of these soft multi-tenancy models is to make it very open ship like, but again this is a strawman. We are up for debate.

C

Whether the question is a tenant wants to provide a service for or other tans, for example, if I won't provide a post grants database for Ohio trends, so the tenant is responsible for setting up all these configurations inside any the tenant in space right, yeah.

B

That's the tenants own private services that he wishes to expose to other tenants, so we wouldn't at least initially need to accommodate that a tenant can expose through normal, not South, endpoints any service. To other tenants me, like I, said there is, we could have the concept of joint tenants it's up to this team, when we want to support that at what point, but again openshift supports that, and that tells me that there are customers that occasionally need that.

B

But for now to keep things simple initially, we would first create the reference model where, if tenant wants to give a service to others, that's private to them and we don't need to take care, but we need to have a reference model for this kind of shape. Services.

B

Remember that when we're talking about multi-tenancy here, these are essentially multi-tenant services, because the single instance of these services is consumed by multiple, multiple tenants. So that's why it's part of state agency architecture now I agree that we may be boiling the ocean here. So we made this decide to differ this point until later in this form, and we are just laying out the things that we may need to discuss, we need to define whether we define it in the first place or in a second phase. That's a separate, separate discussion.

E

um So I really like what you've done here. Sanjeev I will agree with my straight I just want to chuck in one other thing that, because we not to dive into it now but just to put on to realistic, was to think about it because you've got the ingress controllers. There were. How then does DNS integration work in this model? Sorry, that's a.

B

Great point: that's a great point: I haven't thought fully about DNS, but because DNS has I'm hoping that we will find that using our back. We are able to maybe cover all the DNS aspects or just the fact that tenants have network separation, so that may be adequate for DNS. But, yes, we need to think a little bit more about DNS. So it's one of the work items that we need to think through I have some notes on on on the cni in couple of slides, but DNS is something that probably needs more thought.

B

So, for example, if tenant a introduces, a service, we the DNS name, you do not want Canon B to be able to snoop on on to be able to retrieve DNS binding, which is meant for tenant a right. So what what needs to happen is the current DNS infrastructure adequate? We have to investigate and if there is any gap then we have to address it.

E

We don't solve it now, but that's that's one of the. So if we said in in this one we're not having CR days and things, you can still kind of do stuff, like that with admission controllers and just labels and things but yeah.

B

Right, so we will see what can be done with existing objects, and that will tell us you know what new objects we then need to add in future models.

B

So here the sample config of model s1. This is not complete, but this will give you a feel of what we need to define, so there would be custard install requirements right so curated config for the EPS over we disabled, a back disabled static off an able I'll that enable added logging, an integrated set of admission controllers. For example. We need for security policy, always pull images, etc.

B

These were all very relevant for multi-tenancy and we need to make sure that a this is set according to our recommended design in both manual solution, as well as the automatic solution right with the controllers.

B

But will be with the manual solution we get to verify, so we get to verify that this is exactly the right combination of install requirements to support multi-tenancy model and then the baseline we would align with CIS requirements. So she is publishes requirements on everything from what the correct ApS over setting should be to watch the file permissions and so on.

B

So we would define that as part of the more definition we would start initially only with the darker runtime, but at some point we will need to also allow for the possibility of other runtimes like cRIO, cotton and punk rocker. Okay, we start with just docker and we would start modulus one with some common plugins, but it should be minimally dependent on the type of plug-in. So, for example, we could say the next word: plugin is calico.

B

The octave idol is this here right to model a common on-premises cluster running on these peer initially, but the detail, the devil is in the details of how precisely we define the mod these are. These are all the parameters that we need to have the next level of the time on. So there's the cluster level parameters and then there is the tenant or namespace level.

B

So here we will define the curated auerbach like, for example, the new tenant tenant admin role and set it up to to be the namespace manager and also the only role you know we need to be able to not allow other rules have the ability to create namespaces.

B

We don't want developers creating namespaces, because this is equals projects in this model right, so setting up that curator go back model for various critical resources within the namespace. It looks like we should be able to retain the current kubernetes default. Rolls back rolls, so admin edit in view seemed to be fine.

B

We would have curate a small number of curated pod security policies which we would be meaning. This group would believe is the right particularly model to support. Multi-Tenancy and I have another slide on that as well, and similarly for the other things. So, basically, as we fine modulus one will be defining these precise parameters and then subsequently adding the controller to enforce this. But even if we don't have the controller, this would be the reference model. Okay, so just to get you a feel of of the parameters.

B

Here is the same checklist, but a little bit bigger checklist, there's actually more. That was sort of high level points. We would also need to cover things like seeker separation, right grid, spooning, phase storage separation like what would be our curation model for setting up I mean using existing constructs, persistent volumes, storage class quotas for tenant and then to what extent we want this to apply both to the long term as well as to the public hosted model we'll have to see as how much of that we can put in the model s1.

B

So we haven't fully defined more less one. We just looking at the kinds of things that we need to cover right. We talked about services, tenants, meaning that putting that default tenant for having the services. Another thing is for application, automation right, so in this multi tenant model, very often people will want to have em and pillar based architecture for deploying applications to the tenants. But tiller has some known security issues. So what would be a reference model for allowing helman tiller to work in this model?

B

So that's what we would need to capture as well. This is the bigger checklist. We can decide how much of it we actually do, but this is just to give a feeling of the scope of things audit logs, pertinent right audit logging.

B

Maybe this is all covered by just having the ITAR back, but we would need to test it out to make sure that there are no gaps and if there are gaps, then that will uncover new, more contents. And then we talked about things like joint tenants and multi in sprint and other things which for now I said, but not in Model S one. But you know the boundary is fluid, so we could add it there on the radar multiple run times like darker, Sierra and cada.

B

Sending automation similar to you see is proposal for namespace templating exclusive, new CRD right. This is, we said, X naught and s one, but again s 1e, but it could be part of s2 or we could merge it in verse, 1 and then tenancy aware, monitoring and logging. So we haven't talked yet about monitoring and routing if this, if the goal of this working group is look at all aspects or multi-tenancy, that includes monitoring as loudly so prometheus is now supporting multi-tenancy to some extent.

B

So we will need to investigate that and say: ok, how can we integrate multi and intermediate into this name, space based kubernetes multi-tenancy and have a reference model for that same thing for the SK tenancy aware container registry integration, so assuming that harbor is the registry service for such a custom, harbor has the concept. Harbor is the multi tenant capable docker registry, but that multi-tenancy would need to be integrated with the kubernetes multi-tenancy to common, and maybe that is largely common identity and all that.

B

But whatever that work is we need to define how much of that we are going to provide an opinion on in s1 or how much of that we leave it open for people to do what they want, but it's going to be a balancing act between how much we stand, how much we have an opinion on versus how much we don't define and let's take Leo whatever they want, and we have to learn the right boundaries over time. Ok, so so that's a bunch of things on.

B

So let me see how much time I have ok, so we are actually close to the end of the time. So I just run through the rest of it, and you know you can do get it later. So again, on March we talked earlier about THAAD security policies. I looked at a couple of options. You know: rancher has like to standardized for security policy where OpenShift has seven standardized particular policies. So what should we do? So? Maybe the proposal for now is maybe we start only.

B

The curated core security policy is similar to the rancher model and we don't support all this flexibility. Like OpenShift, maybe at over, sometimes we do it, but maybe not initially, and here's the example of the part security policy. I'm gonna skip past this I'm going to skip past. This was well I, haven't added details on how much opinionated we want to be in terms of the weather. The currently recommended default selinux, a farmer policies are adequate or whether we find some gaps this okay, but we start with whatever.

B

Is there and see whether it's adequate and, if not at a minimum?

B

We need to understand it so that we can able to know what is the level of support we have in the current model in model s1 and s2, and that might uncover us new gaps, which we do in future phases network having a reference architecture for the CNI and the network policy, so similar model, as we said before, but one issue is that we cannot use their policies to do both tenant isolation as well as internal space, isolation right, because if you want to do internal in space isolation, then the tenant user would need to have write privileges on the network policy.

B

But if they have write convictions on a policy, they can end up covering the isolation boundaries. So initially, what we would have the assumption of this model would be that tenants do not get the ability to do interconnect. Microsegmentation mental policies are only used for tenant isolation.

B

There are, you know which can do both meaning they can do both tenant isolation and still give the flexibility to dependent to do interact and network policies.

B

Maybe we don't include that model initially, but some implementations might choose to do that. To give more flexibility, we have to define a if you will a least common denominator solution that works with all Co knives, but some cyanide are actually already names paper pair, so they can do more, but we go with the baseline CNI, which is not namespace of it. Okay, so there's some detail here which won't talk through this, but and there was a question earlier about DNS records.

B

So again we need to scrutinize back to see whether the current capabilities are adequate for namespace base tenancy. So here's an example of native connected policy. If you take the least common denominator approach, you can use a simple policy like this to do namespace circulation, but you don't have any egress control here and you also cannot use it for internal space isolation, but maybe that's fine for the initial model.

B

There's a bunch of text here which is very, very subtle, just point to what it is, and you can look at it later. So it depends on how deep you want to go into monitoring the multi tenant container registry service. The equipment is in English and multi tenant lodging. Maybe they differ this to a future model, but it's something that we need to start thinking about if you want to have a complete solution, so action items are next steps right.

B

Of course, we will discuss this offline with Tasha and they will join others. But some thoughts are that we would move this information formally into a working group architecture or a single document that we could all edit or comment on and have that in the wiki page. We will kick off threads eventually for all the models like s1, s2 and h1, but in particular 4x1.

B

We want to make earlier progress because that's mostly using current capabilities, so we should be able to target that for the next two and maximum three months really I'm talking about one to three months: maximum for fully fully documenting that and to limit that scope. Further, we only focus for on premise cluster. We don't focus on cloud clusters, we we don't focus on things like service namespace, maybe initially right.

B

So we reduce the scope to keep it manageable for the for the short term, and once we have some bins, we can expand the scope and then we start working on a third-party test, suite repo and add some initial tests for more s1, and we use that to give a demo on a read out specific architecture and say architecture, hopefully will then come back and say: yes, we agree and you know, you've got a reference model and you know you got you've given a demo using, so we can bless it for the full controller based solution, encoding, soft and hard versions right.

B

So that's all we try to phase it and then, in parallel we work on the ps4 model s2 on each one and get those into the conformance, so I know this is. This is all kind of just speculative, but we are really talking about. You know.

B

The next two, essentially to end of March, to kind of have a reasonable s1 model, have a demo with it review it with the Gotcha. In parallel, we can we work on the other models, as well as the conformance test, and that will happen. We can see how this plays out, and hopefully we can do a demo at Keep, Calm Europe, in which we demo at a minimum model, more s1 and hopefully parts of more SQL h1 also which is in the end of May time frame.

B

So that's about it. Sorry I may have rushed through some parts quickly. Hopefully you got a feel for it silvery storm, and so we think about it. So are your comments and we have to take a comment.

A

Thanks Angie, for going over that, we will share this over the this link to this document over the mailing list and in the agenda as well as a video of this conversation, and so, if everyone can just go through it, add your comments wherever appropriate and then we'll start kind of pulling those in and take and then try to put together a second draft and share that with everybody as well and if yeah so be as detailed as you can and we'll take it from there. Thank you and thanks everyone for joining today.

B

Thank you. You.