Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A Hard Multi-tenancy Solution: Use Case, Challenges and Work - Xiaoning Ding & Qian Chen, Futurewei

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A Hard Multi-tenancy Solution: Use Case, Challenges and Work - Xiaoning Ding & Qian Chen, Futurewei

Multi-tenancy enables multiple organizations or teams to share the same cluster. There are different proposed multi-tenancy models in Kubernetes for different use cases, from soft-tenancy models where tenants trust each other to hard-tenancy models with no trust among tenants. In this session, Xiaoning and Qian will share their use case of multi-tenant Kubernetes, where they share clusters with public cloud users. There is no trust among these users and a strong isolation among tenants is a must. They will talk about the use case, challenges, work, and also experience and lessons they learned from building such a hard multi-tenant cluster. And they will conclude the talk with a demo.

https://sched.co/ZemB

A

Hi everyone and thank you and welcome to talk. My name is jeanine in this talk. My colleague chair and I talk about the multi-tenancy work we did in kubernetes uh chen. Do you want to quickly introduce yourself.

B

And I'm a cloud dev engineer in future way and my focus now is in multi-tenancy work in kubernetes. Thank you.

A

Okay, this is the agenda of the talk. First, I will give a brief overview of the project, including some unnecessary background information and then, for the most of the part, we will walk through our multi-tenancy design and features after that, I will also give a summary about the current project status, the next step and the challenges, and we will conclude the talk with a demo.

A

Ok, let's get started with the overview. First, I want to say: motor tenancy is actually a very broad topic in kubernetes.

A

There are different multi-tenancy models for different scenarios and we usually categorize these different models by their isolation level. If a multi-tenancy model provides strong isolation would say it's a harder model. If the model provides a weak isolation, we say it's a soft mode in for some use cases, weak isolation. Properties are enough, for example, if you are working with multiple teams inside a company, because these teams all belong to the same company, there is a certain level of trust among these teams, so you don't need a very strict isolation.

A

It's probably okay for them to see some resources from each other and they probably can even coordinate on some resource allocations like ip range. But if you are working with multiple, even external organizations, there are no trust among them and you need a harder isolation.

A

That's our user case.

A

In our use case. We want to enable multiple external organizations to share same physical cost and we don't assume any trust relationship among these talents. A typical example is a a cloud provider offers a kubernetes cluster as a service to public cloud users and obviously there's no trust among these public cloud users.

A

So we are trying to address a hard multi-tenancy model with strong isolation.

A

While we will talk about the detailed, detailed design and features in the next slides here, I want to quickly describe our solution from the following three aspects.

A

First, our whole solution is built around the idea of virtual cost, so there is only one physical class and one control plane, but each talent feels like they are using the cluster in in an exclusive way. They are not impacted by other talents and they even do not realize the existence of other tenants.

A

We also provide api backward compatibility, so all the tenants can still use their old kubernetes tools and uh and the clients all the apis are same. They don't need to change any code or yaml files. Everything is transparent. Api is all same and we also provide some self management talent. Admins can manage their own talent without getting help from class enemy.

A

They can still use like the native recognized objects. Like rule row, settings rule bindings, cluster rule quota security policies, they can still use all these familiar api objects, and all these changes only apply to its account to its own talent. It won't impact other talents.

A

All our work is implemented in a open source project called actus. I put the repo uir here feel free to check out if you are interested in the project details after the overview, let's walk through our design and features. First, I want to start with a very fundamental concept. Called space space is a new concept that we introduced in api server.

A

A space is a high level of isolation. Unit beyond the name space.

A

A space can contain any api object. You can single, you can single a space is a subfolder. When you create a subfolder, you can have, you can have put anything in the root folder into the sub folder. So here I put an example. We have two. um We have two objects under the space called t1.

A

If you look at the left side of the example.

A

So this t1 space has a pod resource, which is a namespace scope, and it has a pv object which is non-named space scoped in the uir. Due to some historical reasons, we are still in talent, but it should be species and on the right side, you can see how these objects are facilitating lcd.

A

We basically embedded the space name in the icd key, so all these objects are stored um separately. You don't need to worry about naming conflicts after we introduce the space. um Every api object in api server belongs to one space and only one space.

A

In order for us to quickly find out which space contains this api object, we introduce a new field in the object metadata section. So, every time when you check this field, you can easily find out which space this object belongs to.

A

Okay, so we introduce the space um and each talent can have its own space and in the space he can has, he can have his own uh copy of the resource higher hierarchy and he doesn't need to worry about naming conflicts and uh other any other conflicts with other tenants.

A

But how does space get related with talent and how do they get initialized?

A

Basically, there is one special space called the system space.

A

This space is created automatically during cluster bootstrap, so after a cluster is bootstrapped, there's only one space system space there and this system space contains all the cluster level resources like uh cluster settings and the nodes et cetera, because this resource they don't belong to any particular talent and in the system space we introduced a new api object called talent.

A

I put a sample yama file here on the right side. You can see it's a pretty simple object.

A

It also has a talented controller, like any other controller in the kubernetes. This talent controller watches a live cycle of these talent objects.

A

Whenever a new talent object is created, this talent controller kicks in it will create a corresponding space with the same name of the talent, and it will also initialize this space to create all the default objects like the default. Namespaces uh differ. The uh rules default dns all this stuff, so it will initialize this space and make it look like exactly the same as the original, the kubernetes accessor, and also, if I'm telling the object, is deleted. This tenant controller will kick in and destroy the space and delete all the resources in space.

A

So this is how a talent space gets created and gets initialized, and next I will hand over to chen. She will explain how we build all the other features. On top of this fundamental concept, space.

B

Hi thanks to sunny this is a chat, so I will give more details about our work, as mentioned by xiaomi. Just now, one of the key goals in our design is backwards. Compatibility we would like the user.

B

Oh sorry, I forgot to move the slides, so here the slide I'm talking to so we would like our customer, continue to use current tool like coop cdo and continue to use the resource, emo files and the crd operators, without any code change, without compiling or without port image, rebuilding how to do it. The key chain, the key design here is something we call a short path resolution it makes. The existing apis will continue to work in our system.

B

Short path is something.

B

Same as the existing api, so it contrasts something we call full part the full path. Maybe you have seen it just now in the previous slides. It's such a slot. It's such a url with the tenants specified in the path. So with this path you can ident, you can uniquely identify results in the system and the short path is, without this part part. So it's the same, existing apis, but how to translate short part to full path. So we may have two customers.

B

Two tenants send out two apis with the same urls, but we know they need to go to different space. How to do it? We added we added a component called short path resolution.

B

The short part resolution will infer the target space from the access credentials in this resource request. So after the resolution, this short path will be translated into full path and then you will go to the correct space. So that is something we do for backwards compatibility, as mentioned by just now. We said that we need the credentials, include the user tenant information, so we made some change in authentication.

B

The main change here is, we add, a new field called tenant in the user identity, in addition to current user and group fields. Here I show some examples here, so this one is the new way to generate a cert, and this is a new format for the it's it's for the token and this one is the new format for the password file. So basically, we support all authentication strategies that is working in kubernetes now and the next thing I would like to say it's about authorization in our system.

B

It's pretty similar to what we have now. Actually, the main change is here. We add something check at some check on the user tenant. We make sure the tech user tenant have access to the resource to the to the space of the resource. So the rule here is the system. Tenant can access any space, and but if it's a just a regular tenant, he can only visit his own space. We may have some cross tenants across space visit in the future, but now it's it's the rule that we we have done and you can see.

B

It's actually just add a few lines here and then, after that, we make sure the user tenant is in the right space and then the access control inside space is same as we have now in a kubernetes cluster, so that is access control and in the next slides I want to combine my talk about short path and access and control to give a full story on how a resource request is handled in our system.

B

So suppose there is a resource request arrives and then the api server will retrieve the tenant and user identity from the request first and then the next thing it will do is to check whether or not it's a short path. By looking to whether or not the space is defined in the url, if it's not defined, then it's something we call short path. Okay, then we will set the resource space to the tenant name, that's something earlier. We mentioned it's short path resolution.

B

If it's a full path and we'll do some access control, uh we'll do some authorizing check to make sure whether or not the tenant is visiting his own space. If it's not, and we will check whether or not the cross tenant policy is allowed, whether the cross-tenant policy allows this request.

B

If, yes, then we are here, then it's we will access the results with the full path. It's then, what what is happening next is same as something we will see in the kubernetes, so that's access, control and short path, how they are fit together.

B

Just now I mentioned about backwards, compatibility that we said every tenant can use their existing code existing tools with without any change because they are working in their own space. But there are some components that we realize there are some components they need to work across the spaces like something like a scheduler like a couplet and like all the controllers and the kubernetes.

B

They are now working with api server which supports multi-tenancy, and they have to make some code change, so we did it. What we did first is to make some code change. Client go so we add uh a set of multi-tenancies, the iud apis. The difference is now in this apis. There is a new parameter called tenant and also we make some code chain to support cross space lease watch and after that we make some code change to this components.

B

Use this using this client go new apis, so here I gave some sample change here like we would like for this one it's about visiting pause and for given name space, and now we need to pass in another parameter which is tenant, and one thing I would like to mention here is that the code change here is quite straightforward: it's with the client new client go apis.

B

The change will happen to this scheduler couplet and controllers. It's not big. We have finished it almost all of it and I scheduled a couple. It's working our system now and I think they are 15 or something I think the number is pretty big, but we did pretty quickly. So if you have interested take a look at our code and you can see that code changes quite quite easy and that's client go and controller change, and next thing it's here. I want to mention that as something shiny, just now give a very brief description.

B

It's tenant autonomy as we like to design a system in such a way. It's like each tenant is working with his own virtual cluster. Just like a virtual cluster is delegated to the tenant and the tenant can manage its own cluster without turning to cluster admins, so he can work with almost all the existing apis like row.

B

Row bindings cluster row, cluster bindings resource code, and things like this, and also here you can see we said crd I mean each tenant can have his own crds and one thing here like there are some clusters scope concepts like cluster rules which in existing kubernetes it's it's a clutter, scope, objects. Now in our system, it's transparently translated to er space level objects like here. This two pictures explain this. This difference like earlier, like the cluster rows, it's effective in all the namespace in one cluster, but now it's working on a virtual cluster level.

B

So this cluster role defined by this tenant, will work only in his own space and has no effects on other tenants. So it's something called tenant autonomy, so each tenant have his own virtual cluster.

B

Next and something like yeah, I briefly mentioned just now: multi-tenancy crd, so crd is a very important feature for kubernetes, and now we allow each tenant to define his own crd.

B

They don't need to worry about any conflicts with other tenants and they can use their own crd just just like they own their their own clusters. It's and this in-store seat is only visible only usable to himself I mean to his own space and the nice thing is because our backwards compatibility, they don't need to change their crdml file. They don't need to change this custom resource emo file and they don't need to make any change to their crd operator.

B

They can just work, and besides this, we also realize there are some requirements that at some point the cluster admin would like to enforce the crd, which is usable by all the regular tenants. We also support this scenario. We we call something, we call a system crd and it's such a city that can only be installed by system tenant in the system space, and we would like to mark it and we want it to be marked with a special label.

B

Without this label, then such a crd will be only available to their system space, but once it's marked a special label, then all the tenants can use it just like his own custom, just like his own crd. So here ends my brief introduction. Next, I would like to hand it back to shelling and he will give more details. Thank you.

A

Thank you in the previous slides chen and I talked about the about all the design and the changes in the control plane in the master in the master part like the api server and the controllers.

A

Here I want to talk about some changes in the in the in the data plane. In other words, things happening on on the work notes.

A

The first part is about the runtime runtime isolation.

A

As we all know, regular containers like docker containers. They are actually sharing the same host kernel host os kernel, but when we are running in a strong, isolated, multi-tenancy cluster, a single single work machine can run workloads from different talents and there might be some malicious: u users or codes inside the workloads.

A

So it's not acceptable to share same copy of host kernel to address this problem. We are currently using qatar to support the oscar isolation and we are also actively investigating lightweight virtualization approach to provide better performance while maintaining the isolation level.

A

The next part is storage, most of the modern storage providers. They already provide either account based or volume-based isolation. So there really isn't much iso work to do here in terminal storage, isolation.

A

We only added some talent level storage quarter to manage the storage cost among different talents.

A

The most complicated part is actually about network in the vanina kubernetes. The network model is a flat more flat mode. It assumes every port is reachable from another pod and they all share a single port, ip side range and the service ip side range.

A

But, uh as you can imagine, this cannot work in a hard, multi tenancy model, because in a hard multi-tenancy model, each talent is supposed to be strictly isolated from other talents, and they shouldn't aware of the be aware of the existence of other talents so um that the the pod ip side range need to be uh need to be uh isolated and each tenant should should have the freedom to choose their own ip range without worrying about the ip conflicts and the service.

A

Ip range same needs to be isolated and also each tenant should have their own dns, and all this stuff should be isolated in our solution.

A

We choose vpc network to provide strict isolation, but at the same time we also want to support the other exchange as a long vpc network for easier adoption.

A

In order to make the design consistent and support different network providers. We introduce a new network api object, the name is network and a network object represents an isolated virtual network and each version. Each this network has its own port ip range service, ip range, dns service policies. Everything is, is isolated.

A

With this network object and a network object that can be of any type either vpc, backed or long piece backed and for each type there there will be a network provider deployed in system space just like how a cid and operator works, all the ports, their ports and service and other network related objects.

A

There are annotated to attach to a specific network.

A

If we look at the example here in the red space you can see, there are three network objects default net, a net and net b, so any product service can be attached to any network.

A

If a product service is not annotated by default, it will be attached to the default network, and this default network object is created by the tenant controller automatically during the space initialization.

A

Here I want to give a brief summary of the current project status next and the challenges currently in actos. We already finished pretty much the mostly common used functionalities like the talent, initialization talent, deletion and the commonly used functionality. Objects like pos, namespaces replica sets deployments job stateful set all this all these commonly used functionality, objects and also commonly used uh management objects. Like rule row, binding cluster rule classical binding.

A

Resource quota etc. Now we are working on the multi-network internal next step. We are planning on two things. First, we want to uh support the tenant level or resource quarter rate limit and the perf isolation, since our solution is based on where to cluster cluster mode. Different talents share a same control plane.

A

So this is to prevent a talent from abuse the control plane and uh it won't cause any to make sure it won't cause any performance impact on other talents and the second thing about the cross, talent, access and the find granular access control for system talent, because in the current model system, tenant users can access any other regular tenants.

A

It's like a super user, but the next step we are going to introduce, find graduate access control. So we can reduce the use of this super user.

A

Talking about challenges we can see, there are two challenges. First, our solution has a lot of intricate changes like the endpoint resolution in the api server and some controller changes, client goal changes.

A

uh We are thinking about how we can optimize this and reduce intricate changes and also the second one is about the client goal and controller code changes.

A

We need to add a new parameter called talent for announce our ciud and the list of watch methods, but unfortunately, because gulen doesn't support the method overloading, so we have to create a new function for each of these changes.

A

Next, we are thinking about how we can use some other methods approach like a context-based or implicit context, to reduce this kind of code changes.

A

uh That's pretty much china I want to talk about today, and here is a list of links of our work. As I mentioned earlier, all our work is done in open source project. Called actors. Here is a link to the project, ripple design documents, our roadmap and the detailed design docs everything in public.

A

You are also welcome to join our snack.

A

Channel uh hey.

B

Welcome to the demo.

A

Next, I will play our recorded uh demo.

B

Video made by channel for different tenant, this one is for system tenant, and these two are for two regular tenants which I just created. We.

A

B

In the presentation that each tenant is working like only its own virtual cluster, so when a tenant is created, some resources have already been created for it. Just like some resources have already been created when the cluster, which I mean the kubenetes cluster, is up so here we can take a look like in this tenant, some name spaces, all right.

B

You can create it same here, and some cluster roles have already been created like here and also here, and of course, each tenant can create their own resources say this tenant would like to create a new namespace and you can do it and then we can see yeah it's created and but for this tenant, okay, no new namespaces is create.

B

So everything remain the same. So here we can see the resources. Management between different tenants are isolated, and next thing I want to show is the creation of the deployment, and here I have the department yemo file. Let's take a look at first okay and you can see it's just a yaml file ex same as any yama file. We can see in kubernetes and let's run it.

B

And you can see it's up, so we can have a check here.

B

Okay, you can see the deployments here, but for this tenant, okay, resource creation is isolated. No new deployment here and also earlier I mentioned like cluster role, which is working as a clutter- scope, object in kubernetes.

B

Now it's working at the tenant space level in our system and it's done transparently to the customer, and let me show it so here I'm going to create a cluster role and that's the yaml file, and here we go and we're going to create it.

B

Okay- and here you can see.

B

B

Yeah- and you can see the new row is here, but for this one, okay, you see there is no new cluster rule. So that's we say the cluster scope objects is now working at a space level, and we also mentioned each tenant can create their own crd, and this time we are going to play with this tenant. Okay, so.

B

And then we are going to run it okay, it's created, then we can take a look at the crd here: okay and this crd, just like the crd in kubernetes, you can cheat it like a api resource and this command will work, but for this one, okay, he knows nothing here and also we can create.

B

Custom resource based on the crd okay, then you can check it using this command and here it is, and for the system tenants, as we mentioned it, has access to the spaces of the regular tenants. So he can take a look.

B

At the resources in those regular tenants like he can take a look at name spaces in this tenant. You can see there are four or three which are created by the system automatically and the one we just created and also you can take a look.

B

At the other, tenant and you'll see here, there are only three here ends our demo. Thank you very much.

B

And thank you very.

B

A

Hi everyone- I am this happening chair and I will take the questions asked during the presentation and first I want to apologize uh sorry due to some technical problems. I think the demo sound wasn't playing out, so it didn't work very well.

A

uh I guess we have five minutes to answer all the questions. I will address shane and I will address these questions based on the order it issued on the platform.

A

The first question is uh for mike brazer: does the virtual class pass the cuber conformance test? Yes, that's our target and we haven't performed the conformance test yet because we are still working on some and the multi-multi-networking part.

A

Chen, are you able to see the question list.

B

Yes, okay, like I, can I I'll take next question. That's question number three from sugander.

B

Agrawal, so what's the difference, the question is: what's the difference between space and kubernetes namespace? Fundamentally, space is a higher level hierarchy above namespace, so under one namespace you can have different name spaces and it's possible that you can have two namespace with the same name under under two different spaces. So it's a higher level than namespace.

B

That's I think johnny has slides on this, but basically that's just it's a higher level of hierarchy. That's it.

A

Yeah, yes, as a space, is actually a higher level isolation unit than namespace a space can also contain long namespaced objects like cluster rules or pvs. So it's more like a virtual cluster, it's larger than namespace.

A

The next question is from martial maltese: what's your secretive model? uh Basically, what are the shared resources among talents? Actually, they are in our model. They are first, each space are slickly, isolated, there's no shared resource among spaces, some physical, some cluster resource like nodes. They are in system space and they can be used by each talent.

A

But each talent cannot access this system space directly. They can only be used by the system controllers.

A

What mechanisms are actually doing isolation between talents?

A

First, on the storage side, I mean on atcd all the talented resources are saved on the different key prefix, so they are totally isolated and the second, when, when clients try to access this resource in api server, there's authorized authorization and authentication mechanism, we make sure a talented client can only access the resource object that has been authorized to him and third, in all, the controllers, like a deployment, controller or replica set controller, they are aware of talents when they associate different objects like associating a replica set to a part, the only associated resource among inside the tenant.

A

I think this to these three sites uh build the talent, isolation.

B

Okay, so next one is question number six from bijan. Oh, I hope I pronounced this right name correctly. The question is: uh uh is this behavior you are describing a custom, kubernetes distribution or extension, for example the stock kubernetes?

B

uh Actually what we are describing here today, it's uh the the it's a it's more like a customer comment. Distribution involves something she changed as we described just now like we have some change to the api server and to the controllers, corporates, and also we had to add some new apis to client go. But one thing we would like to emphasize is we have some designs for backwards compatibility. So all the existing crds, all the existing yaml files and all these tools are continue to work.

B

I hope that answers your question.

A

Yeah- and I probably can add one couple more sentences: basically, we are describing a standalone standalone project open source project. It's called actus actus is a base, is derived from kubernetes ripple a code base, but it's a standalone project and is using for for the cloud infrastructure area. So it's um it's not a add-on, kubernetes add-on. It's it's not also a kubernetes distribution. It's a standalone project, open.

B

Source project.

A

The next question is from bram and nat scale. um Does it also work when, when using us service mesh? Yes, we are working on the network solution once the network solution is done, the service match should also work.

A

Okay. I guess we have only one minute remaining. So so probably we won't be able to take uh all the questions. Let me see if there are any other important one, I'm I'm sorry for it, because the platform limited our time just popped out a warning window for the one minute limit.

A

uh One question.

B

Okay, I think for question number 11. Yes, it's open source project. I think that's a quick answer.

A

Yes, it's a this everything's open public. I put a link there and some, so somebody asked if we can share the slides and also because today the the recorded demo video didn't play well. I will put the link to the slides and probably also the demo, video uh on our github github github, a wiki page home page. So you can, you can watch it there.

A

Okay, I guess we are.

B

Probably we are out.

A

The time now and I think we are cut out from the platform from the live.