Kubernetes WG Multi Tenancy, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Virtual Cluster - A Practical Kubernetes Hard Multi-tenancy Solution - Fei Guo, Alibaba

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Virtual Cluster - A Practical Kubernetes Hard Multi-tenancy Solution - Fei Guo, Alibaba

Conventional, the concept of Kubernetes multi-tenancy is realized by namespaces. Tenants access controls are limited within tenant namespaces using RBAC rules. The Pod level isolation is primarily done using network policy. This model faces various problems when applied in production since Kubernetes is far from tenancy-ready. For example: 1) APIServer is lack of tenant-aware flow control. A single tenant may generate large amount of concurrent traffic making APIServer unresponsive to other tenants; 2) Tenants cannot install customized CRDs which requires cluster scope permission; We proposed Virtual Cluster solution to resolve the multi-tenancy problem from a different angle. Basically, every tenant will be assigned a dedicated K8s control plane. All tenant K8s shares a big super master. Virtual cluster is built based on CRDs. The entire solution is open sourced in Github.

https://sched.co/Zek6

A

Okay, good afternoon, everyone today, I'm going to present the virtual cluster, a practical kubernetes hardware, tendency solution.

A

So multi tenancy has been us required by many kubernetes users for a long time from company's perspective, multi-tenancy impact both the infrastructure cost and the rd efficiency, two things that are very important for companies growth. However, in kubernetes uh that multiple users should consume the shared cluster resource in your isolated manner is not easy.

A

This is because, in original kubernetes design, the design was focusing on the single tenant use cases and it works great for single tenant use use cases, but if we want to provide strong isolation in kubernetes, most of the core components, including its control plan and the data plan need enhancements.

A

Some innovations have been done. For example, nowadays kubernetes supports sandbox runtime to provide vm level computer resource isolation and the kubernetes api server provides name space api for metadata isolation, but there's still many isolation. Requirements has not been met.

A

Let us reduce the problem scope a little bit a little bit by looking at the control plane. Only is this possible for the native community to support the following: multiple tenant users run. Their workloads in a shared resource pool are managed by a kubernetes, but each users can only manage their own workloads and pass and no matter what they do. Other tenants shouldn't get to be affected in terms of the security, stability and manageability.

A

In other words, is this possible to provide a complete control plan, isolation in kubernetes? Another problem is: can this be done? Natively?

A

Yes, we can build another path, layer to encapsulate the kubernetes cluster, to provide us complete control, print isolation, but this over introduced non-trivial integration cost for the tenant in order to use the new pass.

A

Unfortunately, the native kubernetes cannot achieve both. In this talk. We will see how virtual cluster can provide complete control, isolation natively so that the user doesn't need to pay any integration cost a little bit. A brief introduction to myself. My name is feiko, I'm a senior staff engineer from alibaba cloud. I work in the cloud native application platform team. I'm currently leading a few communities related projects in the area of service, kubernetes workload, management and edge compute.

A

Here is outline of today's talk. First, I will present the virtual cluster architecture design. Now I will talk about some technical challenges that we face and how we solve them. I will show some experimental results to show that virtual cluster is a practical solution and then I will also compare virtual cluster with some other solutions and give a quick summary about the project status.

A

In the end, I will give a short demo.

A

Okay, before I move on, I want to emphasize that this talk solely addresses, kubernetes control, pronunciation problems and the data pro isolation. Technique techniques will not be discussed in this talk and some of the existing solutions may be referred if they are publicly available. For example, the sandbox container technique.

A

Okay, before we design our multiple mart tenancy solution, we need to be clear about the threat model that we are tackling, so we assume that the tenant users are not trustworthy. Therefore, exposing class scope resources is dangerous.

A

um uh There can be security concerns when doing well. When doing this, of course, um the tenant users shouldn't be granted permissions to operate on the class scope resource as well.

A

The tenant user may generate harmful usage patterns, for example, the denial of service attacks, but sometimes those those harmful patents is not generated on purpose. It can be an accident so uh and, and and lastly, the tenant users may serve other users as well, so it will make their it make it even harder to predict the calendar user's behavior.

A

We assume that the containers are not safe. So, overall, if you look at this threat model, they are typical in cloud environments, but they may also apply in the internal organization as well. Let's see how upstream kubernetes can do about this.

A

Namespace is the kubernetes building api for tenant abstraction, but with a shared control plan using named space. Odin is not enough to provide strong isolation by sharing the control plan, we have a legacy and a classic noise neighbor problem.

A

For example, if tenon a may have a lot of parts and if he keeps listing his parts and the tenant b may get a slower response from the api server, it will become even worse if canon b runs a more important application compared to canon ace application.

A

The kubernetes nav space api does is not a talent aware, which means a tenant may list all the name spaces in the api server, which can be a problem because sometimes the namespace name can may contain the sensitive information.

A

The tenant users cannot install class scope resource, which means they cannot install crds web hooks, create scope, rows, etcetera. This limitation can be a big help mode for many many kubernetes use cases. Overall. We need to find a way to enhance the current namespace based multi-tenancy model in a kubernetes.

A

I think there are, there are many possible solutions to achieve a stronger as a isolation model in kubernetes, uh but when we design virtual cluster, our primary concerns are in four aspects. The first is a compatibility we need to make. We need to make sure our solution doesn't change any api existing api. So in order to reduce the 100 users, integration cost.

A

The second one is the complexity. So we aim to leverage kubernetes extensions and don't change any kubernetes core components for scalability. We we targeted to our use cases to support hundreds or even thousands, tenants.

A

To share one kubernetes cluster, so the solution has to be scaled. The last way is the cost. I think it is okay to have a trade-off between the isolation level and the cost, but the cost has to be as small as possible.

A

Now, let's look at the our design in virtual cluster, each tenant user has a dedicated tenant control plan, which we call the tenant master. A tenant mask includes api server, controller manager and atcb.

A

The tenant master life cycle is managed by a tenant operator which runs in the supermaster. The supermaster is the kubernetes cluster that manages the actual node resource.

A

The tenant user operates in the tenant master directly and that there is a single controller which copies the objects from the talent master to the supermaster for part provision.

A

The node couplet watches the supermaster and create create the actual container using the sandbox runtime for computer resource isolation.

A

Since the kubrick in the node olin can execute the supermaster, we need to find a way for tenant a user to access their parts using login or streaming apis. In order to do this, we add a proxy in each node called vian agent in in the tenant master. There is a version of the object which maps to the physical node that the attendant part is actually running on the version of the object points to the via agent, so that the video agent can proxy all the tenants requested to the couplet over.

A

If you look at this architecture concept wise now, the supermaster becomes a part resource provider and other tenant objects, management and rba control happens in the tenant master and all the workload controllers. Extensions crds operators are installing the tenant master directory based on the user's.

A

Demand: okay: let's see how this design fits fits in our criteria from compatibility perspective, we didn't change any object, api and semantics and we can always support the upstream kubernetes versions. People may ask the differences between a virtual object and a virtual couplet in virtual in virtual cluster. Each virtual node has a one-to-one mapping to the actual physical node, which means the part affinity and the pattern of affinity. Semantics are the same as before, but the virtual cubelet does not necessarily represent a physical node.

A

In many cases it represents a group of nodes, hence the node semantic actually changed in terms of complexity complexity, our thinker only seen 12 api objects from the tenant masters for c to the supermaster, not all of the tenant objects and everything is done using the kubernetes tensions. The complexity is limited for scalability.

A

We find that one sinker can support hundreds of tenant masters without much performance impact, so we will show some experimental results later. Overall, the solution is scalable.

A

The tenant master cost is always concerned, but if you look at the kubernetes design, it is actually similar to unix micro kernel. The api server is concise and small, and most of the logic are implemented in the client side.

A

Also, a one kubernetes master pro tenant user is a kind of standard in many cloud service products such as the aws eks and the article of ask overall, I think the cost is acceptable.

A

Well, the benefits of this design are clear: there are no noise neighbors anymore and when tenants cannot see, other tenants objects at all if one tenant hits the security vulnerability and only that tenants get affected. The tenant user has a full control over the tenant master, which provides the best user experience.

A

Next, I'm going to describe some challenges that we face when we implement the virtual cluster.

A

So the biggest challenge we have is actually to provide a virtual cluster video for each tenant. Let me explain it in virtual cluster. One tenant path has three entities. One part object intended master, a part object in supermaster and a part containers running in nodes created by the couplet.

A

The sinker doesn't change the part name, but it has to change the partner space in order to avoid potential naming conflict in supermaster. Basically, we will add a prefix to each namespace in the supermaster, but the problem is that the user can log into the parts that are running in a supermaster, but we have to make sure the user recognizes his parts running his tenant kubernetes instead of the supermaster, which means inside the part, the service service account dns settings and namespace has to have to represent the tenant. Config configure tenant master configurations.

A

It sounds like we have to change the couplet to achieve this, but in fact we didn't do that the thinker doesn't magic. It manipulates the pod template like a mutation web hook for all the environmental variables service, recorder, secrets, hosa lines and the dns config, so that all the configurations in the tenant master are used.

A

The thing can also ensure the data, the data consistency between the the tenant master and the supermaster. Basically, the tenant master is the source of the truth for the object, spec and the supermaster is the source of the truth for the object status.

A

In the end, the user is not aware of the super master at all. Literally, you don't need to change any of your yaml files when using canon master. It should just.

A

Work synchro is a very powerful controller in virtual cluster. It has a full control over all the masters, but we have to make sure that the sync doesn't generate too many requests to all the masters for consistency check we leveraged kubernetes least and watching mechanism.

A

If you look inside the sinker, they are per object, reconciler, which synchronize the object based on the states in the tenant master, informal caches and the supermassive informal cache. The idea is that by looking by comparing the cache states, we can minimize the pressure that a thinker may apply to all the masters, regardless. How many objects needs to be synchronized.

A

Okay, next, I'm going to pre present some experiment results.

A

We conduct some stress tests uh in experiment. We set up a hundred tenant master, doing concurrent product creation.

A

The number of created parts is increased from 1k to 10k, and there is only one single controller in a supermaster we install 100 virtual correlate to simulate a 100 node cluster, which also means that the times depending node during part creation is skipped because we use uh virtual corporate.

A

This test mainly evaluates the performance of the control plane. The left figure shows the histogram of part creation time in 10k pass case. So this is a very stressful, parallel workload. The part creation time is determined. The one part becomes ready in the tenant master.

A

In the baseline case, all parts are created in the tenant must, in the super master directly. So from this figure we can see that when using virtual cluster about 80 percent of parts are created within the baseline time range, the vc has a longer pairs because of the queuing latencies in the sinker.

A

The right figure shows the workout time of creating r parts for different number of parts. The smaller value, the better from this figure we can see that we can see that the vc doesn't bring significant throughput degradation in the very stressful 10k pass case.

A

There is about 70 percent of slowdown, which I think is acceptable.

A

Now, let's look at the synchronous cost the simplest resource, consumption doesn't scale. This is because the supermaster has its own stupid limit, so the sinker cannot drive the load infinitely by burning resources and the other cues in the sinker are back of rate limited queues.

A

The sinker is stateless uh in the 10k parts case we find that the skate recovery time for the sinker can be is less than one minute when sinker restarts in the worst case, where sinker really becomes a bottleneck, it can be horizontally scaled easily.

A

In a normal unstressful case, the extra latency added by the sinker is usually less than a few milliseconds.

A

Okay, virtual cluster is not the first trial. There are other solutions that address the kubernetes control plane. Isolation problem, I think case 3v- is probably the closest one compared to virtual cluster in k3b, each user has a dedicated control plan, which is a modified case. 3S api server to compare in virtual cluster. We didn't change upstream. Api server, also increased 3v. It follows a per tenant sinker model, which I think is not necessary. uh If you look, if you look at the experiment, experiment results I showed just now. One sinker can support multiple tenants.

A

There is another project called actos which modifies the eps server to support the new tenant apis. I think all the existing plugins have to be changed as a consequence, also in october's. If you use a shared control plan for all the you for all the users, then the noise neighbor problem still exists from high level. Consider wise the single controller and the virtual corporate have similarities, but the virtual couplet uh use simplified provider interface, so it always has some compatibility issues.

A

Okay, last last, let me give a quick summary about this project. So virtual cluster is a multi-tenancy working group project. It passed 99 of the kubernetes conformance test. The failed ones are not supported. For example, there is a one field test case asked for changing the super master sub domain, which is not supported in virtual cluster virtual class, has complete unit tests and annual test cases which covered more than 70 of the code.

A

It supports both the cloud and on-prem kubernetes clusters.

A

The solution has been used in icloud service product and it gains more and more interest from the community as well, so feel free to give a try and then let us know your thoughts, the comments and the requirements, and we appreciate your help.

A

Okay, next, I'm going to give a quick, a short demo for virtual cluster.

A

In this demo, I'm going to use kubernetes dashboard, the left dashboard connects to the supermaster. We have already installed one virtual cluster in the supermaster.

A

As you can see, there are a bunch of new namespaces showing up this namespace is the root namespace the tenant. Must parts are installed inside this namespace? You have api server, controller manager and etc.

A

The other namespaces are synchronized from the tenant master.

A

The right dashboard, classical canon master, and if you look at name spaces, we have a regular kubernetes name spaces plus a dashboard namespace.

A

The default namespace in the tenant master is empty, so the corresponding name space in the supermaster.

A

It is also empty.

A

Now, let's try to create an nginx service in a tenant master using.

A

A

As you can see, the part is actually created in the supermaster and the part status is synchronized to the tenant master inside in the tenant master. We have a stable set called my nginx, but in a supermaster the safer set is not seen, which means the workload management happens in the talent master.

A

Now, let's try to create another part to see if we can access the nginx service in the tenant master.

A

The part is really simple:.

A

And yes, it is created now, let's try to log inside the pod.

A

You can see the engineering service showing up in the enviro, the environment variable and, let's see if the tenant dns works here,.

A

The service name is mynginx, the namespace is default.

A

A

Okay, that's the end of my presentation. I'm happy to go, live on and answer any of your questions. Thanks.

A

A

Hello, everyone.

A

Yeah, so I think there are some. I hope you guys can hear me. uh It seems like there's some technical issue. The the demo video is not showing up in the screen, um but I think I upload my whole slides uh in the.

A

Thank you. You will now be placed into conference reported.

A

A

A

Is the phone bridge not coming.

A

A

A

Okay, sorry, I think there are some technical difficulties to play with the demo video. uh Let me see how kind I think I need to work with the team to make sure that the record, the recording the session has a video shows up. um Another thing is, uh I think, in the in the in the session page I can I try to if I can upload the um the slides back directly and the video is embedded in the slide deck. So if you guys have time you can just try to play it locally.

A

If I download the ppk, let me see if I can do that after session, but by now I can try to answer some of the questions that some guys already ask.

A

I think mark uh mark just mark caller. Just ask the question: how do you manage the network isolation to actually prevent unwanted traffic? um I think that's a great question for our the in the multi-tenancy area, but, as I mentioned in the flight, so this slide doesn't cover the uh the data plane, isolation. uh Of course, the network solution is in part. I think there are parts they combine, both the control pronunciation and vertical isolation.

A

I uh and I do believe there are a lot of ways to try to prevent the r1d traffic in a multitesting model, but I didn't include it as a general practice best practice, because the network startups in all the different vendors are so different. For instance, in some vendors car vendors use an epc to do. You know a narrow isolation for individual parts, so in real scenario or even the cognitive service, the upstream community service may not work because the cookie proxy has to you know, access the node in a network stack.

A

um Basically, I try to make things a little bit simpler. There are a few things that I can do. First, if you, if we use the vpc model, uh one way that we have tried internally is that we kind of change the kuvi proxy to inject rows into the each. You know parts directly, instead of in part namespace directly, instead of the whole network. Namespace, that's one way you can do uh if you, if you don't use the vbc network, um something that we can do is we can use the upstream.

A

You know network policy to let cra csa cna plug in to do the network. Isolation between the namespaces, the nice thing of virtual cluster is the other tenant name. Spaces are synchronized to supermaster, so exactly that is the one you were mapping between the tenant, namespace and the supermaster namespace. So uh one way that I see some of the vcu user has tried is they can come up with their own network policy, uh create throws policies into the tenant, the master and the supermaster double sink.

A

The policy and the cna plugin can honor those policies um then to uh to provide the network isolation between them spaces. As a as native kubernetes does. uh I hope I address some of your concepts, but if you did have more concerns, let's check later in the slack channel about the network isolation, there is a one question from the kathy uh who asked is: what's your class to use in production by any any users?

A

So currently we have virtual class has been uniting uh some internal, uh the the product, but uh it has not been exposed to any use. Yet so in our model is the service product. So there is a small, you know part layer to encapsulate the virtual cluster. So the the end, the user, doesn't directly access to the virtual cluster, um but I don't think there is a uh a technical difficulty is to impose the virtuality and the user because in theory many of the existing service products kubernetes does all the same thing.

A

uh Each tenant has its own tenant, uh the virtual, the tenant master and uh that's the interface that we can provide um owning additional concerts. That I can think of is the um the privileges, the recovery you think about what are the rights you know service account kind of what are the right permissions that these channels can operate on the virtual cluster, but for not exactly the security perspective, most likely for the stability perspective. You don't want tenants, you you, you know easily to to make some mistakes and break down this virtual cluster.

A

um The tooling wise. um I see some challenges on the tooling. So as far as I can see now, the biggest challenge is the dns um the we have to in kernel model. We have to make slightly change. If you look at our uh github page, I I I explained: how did we just store the tenanted dns? uh That probably just has a one line.

A

Change to the upstream uh coding sg in order include cooperate uh with this model for the to make sure the tenant dns works, um but that'll be we provide a guideline in the github page and they should. It is very straightforward and you can easily understand what I'm trying to do in there in the queue allow the dns work.

A

A

Oh okay, I gotta. The latest question is um uh the question is from way uh the question is. I cannot link the multi-tenant project to solve this concept. So, okay, so do you have any thoughts on it? So let me explain a little bit so, in my opinion, the service in the service contact subject the problem, the whole problem, the the idea is, we will have a node which is shared by multiple tenants. So the the tenant users doesn't have the the ownership over the node resource. They only have ownership to the part resource.

A

So that's the whole idea of having this. You know uh virtual cluster objects. Basically, uh we are allowed in a supermaster, which is we which we should actually imagine the node resource, but the the node resource can be provided by many many tenants so in the older model.

A

So in all the multi cluster models, each channel will have a dedicated cluster but rose battery clusters. Also connect to a dedicated approval, node, so rose nodes can only use by one of the tenants in order to provide strong isolation. uh The problem is the: u. No node utilization cannot be optimized, because if the tenant doesn't use the rules no resource, the nodes will become just idle. um The the whole purpose of this service model is trying to improve the node utilization.

A

Make sure you know the the the hardware resource can be used by many tenants without any uh interference or any. You know, uh isolation concerts between those tenants. So that's why we start. We start this project and if you look at our model, um the node we do expose, unlike virtual coordinators, so the node is completely abstracted. uh We try to you, know maximize the you know the the the node semantics in the kubernetes. So that's why, in our virtual cluster, we have a virtual node object which actually maps to the real node.

A

So you can see the everything that you probably about load capacity, no name. uh What are the parts running in the node? So those information are available so uh from the tenant's perspective, uh but the but the users just um can cannot say that I'm dominating you know that nobody else can use it at this moment.

A

Node is just a part: research provider, there's no ownership for the node. So notice ownership is belongs to the supermaster. So that's uh so that's a that's my view to you know, mapping this project to your service concept. uh I hope I address your content. So if you do have more questions about this, so uh we can check later in the.

A

Channel, um let me.

A

A

Okay, that's all the questions I have got so far. um Let me see if I missed anything.

A

Yeah, I do apologize that the the the demo video is not showing up um yeah. We have some problems. While I do the video recording, uh I hope we can resolve it uh later and uh especially if you have a replay um in the youtube somewhere and that demo will show up. uh Also, I try to you know see if I can upload the powerpoint directory in the project page um all right.

A

It's almost the time and thanks a lot for joining this session, and uh um I I'm going to be online on the select channel.