Cloud Native Computing Foundation ServiceMeshCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Single Mesh vs Multi Mesh- Tradeoffs for Enterprise Multi-Tenant Su... Ding Shaojun & Zhang Huailong

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Single Mesh vs Multi Mesh- Tradeoffs for Enterprise Multi-Tenant Support- Ding Shaojun (Iris Ding) & Zhang Huailong (Steve Zhang), Intel

Service mesh is an infrastructure layer for micro services. It provides functions like: service discovery, traffic routing/shifting, security, observability etc. Multi-tenant is very common in kubernetes clusters. So how to better utilize service mesh capabilities while providing enterprise multi-tenant support in kubernetes clusters? In this talk we will walk you through below items and show you some tradeoffs for different service mesh options. 1. What challenges does multi-tenant bring to Service Mesh? 2. What service mesh options are applicable for multi-tenant support? 3. What does single mesh and multi mesh mean for multi-tenant and what are their Cons & Pros? 4. Performance comparison for different service mesh topology on multi-tenant clusters.

A

Hello: everyone, my name is iris. I'm a software engineer from intel today. I will, together with steve, to deliver this topic to you, hello, steve.

B

Hi alexis and everyone this is steve, a cloud software engineer from intel. I am very glad to address this presentation with aries here. Thank you.

A

Okay, let's first take a look at our today's agenda. We will first introduce some background information about our topic. Then we will tell you what multi-tenant challenges will or will be for service mesh.

A

Then we will go through several options that you can use to to achieve multi-tenant in service mesh same way, we will go further to tell you how you can safeguard the multi-tenant in-service mesh with intel sgx at last. We will do a demo about one hour, our solutions and do a quick summary about all the solutions we have shared in this talk.

A

So first, let's take a look at the background information. So, as you all know, edge computing is very important. Now, especially in 5g area, it provides functions like 5g, slicing packet, processing for cloud native functions and applications.

A

So in the edge computing scenarios the resource is very constant, so there will be multi-users running in the edge. So we need to provide a way to make sure the resource is isolated between the multi-tenant.

A

Also, service mesh is acting as an infrastructure layer in edge and cloud, so you can offload all the traffic management security and the telemetry tasks into the sidecar. There will be also uh there will be a central control plan that will manage all the configuration information for the sidecars, so we need to take care, multi-tenant in-service mesh, to support the edge and cloud scenarios so after we want to. If we want to support multi-tenant in-service mesh, what the challenges we need to address.

A

So we, firstly it's operational isolation, so you need to make sure the service which belongs to one tenant doesn't influence other tenants, the others is config isolation. For example, if you make a traffic management rule, you need to make sure it doesn't influence applications running under the other tenant. The other part is traffic isolation.

A

It means uh you need to make sure traffic targeted to one tenant will reach the tenants correctly and it doesn't reach the non-related attendance applications on the other part is identity isolation.

A

So if there are multiple users coming in the service mesh applications, we need a clear way to identify between these tenants to make sure the correct request has been handled by the correct tenants application.

A

The other part is, you know there are multiple telemetry data in the in the service mesh, so how to make sure all this data can be isolated between tenant is also a problem or challenge.

A

So if we want to mount support multi-tenant in service mesh, the straightforward way you can think out is to make sure every tenant has his own mesh and own cluster. So in this way our isolation will be achieved natively because they are running in different clusters.

A

But, as we mentioned before, in the edge computing scenarios, resource is very limited, so it's always impossible to make. Every tenant has his own mesh or own cluster. So we went further to think about. Can we support multinational in single cluster, but in multiple mesh? So this is the picture that for this solution, so you can see there are two tenants 10 to 1 and 10 to 2. Both of them has their own mesh, but they are running in the same cluster cluster 1. so um because each every mesh has their own root ca.

A

So it means the traffic between them will be isolated.

A

Also in istio there is a discovery selector that you can utilize to achieve service discovery, isolation for mesh in the single cluster, so you can see in the left part when you install steel, you can specify which namespace you are interested to watch to to make the sdod watch. So the the left side is an example that the easter d will only watch name space which, which has the label environment equals id. The right part is a is a sample that for the accounting department.

A

um It's this, but but one thing we need to mention here is actually in each upstream multi-mesh in single cluster. Isn't that isn't supported, because um in the in every name, space easter d will create a root uh root and create a secret to host the root search for all the all the other tenants in the in?

A

So we actually to support this solution. We actually has a patched solution to respect the revision in your service mesh installation.

A

So if you are interested, you can uh find the link about the past solution at the end of the presentation, but this um uh this solution. uh If you want to use this solution, it also has some upgrade issue.

A

So in the future, if you want to upgrade your easter in a new newer version, it will have some problem, so we think it further to see. Can we support multi-tenant in single cluster single mesh? So we need to solve the challenges we mentioned previously, one by one, the first one. It serves discovery, isolation, so in istio it provides a custom resource sidecar, so you can use it to achieve the service discovery. Isolation in the left side you can see there is a sidecar custom resource it.

A

It will allow applications running under namespace 1 to access applications running under namespace, 1, nemesis, 2 and easter systems for all other namespace. It cannot access.

A

So the the right side is, for example, uh in your mesh in even there are not in your mesh when there are multi-tenant um there might be some global service that you want all your attendance workloads to to access, for example, some logging service. So in this case you can define a global, accessible service. Here you can see in the eagles host part it is defined and star star, so it makes makes it a global service.

A

Then, for the config isolation part, you can utilize the export to field in virtual service destination, rule and source entry. So using this way you can achieve config isolation and then the last, the the next part about the traffic isolation. It still has a custom resource which is called authorization policy. So the best practice here you is to first deny all the traffickers between the mesh. um So then, you can gradually allow the communications between the name spaces. So you can see here. Example.

A

It allows um applications running from names, namespace 1, to allow to access applications running under namespace 4..

A

So this is a picture to demonstrate uh other options we just mentioned. So in this picture you can see there is only one mesh there. There are two tenants tenant, one who has who earned the names based team one and attended the two who owned a name service team. Two between the uh between the two mesh between the two name spaces communication is broken um wire. The authorization policy also uh it provides uh config and the service discovery, isolation through export, 2 and a sidecar. But this solution has some obvious disadvantage.

A

So, first, um because you you want, you need to use the authorization policy to to enable the traffic isolation. It will add additional outback filters to your email, sidecar and the other part is for example. If you have new tenants go on to onboard to your service mesh or you want to deploy new workloads for your attendance, then the authorization policy might need to be changed to consume all these new workloads.

A

The third part is that there are. There is no identity isolation. So, for example, if there are multiple user requests coming, there is not a batch, not a good way to distinguish them from the different tenants workloads. The last part is actually the house applications running in the mesh. Even they are belong to different tenants. They are sharing the same root search. So this is almost unacceptable for most tenants because they don't want their their root cert to be mixed with other tenants.

A

So to solve the problem, the first one, the identity, isolation. We have this solution, uh it's called us service, configurator, it's a um intel project, but I put the in github link here. So, if you are interested, you can click here to take a look. So um the basic idea for this project is um because, starting from easter one, nine easter support external authorization and there is a project called earth service. It's a histo ecosystem project.

A

So using this these two combinations, you can define uh different oauth providers for different service according to different service, url or parameter. So this also opens down for us to support multi-tenant.

A

So in our project to support more tenants better, we have defined a custom resource which is called the chin, but in the chain cr different tenant id a different tenant at me can define their own chain crs.

A

Then that there is a chain web hook. It will do some validation to make sure the chain cr is valid. Then, after that there is a off source controller. It will monitor all these chain crs and will combine them together to app and update the chain information into the config map, which will be picked by the os service dynamically.

A

So using this way we can support multi-tenant dynamically as well, then how to solve the problem that uh every tenant has the root symbol to see a so I will hand over to steve to take you through the journey.

A

B

Thanks aries and I will go on with option 3 about the multi-tenant in single mesh in single clusters and at first I will describe the image in the slide.

B

There is one cluster with single mesh and multi-tenant contain the one with service a and tenant to d, with with uh 10 2 2 with devsky, and all these series are under the controls is to in the middle of the image there. There is the kubernetes api server in the right side of the image. There lists some sign solutions for certificates of tenants.

B

The first choice is sign-in operator, which means that you can develop, develop your own operators to doing this to do the signing work and the second choice is about the search manager, which is the open source project, and it has a complementary ecosystem and you, you just need to use it, and the set choice is that you, whatever you, can think about it? But all these these sinus solutions needed to be watching the capabilities csr all the time.

B

So for the next time next step, I will give more details in about this signing process.

B

The first one is that the service of the tenant will send these to cr csr to east to d, and each duty has the rs registries. Authority.

B

Server should name for ra server, and I server will do some will take some actions, including validate, is to csr from the from the service and then retrieve the some information from this to csr and then organize unnecessary information to formal community csr.

B

The necessary information will include such as a service name from the service and some involved variable such as search signer domain info, which in which is configured in the configurations when installing the issue- and I will give them more details later.

B

And the solution then we'll sign to well side the community csr when there, yet when there is a community csr and that is too deep- well be notified and updated. The kubernetes csr once the community sales are had been signed.

B

Oh okay. After all of these first steps, the certificate certificates of the canaans have been already now. Therefore, east udwell can dispatch this certificate to specify the city sidecar.

B

So that's all for the entire process, and you can see that there's. No, nearly some uh advantages, such as different lucy for tenants, can be supported and no additional field to added and no additional config changes. If new work notes added.

B

B

In this slide, you can see that there are some clusters and one tenant such as, for example, tenant a attendant one or tenant two. Each of them have the multi clusters and multi multi meshes.

B

However, all of this service in one tenant can be communicated with each other, because all of them have been signed by the same signers, which created by their signer operations- and this is extensions and and uh and in this situation it can benefit from our solutions.

B

Okay, for the next slide, uh I will introduce about self-card multi-tenants in service mesh and then as we as we know. Now, there are some much tenants and some service meshes, so there are some many additional private, private, private keys for age, 10 and 7 service.

B

So how to manage this private key service? It's I think it's not not popular properly, it's on properties to store them in the configuration files or some config maps and securities like that, so we import in intel software guard extensions. Sgx sgx is a system of architectural enhancements defined to help protect application. Integrate and conf convey confidentiality of data and to withstand certain software and hardware attack attacks, and it can have had the following items for protecting data. The first is protect against software attacks, even if os drivers, buyers and hyper riders are compromised.

B

A second item is secrets.

B

Secrets remains protected even when attacker has full control of all the platform, and this set item is prevents attacks like memory box, looping memory, tamping, temporary and cold boot attacks against the memory contents. In our run. This. The fourth item is provides hardware based as testation capabilities to measure and verify vanity code and data signal signatures.

B

So in our e in our in our solutions, multistate.

B

Sgx will provide a trusted computing enclave where data and applications are protected, independent of the operating system or hardware configuration itself and every tenant will provide its own provider key in their enclaves and increase cliffs will provide some handlers to user to get this rari key and as that, as, as you know, sdx enclaves will help provide more security transfer between processes in an os.

B

So there there there are some very security, accelerate isolations between the confidential data and the the consumed data client.

B

Okay, I think this the next one, one more time, if, if you want to, if you want to know more details, you can click the you can see the more details from the intel homepage and the next time and the next step. I will do some demo shoes for us at first, let's make things either.

B

That means one one tenant with one namespace and each name. Each tenant has two services, http bing service and sleeve service, and I will show I will share my screen for this demo.

B

Yeah as you as you can see before we, I will run the demo demo script before we do something we need to have one clusters, the class in this. In our scenarios there are one muscle node on our connote, and you can see it's ready now and then we we need to make sure that our signer operation solutions are is working well in our in our scenario, we use the set manager and our signer solution.

B

Then we need to at first we need to create the self-signed class issuer of setting manager is a concept of set manager and there, therefore, the namespace bar full and sql system buffer and bar and foo, is our user username space and use the container space, and then we need to install is to before we is before we install a install is to nest, goes through the configuration file for the issue.

B

There are two sections for this for this. The first is the environment. As I have mentioned, there is the environment variable search designer domain, and we set the set uh class issuers set managers as a at the venue. That means we it will. It will use the set manager as a designer and a external ca and in the second second sections about the overlays, we need to update the s2d cluster cluster role is to system.

B

We need to gain more privileges for the set manager to sign to sign them: csr communities, theater, okay, let's install the institute.

B

Installation is done and now, let's deploy in the workloads, as we have mentioned, every every name space had one tenant and with two service services, http service and sleeve service, and the same same things in full name, space.

B

Okay, everything is there now. Okay, now necessarily verify the network connectivity is between inside every namespace. First in in bar, as you can see, we will call we. We will call the http bin service http service, embarrassed from the s sleep bar.

B

We get a response from the http bim bar service, and it is available. It's the same things from in in the full name space flow. We also got got to get the response from the http being full. Now we also supplied.

B

We also support the another ways to do such network connectivity is verified, verification, it's a is to ctl, and you can use this command. You can use this command to do. Such verification is to ctl proxy config, ruci, compile and the ports and name space and two parts and with their namespace, and the result is that both of these service is available with each other.

B

This is the inside. In the inside name, space, another is about the uh across the name space. You can say we check the network connectivities from snip dot bar to the http bing, dot, thoughtful and the result is a connection arrow accurate.

B

And the same thing is the same arrow when we do when we from sleep dot, food to http, dot bar and, let's see, also see the is to ctl result. Okay, they are unavailable. So that's all there.

B

That's all the very file inside the onenamespace and across the namespace.

B

And we also, we also do a summaries for this all of these solutions and in all these solutions we have some uh comp comparative horizons between them and the first, but the the more important in that I also I want to highlight is that the third one assuring the slides in the second options, single mesh, single classroom via c multi, c external also and sjx we have mentioned there- are different routes. Besides the differences you see for tenants, no additional field added, no additional config changes. If new or knows added, we also provide identities.

B

Isolation by the external us, auth, authentic, authoritation and another is the to make sure the more secure, but the the cause is that there is additional customer say needed. So I think that's all for our today's presentation.

B

A

Thank you, bye.