Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dedicated Infrastructure in a Multitenant World - Carlos Sanchez, Adobe

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Dedicated Infrastructure in a Multitenant World - Carlos Sanchez, Adobe

Running multitenant Kubernetes clusters is challenging, particularly when different tenants require their own dedicated infrastructure. At Adobe Experience Manager Cloud Service we built solutions to provide customers with their own dedicated infrastructure, such as ips, DNS, VPN connectivity,... while running most services in multitenant Kubernetes clusters. We will share how we built our solution making extensive use of Envoy to run networking tunnels between Kubernetes pods and customer dedicated infrastructure, enforcing encryption and mutual authentication using certificates. This allows, for example, different pods to have their own dedicated egress ip instead of the cluster's, or connections from pods to multiple customer on-premise services using VPN. The solution is provisioned automatically using Terraform, Terragrunt and other services.

A

Hello, my name is carlos sanchez and I'm here to talk to you about dedicated infrastructure in a multi-tenant world.

A

I work at adobe and I'm going to talk to you about some of the things we are building there in one of the services, I'm a cloud engineer at the adobe experience manager, cloud service and I'm a long time open source contributor. I started the jenkins kubernetes plugin that allow you to run jenkins agents on kubernetes, but I've been a long time contributor at other projects like apache, maven, puppet and a few other things just so you know I'm gonna do a brief introduction on what adopt experience manager is.

A

This is a content management system um that does digital asset management, digital enrollment forms and it's used by many fortune, 100 companies. This is the product that I work on at adobe and my team.

A

It's um from a technical point of view. It's an existing distributed java, osgi application that uses a lot of open source components from the apache software foundation.

A

It also has a huge market for extension developers that write modules that run in process on adobe experience manager, and this is important. uh You'll see later we are running a adobe experience, manager cloud on kubernetes and this has been launched over a year ago, and this is all running on asia. We have more than 14 clusters right now and growing.

A

Multiple regions, united states, europe, australia, japan and wikipedi. More and also something to note is that adobe has a dedicated team managing clustering clusters for multiple products. So we are one of the customers of this other theme, so we get the clusters and this uh defines what we can do in the clusters right. So we don't have full access to to the kubernetes clusters. Important thing customers can run their own code. We have limited cluster permissions for security, and another requirement is that traffic leaving the the classes must be encrypted for for compliance.

A

So again, it's going to be uh interesting later on, as, as I explained, what we built, we use namespaces to provide scopes. So the new spaces increments, give us network isolation, quartus permissions, and if you want to know more details on what our kubernetes setup, you can watch my kubecon 2020 talk and you'll find there that I went over all the things that we have built on top of kubernetes to to support the am, and this talk is gonna, just focus on the on what we are doing with with envoy.

A

The requirement we got from customers is that they wanted to have a dedicated infrastructure, a dedicated infrastructure boiled down to dedicated aggress ips, so traffic going out of the cluster.

A

They wanted to have a specific ip that is not shared across other customers for multiple reasons, for instance, getting rate limited when hitting different api servers, maybe filtering at the network level in some firewalls or whatever. There's there's a few reasons to do that, and they also are interested in having private connections to their data, centers or other cloud services.

A

Things like virtual networking network, peering private link, express route direct connect, all these services that cloud providers offers offers customers to to set up their virtual networking across across multiple locations and vpn another one, whether they want people to access the service through vpn only or connect to their internal systems through some vpn.

A

So this came as a requirement sort of and we came and and worked on, iteratively on how to how to best support this.

A

The first version we built was running squid proxies in virtual machines that were connected to the internet, and these virtual machines would give traffic going through the squid servers their own and dedicated ip address. So how that looked like um we have kubernetes. On one side, we have bots running adobe experience manager inside the kubernetes. Cluster traffic is going out through the through the load balancer of the cluster, and this traffic goes into a virtual network where we run multiple squid vms skill sets how to scale sets one for each customer, one for each tenant.

A

So the traffic goes from the cluster into the virtual machine, squid servers, and then it goes out of the load balancer in front of the squid machines.

A

The load balancer is both for incoming traffic to the squid and uh outgoing traffic, so that load balancers uh has uh has a private, not a private, a public dedicated ip for that uh auto scale set of virtual machines. So we run out the skill set uh because we don't want to have just one squid server and when it goes down or whatever happens, or if we want to run upgrades.

A

Obviously we want more than one running and we want it to automatically scale on demand if there's a lot of traffic, so again traffic going coming from kubernetes into a load balancer that is in front of a squid servers with vms and then from there. It goes out to the internet through the load balancer that has a dedicated ip address.

A

So that's what gives it the load balancer is what gives it the dedicated ip and there's a skill set for each tenant.

A

The java virtual machines are configured with the squid as an http proxy, and this becomes transparent for the customer adobe experience manager. I mentioned before it's a java application. You set some system properties and then every connection that goes out to the internet is going to use that proxy. So all the traffic is going transparently into the squid vms and then out to the to the internet.

A

We use kubernetes network policies to prevent one tenant to access a squid proxy from a different tenant. So that's how we limit or prevent that one tenant uses the dedicated ip of another tenant.

A

And, as I mentioned, each tenant gets auto skill set and their own load balancer, with the with a dedicated ip.

A

All the virtual machines run into a virtual network that is paired to the kubernetes cluster virtual network, and so this gives a high speed, low, latency private connections between both virtual networks.

A

What is good about this solution? It's a simple and transparent jvm configuration using the system properties that java supports and the v-net bearing the virtual network period makes the traffic all the traffic private.

A

On the other hand, we have some issues like proxy authentication. Authorization is not really well supported on java or other or squid, so we need to use network policies to to prevent one tenant traffic going throughout different tenants, uh squids squid vms, and this complicates a bit the setup.

A

It only works for http and hps protocols and if, uh if somebody by mistake were to use http protocol, uh the traffic would not encrypt it, but this is kind of a problem in general that you don't want.

A

I mean people should not be using http and it doesn't support other use cases like virtual private network or private connections, because it's uh everything is running on on the same virtual network, all the proxies all the vms are running on the same virtual network, because that makes it easy to do the peering to the kubernetes network, but in turn makes it hard impossible to do other other use cases.

A

So we came up with a second iteration using envoy, so we it's a very similar architecture, but we are running envoy on the virtual machines and also running envoy as both side cards to other experience manager, pods containers.

A

So the architecture is similar. We have on one hand the coronarius prox kubernetes cluster. On the other hand, we get, we have a virtual network, and in this time we give one virtual network for each tenant a dedicated virtual network for each tenant.

A

We have a virtual machine, iot skill set as before, but this time running envoy. For the same reasons I mean we don't want one vm to go down, and this this also runs across multiple availability sounds and it can auto-scale. If there's a lot of traffic.

A

Having one v-net uh for each customer allows us to also uh have more open opportunities to grow the the architecture and support more use cases. So each tenon gets the virtual network, an auto scale set and load balancers.

A

On the, on the right hand, side, what you can see is that the virtual network can be privately connected to the customer network, so cloud providers allow you to have v-net peering, allow you to have vpn or any of the private link express routes. All these things at the virtual network level.

A

So you could you can do it as a service. You can just tell azure iws google connect this virtual network to something else that the customer already has, and so this allows us to have all these other use cases that in the previous iteration we didn't have the load balancer in front of the virtual machines, uh as before gives the a dedicated public egress ip to the traffic going out of envoy.

A

So once the traffic is on the virtual machines, it can go out to the internet with a own dedicated ip, something that we already had in the previous version, but we still support, put support.

A

We have a private load balancer that is connected uh also to the customer side network, so this load balancer all the traffic coming in from envoy into the customer network, with whatever of the connectivity options that the cloud provider has. This would give all the traffic a dedicated private equals ip.

A

So, on the previous load balancer, we would get a public ap of all the traffic going into into the internet, and on this case uh we would get a private ip, dedicated private ip for to all the traffic going into the customers network. Private network.

A

The jvm is configured as before with http proxy in, but this case instead of pointing to the vms, is pointing to the to the envoy sidecar running in the same pod. So the am container the adobe experience monitor java container uses the localhost envoy sidecar as a proxy for for the transparent forward.

A

Then the traffic from that envoy, the sidecar envoy to the ember running on vms, we do a http 2 tunnel for all the traffic that is encrypted and authenticated with a mutual tls.

A

So this allows us to do full encryption from the pod to the virtual machines running envoy on the on the scale sets.

A

So, what's good about this solution, we have the same pro as transparent configuration on the java virtual machine using http proxy system properties, so that part is transparent to the customer. We support any protocol.

A

Not just http http is very easy because on the headers envoy can inspect that and know where the traffic is going, but for any other protocol we can support it by adding envoy listeners in different ports in the sidecar container, so the traffic from those different ports can be routed to the correct destination.

A

All the traffic going out of the pod into envoy into the outside of the cluster is encrypted. So we, if there's any mistake by the customer or something that that is not encrypted. This makes it automatically encrypted.

A

The virtual network, dedicated vehicle network per tenant, allows configuration of multiple options on the on the service level of the cloud provider.

A

Vpn private connections express route, private link, all that and is just a matter of configuration on the on the service of the cloud service.

A

Mutual tls prevents unauthorized connections and like one tenant, connecting to a different tenant envoy virtual machines, so something that we were doing before with network policies. We can do it now, just using certificates and it's much simpler to to manage and more secure.

A

There are some drawbacks like vpn and private connections like require that the v-net has a non-overlapping ip range with a customer private network so requires some interaction with the customer to know what are their range of ips and the certificate management may becomes a bit complex because we need one set of certificates for each tenant for both the sidecars and the virtual machines.

A

So we have a certificate authority and we emit certificates for each tenant, and then uh we we have to emit uh for the sidecar for the vms and the different vms and all the pods and so on, and then we have to rotate them, handle expiration. All these things and that kind of becomes a bit.

A

Complicated, how did we configure all this with envoy?

A

Envy is very powerful, um so what we did is uh look at the features supported and we built this solution that on one hand, you have the invoice sidecar running on the pod and on the other hand, you have the envoy running on the on the virtual machine.

A

So from the container running adobe experience manager, we have http traffic that goes into the envoy sidecar as a using it as a proxy. So using the connect header- and this is one listener using tcp proxy filter envoy, tcp proxy filter, both http https and http connect give some void destination. So all the traffic going into the sidecar through this listener um envoy is gonna encrypt it and tunnel it uh on tcp over http 2 and uh using mtls, and it goes to the envelope running on dvm.

A

The invoice running on the vm examines the the connect header and then sends the traffic to the internet through a dedicated public ap for that tenant or to the private network. If that's the case through the dedicated private.

A

App for other ports that are not http, we have one listener for each port and the destination. We have to hard code it into the envoy configuration on the tunneling config part, so this tells we have a set of rules.

A

Basically, if if traffic is coming on port 10 000 and then send it to smtp.example.com port25, and if traffic comes to mysql to report, whatever 1001 send it to mysql.example.com and then on the other hand, the invoice running on the vm then knows what to do with that and sends the traffic to the right place through the right.

A

Interfaces on the envoy configuration we have uh also the cluster configuration and on the pod side on the sidecar. We have just a pointing we just point to the envoy running on the vms, to the load balancer in front of the invoice running on the vms, and we configure the tls transport socket to do the mutual tls.

A

So some examples, some code, examples the listener for the proxy. You can do it like this. It's listening on port 3128 and the filters is just tcp proxy and pointing to cluster cluster underscore zero.

A

Other ports, not http, you can do the same thing, but just in the tunnel and config just put the hostname of the destination. So any traffic coming through this listener is going to end in in that hostname that you are setting that you can put the hostname on the port.

A

On the invoice sidecar, the cluster configuration it's uh just pointing on the lbn points were just saying what, where the envoy vm load balancer is listening. In this case it would be envoys underscore vm and por 443, for instance, and this would go to that load balancer and that load balancer would send the traffic to to envoy and the transport circuit configuration. This is where all the mtls is happening and you set what's the certificate and the private key of that envoy and the trusted cas.

A

So as long as both both sides have a certificate trusted with that ca and boy can can connect to that and envoy will present envoy on the sidecar will present its own certificate to the other side.

A

Now, on the vm side, uh we have one http connection manager listener that has the connect upgrade and uh takes all the traffic coming through through this one port and one port. Only we don't need more than one and on the clusters we just configure a dynamic forward proxy and then this is uh this: handles all the connections, all the oh sorry, all the destinations for all the traffic.

A

We don't have to have more than one cluster the listener side, so listening on port 443 and the connection manager, http connection manager, filter and then what it does is the d-max the multipliers, the all the traffic coming through through the tunnel. So this is configuration that you can find in the envoy examples.

A

uh Basically, it's telling uh the upgrade is connect, so everything that is coming through there send it to the destination set on the connect headers and just a small detail that, if you want to use uh this is a proxy with http, not http with https the other. The first route just is enough: uh the route that matches on the connect, um but if you have traffic coming in through h, uh using http and not https, sorry not coming but encapsulated that were originally for http.

A

You need the second rule that matches the slash and sends it to the to the same cluster. So this is just a small detail, so you just need to this one. These two routes to match both http and https.

A

The filters is just the router with allow connection, allow connect on http 2 protocols and the upgrade type connect just to handle all the traffic that is coming through the tunnel and on the transport circuit. This is the other side of the mtls uh connection, connectivity and and security, so same as before. We set the envoy certificate public key and the trusted certificate authority and a small detail here is that we say much subject: alternative names to a specific subject, alternative name so only certificates that have this sun.

A

In this case envoy sidecar can connect to this envoy, and this is where it's defined, which tenants can connect to this envoy, so you can have certificates with different sounds, and each uh each song can only talk to its counterpart envoy in the vm. So each set of envoys has their own line here, their own match, subject: alternative name and then envoys on the side. Cars have different certificates for different tenants. Different sounds different. Subject: alternative names for different tenants and this is what validates that is not receiving.

A

Connections from somebody else on the cluster side is just a simple dynamic forward proxy. So we don't have to define uh what are the destinations. It will figure it out based on the on the dns name and that will uh just forward all the traffic some resources. If you want to check examples on the envoy, proxy.io uh documentation, site yeah, you can read about the http upgrades where it explains how to do all this tunneling thing. Tls and double proxy is what allo explains: how to do. The mtls authentication between to envoy proxies some advice.

A

Let's say about debugging on on cases like this, the tls connection errors only show up in the connection component, the backlogs. So from the client point of view, you only see that the connection is dropped or the connection is not established and sorry. The connection is closed and you have to go and look into the debug logs debug logs for the disk connection component.

A

Then you will see things like if the certificate subject, alternative name doesn't match the one that the envoy in the vm is expecting. On the other side, you will see things like certificate. Verify failed on the virtual machine side. So this is where you can see. Okay. Why is the connection being closed? You can go to the vm side and if you see this or on the sidecar side, you would see something like alert certificate unknown and then you would know. uh Okay, I made a mistake on the configuration or something like that.

A

Otherwise, envoy is not gonna. Give you a lot of information and in the default uh login levels. So that's it. I hope you like it. So if you are uh looking to do something like this having dedicated infrastructure in your kubernetes clusters that are like shared across multiple tenants envoy is a very nice solution to the networking side things and uh for us it allows us to do this. uh Whole dedicated eyepiece, dedicated virtual network connections dedicated uh like private link, express routes vpn.

A

All these things are basically are put together thanks to envoy, allowing us to do this, encryption, end-to-end and routing of the networking and dedicating infrastructure to to each specific tenant. So it's a it's a very interesting project. uh We I've liked it a lot so far and I hope you get some ideas from from this talk.

A

Thank you and you can find me on twitter at csunchef.

A

Thank you have a good day.