Cloud Native Computing Foundation EnvoyCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dedicated Infrastructure in a Multitenant World - Carlos Sanchez, Adobe

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Dedicated Infrastructure in a Multitenant World - Carlos Sanchez, Adobe

Running multitenant Kubernetes clusters is challenging, particularly when different tenants require their own dedicated infrastructure. At Adobe Experience Manager Cloud Service we built solutions to provide customers with their own dedicated infrastructure, such as ips, DNS, VPN connectivity,... while running most services in multitenant Kubernetes clusters. We make extensive use of Envoy to run networking tunnels between Kubernetes pods and customer dedicated infrastructure, enforcing encryption and mutual authentication using certificates. This allows, for example, different pods to have their own dedicated egress ip instead of the cluster's, or connections from pods to multiple customer on-premise services using VPN. The solution is provisioned automatically using Terraform, Kubernetes operators and other services. We have previously shown the architecture we built to run this service, and we will provide updates on what worked well, what didn't and the lessons we learned running it in production.

A

Hello, my name is carlos sanchez and I'm here to talk to you about dedicated infrastructure in a multi-tenant world. Why would you care about this? Our use case was. We have multi-tenant kubernetes clusters where each tenant may want to have dedicated networking features like dedicated ips, dedicated network connections or vpn connections. Things like that, and we built this solution on top of envoy and I think it's uh for us.

A

It's been a great success to be able to do this, while keeping multi-tenancy on the kubernetes cluster and creating this infrastructure with with emboy kind of driving it, and it's been great for us. So maybe it's it's good for you too.

A

So let's go I'm a cloud engineer at the adobe experience manager cloud service, that's the product, I'm going to talk to you about today and where we built this at adobe, and my background is a lot of option source work over the years. I'm the author of the yankees kubernetes plugin, I started years ago and I'm a long time open source contributor at open source projects like jenkins, maven, puppet and so on.

A

So brief, introduction about the experience manager is a content management system, digitalizes asset management, digital enrollment and forms and is used by many 14 100 companies. This was even before the cloud service was was created and it was already a distributed java, osgi application that used a lot of open source components from apaches of our foundation, and it has a huge market of extension developers writing code for aem, so adobe experience manager on kubernetes.

A

We took what was already there. We started running it in kubernetes like a couple of years ago running on asia. We have more than 18 clusters already across multiple regions. U.S europe all over the world, and an interesting point is that we have adobe has a dedicated team that runs the clusters for us and for other products.

A

So this is also limiting the amount of customizations we can do to the to the kubernetes clusters.

A

A very specific use case is that customers can run their own code, um so this uh the customers can create their own extensions and run them on the cloud in this in a.m.

A

That's why cluster permissions are very limited for security and we have to enforce like traffic leaving. The clusters uh has to be encrypted.

A

We use namespace to provide a scope on network isolation, quartus permissions and trying to keep every tenant separate from other tenants.

A

If you want to know more about these details, you can watch my kubernetes stock, uh my kubecon 2020 talk and where I last year, when I went through more details today, I'm just gonna focus more on the envoy details and on this networking infrastructure that we set up.

A

So for the dedicated infrastructure part, as I said, customers want to have uh their own like egress ips, because by default all the tenants running on the same on the same cluster would get the cluster igor's ips.

A

Maybe they want to have private connections? uh V-Net, peering private link express route. All these things that the cloud provides you that cloud provider does or they want to connect through vpn to their on-premise or other cloud assets.

A

So we built the solution on top of envoy, we run envoy on virtual machines and we run envoy on both sidecars inside kubernetes and that's a bit of how the whole system works on on the setup or the architecture we have like each tenant. We give them a virtual network and a virtual machine how to scale set and load balancer. These virtual machines are running envoy.

A

The v-net can be privately connected to the customer network, so if they wanted to have this vpn or if we wanted to support express route things like that, we could use the cloud provided features and products to connect this v-net to the to the customer. So the v-net, where the envoy vms, are running the load balancer in front of the virton machines, uh provides the a dedicated public rear sip that is only for those vms.

A

So all the traffic going out of those vms will have a dedicated egress ip.

A

The private load balancer on the on the right hand, side gives the dedicated private but egress ip for all the traffic that is going to their customer network through vpn or dedicated connections.

A

On the kubernetes side, we have the java virtual machine configured with an envoy sidecar as an http proxy, and so the traffic from that via jvm, is going to the invoice sidecar between the sidecar envoy and the envoy in the virtual machines. We have an http 2 tunnel that is encrypted and authorized with mtls.

A

So what are the benefits of for us of using envoy? It is uh for for the java virtual machine. We have a simple and transparent configuration, uh we're using hdb proxy system properties, so this is like java level configuration that will apply to most or, if not all, the http connections that got out of the of the jvm, and we can also support any other protocols, not in http, by using different listeners on the envoy, sidecar and all traffic from that pod into the vms is encrypted.

A

The virtual network on nature allows us to configuration of vpn private connections at cloud level as a service and there's nothing else. We have to to set up or buy infrastructure and anything like that and mtls prevents unauthorized connections and also one tenant connecting to another tenant envoy. So this is how we can isolate.

A

Some of the issues we hit or or the main issue is that we have to manage as one set of certificates for each tenant for side, cars and for vms, and that comes with like rotation, expiration and all the things that we have to do with with certificates which becomes complex over time.

A

How is the end voice sidecar set up, so we have one listener, with a tcp proxy filter for http, on https by using http connect so using it as a hdb proxy that gives envoy the destination where that connection should go for each non-hdb port.

A

We have one listener that hard codes that the destination on the envoy configuration under the tunneling config, and we have one cluster that just points to the envoy load balancer, that is in front of the envoy vms, and we have the tls transport circuit configuration there so quickly through some code uh we have uh the filter chains and uh we are listening on port three one, two: eight, uh where we just get all the http proxy traffic and we send it to the cluster cluster zero for non-http.

A

uh We just put tunneling config host name, what's the destination host, that we want to send every all the traffic that is coming through, that port and on the cluster side uh we just put the in the socket address the destination address uh the envoy load, balancer that it and the port that where the load balancer is listening, so those are the main, the main configuration options and on the tls configuration we use. We point we point to envoy to the sds configuration files for the certificates and the certificate authority.

A

What about the envoy vm side? How is this configured? We just have one connection manager that listens with a connector break, so every traffic coming in and we will check the connect header and then it knows where the destination is.

A

We have one dynamic forward proxy for cluster for all the destinations. We don't have to com to hard code, any destinations, so neither on the listener side or on the cluster side, so on the listener side, http connection manager and port 443, and this is accepting the connections from the invoice sidecar, and uh we also need to configure a bit differently. What https and http is and the braid type is connect. So all the http 2 connections coming in from from one name through the tunnel they get sent to the to the right destination.

A

And yeah upgrade type is connect and yeah. We just set the dynamic forward proxy dns setup and all that.

A

Again for the tls configuration we point and void to the sds config files, um both um for the certificates and the ca and very important. We require a client certificate. We set that request client certificate to true, so it uh it is effectively mtls.

A

And on the cluster side, it's just the dynamic forward proxy. As I mentioned.

A

For sds configuration, we have a separate file because uh this way allows us to reload the certificates we have having to touch envoy and we will automatically read the the new certificate files when there's uh they changed and we set the match. Subject: alt names so we use the sound subject: alternative names to match uh exactly the tenant that we want to allow in those set of vms. So the tenant that is set on this side has to match the tenant.

A

uh The certificate song that has been generated for the for the sidecar, so both sidecar and vmm voice certificates have to have the same subject: alternative name for certificate rotation we use fault agents to generate short-lived certificates, both on the sidecar and the vm.

A

We have the certificate authority store involved and we have a password-less authentication from kubernetes and azure vms to the to vault. So we don't have any. I mean it's using the the apis there.

A

The certificates are configured, as I mentioned, in separate sds files, and this ensures that they are automatically reload uh unchanged. So when the voltage in rotates the certificates envoy automatically notices and the ca is the same same setup just two different files, there are some alternatives. You can use cert manager on kubernetes side to do that, rotation for the sidecars or you could use a spfe.

A

That's another option. We just didn't want to complicate our first implementation, but that's something we also consider.

A

What about employee debugging? This is something that is probably the most useful advice that I can give on all the issues we hit a lot of tls connection errors only show up on the debug logs under the connection component and when there's a tls connection error. The client, like the sidecar side, only sees the socket closing messages and doesn't get any feedback on why it is.

A

The connection is not, is not working, so some errors that you should look for um so on the config side, uh level warning at least this is one, is level warning that you weren't gonna, get a key values mismatch if the key doesn't match the certificate, but all these other ones are only on the debug level, so alert certificate expired.

A

Okay, your certificate is no longer valid. Certificate, verify failed or certificate unknown. It means that the certificate subject- alternative name- does not match the expected one, and you only get this on the debug logs.

A

So if there's an example, if the certificate sun doesn't does not match, uh this is the the log errors. You're gonna get uh certificate, verify failed on the vm side, and this one shows on the sidecar side, as this one does show as a alert certificate unknown.

A

How do we use all this envoy setup so for http and hbbs as a normal hdb proxy for other ports? We just open a port in localhost in the pod, so in the sidecar and it's available for any any container in the pod.

A

On java, we just set this instant properties that are the default ones uh that any or most of the java clients will uh honor.

A

So this point to localhost and the port, where everybody is listening um and there's a specific case on java, like the apache http client ignores, and you have to explicitly tell them- tell the client to use the system properties. So this is something we've we've seen.

A

Some issues with long-running connections and typical connections for database, like where you have connection pooling uh or lo and run running connections, something we have to do is configure max stream, duration and stream idle timeout, because otherwise our connections were getting uh disconnected very early and we had also to increase the load, balancer timeouts, to make sure these connections didn't get dropped but yeah. This is uh something that you can tweak and you should. If you have long running connections the defaults, don't really they drop the connections very early for the connection pools.

A

We advise customers to use validation of connections before use like in java jdbc drivers. This is just a configuration option where you say whenever you pick a connection from the pool just execute this query: to verify that the connection is good.

A

And some tweaks, we have done yeah the max stream duration, extreme idle timeout. These are set up to increase the to prevent the connections for being dropped.

A

On http, you can also proxy connections through this invoice.

A

One case we've seen is like httpd sends the proxy request, with a connect, 1.0, http, 1.0 header, and so that we saw envoy was dropping these connections and we had to set these hdb protocol options except http 1 0, that is by default, uh disabled.

A

So that's something that took us a bit to figure out some more resources you can find about how all this is set up uh very interesting reads: uh the app http upgrades on how these http 2 connections, tunnels, work, tls for all the mtls setup and the double proxy example on the sandbox.

A

Those were very very helpful to understand how emboy would be configured for these use cases.

A

So that's all from me. If you have a use case like this, then I hope this helped you and show you what you could do with envoy what you could do to have multi-tenant kubernetes clusters and also have a dedicated infrastructure like networking uh this. This v-net solution, providing a v-net for each tenant, allows us to use any of the cloud providers um products to set up a vpn. We could set up a dedicated ips. We can set up an expert route, the private connections, anything like that, any cluster.

A

Sorry, any cloud provider will allow you to do that once you have a virtual network and then the configuration with envoy and all this architecture with envoy allows us to breach the the gap between the kubernetes clusters and the virtual machines with ember running on this v-net. So this uh this is one solution that we are running now and it has solved a lot of these use cases for us. So I hope it's it's also useful for you and I think I can answer.

A

I want to be able to answer some questions. Thank you and have a good day.