Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Mutual TLS Adoption Made Simple, Safe and Secure - Lizan Zhou, Tetrate & Jianfei Hu, Google

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Mutual TLS Adoption Made Simple, Safe and Secure - Lizan Zhou, Tetrate & Jianfei Hu, Google

Rolling out mutual TLS to service meshes is challenging. In the real world, service mesh adoptions are incremental. Services deployment are heterogenous, consisting of workloads with or without sidecar, able or unable to speak mutual TLS, on both client and server side. Coordinating the mutual TLS rolling out with service mesh adoption is hard. In this talk, Jianfei and Lizan will explain the lessons learned from the last several years experience. Specifically, we walk through Envoy innovations to address the problem on both client and server side: TLS sniffing on the server side and per endpoint mutual TLS labeling mechanism on the client side. In the end, we show how these techniques combined together to offer a frictionless user journey to adopt mutual TLS safely.

https://sched.co/Zesi

A

Hi everyone: today, we are going to talk about mutual tears. Adoption.

A

So here is the introduction for.

B

Ourselves, hello, my name is rizzan jo. I work at tetroid, I'm an envoy maintainer and also in istio networking and security working group.

A

Hi, my name is jim fay and I'm working at google. My work is also mostly focused on. Is your security and working.

A

Group, so we are going to talk about what is mtrs and why mutual tiers is very hard and what customer challenge is going to face um and based on that challenges, we are going to explain some of our approaches to how to optimize and improve user experience, and in the end, we are going to talk about the uh summary and also some lessons. We learned during this uh api improvement for mts option journey and last section when we have time you're gonna have a q a session.

A

A

um First, we're gonna, explain why what is mts and why it matters um so mutual kiosk is a extension of the tis and it's essentially a security check on both client and the server side.

A

The client is checking the server's uh identity by a x509 certificate and the server is also asking for a certificate from a client. The these we are. These steps is secured to service to service authentication and the transport layer, also based on the x5509 certificate information. The server can derive some client information like namespace and services account to do some further authoritation check.

A

So, if mkrs is good, why what makes it very hard to adopt? Let's take this as an example on the left side, uh we have client and a server a full service, and the traffic are in plain text.

A

After the customer adopts the site car by injecting the envoy to the application, we can now establish a mutual ts by adding this in between the server and client envoy. So this way you don't have to change your client and the server application code at all. You get mts for free everything is transparently.

A

But in the real world, things are not that simple. So if you look at this diagram, you can see on the right hand, side. The bar service actually has v1 and v2 uh two versions, and the v1 version is a legacy release, for example, and it doesn't have the envoy cycle rolled out. So if the client envoy is only sending the mutual ds traffic for the bar service, it's gonna fail as they need to reach out to the bar v1 port endpoint.

A

And similarly, on the counter part of the client side story, we have a similar situation. The client one does not have the invoice card and the client too does have when the server full service envoy only accepts the mts, then the plaintext traffic request from the client one will start to fail.

A

So, to make things even more complicated, you can see both things can happen at the same time. So we have a partial sidecar brought on both clients in the server side, and you can see two of the requests kind of situation out of the four just be fail.

A

So how can we solve this problem? One way to think about is well if you cannot accept the mutual test between the client and server and way unless envoys are everywhere. Why not? We just wait right on the um we start from the plain text, by not doing any of the muted at all on the left side, where you start to ramp up the invoice injection in the your service mesh, and he only starts to do mutual tears once you finish the um cycle route on everywhere, and at that time you track this progress.

A

The then flip the entirely from the plain text to mutual tiers and julio all at once. However, in the real world this is almost impossible to do. We all know the complexity of a enterprise deployment that you always have some kind of corner case, where some team's service would not be able to adopt new technology.

A

Sidecar can just not be put there, you cannot. uh This will never happen. Basically, so um ucl is doing initially is doing some initial api improvements on this. In this example, you can see the full service, both its client and server application have already finished the envy cycle route.

A

So um in that case, we don't have to wait for bar clients to be finished. We can just adopt mutual gs for full service, so we have a two apis on the eq side, while it's called destination rule, which is essentially a api configure. The client envy cluster behavior, for example, like load, balancing like a circuit, breaking and also eco mutual tiers, can be configured here.

A

We have a in this case. We specify the service host as full and the ts mode as issue mutual to let the client envoy sending mutual tears traffic on the server side. We have an authentication policy, and this policy also specified the service as full, and a single boolean flag has to enable the mutual tears.

A

uh This here is our very basic approach, our baseline of how to solve this problem. Now lisa will start to talk about how istio and envoy make some improvements.

B

So in the server-side improvement, this is in the case of the client-side car is partially deployed.

B

We do the ti sniffing that detects the tos and plain text traffic, so, in this case, server will be able to serve both case, whether the clan have unvoiced, sidecar or not.

B

In the istio api, this is expressed in permissive mode and compared to the strict mode. The server only accepts mutual ts config in envoy config. This is shown in the snippets in the slide. Here we have the tos inspector listener filter and we have to define two filter chain with different filter match one matches to the tos traffic and another matches to the plain text.

B

B

So how it works. Basically, the ts um sniffing the ds inspector take a look on the first packet that sent from client to check whether it's a clan halo in tis. If that's the case, the tos inspector will extract the some tos metadata search, whether it's the os. What alpn it is us used.

B

uh What the sni in the client, hello message, there's some drawback of this approach, for example, for some of the server first protocol, such as mysql this case, the first packet won't arrive when ts inspector is working, but the traffic is not forward to upstream server.

B

So in this case, we introduced some timeouts in the listener filter, but it still adds a timeout to the such protocol.

B

When timeout happens, the server side avoid forward the traffic as plain text and then server respond with the first message back to grant. It keeps clients such as mysql works but add latency to the all the other to the time. Timeout.

B

So, in this case, um we we address the two different needs. One is permissive. One is strict, that is the ins. Once you're done with the all row out on the client side, you can switch the tos policy to strict mode. In that case, the server won't. Do the ts won't detect plain text traffic.

B

That means you are now fully secured in the full service that won't get plain text. Traffic.

B

So in istio site configuration this, is this express as destination rule and authentication policy?

B

So once server side rollout finished, the clan destination rule should be still mutual and the permissive should be changed to strict.

B

This also addressed some config complexity in the previous, like the manual config approach, since the install destination rule is an override base in this example, if you have some fine-tuned connection timeout in full service and you apply the cluster while it is still mutual, it won't apply to full service, so it will break there as well. You have to do something like the bar service to have the traffic policy and also you have to apply that to the other subsets as well.

B

This is an uh complexity of the istio destination rule itself, but this makes mutual ts much harder because you have to know about the odd override and applying them correctly.

B

So in the client side.

B

This is for the case that server doesn't have the fully uh sidecar deployed.

B

This is based on the new feature we introduced in envoy, called endpoint.

B

Labeling uh transport matches was added to the cluster config, which defines a can defined multiple transport socket, which basically allows you to specify multiple tos, config or plain text config and, in the end, points that delivered for the for that cluster. Just a label um called mutual ts ready.

B

So when it is true, it matches to the transport socket defined in the cluster to initiate the tis connection to those server workload with sidecar deployed for those without server envoy deployed, it still uses the raw buffer means. It uses the plain text, communication.

B

So combine those two server side and client side approach you can see. We addressed all the situation here that whether client have envoy or not whether it's partially deployed or fully deployed all the server and client can talk to each other for the for the legacy case. Like both side doesn't have envoy deployed, it can still communicate.

B

B

So we start so as a journey of the adoption we start with. You can start with premise save mode by default, and then this happens automatically when you start deploying sidecars to your server and client. Once your rollout is done on this on the service file side, you should be able to switch that to strict mode.

B

Then you are fully secured on that service. Now I'm going to uh pass back to genfe to tell more about the lesson we learned from this.

A

Thanks lisa for explaining what kind of the details and the mechanisms for improving the issue: mutual trs adoption. So now I'm going to talk about some lessons when we learn go through this journey ourselves.

A

The first of the first that service mesh, sidecar raw, doesn't take all at once. It takes time people try new technology gradually and there will be quite some periods when your service mesh consists of both workers, visa and vis outside car, and this could take quite some time. We need to have good user experience in terms of the configuration amount you need to do for you to adopt mutual deals.

A

That brings up to the second point. We one way to look at this problem is that customer can solve this problem on their own if they just reading to write more kind of configuration.

A

For example, one of the alternatives as our compared to our approach is that you can create some subset in the destination rule and the one subset corresponds to the server workloads without any sidecar and one that has the mvos icon. You can also specify the percentage weight between this subset when doing the load. Balancing and, of course this is very manual and introduces a lot of config toy problem.

A

And the next point is that don't forget about security, even you just want to focus the transition and make user experience good. uh That's why we realize there's two modes in the beginning. You need to need permissive and make things very smooth, but once you're done with the second route, since you want to be fully secure, only strict mts is what you want um in the end.

A

One last point is that the real world traffic is really complicated and we hit all kind of corner cases when we design these mechanisms, for example, on the server side, then we are doing the permissive by tr sniffy we found. Sometimes the legacy client can send a https request to the server and if we are just doing the transport circuit match transport tier sniffing based on the transport protocol as trs that will match to the wrong filter chain and trying to terminate a https traffic with eastern mutual configuration.

A

And, of course, since we start to fail um on the client side, we have here some similar issues: um the endpoint labeling, a solution, only works for the endpoint discovery, typed moa clusters and, and we have other kind of clusters like original destination and sorry. This is the type of on the slides that we.

B

A

Dns resolved kind of clusters. Those kind of clusters doesn't go through the end. Endpoint labeling selection code pass in the workflow. So you cannot have this automatically detect endpoint mtrs, ready, behavior.

A

Next is our just to summarize, or we present today um we solved this mts adoption problem by two parts. The first is the server envoy. We make it both support, mtrs and the plain text. This is done by the envoy level ts sniffing in as a network layer filter and on the easter api side we introduce permissive and strict for two kinds of needs.

A

On the client side, we introduce the make the client envoy able to send both plaintext and the mutual ds traffic based based on the server endpoints labeling results.

A

The outcome for these improvements is that we remove the client side destination. Rule configuration totally if you just want to get mts done, and these two things combined together is that you start from the permissive and the auto materials in the beginning. All you need to do is one switch from permissive to strict for your entire mts journey, which is much much more simplified all right and today, that is our today's presentation uh here now is the q, a sessions feel free to ask any questions.

A

C

um I um my name is gandhi.

A

I see a question of posted on the room that is asking. Is it easy to renew the mta certificate in the whole cluster in istio's case, that we have the entire public key infrastructure to responsible for key rotation and deliver it from the control plane to this from the control plane to the envoy sidecar?

A

So the key certificate of the workloads are rotated every for a short period of time.

A

C

Another question supposedly.

A

The original question is if some of the apps use the mts already and then some rely on anyway. What is the easiest way to support this traffic pattern?

A

Plain text will not allow during transition, um so uh on the server side that we uh for the workers that have the envoy sidecar, we have uh the aipm match with instill dash our own erpm, so that that won't conflict with user, their own mps that offer you a support, both the application that has speaking the its own mps versus that we are trying to terminate nts, but uh we cannot control the workload that does not have the mts uh with a car.

C

C

So yeah I can answer the next question: um can use of mutual tears lead to lack of visibility since the traffic is now encrypted? There's no way to do traffic inspection on application traffic? Is there so yeah? There's a debugging techniques still can be used with envoy to inspect application traffic using android debug feature, or you can still try to see what the traffic in plain text between side, car and the uh and the application by inspecting the local traffic traffic, but by inspecting local host.

C

C

Okay, I think we.

C

Okay, there's a follow-up for that one. Can traffic inspection be done for encryption application traffic? um If, if your application encrypts the traffic itself, you should you need to somehow by using application specific logic to export the trs keys?

C

I mean the encryption keys. There's some way to do that and having the chat like um tcp down for any traffic capturing stuff and combine them. You should be able to do the inspection.

C

So, yes, it will be more steps to inspect the traffic when things are encrypted.

C

Okay, we will um on slack for uh more question if you have more um we'll monitor slack for more questions. Thank you.