Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio New Workload Identity Provision Pipeline Based on Envoy SDS - Quanjie Lin & Diem Vu, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Istio New Workload Identity Provision Pipeline Based on Envoy SDS - Quanjie Lin & Diem Vu, Google

Istio introduces a new workload identity provision system based on envoy SDS (secret discovery service) from release-1.1; as the main developer who works on this project, my talk covers: 1. Background topics like what is envoy SDS, the motivation why the new system is introduced; 2. High level end-to-end architecture, deep dive into some design decisions we made during development; 3. CNCF projects we leveraged during development (kubernetes, envoy, helm, spiffe etc); 4. Real enterprise customers’ user cases that built on top of this new system in production; 5. How to plug customer CA into the new system for your user case. From this talk, audience will get better understanding of designing/using service mesh’s identity system from first-hand development experience, and how to build a system by leveraging CNCF projects. [Note: I could demo if time allowed]

https://sched.co/MPfJ

A

Hi everyone thanks for coming to our sessions. My name is diem. Can you hear me? Okay,.

A

My name is dim and my colleague John cheese. We are working at Easter still security at Google. Today we present how we use am going SDS API to securely and seamlessly provisioned identity provision for workload in communities so before getting to detail. I try I want to give some background about history, of course, just a quick hand, how many of you have already used SEO like experimental, good, quite good number, and how many using this for mutual TOS see sure not so many, but still good.

A

So, let's quickly go through some definitions. We heard about so bit mesh a lot today, I guess and so very quick one Centon definition. Soviet mesh is infrastructure running on top of networking layer, provide a transparent and language-independent way to automate application network functions, whether is the load balance for telemetry routing and security.

A

So what do you do? First of all, which do is of the soviet mesh one of the popular one. Today I guess, but more than that still is a platform. Israel was designed the way you can plug in any kind of solution for your telemetry, for testing for a second row, etc.

A

It's the okay I mean the the value proposition. We still can be summarized to three categories. Any form of severity is still running on the network layer. So it's able to see everything happened on the network, so we can collect statistic metric and sends up information to the telemetry backend of your choice. You know SEC driver for materials, etc.

A

Thanks of that user, developer can focus on the application within a lousy rather than operation operational agility. This is about how the ability to to route your traffic through your mesh rather traffic through your mesh. You can't do cannery row well, lou boudreau! Well, you can set port for your services.

A

You know you can inject fault to test your impact for into network, to test your service resilience and so on and last but not least, is security. We still provide comprehensive security solutions for your applications. Out of the box. We provide the service to service encryption and strong identity using based on mutual TOS with authorizing authorization policy. You can the fire fight brain as a control to your to individual service or even individual requests.

A

So here's a diagram of each new architecture- you have your service running your pot, a gray box over there, so it made of this stuff is still dropping next to your service or and at the proxy container running next, on the same network, space of your service.

A

That hexagon was there is the leaf and boy I. Can't it's a very lightweight, very fast, high performance proxy running next to the service is able to intercept and filter out the traffic in and out of, your application, provide a layer up to control the traffic to your application.

A

So this is a programmable proxy. You can program it by giving a static configuration data and to employ to load at runtime to load a starting time, but if I can do better than that, it offer a comprehensive set of API discovery, service, API or who is actually as facade and nikuman implemented, service or services provider offers this API to provide the configuration at runtime, and that's where it still came in is toppled by central plain central control, plane pilot. Basically, the plantation of the XDS is.

A

Responsible to send the group of configuration to avoid to enforce you know the security or networking policies that you want to apply mix the receives the traffic. Sorry, the telemetry data came out from employ and send it to the backend of your choice and Citadel is the component that manage key and certificate that power. The mutual TOS is.

A

Take is, is they receive the key and certificate for every service? Allow committee service allow basically, the identity model that is Joe use, distribute that to the service that needed. After that pilot hand, program piloting, Chow employ here. The pieces should be used to connect with other services I/o. What is a subject or name that is you expect from the server to in foster mutual TOS and install secure, naming the short circuit naming.

A

So, let's hold it a little bit deeper into how exactly see today. I will do this for 1.0 before I mean today we are is go right, 1.1 and before 1.1, the key insert is ended by Citadel and mouth into five sacred mount into workload. So basically, City I was set up to listening to the API server for every service. That kerosene equate a key cert self/psyche, sir, and put it back into community secret when a pot is deploy.

A

The cycle. Jackson is another process of that injected cycle into your pot is ironic, ending the history of cattle command of Mir webhooks.

A

It injects the proxy as well as its mouth secret volume, corresponding to the secret that corresponding the surface occurs at running the pot and and then after that, as I mentioned earlier pilot cow proxy to use app and it's a bit the mutual TOS to connect to other services.

A

So the few problem is this approach.

A

First, every time Citadel rotate the key and sort what it does is create a new key, a key sort and put in the secret. The small program running is like proxy container restart proxy in order to reload key third, so it caused a little bit interruptin, not too much but still not desirable. The second problem, which is more serious, is the key, is generated in Citadel and then sent to workload. The other. Why even the secret but is, as you know, is increase the attack surface or comprise a key and third is also about that.

A

The key and solitary star in communities secret mount into the workload as five miles. So it's a container it compromise. The chiesa can be assessed easily. So my colleague will explain how we fix all this problem with SDS.

B

Okay, thank you TM. So later, let me explain the new identity, the workload identity provision workflow that we introduced is.

C

D

B

1.1 so before we dive into the details step of the new pipeline, let's skip some background of what is armed with a secret discover service, since this is serviceman channel. Probably many of you have heard like existing and wait the discover service like LDS, CVS or RTS. Basically, it allows the UM water to fetch. The configuration like listener trust her routing those come from configuration dynamically from the conch opening.

B

Similarly, is the secret. The discover service allows the unwed to fetch back the secret, the private key and certificate information from the SVN server without restart so because we load the key and assert in the memory. So it's more secure compared with the previous fairmont approach. Also, this will allow us to plug multiple CA like today. We already support the Citadel and vote and Google many CA, so you can basically plug your own CA into the new system, so we will cover that in more detail in later slides.

B

As you can see, this is the SPS. Api interface is very simple test: the tube API method. Why is the stream secret? The other is the fetch secret. The only difference between this API is the stream is push the paste and the fetch is poo based. So in current is your implementation. We use the stream secret API because, whenever the significant head get update, the SD observer can push the new Civic head timely to the SDF client.

B

So next, let's take a look step by step about the new identity, our provision pipeline. The first thing you notice from the left side is we introduced a new component in easier 1.1, which is recorded note agent, so this is ACS server. It manages order certificate. The key and a certificate for order. Is your cell car running on the same node? So you may ask, since we already have pallet that running as the like, the LDS sidious palette will generate LDS and cities and RDS and push the CUDA. Is your Celica? Why we?

B

Why we not just use pallet as the SDS the server the should answer for that is for petrol secret. You, as you can see from this model, because the ACS server and ACS client they run on the same VM on the same node. So basically, the STS is over generate the private key, and this private key never leaves this node so compared with put it like pilot handle. This. This approach is much more secure.

B

So let's take a look at the pipeline, so first, when a service join, the mesh pallet is able to detect that so parroted. The first thing is palette will send the listener config and since the Casa config to the said car so can see, and the SBS config basically is inside listener. Config and this classic config, as you can see from the red site, is the central config of the classic config I can see you can see inside the classic config.

B

There is a section what we call the ACS config and in the ACS config we specified the UNIX domain socket so basically the STS client and the HDL server they communicate with each other through the UNIX domain socket. So the first thing is palette will send the SS config to the onward to the is your set card. So after the esau said, car get Stacey's config it will construct. The ACS request. Basically, is the stream secret?

B

If yeah, we just saw in previous slide and send this request to the ACS server, basically is the node agent in this picture. So in this line in this request it will carry our security secure token. So in most case, this will be illuminated. Service account shot.

B

So after the notation catch the STS request, it will generate the private key and at the same time as the server will send the CL request the certificate signing request to the CA. In this picture the CA is Citadel, but, as we mentioned today, we already support the customer CA which how to do. That is what we will cover that in more detail in later slides.

B

So another thing was to mention is the CSL request also carries the security token passed from the SEO said car, so we needed to send this CSL request through the secure Channel.

B

So after Citadel the CA get the assigned certificate request, it will first validate this security top secret token, the Covenant service control against the API server. If this is a wedding chart, sadhana will sign the certificate and send it back to the ACS server and after the ACS server get the signed certificate, it will send the private key and certificate back to the unwitting Celica so that the selkirk get the keys and pair and it can use it to start the mutual commercial test communication with other service.

B

So basically, this is how the workload identity provision pipeline work in easier 1.1. So, as I mentioned earlier, one major benefit of this approach is there is no traffic interruption during the key search rotation? So next I will show quick demo to to show that.

B

Ok, so this is what I record last a Friday. As you can see, we have a kubernetes cluster and in this cluster we have the e Co control plane deployed. Also, we have two sample app installed. They are HTTP and sleep, so HTTP beam is the kind cell app and sleep is the server status app. Both of them have East. The ascetic are injected. Also I config, the certificate rotation happen every five minutes by the way. This is just a photo more purpose. We suggest to use longer certificate of rotation in your production, environment and.

B

Next I will try to get back the certificate used by the server that use the PI the HTTP pin so in this demo. I do that by remote to the sidecar container of the cry nap. So first, let me try to remote to the is your proxy container of the the sleep app.

B

Okay, so now I'll see the Easter proxy container of the console. App next I will use the OpenSSL command to fetch the certificate used by the HTTP VP. As you can see, after I run this command, it will give me the server-side certificate. As you can see, this is the decode format of the server status certificate. So once in words to mention is there to time stamp there when it has the timestamp when the certificate is issued? Also the timestamp the certificate is going to expire.

B

Also, the same field is the identity use the paddle server. As you can see, this service is running in the photo namespace and in Tehran as HTTP ping. Is a service account, so the rotation happen are five minutes by the way on the right side of the screen. I have a test the script, so basically, what this script does is in sending some coke amount from the cryostat app to the service, adapt to the HTTP and activity running in the wire loop and the print out the response code.

B

So before I started the probe, let's make sure the significant er rotation haven't happened yet, as you can see it, co use the auto certificate now. I start keep running this the probe script and, as you can see, the response code is always 200. So let's keep it running for another one or two minutes. The certificate rotation should be happen like very soon. As you can see, it's going to expire in 2 a.m. and the current timestamp is 158, so the rotation should happen.

B

You are two minutes and let's keep an eye on the red side of the screen. Let's see if there is any traffic interruptions happen here during the certificate rotation.

B

Another thing worth to mention is the certificate rotation should happen before it expire. So the server side should be able to handle that okay, let's keep it a run and we will run the opposite again to see if we get the new certificate from the server.

B

Okay, as you can see, right now is the other old time stamp.

B

Okay, I ran to again, as you can see, the time stamp get update, which means now the SES client get a new certificate from the server side and then, meanwhile, on the rest of the screen, the response code is always 200, which means during the certificate rotation, there's no traffic interruption.

B

So before this demo are finished, let's make sure in this environment the mutual TS is enabled between service to service communication. So we can do that by first check the mesh policy, which the authentication policy and we can dump the mesh policy and see that the mutual tears is enabled in this cluster globally.

B

Another way we can double check the mutual tears is it's in a bodies we can dump some on with the configuration, so, for example, I can download, dump the server-side on wood. So let me put forward the unloaded configuration to my laptop and we can see how it looks like.

B

So if I refresh here and I'll search for active listener, as you can see in the filter chain, we do have the TTS context settings there, which means, whatever the server said, accepted the traffic. The well used a certificate to authenticate the client site so which means in this environment the mutual TS is enabled and used okay. So this is the demo part. So I will return back to the to the slice.

B

So in the previous demo we show that there's no traffic interruption, you're in certificate or oh rotation. So next, let's dig into a little more about how does how does certificate the rotation is handled under the hood? As you can see from this picture, the first, the request flow is what we already covered in the previous slides.

B

So basically, the client side will send follow the on with ACS API, send a stream security request to the ACS server and the server will send the CSR to the CA and get back the signed certificate and returned to the said car. So this is the steps we already covered in the previous slides and how we handle the significant rotation is in the server side. We cache all the unwed connections and for each Android connection we cash the signed certificate and private key and that there is a timer job running periodically at the server side.

B

So it will basically iterate all the cash the secret and check if this is going to be expire soon. If it is the server side, we'll construct the error response and send to the sell car and said, hey, we'll reconnect it with the updated hooker. So this is for the scenario that your security token is going to change in each certificate rotation. For some token, the token is is the same, so in this case thus SDL silvertone the necessary new to send the response to the client we just sent.

B

This is our request, with the token to the C again to get back the update, sign a certificate so that it will update with the secret cache and send the update our certificate to the crime side, so that this is how the certificate rotation happened and the hood.

B

So in the previous slides we basically focus on the mutual years between workload and workload, but how about Kanchana. So there are two user case we need that. You deal with the control. A first example is when the SEO said car first start, it will talk to pilot to fetch back the dynamical configuration like listener cluster configuration from pilot, so we need to secure the connection between workload and pattered.

B

Another scenario is the control plane component may need to talk to each other, while example is in East EO 1.1. We introduced a new component called cally, so basically pilot and mixer talk to galley and fetch back the policy CRTs. So we need to secure the connection say between Kelly and pair pallet and mixer. So in today's SEO we still use the old approach, that is, the secret file months on the ischial control plane so which will cause the traffic interruption.

B

That is not the ideal solution and the way we are going to replace that with the ACS so that we don't have any traffic interruption in the SEO control plane, and this is a diagram of how to do that. So, basically, in each of the cell car of Israel, we picked a bootstrap file into the proxy, the second stalker image. So in the bootstrap file we have the onward static config. So basically how how the holder is your cell car start sequences?

B

It will first bootstrap from that static, config and talk to payload French packet and Amiga config and start using the Denton Emma config to perform other functionalities like a routine blah blah. So to secure the SEO controller, we are adding the SES to the bootstrap file so that when the cell first started it will talk to the SDS server to fetch back the certificate information so that the connection used between the co-worker, all the cell, car and pallet there will be secured through the SDS.

B

Also, the workload I can also talk to mixer to send pass. Senator check and report call. So this connection we're also going to secure through the ACS.

B

So this is about the control plane. So another thing worth to mention is how to plug your own CA to the new system. As we mentioned in the previous lies in easier 1.1. We already support her citadel about and Google menu CA as the essential dossier, and there is another PR that is trying to plug the Spile as the seer. So, basically, if you want, you can plug your own CD into the system and how to do. That is quite simple. So, basically, you just need to write your see.

B

A client just handle the CSR request and put a configuration like your CA endpoint. So I put a link in the slice to point you, the sample code and the sample configuration that use the PI Citadel and about so. Basically, you just need to follow this example and write to your own, see a client and the configuration. So if you hit any issues during this integration feel free to open issues on the pill hub, so I and my teammate ever happy to help I.

B

Think that's all we have for today. Is there any questions? Okay,.

C

Thanks for the extensive details, first, the protocol, which the note agent speaks to DCA- is it some open protocol? Is it your own? Is it ACMA? Is it something else in.

B

Today's implementation, we defined our own the interface. So basically, if you want you plug into your say, probably just follow that interface or read somewhat Apter. If you're kinda see I, don't expose ya like that, but it should be very simple: yeah thanks.

D

Who will install this node agent and we mean it which face install will be not even installed so.

B

Today we have a simple in the issue: dot IO the official web website- how to do that so, basically, when ICS and not agent is enabled it will install when there is your control plan is installed. But it's not a bad effort is not an imperative for now. So we have a page SEO total how to do that so feel free to raise github issues. If you hit any issues during the installation.

C

That could be applied on gateways as well.

B

So the question is how we leverage the ACS on you, so you guys get away so mmm there's a so did this session is focus on workload, I'm, not sure if you have attended the previous session that happened before this. We have another session that focus on. How is your English gateway to study the management using the Android ICS, so you can check out the you co to our website. There is a page also. We can check the video later after it's publish.

B

Okay, any more questions.

B

Okay, I think we're good. Thank you. So much.