Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Envoy SDS: Fortifying Istio Security - Yonggang Liu & Quanjie Lin, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Envoy SDS: Fortifying Istio Security - Yonggang Liu & Quanjie Lin, Google

In Istio 1.1, Citadel Agent is introduced to dynamically provision x.509 certificates and private keys to workloads through the Envoy Secret Discovery Service (SDS) API. Running on Kubernetes nodes as DaemonSets and standalone on VMs, Citadel Agents improve security by making sure the generated private keys never leave the node and can be securely delivered to workloads via UDS. Citadel Agent also offers flexibility on local workload identity attestation and various adapters to integrate with custom CAs. In this talk we will demonstrate how SDS makes this model really efficient, and citadel working independently from other Istio components for both K8s and non-K8s workloads.

https://sched.co/MPe0

A

ah So good morning, everyone welcome to this session. My name is Tran jelling, and this is my teammates Oliver Leo, so we're both from Google is your secretary team. So today we are going to talk about. How is your leverage on with SDS the secret discover service to fortify? The is your security features, so first I will start have a brief introduction about what is easier and easier security features we offered, especially on the authentication flow. Then Oliver will take over choose talk about in easier 1.1.

A

How we leverage the on with the secret discover service under is your gateway certificate management. So, during this talk, if you have any questions or feedback feel free to, you know to ask.

A

So first, let's have a talk about SEO, so I guess probably some of you have heard or even used easier before. If not, you definitely have hurt like service match or the mackerel service pattern. So, in recent years the mackerel service pattern becomes more and more popular. The advantage is very obvious. We can have smaller teams using the program language that they feel comfortable with and focus on their business logic.

A

Also, we can have faster development and faster deployment. However, this pattern also brings many new problems that we need to solve. For example, if I have an environment like have hundreds of service running, how can do the service discovery?

A

Another user case is, if I have a new feature once you roll out I want you out, like certain percentage of user traffic, to do some AP testing and how to solve the problem. So this is the first, the problem that's easier. Well, so for you, that is the traffic management. The second problem we need to solve is with so many services running in the cluster.

A

It would be great if we have offered the operator essential place to monitoring the service translators if the availability job in a certain service, we could give the operator some tools to dig into the logs and find a problem. So this is the second probe easier is going to solve for you, the monitoring and the telemetry. The third part is we were. What we were folks today is about security.

A

Think about it in your network that you don't have any security feature enabled, which means each service can talk to you, the other service VM or the Potkin compromised. It can steal the sensitive user data. It will be a disaster. That's why we will focus about the issue today. Is your security feature today?

A

So next, let's take a look at the easier. What the overview architecture, as you can see. From this picture, we have two parts: the upper part is the ischial data plane. Basically, in each user service we have the easier said car running by side of it, so this part is transparent to our application developer, because it's your cell car injector will automatically inject the unregistered car. For you, so the Amida Celica after it gets is configuration from the Israel control plane.

A

It can do many functions like it can do the routing or when it routed the traffic to a stream cluster. You can do the out light detector or the error handling. Also it can do the visual tears the tears termination for you. So this part is about the data plane. As you can see from the lower part. We have three major components in issue in it's controlling the first is colored. So basically, a palette has a service registry.

A

It keeps information about all the services service instance and point all those informations in real time in his memory, so that it knows about the service mesh topology at the real time. Also panel is able to accept the policies, for example the security policies and combine the service registry, and the policy palette was able to generate and push the Selkirk configuration to each of the celica running inside mesh. So ahmud already provide the discover, a service like LDS city s, RDS, listener, cluster, routing and point the discover service.

A

So parent will follow the API interface and push the Android configuration to the cell car and SBS is what we will is another API interface that would recently bring that we will talk about detail later and make sure you consider it as a telemetry component. So basically it's a attribute process. Engine order on cell car in the mesh will send metrics. For example, the sauce service name destination subs IP address to the mixer mixer will process those attributes and send it at you.

A

Different telemetry back end like the sector over, so that our operator have a central place to monitoring the service translators. The last thing is the CA, so we have seeded around as the CA in SEO 1.1. Basically, the CA is responsible to assign a certificate and all the components who will trust the dossier in the mesh so that after the guide the signed certificate, they can use the certificate to authenticate each other. So we will cover this in more detail. Later nationalists took like take a look at the easy of security features that we offer.

A

As you can see, there are so many terminologies in this picture, but from a developer's personal view, there are two major problems we want. We want to solve. First is how to secure the service service-connection.

A

The second problem we want to solve is the end-user: how to secure the end user request so for the first problem service to service pro sonication, we first need to have the identity for the service. That's why we have the identity provision workflow, so we basically provisioned the public key certificate for each service running inside mesh. So after the service has the identity, we have the authentication policy so that we can enforce, say: I can have the mutual test enforced at match level or at namespace level, or even at a service level.

A

If I'm in migration, Staters I can configure, my authentication policy accept both the encrypted traffic and the plaint has the traffic that is promising mode. Another thing is access control, so, let's take a step further after we do. The authentication I want to specify some authorization policy. For example, we can read her or engage in policy to specify that we allow service a have read-only access to service, P or service peek have the readwrite permission to service C. That is authorization policy do so. This is about service to service communication.

A

Another part is any user request, so there are two scenarios wise: the end user can access the service inside mesh, for example, operator an area using case is we can have some external user that try to access from outside mesh, and for this case we have the authentication policy are enabled on the issue: ingress gateway. So whenever an user requests coming, we can do the authentication for this end user at the ingress level. If this is a malicious user, the connection will be rejected. Otherwise the request will be forward inside mesh.

A

So this is tube categories or problem. You try to we'll try to solve using the SEO security features. Next, let's talk a bit about the authentication flow, so we talked about the certificate. Many times improve is less. So what is actually is a certificate, as you can see, I put here at ecology. The format of the certificate- basically, it has two things we want to notice here. First is a timestamp, so each certificate is a lifetime. It has issued a lot time stamp and it has a expire timestamp. So, basically one significant expire.

A

We need to rotate the certificate. That's the problem that the Android SDS will try to solve, for you only we'll cover that later. Another field in the certificate is the subject alternative name. Basically, this is the identity of the service. As you can see, it's in the specie ID the service running in the default namespace and the running as the deferred service account. So this certificate is signed by our CA.

A

So all the component in the mesh will trust the CA so that, after the service have the certificate they can use it to authenticate each other. Also, the certificate can be used to do that to encrypt the traffic so that the traffic between service, your service and not print has anymore.

A

So this is the cell car, so we also can have the certificate that under is your ingress gateway, so that whenever any user, the any requests the coming. We can also indicate the backend service use the certificate on the issue, ingress gateway. So next Oliver is going to talk about how we manage the certificate used by the ingress gateway in one is your 1.1 and in our lesson recently release? Ok, thank you.

B

B

Hello, everyone, so this is Oliver. I'm gonna talk about a little bit about their SDS and things before SDS, and also show a demo for using certain manager plus the last encrypt service to provide a certificate. So what's their old dates, the legacy cert management with secret volume mount is what we used in 1.0 and 1.1 by default and in this flow we are relying on the Citadel and the secret volume mount for delivering their its internal certificates to their workloads.

B

So, basically, for the internal certificates, we have a citadel attached, which has their secret, which is their service accounts created in their cluster and for each service account.

B

It will generate a secret and key pair and put that into secret, and the API server will automatically do a super double amount to every of their workloads, so that workloads will be able to access the key answers through the local files and for the external source, the admin order set manager or any progress that you have can write to the API server the secret and put the key and external key under certs into the secret and in the same way the gateway can mount them and use it through the local files.

B

Their drawbacks of this approach includes one performance, which is the Amboy host restart is required whenever you rotate the key answers and to security concerns. The first thing is: their keys are transferred through the cluster network, even though this is secured by POS, but it's do something that we want to avoid. The second security concern is key and certs stored as the files under pot. So if you have like a malicious process, that process might be able to access the file, so what is SDS SDS is secret discovery service introduced.

B

In my it's an API to manage the x.509 certificates and kids for my so that the key and certificates can be dynamically delivered to my and the key uncertificated locations will happen on the fly without restarting envoy, so the flow will be looking like they're following chart, the Envoy. We are load their certificate and key through SDS from the SDS server and the intern. The SDS server will talk to some upstream certificate issuer.

B

In this case it might be Citadel, it might be a cat server or it might be any custom CAS and the protocol that you define here can be customized.

B

So let's talk about especially how the SDS helps us to manage external certificate for the ingress gateways. So this covers a lot of use cases for our customers. They want to say they deliver deployed in Grasse gateway with many of their certificates external certificates, because they are running a couple of different services in the back hand, and it might be a multi-tenant scenario, so in that case they may be able to. They want to be able to like dynamically put more, certainly in the grass gateway and rotate them or remove them at the same time.

B

So how to do that? The me, or whatever asset manager or process, that they're the main ones, can put their eternal key insert into their kubernetes secret.

B

That's the first step and then once the secret is there the gate, linguist gate whale will get their configuration from Pilate, and in that configuration the palette rot how it okay gather SDS configuration, get SD as he answered from this local web address, so that this local web address is actually the ingress gateway agent, which is running as a container in the same pod.

B

The ingress gateway is running, so the gateway knows how to talk to their agent in vasana SDS request asking for the key answered, and then the agent go retrieved the key answered from the secret on the API server, so the API server will check if it has the permission. If, yes, they were returned the secret to the agent. Otherwise about return on arrow. The agent in turn will respond.

B

The SDS request by sending either the key answered or an arrow to their gateway once the Gateway gets the external certificate, the external callers are able to call this endpoint through TOS.

B

To recap, the one pager is brought by SDS includes enhancing the security. The private key never leaves the note, because the private key are generated by their agent instead of some remote service.

B

The private key is not stored in local filesystem, because it's always in memory. It's not through the local files. The volume mount there's no interruption to the traffic, because the search rotation will not require a void to restart, and the last thing is, it holds their certificate management flow from the underlying infra red sensor.

B

We are relying on the communicates secret volume mount which relies on kubernetes feature and now, since we have SDS whatever service that you expose as their SDS server can be used here, and also this agent allows you to customize api's between the agent and your uploads certificate issuer as well. So there are, it makes it very, very flexible for you to deliver the certificate and generate this certificate.

B

So how to integrate with such manager and the lesson quote: I guess a lot of people are interested in this topic. This graph shows you their overall workflow, as I described earlier. Their gateway talks to their gateway agent through the SDS. That's the first part, and the agent will talk to the surf manager right. This time is not, it has server.

B

The subtle manager is a part that is running on the cluster and what it does is it takes a couple of configurations and it will be able to provision the certificates for say, DNS domain, and then it will all be able to send this to their let's encrypt service. It's a bunch of challenges, a description according to their acne protocol and then the last sink rotisseries will basically verify if you are only in that PS.

B

If, yes, it was sent back their certificate generated and signed by public PKI, and then a certain manager will be able to deliver that secret to their gateway agent. The gaiter agent in turn returns that to the gateway through the SDS, to be more to offer more practical reference. I have created a demo.

B

It's recorded sorry yeah, so this demo shows probability less encrypt certificates for an ingress gateway through the SDS and circuit manager. Alright, let's start it.

B

Starting from the very very beginning, we first set up their tcp cluster project right.

B

And then we download the binary currently is one point, one point: seven as default and.

B

Then we add some ham rebel. This is all the typical process that you have to do. We are using its deal and this stage and deploying their inquest gateways with the SDS and as well as their cert manager, take some time by the way, I clipped a cup of select time there, because I want to reduce the time of demo all right.

B

So it's deployed, let's see the ingress key, so the surf manager here, the inquest gateway, has two containers, even though it's waiting for the other container to start up, but you can see it has two which means one of them is in my the other way as their agent and I am looking at their external IP for the ingress gateway. Have that IP I will record it.

B

That's the external IP I want to make that external IP permanent, so I do this create the address and then it will be preserved for me, I won't lose it alright, so now, I have their external IP and I will create the demo namespace to deploy a demo app. That I will interact with. Basically, this demo app just shows me the headers. It receives it's very simple.

B

All right have the demo namespace and get the part it's HD living.

B

Now I will deploy the gateway. The Gateway now has 80 and 443 two parts, one for I secured, the other ones are secured and in for the secure, while it referred the secret to use for the virtual service, it will check their host and also for that host. It will redirect to there with internal service, which is their HTTP, be I will apply the virtual service as well. Now the Gateway is configured right.

B

The for Post report will not work because the certificate is your Appetit.

B

Now what I'm doing is I will register a DNS domain here, I'm using Oliver org alivelu top org, which I provided through GoDaddy and I have configured that to use their name service servers of their DCP.

B

So you have to go. If you do that, you have to go to whatever your provider and configured the name servers to be point to the name. Servers set here added a a record which is pointing to this IP.

B

This is called demo dot, Oliverio, dot, org all right and then I added another one which is a cname to redirect other sub domains under demo, dot or liberal dildo org to demo dot. Only with your daughter org. This is not actually required in this demo, but I.

B

Just anyway did it so remember this name servers you have to add that to your provider, yes, provider and then I wait a few minutes and then I checked their DNS, its propagated to most of the DNS servers and then I check, okay for their HTTP I'm able to actually get to this endpoint, that's cool, so the HTTP part works and then what what I will do is I will create a service account for their cert manager, and the cert manager will have that JSON file for the service account.

B

It will use that to talk to their less encrypt less encrypted will have that service account and it will use that service account to manipulate the DNS server here under Google, so be able to verify that you own the DNS server. You you own the DNS name so I create a DNS at Amin here row so that they're less incra to I'll be able to use a service account to access the DNS configuration I download. The key.

B

All right, that's a serious amount. The key is there major part, so what happened to it? I will copy that key here. So Jason are the first one is the key. Second one was created like before.

B

I will rename that as a cloudian ascii table Jason right so I have the key now I'm going to configure the search manager.

B

I. Add that key into the secret of the cluster, so that the certain manager is able to access it.

B

See the kids see it's here and it has a special metadata which is indicating the project ID. Okay, here's their. Let me pause a little bit here: here's their configuration for their search manager, it's the back aides had acne protocol specified and that's their let's encrypt server address and the this email is their account in their last encrypt service, a user private key secret graph here, using their Athena's 0 1 standard in let's encrypt.

B

Basically, it called it has the provider which is cloudy s and then refer to their secret, which is what I put here a cloudy areas that kid of Jason in their cloudy a secret right. The project is called Q count. 19 demo.

C

B

B

And I create her secret, so the secret is basically like this and it was a little bit here. Secret name is: let's encrypt demo: that's the secret! That's going to load further four four three part under ingress gateway, the usual reference is, let's encrypt that DNS. It's not very it's for here, common name in a certificate as well as right here, PS names as specified come here. Those are to be appeared in their certificate and then acme domains as we all. It will verify those domains for the certificate right.

B

See how much time? Okay.

B

So after I deploy the secret, the surf manager will start to work. It will try to create the secret and then talk to the last encrypt to provision the certificates see I was checking the secret status. It created the order for resource. It also creates a secret what it does it. It generates our key locally and then the cert is still empty, because it's waiting for their certain manager to deliver that search for itself.

B

Okay, then I'm checking the log of the cert manager. It takes some time a few minutes to do that and if you check their GCP, you will see that txt type record here which is sent by their let's encrypt for the challenge. Ok, now we have the certificate provisioned and I can see from the log here fans that has changed for a certificate demo, cert condition already forced to true. So this means I have the certificate.

B

Let's check the secret again, the TOS cert is written. The CSR is not important here because it's only used for meet with us. Currently we are doing this demo. We are only using TOS. So let's change this to HTTPS to verify. If it works. Ok, it works, see the deadlock and then we check the certificate. The common name is star dot demo, oliver north of cork, and it's just created by the way I created this commanding. So the time stamp is on Monday subtitle turned him name being as type correct demo, dot or w.org.

B

Alright, that's it for the demo.

B

I guess that's it. We have about three minutes left any questions.

B

C

Of the agent yes for pushing the SDS forces, which is what the rest sto does it pulls it from Pilot or pushes has pilot push it. So what was the choice of having an agent there versus oh.

B

Right the pilot pushes so the pilot pushes their IDs and XPS all the information for basically not related to the secret part, but it will have a reference there for there, the address and their name of their secret that you want to get and for that cluster definition. You want to talk to that.

B

For example, you want to talk to that address, to get this secret and that secret uses is used for retrieving their key answered and its interaction, and then once their envoy gets that information, it will okay, look at that access that and point that address and then Sunday SDS request to it and retrieve that certificate. So that's this works in a whole like yeah.

B

Other questions.

B

Alright thanks everyone.