Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Certifik8s: All You Need to Know About Certificates in Kubernetes [I] - Alexander Brand, Apprenda

Description

Certifik8s: All You Need to Know About Certificates in Kubernetes [I] - Alexander Brand, Apprenda

Certificates are an integral part of a secure Kubernetes cluster deployment. They are mainly used to secure the Kubernetes API server using TLS, but certificates (and keys) are also used for other cluster functions such as client authentication, encryption of secrets, TLS bootstrapping, and the generation of service account tokens.

Certificates pose interesting challenges to cluster operators. What does the certificate setup look like in an ideal scenario? How long should certificates be valid for? When nearing expiration dates, how can certificates be rotated to ensure the cluster remains operational? These challenges must be understood when it comes to deploying and operating a Kubernetes cluster.

After this talk, you should have a better understanding of:
- How each cluster component uses certificates for secure communications
- How certificates can be used for authentication, including service account tokens
- How the Kubelet TLS bootstrapping process works
- How to plan, generate and deploy the certificates required for a secure cluster
- How to rotate certificates that are nearing their expiration date

About Alexander Brand
Alex works on the Kismatic Enterprise Toolkit at Apprenda, making the deployment of production Kubernetes clusters easier. He has been involved with Kubernetes and related projects since early 2016. Before Apprenda, Alex attended Queen's University in Canada, where he majored in Biomedical Computing.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So thanks, everyone for coming super excited to be here. My name is Alex brand I am going to be talking about certificates and how we use certificates in kubernetes and all you need to know about certificates in kubernetes.

A

The only caveat is that I thought I would have enough time to talk about everything certificates, but half an hour is really not enough so that all there should really have a star and what we're gonna be talking about just a smaller scope, we're specifically going to be talking about how to use certificates to produce a secure, kubernetes cluster. So just to give you a bit of background about myself and about aprender, we've been working with kubernetes since early 2016.

A

When we got involved with the community, we actually created the sig windows, so the special interest group, that's leading the windows implementation of kubernetes, so you can run Windows containers on Windows and then I personally have been involved with maintaining our open source project called the cosmetic enterprise toolkit, which is a set of tools that allow you to manage Orchestra, operate and bootstrap kubernetes clusters on Prem or on the cloud.

A

So our initial implementation of cos Matic was released in November 2016 when our back wasn't quite there yet so our security model, we were secure from the from the get-go, but our security model was very, very simplistic. So every single component had the same identity. All the users were using the same, the same users.

A

So after our back came out in April 2017, we actually decided to adopt it because we thought obviously it's a it's just a better model, so we adopted that model and that meant we had to just revamp how we generated certificates in cosmetic. So this talk today is going to be informed on those experiences and what we had to do to revamp the generation process and how to produce a secure cluster.

A

So this is the agenda, as you can see, there's a bunch of stuff in there. Hopefully we have enough time to go through it, but basically you know we'll talk about a couple things in that you need in kubernetes and that are related to certificates to have and to run a secure cluster, so I know we're in a security track, but I just thought. I would do just a quick certificates, refresher just in case yeah, so the certificates basically allow parties in a conversation to authenticate each other.

A

So if you think about a client and a server relationship, the client can authenticate the server and just make sure that it's talking to who it thinks it's talking to and the server can also authenticate the client using certificates. So this is all this all depends on the on. So one way of doing this is using a third party that both the server and the client both trusts. So if they don't know each other, they don't have a previous relationship.

A

They can actually trust a third party that issue certificates to be able to trust each other. So that is where the certificate authority comes into play. So the certificate authority is actually going to issue certificates to all these parties and given that they trust the certificate authority, they can then trust each other.

A

So, if you think about when you access your bank's website, for example, you want to make sure that you're talking to you're actually talking to your bank servers and not some random server on the internet. So what you do is you ask your bank for a certificate and then you validate that certificate and given that that certificate has been signed by an authority by a certificate authority that you trust, you basically trust that you're talking to the the banks server, so that takes us to kubernetes. Why do I need certificates in kubernetes?

A

Why do I even care about certificates in kubernetes? So how.

B

A

You are running a kubernetes cluster or in an Operations team. Very nice. That's awesome to see how many of you are familiar with how certificates are used in kubernetes and exactly awesome, very cool. So, as we all know, kubernetes is a distributed system. This is a very simplistic diagram of some of the interactions between all the core components in a cluster. So we have the API server in the middle.

A

The API server is the only component that talks to XE D and then the API server has a bunch of clients, including the scheduler, the queue proxy, the control, the manager and the cubelet. These are basically the main, the core components of your cluster and given that all these components are running on different machines that are connected through a network, you just want to make sure that all these interactions are secure and that all these interactions are basically all the components can authenticate each other.

A

So I'm gonna walk you through how we can build a secure cluster using certificate. So the first thing we actually need is a cluster certificate authority, so the cluster certificate authority is going to be the trusted root throughout the entire cluster. So all the certificates that are used in the cluster are going to be signed by the cluster CA, and this is what's going to enable all the components to be able to authenticate each other.

A

So all the components are going to trust that CA and then, whenever they they're presented with a certificate, they actually can trust it because they've been signed by by the CA.

A

Ok. So now that we have the cluster CA, we can actually start securing some of our interactions. So the first, the most important interaction is the API server. As we all know, the API server is the main entry point to the entire cluster.

A

So we want to make sure that we're exposing this endpoint or the server over HTTPS and to do that we need a servant certificate and the corresponding key and as you as I mentioned before, the certificate is going to be signed by the cluster CA, which is going to allow all the components that are using or that are communicating with the API server to be able to authenticate it.

A

So there is an important gotcha here with with certificates for the master, and that is that, when you're setting up multiple API servers, when you want to have an H a cluster, you actually want to make sure that the load, balancers IP, address and DNS name is part of that certificate, because otherwise, whenever a client tries to talk to an API server through your load balancer, the client is actually gonna. Try to validate that certificate and it's going to complain.

A

It's gonna say you know what the common name that's on the certificate is actually not what I'm trying to talk to. So it's going to complain. So, just whenever you're setting up multiple masters just make sure that the certificates have the load, balancers, DNS, name and IP address. This is usually taken care of by your favorite installation tool. They usually have a configuration parameter to just set extra extra extra names in the master search.

A

So there's actually another API, which is really an internal API, is just an internal private API as of now, and that is the qubits API, which you also want to expose over HTTP and then again to do this. We need a servant certificate and the corresponding key, and this API is mainly consumed by the API server when it needs to get logs or when it needs to get metrics from a specific container and also when it when it needs to exact into a pod.

A

So again, this cert is signed by the cluster CA and that's how the API server can actually authenticate the cubelet and and make sure that it's talking to the the qubit similar to the API server, the cubits API, is also protected by authentication and authorization. So this means that clients of this API also have to present client certificates. So whenever the API server has to talk to a cubelet, it actually needs a client certificate, and that's why the API server actually has a client cert 4 we're talking to the cubelet.

A

So, as you know, as you'll notice, I have little links at the bottom. So if you want to look into it more there's some more ducks on it. Most of the slides having.

A

Coolant so we talked about how to how a client can authenticate a server. So the next piece is: how can a server actually authenticate a client? So that's where the client cert authentication strategy comes into play, so the API server and the cubelet actually can authenticate clients using this strategy. So this is mainly used by kubernetes components, although you could use it for user off. Although there's some important caveats, but the way this works is that any request that presents the client certificate that has been signed by the cluster CA is going to be considered.

A

Authenticated part of that authentication process is to obtain the user information and the group membership of that user, and that information is actually obtained from the certificate itself, so the user is obtained from the common name field of the certificate and the groups are obtained from the the organization field of the certificate. So, as I mentioned before, this authentication strategy is mainly used for authenticating kubernetes core components, so each one of them is going to have its own client certificate because each one of them is going to have different access levels to the cluster.

A

So you want to have each one of these components to have their own identity and the way to do that is to have certificates with different common names. So, for example, the scheduler has its own certificate, with the common name set to system cube scheduler you'll notice that the cubed cert is actually special and the main difference there other than it belongs to an organization. Is that the host name where the cubit is running is actually part of the certificate.

A

So that is very important, because you want to make sure that each cubelet on your cluster has its own identity. Why is that important, so beef, so that? So the reason for this is that it allows you to enable a couple features in kubernetes, which are called the node authorizer and the node restriction admission plugin. So what these do is that they actually limit access to the API server from the cubelets perspective. So without these features enabled the cubelet is actually capable of seeing and writing and modifying every single resource on the API.

A

So you- ideally you don't want this. You want to follow the principle of least privilege, so, instead, what you want to do is by having each cubelet have its own identity. You can enable this feature that will actually limit the access to the API to resources that are actually related to that node or to that cubelet. So, for example, instead of the cubit being able to modify every single node resource in the cluster, it can only modify its own resource in the cluster, similar with secrets.

A

Instead of being able to access every single secret in the cluster, it can only access secrets that belong to pods that are bound to that node.

A

So we just went through all the certificates that we need and how do we actually produce these certificates, so we can either manually create them, so an admin can go ahead and produce all the certificates issue, all the certificates that we need or there's actually an API in kubernetes that allows us to generate certificates.

A

So, as of recently kubernetes offers an API that allows you to request certificates through just you know, using the regular serger, Brunetti api's. So the way this works is that a client creates csr or a certificate. Signing request and shoots is all shoots it off to the API server once that csr is created, it actually is going to remain in a pending state until someone manually approves it. So the idea here is that you don't want to actually randomly approve and generate certificates, because these certificates are going to be signed by the cluster CA.

A

So, as we talked about before, every single component in the Buster trusts that CA, which means that is going to trust any certificate generated through this API, so the CSR actually has to be approved for the certificate to be issued. So once that CSR is approved, the certificate is issued and the user can actually download it. So I actually want to show you how this works so yeah, let's dive into a quick demo and I, have a kubernetes cluster running here on my machine.

A

It's a 1, 8, 4, cluster I, think yeah 1, 8 4 and the first thing I need to do is to generate a CSR, so I've actually generated that previously just in in the interest of time- and you can see here that I have my certificate request and what I have to do is to actually send this over to the API server. So I actually have to wrap this in a CSR resource in the kubernetes api.

A

So the kubernetes api has a csr resource and you know similar to a pod or any other resource in kubernetes. You add some metadata, so I can call this CSR and set the name to cube Khan and then I have my spec and the most important piece here. Is the request field there and that's where the CSR is going to be included in this resource so that the API server can actually store it? So I'm gonna go ahead and copy this.

A

Yeah yeah I'll, take questions in here and I can just post this to the API server and you'll notice here, I'm using cat, but it'll just read the CSR and base64 encoded and then just include it in the resource, and then there we go so I've just created. My CSR and I can actually get these this resource, similar to how I can get any other resource and the API so I can go and do cube. Ct I'll get CSR and you'll notice that it's in a pending state.

A

The other thing to key to keep in mind and to and to learn from this is that there requester of the CSR is actually stored as part of the resource, and this is important for what we'll talk about next, which is how the Cuba can use this API through request certificates. So I've created my CSR and again, I have to approve the CSR, so I'm wearing an admin hat right now and I can go and approve the CSR.

A

So the CSR has been approved and if I get the CSR again, the condition has changed to approve and the certificate has been issued. So as a user I can go ahead and download the CSR. So if I actually describe or get this resource in yamo you'll notice that the certificate is now part of the resource, so I can actually extract this the certificate using adjacent path.

A

So that's in the status field, a certificate field, yep I gotta get my coupon cert CSR, and that is basics before encoded, so I can actually base64 the code and there's my certificate, so I can actually start using this certificate. So you can imagine building workloads or building controllers that live on top of this API and that's one of the interesting things of kubernetes that you can actually use all these API is to build stuff on top.

A

So that's the quick demo of how this process works, and now we can talk about a specific use of this API and that's how the cubelet can actually use this API to bootstrap itself and to rotate certificates when, when they're it certs are nearing expiration.

A

So as we as we talked about before, the cubelet needs a client cert to talk to the API server, and it also needs a serving certificate for its own API. So, instead of having the admin having to generate all of each of these certificates, one for each cubelet and all the other serving certs, the qulet is actually capable of requesting certificates when it starts up. So this is again.

A

This is built on top of the certificates API and also the bootstrap token Authenticator, which I won't go too much into, but it's basically a different Authenticator that that authenticates clients based on a short token and these tokens are basically used during the cluster bootstrapping process.

A

So this is how this process works. So, let's assume that we're bootstrapping a cluster and there's an API server and the controller manager. It's all there all the way running, so the first worker node is starting up and the qubit is registering with the API server and it's going to realize that it doesn't have a client cert. So the first thing it's going to do is it's going to create that CSR resource that we just learned about, and it's going to use the bootstrap token that is used for bootstrapping clusters.

A

The importance about the bootstrap token is that's is that it's going to tie the request back to a cubelet so, as we saw before in the demo, part of the csr resource is who requested the csr. So in this scenario the the requester of the csr is going to be a cubelet which is going to be important in this flow. So once that csr is created, the API server is actually going to issue a watch event to one of the controllers that what that's watching the CSR's and there's a controller.

A

That's going to check whether the csr was requested by a cubelet, given that, in this case it was the it's going to automatically approve the csr. So as we saw before, CSRs have to be approved. So there's a controller that automatically approve CSRs that come from cubelets and then there's another controller that is watching CSRs that once there's a csr that's been approved. It's actually going to go and sign the csr to issue the certificate and it's going to go ahead and upload it to the api server.

A

Once the certificate is up in the API, the cubelet can actually go ahead and download that certificate, so it can go and, as we saw in the demo, we can go and download the the certificate using the API.

B

A

Going to after after that, it's going to be able to use it for further API access, so just to summarize the most important steps. The first thing is that the Keele it creates the csr using that bootstrap token. Then there's a special controller called the csr approving controller that is going to approve the csr automatically, because the request came from a cubelet and then the csr signer controller is going to go ahead and sign the csr, which will then allow the cubelet to download the certificate and start using it to talk to the api server.

A

So that's how the qubit gets the first certificate or that's how the cubelet bootstrap bootstraps its own certificate, but what about rotation? So you don't want to get paged at two o'clock in the morning if, when your cluster dies, because all your cubelets fail to talk to the API server, because the certificate expired, so instead of you know getting paged the as of kubernetes api as of kubernetes 1:8, the cubelets can actually request a new client certificate when the one they're using is nearing their expiration date.

A

So, similarly, the cubelet can also rotate the serving certificate and that's actually an alpha right now, there's a couple of kinks to work out but yeah. If you want to share, if you want to follow the progress on that issue, I, like the issues down there.

A

So that's basically how we can create a secure, kubernetes cluster deployment. That's how certificates are used at the kubernetes component layer, but, as you might imagine, certificates are also used for different things on the cluster.

A

Once you have your your running cluster, so one of those those things which is actually which is actually a missing as something that's missing, is the ability to revoke certificates using certificate, revocation lists, so just something to keep in mind if you're, using certs for user off, you currently cannot revoke certificates, so they're always going to be they're, always going to be considered valid until they actually expire.

A

So if you want to learn more about this or if you want to join the discussion or if you need, if you think you need this feature for whatever reason, definitely check out this issue on on github, so.

A

Today we talked about certificates at the kubernetes component level, so how the components themselves use certificates to authenticate each other, but what about workloads? So if you want to expose workloads over TLS, you can actually do so using ingress. So ingress has the ability to expose services that use TLS and the way to do this is you first have to define a secret that includes a certificate and a private key, and then you have to reference that secret in your ingress resource.

A

So, what's going to happen, there is that once your ingress controller, realizes that there's a new ingress resource defined it's going to go and use that secret to configure that TLS listener.

A

So if you, if you don't want to do this manually, there's actually an interesting project called cube Lego, which can actually automatically generate all these certificates using the let's encrypt API. So if you want to check that out, that's pretty cool and then, similarly at the workload layer, there is actually a working group that is working towards allowing containers to prove their identity outside of the kubernetes clusters. So inside the cluster. Each workload has a service account which they can use to talk to stuff that's running on the cluster, but what about accessing external services?

A

So there's? Actually a working group called the kubernetes container identity working group, which is working towards this idea of being able to use some mechanism that is going to allow containers running on the cluster to be able to access external systems that are running off the cluster.

A

The other potential use case, although there's a bit of overlap with it, something like Sto is potentially kubernetes could do service to service. You know mutual TLS everything out of the box. So if you want to learn more about this, these are the notes of the working group and they have weekly meetings. So I definitely encourage you to check those out so just to quickly summarize what we talked about today, we went through how we can produce a secure, kubernetes cluster and how certificates are used to create a secure cluster.

A

So certificates are key to the function of the functioning of a secure cluster and the main reason for this is that kubernetes again is a distributed system and you want to make sure that all the components are talking to each other in a secure manner and that's exactly where certificates come into play. So certificates allow all of the components to authenticate each other and to establish mutual TLS, and that is what's going to produce the the secure cluster. We also talked about the API that kubernetes offers for generating and requesting certificates.

A

So we we saw a specific use case of this, which is the cubelet and how the cubit is capable of requesting and generating, and we and rotating those certificates and again this is an API that can be consumed by anything other other than just just a cute look.

A

So this is just a quick table that gives you an idea of all the certificates that are in use in a kubernetes cluster. Obviously, at the component layer, so I mean we talked about all the the API server. The controller manager, scheduler cubelet queue proxy and I hope. This gives you an idea of how important certificates are and how careful you have to be with certificates. So again, you don't want to get paged at 2:00 a.m.

A

in the morning when one of these certificates just expires, and your entire cluster is down so I hope that gave you an idea again of how certificates work, how they're used in kubernetes and if there's any questions, I'm happy I'll be around and how did it take them.

B

A

No, so that's a great question, so the actual component that signs the certificates is the controller manager. So the controller manager does need access to to the private key of the CA.

A

Yes, it's fairly sensitive, so it really depends on how you're setting up your cluster. So some installation tools run the controller manager as a workload on the cluster. So you could use a secret for that or something like that. Yeah yeah, yes,.

B

A

So the the cubed API.

A

Yeah, so the Cuba needs a client search to talk to the API server. But then, if you want to talk to the cubelets API, you also need a client cert yeah. So that's a great question. So all the issues that I've seen around the cubelet API, the cubed API, is really an internal API right now it's undocumented. So it's not really maintained or yeah an API. That's going to remain stable, so the recommendation is to not really depend on it, but I think there are people that are that are using it. Yeah.

B

A

A

Right so when you you're on your bootstrapping, your see it when you're sending your first csr I think yeah so you're using a bootstrap talking at that point, but I don't think it'd be encrypted.

A

That's a great question: I'm, not sure I can definitely look it up and come back to that. Yeah.

B

A

So the the majority of clusters that at least that I've worked with have a single CA. So there is one CA.

A

So the CA is a single CA in the entire cluster, the the yeah I guess so. You're talking about signing the certificates when they're requested.

A

Yeah in technically the controller manager, but yes, the the kubernetes api and the yeah exactly yeah make sense. Yeah.

B

A

The CA continues to be the same CA, so there is one CA for the whole cluster yeah yeah.

B

A

A way to automate the generation of keys, the private keys, so that really so, for example, in the community, there's cube ATM that takes care of all of this. So if you're thinking of creating a cluster cube ATM is a good way kiss Matic. Our tool also gives you that capability and it gives you a couple other things. So we take care of that. We take care of all of that, for you, so you're gonna have to do that. So yeah yeah.

A

I think so I'm not super familiar with that right now, but I believe there is an effort to integrate with external key management. Software I believe there is some integration with vault there's I there actually is. You can actually use vault to store secrets today, I'm not super familiar with that so I yeah. Oh there's the next talk there we go yeah.

B

B

A

For the Masters, so the master certs are actually not generated using the certificate API, so you actually have to create them beforehand, right yeah, so either manually or the installation tool actually takes care of that for you. So, for example, kiss Matic will take care of that or cube a the M will take care of that, and then you also write yeah. You set the Sun for the LV. Exactly yes, sir.

A

That is a good question: I, don't know yeah I'm, not sure yeah, yeah, yeah I would expect them to be after after they're created. Yeah I would expect them to be clean and clean up yep. Yes, sir.

A

So it really depends so we we actually issue in cos. Matic, our CA is I believe two or three years and the certificates that are produced by that CA are one year.

A

No, so as of right now, the only one that gets rotated automatically is the the cubelet one yeah yeah yeah every. Whenever they they're close to their expiration, they they, the cubit, is gonna rotate them. Yes, sir.

B

B

A

Not necessarily so you can actually use your own CA, you can provide your own CA to kubernetes and your installation tool. So, for example, in cosmetic you can actually bring your own CA and we will use that CA to produce all the certificates that are that are needed for the cluster.

A

Yes, yeah, so yeah there's some implications. There definitely yeah.

A

To upgrade the CA, that is a great question, I believe from what I've seen you can only use one CA, so you would incur downtime there I think yeah.

A

I believe that should work yeah. That sounds like a potential workaround there yep.

A

Sorry so he was suggesting adding the CA to the machines trusted route list so that the the the binary just trusted, because the Machine trusted.

A

So, in the multiple master scenario, each master is going to have its own certificate and then the the consideration there is to just make sure that the load, balancers, IP and DNS name are part of that search so that your clients, you know, can validate that certificate.

B

B

A

So you can bring your own CA I'm, not sure. If you're talking about like integrating with another system, yeah I'm, not sure I, don't think so. I think you actually have to provide like a file-based CA, so yeah at this point you have to do that yeah cool! Thank you so much and enjoy.