Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: You Like It Or Not; You Need It! - PKI And Certificate Management - Shweta Vohra, IBM

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

You Like It Or Not; You Need It! - PKI And Certificate Management - Shweta Vohra, IBM

Speakers: Shweta Vohra
PKI (Primary Key Infrastructure) and Certificate management are must to have feature in production as they provide critical security measure while communicating over network within or outside your Infrastructure (VMs or Clusters). Production environments are increasingly dynamic and heterogeneous with micro-services, service mesh, container orchestrators, and cloud computing. When you use Certificates for Kubernetes clusters it’s still a simple problem. However, when you design and implement it for microservices and service mesh that's where the real fun begins. I disliked this topic so much, avoided it for long and then after messing it up, learnt it the hard way. All this to discover it can be understood with little patience. In this session I will be presenting basics about certificates infrastructure, demo, followed by 5 must know about certificate management for every software application creator/owner/maintainer and how to handle it wisely with open-source tool like Spiffe/Spire without getting intimidated with the complexity it brings.

A

Hello, cubecon and Cloud nativecon hi to everyone, I'm shweta, bhora I am an architect in Winter, blogger, AWS, Community, Builder, open source, Enthusiast and contributor at times before, I start with this session, I must say what a phenomenal week this has been. We all have gained, observed and interacted so much it's time after today to go back and see who and reflect back what we have gained in this amazing cubecon and Cloud nativecon before we go back today.

A

It's time for this session, and it's about you like it a lot, you need it pki and certificate management. Let's get started.

A

The contents which I'm planning to cover in this session are vocabulary and repressure about pki and certificate management vocabulary for those who are new to this topic and refresher for those who are might be aware of, but they might learn a few things new from this.

A

We are also going to talk about a case study which is common, yet complex production, case study and a demonstration, and we will be ending with the five pki design decisions that you must know.

A

Okay, so here we are with the vocabulary and repression.

A

Pki public key infrastructure: what is pki? Let's understand that, always when you communicate over Network, you have two endpoints. This could be client and server. Now these endpoints talk to each other. Let's say you are talk. You are using a internet banking or a bank site where you are doing internet banking, and this happens over secure, socket layer or transport layer, security is required and that protocol is used.

A

What this client and server, how they trust each other and to do what do they need? There are two things primarily any end point or two endpoints need from each other so that they can smoothly talk to each other. The first one is the trust. Establishment client needs to know that the server I'm talking to or let's say, the bank site I'm talking to, is secured enough. That I can share my data as information and once that trust is established, you need to exchange the encrypted data over Network.

A

Now this happens day in and out when they indirect over that work. But what happens behind the scene? How does pki come in picture? Let's look at that to have that trust established. Your server needs some authorities, the first and foremost the certificate, Authority registration, Authority and the verification Authority.

A

What the server does is that it requests the registration authority to issue and assign certificate to the server using which it can give trust to the client that, yes, it has a digital certificate and it communicates encrypted data over Network. In turn, registration Authority takes that request and takes it to the certificate Authority who issues that certificate to you, the certificate Authority signs that certificate for you and gives you two things.

A

First, it gives back to the server a certificate and a private key, so this private key remains with very few entities who need to have hold of that. However, on other hand, certificate Authority also gives a certificate plus public key, which can remain anywhere available on the network, because it is public and verification, Authority keep it with itself and can utilize when needed.

A

Now what happens when client needs to or browser, let's say, for instance, needs to see that that this bank site is a valid site and I need to trust on that. That client can request to verify the signature from the verification, Authority and verification. Authority replies back saying that signature is okay or not okay. So what this whole thing gets established at the back or the behind the scenes, then client uses the encrypted data. Oh sorry, key public key to encrypt the data and, on the other hand, server decrypts.

A

The data by using the private key. So in between there is. There is no one who can possibly. There is no one who can decipher that data, because public key is available to all, but to decrypt the data. You need that private key and that's what public key infrastructure provides. You, let's and one more thing before I move out of the slide that certificate and DB store, because this whole thing remains somewhere on these servers, which we have seen and this certificate TBN store, gets utilized.

A

So what does pki provides you? It provides you cryptographic, identity to the endpoints before I move on I want to emphasize on that. It need not be all the communication over public network. It can be any endpoint, for example, one cluster talking to other cluster one. The Microsoft is talking to other microservers, so it it can be any endpoint and you can utilize pki for communication over Network be public or private, so you have to have a cryptographic identity.

A

They need encryption of data over Network so that the data remains secured, how it provides, but pki provides it by governing issuance and management of digital certificates and key components. As we have seen in the previous slide, you need digital certificate. You need Certificate, Authority registration, Authority, verification, Authority. Sometimes these three authorities may be same or different depends on situation to situation and also you need Certificate database or store which can be referred as needed.

A

So pki is a framework, not a specific technology. Please keep in mind. It provides you all these things by utilizing certain digital certificates and infrastructure. To achieve that.

A

Why moving on certainly cryptography, because public and private key difference is something which we want to point out here and we need to understand what is this cryptography cryptography is the sign of secret writing with the intention of keeping your data Secret this cryptography or the algorithms which get used are classified into symmetric or asymmetric cryptography in symmetric cryptography? What happens you have a private key which gets utilized for encryption as well as for the description for the decryption of the data.

A

Now the same key needs to be very, very secure, and that's why it's called private key encryption or symmetric encryption on other hand. You have a symmetric cryptography where you have two different Keys one is public and one is secret key or the private key, which remains only with few entities and public Keys publicly available.

A

Now symmetric cryptography is simple, so faster but less secure. However, on the other hand, a symmetric is comparatively slower, but tough to break, and there are algorithms like RSA, diffie-hellman, ACC algorithm and many more now, pki uses public key cryptography framework, not the private one. So that's why public key is the name given to the pki.

A

Moving on to all this is your gist. Is the digital certificate? Now, let's understand what is a digital certificate? A digital certificate is a digital document that binds the identity to a public key infrastructure.

A

Now you use it at many places in Internet Protocol, including TLs SSL, which is the basis for https the secure protocol you use for browsing the web, for example, if you use any browser, you see that there is either a lock comes or a um open lock, also at times come, which shows you that it is secured or not secured.

A

Now, digital certificates, you need to generate. How do you generate these? There are many options that are widely available and those uh options adhere to the international standard, which is x509 standard.

A

Some tools which I am listing here is open, SSL, Libre SSL. You have third party Ras and PAs like let's encrypt digicert Etc. You have also cloud services these days, providing, for example, AWS certificate manager, Google certificate manager and the there there are many more which provide these digital certificate services.

A

Now, what is the Key activities around the digital certificates, because it's not that you just issue it one time and you can use it forever. It is. There is Generation and issuance, which is required, distribution, deployment of those or installation of those certificates. That's when these certificates gets in use at times, you need to revoke them. When there is um the certificate gets hacked or due to some changes, you need to pull it back.

A

You also need to renew and rotate these certificates time to time and last, but not the least certificate scanning and monitoring is essential, which comes under the digital certificate life cycle management.

A

While uh we are moving, let's see some sample certificates, this one is from amazon.com. It has on the dim on the Windows. It looks like this and you can run cert manager.msc to see on the Windows box on command prompt to see this kind of certificate I'm, showing a sample certificate for amazon.com, where it is issued to amazon.com issued by digisert. It has period of validity. It has some hashing information associated with that now this is on Windows box. However, on Linux box, it looks like something like this: a text file.

A

It has a now again a issuer, a subject in this case both C and CN, stands for common name. Common name is same. It means it shows that it is for a. It is mostly a certificate Authority or a root CA. It also has public key information, as we discussed that public key information is easily available and accessible. It also comes bundled at times with your certificate information and the last here which I want to show is that subject key identifier and Authority key identifier in this case, if You observe this is same.

A

However, if your certificate is signed by some other authority, then these Authority key identifier will differ from the subject we identify. So that's how a certificates look like.

A

Okay, what are those files? What are those encoding formats and files which uh certificates come in under x509 certificate? There are various encoding formats, for example, 10 pkcs format, pem, and there are both but Pam make the popular one I'm, showing the files which it produce these green boxes, as you see, either the dot pem or the dot, cert or dot Ser file which it produces. Are these certificates and Dot key which it produces, is the private key file and it looks like something encrypted data between, for example.

A

If it is a certificate file, then it will be begin certificate and end certificate on other hand, if it is a key private key, it will be written like begin private key at end private key. Now how to how does these certificates get generated? There is always a request, certificates, signing request which goes from a server or an end point in a form of dot, CSR pipe, and this request. Csr stands for certificate signing request.

A

This file looks like begin new Certificate request and end new Certificate request. So by reading this file simply opening that file- if you see this understand that this is a certificate signing request, the certificate signing request goes to an authority which could be root. Certificate. Authority or intermediate certificate Authority, which we will understand in a moment once this request goes to the certificate Authority it signs, it digitally signs it. You get this certificate and private key files which you need to keep or install at appropriate places to make use of it.

A

So before uh we move on to the next topic, there is one more important thing for us to understand that there is a hierarchy of certificates, root, CA, to leave certificate and that's called the certificate hierarchy or chain of trust. It can be single layer or multi-layer cas. Why we need it we'll talk about it, but let's see how does it look like I've pulled a certificate for amazon.in where you have root CA digital Global root G2, which is providing the certificate.

A

Then you have intermediate CA, which gets signed by this root. Ca intermediate CA, then helping you sign with this server certificate. So, like you have Amazon dot in amazon.uk, amazon.us or whatnot, you can have multiple such server certificates, which can be installed on those servers, and you can view this hierarchy and that's where the chain of trust or the certificate hierarchy is gets built up for the actual environments or the pka environment.

A

While that was about the vocabulary and refresher, let's delve into a production common, yet complex production scenario, but how this? How does this pki looks like in actual well in all uh complex scenarios? You have either a corporate data center or a cloud or a cloud to Cloud, where you need to talk between these entities and serve outside. Also in in all these communications, you have a compute layer which could be like your VMS ec2s or a serverless compute.

A

On the other hand, you can have a cluster container orchestrator cluster, which can be kubernetes and to add to the complexity. We we have Network layer which helps you talk to all the communication happening over the network between two different entities or sometimes different different domains.

A

So this network proxy layer typically contains either a API, Gateway or elb or Ingress Gateway, and what else you have a service mesh these days we have microservices, which are talking to each other, so there can be various microservices which which reside in data plane. They talk through control, plane to your external world and all this gets called at various points by internal users by application and interfaces by external users and there's all the time communication happening between these systems.

A

Now to make this communication secure what you need, you need pkin, digital certificates and where all you need, you need all these yellow certificates, which you see. These are some simple places which I am showing. For example, you need to secure your API Gateway. You need to secure your load balances if you are using, you need to have microservices having your the those identity um and we just secured. Then you, your ec2s or VMS, need these certificates. Your cluster needs certificates and whatnot. Even your public websites need these certificates.

A

So what we need to do? How do we make this secure? Now, let's come to the what infrastructure or what pki framework you need to establish around it, so that this becomes a secure network with the digital certificates and communication happening within the encrypted format. Foreign.

A

So, first and foremost, you need for the external, so here I am calling it external and public CA because for having certificate, issues and maintained. So all your caraba functionality at times is given by all these uh various tools or the service providers like GoDaddy, let's encrypted to certain what not they provide these root certificate, Authority, which issues certificates to your websites and other entities which need to interact over internet then, typically inside our secured Network, we don't rely on these external certificates, so you need another root.

A

Ca within your secured Network could be a corporate data center or or it could be, a cloud which is your private cloud and that's where you keep a root certificate, Authority, which issues the certificates and it gives the certificates to the prime intermediate Cas or subordinate ca's. Also, we called so, for example, ICA could be your AWS certificate manager, where you sign your this particular subordinate CA. Using this root, CA and then AWS certificate manager issues those certificates on your behalf and distributes it to elb or English, Gateway or whatnot.

A

Similarly, a kubernetes cluster might need a different IC altogether, so that once it gets its identity signed by Roots a then it distributes these. The leave certificates to different applications, as per need theme, is for your service mesh. There can be many microservices which keep on coming and going and there's a whole life cycle around those services. So you need some subordinate or intermediate CA, which issues those certificates to your the micro services or the leaf entities or the end points.

A

So once it is in place what you get you get all these certificates from these Cas signed installed, and once this is installed, your communication over the network is trusted identifiable and encrypted. So um there is no chance that anybody can creep into your system and reads: data and easily can take away your data.

A

So this is what we have seen. You have a public CA which can be these tools like digicer, let's encrypt and all which provides and used for external facing website and interfaces. Then you can also have private and intermediate cas reflected with these uh symbols. In the previous diagram, you can use open SSL to issue the insert manager, spiffy spire, and there will be many other tools.

A

Then you also have these leave certificates, which are the end entities or those digital documents which we've been talking about so far, and these These are what provide the actual security to your system.

A

Let's see a small demo which I have already put up in place, but let me quickly show you that how this looks like.

A

Okay, so I am moving to this folder or let me put it in format, which is clearly visible.

A

Okay, so let's say you have developed a root CA. Now this root CA is a cell signed certificate and a key pair where nobody else signs the root, CA, so root CA needs to be much more secured and you get ca.cert and CA dot. Key I have used Root ca.com to generate these files. There is a some configuration, let's say, I'm using open SSL, which I have used for this particular case. It needs a lot of configuration so all the time providing that configuration over command line I have used it root.

A

Ca.Com, where I have put all those configuration and I have generated these certificates and key now, once my root CA got generated, I have got my intermediate CA signed by this root CA. So what I did I got? The CSR generated certificate, signing request using which my root CA signed it and provided me with a third which is my intermediate CA certificate again, because it is a certificate Authority. You need a configuration, a lot of configuration to be provided over the command line, so I have encoded it in this file.

A

Now, once these root, CA and intermediate Cas are ready, what I did I start started? Creating my server certificates or leave certificates so again, I need a CSR. This time my CSR got signed by my intermediate CA and I got my server certificate generated. So this is how it looks like, but in no production case you will see it in at one place. Visit, definitely spread it across the authorities and the servers, but let's quickly also look at the format of few of them. For example, let me show you um open SSL.

A

A

And I am entering here, let's say my second yes here I'm, giving it um my root CA and my CA dot, cert, yes Okay. So this is how my certificate looks like, as we have seen earlier. Also, my issuer and my subject in this case is same if you notice, because this is a root CA and in root CA case only. You should ideally see that your issue and subject are same because it is self-signed root.

A

Ca, then you will also see that you have subject key identifier here and Authority key identifier are same, which means that your key here, if you notice, is exactly same again. The reason is this is root C8. But if we see this for any other certificate, let's say now I'm opening it for intermediate CA and where is my certified okay? So this time, interestingly, If You observe here that my issue is different, then my subject, my subject is ica.demo.com this common name and my identifier subject.

A

Key identifier is the newly generated identifier for this particular intermediate seat, but Authority key identifier is the one which we saw in the root CA. So that's how the certificates get generated, but what happens when you have your kubernetes cluster and istio certificates? Would you be generating certificates like this for each and every microservices?

A

Well, that's a lot of work and we cannot do it like that and that's where, let's, let's go back to our presentation where I'm providing you a tool, there can be more, but spiffy inspired are coming up really well, where, uh let's understand this, piffy is secure. Production identity framework for everyone, which means Biffy, provides you a framework to get those secured identities and it's a framework now Spire is a production ready implementation of these PV API. So they usually, you would see that spiffy Spire word gets used together.

A

These are both open source graduated projects. Here, spiffy provides you with the facility like. First of all, you need to register your workloads, identify your nodes and workloads, so registration feature then Federation when you have let's say two different domains or two different clusters, which is let's say one is running on Google another one is running on AWS on a data center to a cloud you need to Federate those domains.

A

You need to establish trust between those domains and that's where it provides you with the feature for the federating, the Clusters or the nodes and domains.

A

Then it also gives you the feature to have your workloads identified and identifiable information given to those workloads and time to time it provides you the attestation of those nodes and workflows how spiffy provides it. It provides it by exposing the apis it um to whom it provides. It provides these features to the actual workload and the nodes. So first thing comes as the node, where your workloads run and these nodes get a spiffy ID by exchanging few things from your spiffy apis.

A

So this spiffy identity looks like something like this piffy, where your trust domain name, your namespace and your essay stands for uh service account, which it belongs to. So that's how it differentiate that Which principle it identifies to. Then you have your workloads. All the workloads running inside your nodes, get our identity, so secure verifiable identity is the name for that which uh spiffy calls it like SV IDs.

A

These svids are nothing but your certificates x, dot, 509 artifacts, which it provides and how it does that it does it by having a Spire, server and Aspire agent, because remember Spire is the implementation and spiffy is the framework of all that API is what it exposes, and once that is in place, what happens as I said? First, you get your spiffy IDs and some selectors along with it once that comes then your workloads get identified. So you need to have certificate. Signing require requests.

A

You have to have uh to get those SV IDs from your again specific apis or the expire server gets it generated through these usage of these apis and you get those nodes, give CSR and get SV ID, then spiffy bundles, which never leaves the node. So you have. These information comes up to the node and nodes time to time, distribute it and gives it to the various workloads. Now it doesn't matter, workload go down or up and keeps coming.

A

You have your secure, verifiable identities, and on top of that, you need not worry about renewal, let's say or the rotation of those certificates it. It gets taken care by spiffy inspired through the field all right. So let's talk about five pki design decisions that you must know.

A

First and foremost, you should know how to design your certificate chain or hierarchy. It can be a single root, CA Authority, where single root signs the your icas and other certificates and takes that ownership. It can also be multiple intermediate Cas to protect your root CA.

A

You can also have a subordinate, CA or ICA, which you are delegating to, and this in turn signs the other size, icas certificate, signing requests other way of doing it in complex systems that you have a geographical or departmental or domain distribution where it is segregated icas at geography level, it can be segregated, icas at your department or domain level, which in turn take care of your leave certificates, distribution and certificate life cycle management, but to keep in mind that wise thing is to keep design your certificate hierarchy based on size of your organization, criticality levels of security required and operational capacity.

A

You have because, ultimately, more the hierarchy more the burden. You have second point where to terminate your certificates by termination, we mean that, um let's say, if you're terminating your certificate at API Gateway, then, which means that Beyond this point, your data would be traversing.

A

It will be decrypted at this layer and data will be traversing in plain text, so it will not be end-to-end encryption, but encryption will end here in most cases where it it can be used that terminate it that API Gateway load, balancer or increase controller layer, which means the network proxy layers or in other cases uh you might want to end it at compute computation layer, which means that the VMS or ec2s or at the pot containers. Now this depends on that. um How much of compliance? How much of data in transit?

A

Do you have a requirement of end-to-end encryption requirements? Then? Yes, it makes sense to do it at compute layer, but in other cases to keep it simple, because it comes up with a lot of computation and performance requirements as well as the operational needs to maintain those certificates which you generate. So it's better that you end at Network proxy layer, because ultimately it then routes it to your internal secure network, but look at your requirements and then design that what is your requirement specifically third thing to keep in mind the TLs version mismatches.

A

So at times we have uh PLS 1.2 to 1.3 or TLS versus mtls need to coexist, um for example, in this instance, if you have to coexist with TLS and mutual TLS, where TLS is one-way, trust, verification and on other hand, mtls is two-way trust handshake. So sometimes you need to have both coexist. For example, in your in case of istio service mesh, it automatically configures your workload, containers or sidecars to use Mutual TLS when calling other workloads. So how would you coexist with these variations? So in such cases know the ways to exist.

A

For example, use mtls mode for peer authentication modes can be like strict permissive or disabled. Where strict is that strictly you have to have mutual TLS to Mutual TLS communication. Permissive could be that it is optional, it can be relaxed and disable can be that you can totally disable Mutual TLS and keep it simple.

A

So that's about the third point now. Fourth, an important point that certificate, revocation methods and design, so there are ways to revoke um revoke your certificates and keep a list of it, because when you do the revocation, you also need to inform various uh entities or verification authorities that, yes, these certificates have been revoked. So how do you do that simplest way of doing? Is that keep a certificate revocation list whenever you are evoking a certificate?

A

And this can be automated or manual process, where you keep checking whether any certificate is removed and take action appropriately, either manual or automatic another way of doing it in online certificate status protocol where web browsers keep a tap on that what certificates are revoked and they they change I mean whenever they get to know about this revocation of certificates. They maintain that list and they renew the site. I get the new certificates from these cas es.

A

The the upward version of ocsp is ocsp Step link where browse web browsers get this information rather than asking from CS get it stepped whenever they there is a TLS handshake happening now why? It is relevance because here are crl. Design is often overlooked aspect of pki design, so you should know how to design it.

A

Recommendation is for internal private CS use crl to keep it simple and use customer automation, but in public Cas, where there is facility of such protocols, then definitely you must use of that.

A

Fifth Point certificate, Automation and monitoring again many a times unplanned area, and do you need to keep a balance about what and how much to automate in operations? Few suggestions create certificate templates to keep it same within your organization and always adhere to certain policies and configuration utilize certificates, rolling deployment to avoid downtime during certification, renewal or expiry.

A

You can use tools like spiffy inspire. You can also use tools for monitoring, tracing and alerting for better and easy certificate management.

A

For example, grafana gives you a nice little dashboard of certificate when it is about to expire or when it is already expired.

A

um Gives you this animation little beautiful animation, where it tells you that which all communication is secured through TLS and certificates and which is not so have your own mechanism, but tools can be any but utilize these tools for automating monitoring and analyting.

A

Yes, we have reached to the end where a lot we have covered uh thanks for listening patiently, it's time for questions and please do leave your feedback by scanning this QR code. Thank you. So much.