Cloud Native Computing Foundation ServiceMeshCon NA 2022, 1 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Your Wish Is Our Command: Declarative Certificate Management in a...- Jackie Elliott

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Your Wish Is Our Command: Declarative Certificate Management in a Service Mesh - Jackie Elliott, Microsoft

Certificate management is hard. Integrating your certificate management systems with your service mesh is even harder. The ability to rotate certificate authorities and make dynamic updates to certificate settings is critical to providing a production ready service mesh. However, performing these updates and rotations is often a manual and error-prone process that can result in significant disruption to business operations. In this session Jackie will walk you through an approach to solving this problem: a custom resource designed to declaratively manage certificate configuration for the mesh. You will leave this talk with a strong grasp of what can go wrong and how to coordinate all of the moving parts of managing certificates in a service mesh.

A

All right, oh, we have Jackie lined up now, so please join me and welcome Jackie to the stage to talk about your wishes over command declarative certificate, Management in a service match. Thank you.

B

Okay, hi everyone I'm a software engineer at Microsoft and I'm, a member of the open service, mesh team or osm. For short, our team wanted to simplify certificate management within the service mesh for users like mesh operators and developers. So in this talk, I'll share some terminology: the problems that led us to seek out a solution. The solution that we implemented and an example of root certificate rotation with our solution.

B

So, let's level set on some common terms that I'll be using throughout the presentation so for certificate provider, I'm referring to the private key infrastructure or whatever the service mesh, is communicating with in order to issue certificates. So this could be assert manager or hashicorp vault for certificate. Authority I'm, specifically referring to the chain of trust for any given Leaf certificate and then certificate management specifically, is the management of certificates within the service mesh.

B

So osm chose to focus on the problem of certificate management because we recognize the importance of certificates within the service mesh. One of the main abilities of a service mesh is to help enforce, authenticated and encrypted Communications between services. So this enables users to help strive to achieve a zero trust environment and focus on their business applications rather than on tooling, to support certificate management.

B

So, despite the importance of certificates, we found that managing those certificates once they're in the mesh is not simple. Performing critical operations requires operators to have deep technical knowledge. For example, rotating a root certificate is a often a manual and air prone process and then specifically for osm migrating certificate providers. Maybe from our you know, internal self-science EA to sort manager to better meet business needs is not a standardized process, so failures in any of these certificate management processes have high impact and can result in downtime.

B

So, in order to simplify certificate management, osm took a declarative approach, so the main goal was to move the complexity from the users onto the mesh. The user is responsible for specifying their desired state of how certificates in the mesh should be used and we'll go into how to do this next and the mesh is responsible for making that desired. State a reality so by coordinating provider, migrations and a certificate, Authority rotations.

B

B

A little preview all right, so how does osm support users communicating this desired state?

B

So we created a custom resource or mesh root certificate, MRC, which you can see on the left of the slide to help centralize certificate management specifications. So this resource is either created by the mesh on Startup, using user provided settings on install or it's created by a mesh operator prior to install the MRC is created to initiate and facilitate rotation. But we'll come back to this later. So what can the user describe with this resource?

B

So the provider field in the MRC contains information about the certificate provider the mesh will communicate with to issue certificates. So osm, like I mentioned before, currently supports sort manager, hashicort volt and our internal self sign CA, which is stressor.

B

So the role the role field is the key aspect of what makes the MRC resource declarative. It describes how the user intends for the certificate Authority, which is accessed by the certificate provider, to be used within the service mesh.

B

So by inspecting this field, uh the service, mesh or MRC controller can determine how certificate should be signed and which certificate authorities should be distributed for validation,.

B

Okay, so the Simplicity of the Roll field is best demonstrated in an example of root, certificate, rotation or a certificate provider.

B

So let's imagine the CA in mrc1 on the top is has been compromised, so the operator has already created mrc2 and this it has an inactive role, and this is going to be containing the ca2 that will be rotated in in place of ca1.

B

So since mrc1 is still active service, a and service B will have certificates signed by ca1 and and their validation context will only contain ca1 and just backing up a little bit. So the active role this is. This indicates that the certificate, the certificate Authority should be used to sign certificates and should be added to the validation context.

B

A passive role indicates that the certificate Authority should only be used for validation and then an inactive role indicates that it should no longer be used within the mesh, and this is either in this case when you've created an MRC, and you have yet to rotate it in or you've already rotated the certificate Authority out. So that's what an inactive indicates so and then again notice, as we move through this rotation that the green check mark remains throughout, and this indicates that the connection between you know workload a and work workload.

B

B is never um never disrupted.

B

Okay, so to initiate the rotation process from ca1 to ca2, you can see in the red that the operator updated the intent from passive or from inactive to passive and they're. Just this is with a just a simple Kube CTL apply command.

B

All right, so the update is picked up by the MRC controller, which inspects the state of mrc1 and mrc2. So since mrc2 is in a passive state, ca2 will be added to the validation context of both workloads.

B

The MRC controller takes on the responsibility of updating these validation contexts in the mesh and upload updating Envoy, which is our sidecar. Without any downtime, and because of how Envoy is configured, these truss chains can be updated and reconfigured without downtime.

B

Okay, so to progress to the next stage uh or of the rotation, the new mrc2's role is updated from passive to active and the old MRC ones. Role is updated from active to passive. This change signals that ca2, instead of ca1, should now be used to sign certificates.

B

So since mrc's one's role is passive, ca1 will be included only in the validation context and will no longer be used to sign certificates.

B

Okay again, the MRC controller picks up on this change and enrolls and begins Distributing certificates signed by ca2. However, during this transition period, some services will still have certificate signed by ca1 and they've, yet to receive the update and others will still will have the certificate signed by the new ca ca2.

B

Also, although these workloads have different certificate authorities, they are still able to validate each other validate validate each other, because the validation context contain both trust chains.

B

So again, throughout this entire rotation, the connectivity between the two Services was never interrupted.

B

Okay so osm sought out a simplified solution to certificate management, experience to reduce manual and air prone processes for high impact scenarios. The mesh root certificate is a declarative resource that gives users control of certificates in the mesh and puts the responsibility of updating those certificates and distributing them on the service mesh itself.

B

So and then, as we saw in the rotation example, if we're migrating from one CA to another, the MRC resource allows you to easily make that Migration by simply updating the roll field and the pattern in which these roles are updated, which I didn't mention is enforced by a validating web hook.

B

This pattern ensures that connectivity between the two Services is never interrupted, as we saw with the green check mark and the osm team is committed to providing an excellent certificate management experience and we are working to expand the capabilities of this mesh root certificate resource to further improve the operator, experience and yeah. We hope that this is a simplified experience for certificate management.

B

So we went through the terminology, we shared the problem, our solution and we demonstrated our solution in an example of root certificate rotation. So, if you're interested in joining more check out osm slack with a very helpful ID and also check out our GitHub and I'll, be at the osm booth all week at kubecon, so I'd love to I'd love to chat. Thank you.