Cloud Native Computing Foundation Production Identity Day: SPIFFE + SPIRE North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Integrating SPIRE and Istio - Daniel Feldman & Glaucimar Aguiar, HPE

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Integrating SPIRE and Istio - Daniel Feldman & Glaucimar Aguiar, HPE

The popular Istio service mesh simplifies many aspects of microservices deployment in Kubernetes, including providing gateways, load balancing, fine-grained authorization, and end-to-end encryption. However, Istio is not natively compatible with SPIRE identities, instead using its own centralized certificate generator. At HPE we developed a modified Istio that uses SPIRE to provide all identities. We have demonstrated this in proof-of-concept environments and are working to deploy it in production, and we are working to upstream this feature into the Istio mainline with help from many key Istio contributors. In this talk we will discuss the motivation for the project, and where we are going with it in the future.

A

Hi, I am glas maragiar and first of all, it is a pleasure to be here and present this project today me and my colleague, dan feldman, will talk about, um probably know them. So we'll talk about the work to connect east to service mesh and inspire, and at the end we will demo our progre our progress so far. So hope you like it and have some fun.

B

In case you didn't know, istio is a service mesh for kubernetes. That means it adds a sidecar proxy alongside each service in your kubernetes cluster, in order to enable advanced networking features like api gateways, load, balancing and mutual tls encryption between services.

B

For that mutual tls encryption istio has always used spiffy certificates, but istio doesn't have a lot of the advanced features of spire like attestation, a workload api, spiffy, jots and spiffy federation. On the other hand, everyone is using istio. Its adoption is, is really widespread in the cloud native community.

B

We don't want to have to force people but to choose between using istio and using spire. Instead, we want them to work together really well and that's the purpose of this project.

B

Before we started this project, we embarked on a number of different, uh easier approaches that didn't involve modifying istio in order to make istio inspire more compatible.

B

The first approach was tricking istio into using spire certificates by essentially hijacking the traffic to the istio certificate authority, and this actually worked in early versions of istio, because those certificate requests to the istio citadel certificate authority, weren't validated in any way. So you could actually just uh put some kind of a proxy in the way that would intercept the certificate requests and and put in a spire certificate instead, uh but that hasn't worked in istio for over a year now, because, finally, those istio certificate requests are actually encrypted and validated.

B

Next we tried using the same upstream ca both for istio and for spire. This gets you some level of compatibility, because then you can have some outside services that use spire certificates and some inside services that use istio certificates and then uh because they're, both spiffy certificates. They have some level of compatibility with each other, but you still don't get all the advanced spire features like attestation and the workload api.

B

Next, what we tried using an envoy sidecar, a second on voice, side, car running inside istio that translated istio certificates to spire certificates. uh So this basically doubles the overhead of doing mutual tls encryption because you have an envoy proxy that decrypts the issue encrypted traffic and then re-encrypts it with spire encrypted traffic. This works it's a hack and we wouldn't recommend anyone use it in production, but it does get you to uh achieving part of the goal and then finally, we also attempted to reconfigure the envoy configs. The istio generates to reinspire certificates.

B

Instead and again this does work.

B

It causes some problems with communication with istio daemon and with istio gateways, and also it's a hack that probably no one would run in production, uh so what we really needed after after trying all these approaches was a modified version of istio. That knows how to talk to spire itself and get spire certificates.

B

So our goal for this project is to modify istio itself to optionally use spire certificates instead of istio certificates. So there are no changes to spire as part of this project and there are some minimal changes to istio in order to talk to spire.

B

So we started by developing a proposal and sharing it with istio community gathering feedback from the istio community and then once we're complete. Once we have this working, 100 and tested and documented, we are planning to work with seo community to get it into upstream istio.

B

What does integrating spire with istio really buy us? First of all, and most importantly, it gets you at a station based on hardware or cloud identity. Istio doesn't have this by default, uh but with spire and istio, working together, every node, that's added to the istio cluster. Every workload that's running on that node can be attested down to its hardware identity, using tpm or its cloud identity using instance metadata next. This will help with integrating with non-cluster workloads.

B

So if you have a service, that's running completely outside kubernetes or just outside istio, but still inside kubernetes, it can get spire certificates straight from spire and then workloads running inside istio, which are getting spire certificates uh through the modified version of istio. Then they can talk to each other.

B

Next, you get easy federation from the istio spire instance to outside spire instances or other oidc federation. Maybe you can federate with an oidc endpoint in order to access cloud apis natively. So that's really helpful.

B

Another security feature is that because spire allows you to keep your root keys outside the cluster and also automatically rotate them frequently, it's a little bit more secure uh istio by default, generates the root keys locally and keeps them inside the cluster, and then it really never rotates them. So with spire. You can keep all that key material outside the cluster and then finally, workloads can access the workload api, which means they can get other certificates, not just the istio certificates, but maybe some other certificate that you want to assign them.

B

Maybe some trust bundles, maybe some jots other things that the workload api could provide in the future and istio workloads would be able to access that directly.

B

In summary, what we're doing as part of this project is replacing istio's internal certificate authority with a new integration path that allows connecting istio to external certificate authorities, and then we have glue code that allows connecting spire to this new certificate authority api.

B

You no longer need to use the spiffy workload api at all, because all the incoming and outgoing traffic will go through the istio proxy, which will automatically use the the workload api under the hood and be able to access all the spire certificates.

B

But you still can use the workload api if you want- and this is a big advantage because mounting that workload, api socket path can be challenging in certain situations where you have limited permissions on the kubernetes cluster and finally, there's a whole team working on this at hpe, and not just us and hpe- is uh committed to developing this new feature and getting it upstream into sdo. As soon as we can.

A

Thanks then, let me go a little bit deeper here in this diagram. We see a representation of the main components apis and protocols being used within is too sexy. Mesh ecosystem for communication and identity distribution envoy the sidecar used within digital service mesh, as then mentioned, communicates with each other using mtls issue. Agent communicates with east2d in the control plane for fetching identity for itself and for the sidecars it does that using a csr api- and this is not a good fit for inspire or workload api.

A

So because of that, the path we decided to take was to teach study and this new agent how to fetch identity from inspired agent. To do so, we implemented a configuration that allows istu to fetch identities from an implementation of the spfe workload api in this diagram represented by spy agent as an alternative to using csr api to fetch identities from east to d directly.

A

So we have two options available and for those of you who may be curious, this was done using an interface, an existing interface in these two called secret manager. So the chains are localized very beautiful and elegant. I would say, and the communication between the sidecars was not changed: mtls is still in place using espire issued certificates.

A

In this slide, we can see the fetch identity flow, using istio default configuration and the alternative developed by this project, using an implementer of the spf workload api represented here by spire. As you can notice, the change does not affect envoy and how it integrates with is2 agent. It only affects easter agent, fetching identity and in the default flow in the default flow, is the agent generates key, and csr and contacts is 2d using an assign request, which is to the response signing the request.

A

The certificate acting as dca on the other side when is2 is configured to fetch identity from from expire is 2 agent does not communicate with with is 2d or nor use the csr api. It connects to inspire agent using workload api to fetch identities, although we did not represent it here. Estudi also interfaces with spideragent for fetching its own identity and that's the magic explained.

A

Okay, but that's not all folks, there is more to come. The list of tasks has just started and includes, but is not limited to federation uh tbd, which forms of federation we are talking about. Is the federation inspired federation both together, if if it even makes sense, but anyway, custom spf id supports for those of you who know? East will probably remember that it is only happy if spiffy ids follow a predefined standard which has namespace and serves the account identifications.

A

However, if one wants more flexibility or existing inspired issued identities that does not follow these two is to be standard. They are in trouble so and last, but certainly not least, we are looking to upstream this change, as then already mentioned before. So enough talk, let's go to the show.

B

And now it's time for the exciting part of our presentation, the demo of istio inspire working together today, we're going to be showing istu, inspire, of course, but also torniak, which is the user interface for spire. That's under development and kiali, which is the user interface for istio that comes with istio.

B

The application we'll be demonstrating is called bookinfo, which is the built-in istio demo app. It consists of a product page written in python, a reviews page written in java, a details page written in ruby and a rating service, that's on the back end written in node.js, and they all have to talk to each other.

B

There are three instances of the reviews page. So, each time you refresh the page, it will actually show a different review.

B

Each service in istio gets a sidecar proxy, so each one of these will have its own envoy, sidecar proxy and because we're using istio spire rather than regular istio each one of these sidecar proxies will be getting its identity from spire.

A

So yeah, so thank you then, and let me just show you then this environment, in which we have the estuaries bar integration running as they mentioned. We have book info application running as well. This is the the main page of the booking. Oh before I move on. This is running on aws and eks cluster. That is stood up for this demonstration and this is the book info. So we have the the book details. This is the product landing page and we have the book reviews here.

A

If I reload the page, we will see changes going on in this book review piece of the part of the the page which are provided by the three different reviews, uh workload and versions. That then mentioned before so looking at kiali dashboard, we can see the traffic.

A

Going through is to ingress gateway through the product page through the other pages like and workloads like details, reviews and ratings, and let's take a look at torniak, the the ui for for expire.

A

Looking this first page here is the torniac dashboard. We can see that there are some information here like the two two agents that are running one for running on the control plane and the other one, the other one running with the the single node.

A

Oh, I forgot to mention this is a two nodes cluster one for the control plane and the other one for the data plane, and we have the two agents running here: one serving the the easter egg running on the node and the other one serving the s2d, and we have all of the entries that are registered to inspire I'm going to move to these other view of the entries, which will be easier for us to identify some of the entries.

A

So we have, let me just scroll down to show you the the interesting ones which are the ones related to the uh book info application. So we have uh the one related to the uh the book info details, workload, the book info ratings workload, the book info reviews and we'll have three different entries for the reviews and the one for the product page itself.

A

So so that's it. In addition to the the uh other entries that are common for the spar infrastructure.

B

Thank you very much for attending our talk today. If you have more follow-up questions, please free feel free to email, either me or glossamar, and you can also find us on spiffy slack all the time thanks bye. Thank you.