solo.io Hoot Livestream, 3 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Spire Integration with Workloads on K8S & VMs

Description

Interested in using Spire as your identity provider for your Istio service mesh? How does it work with your workloads running on Kubernetes and/or VMs? In this hoot livestream, Max Lambrecht and Christian Posta will join Lin to discuss all things you need to know about the newly added Spire integration to Istio 1.14 release.

#istio #spire #servicemesh #kubernetes #cloudnative #security

00:05 welcome + speakers intro
2:39 service mesh and Istio news
4:40 discussion around Spire integration with Istio Service Mesh
15:28 Istio Spire integration architecture
22:20 Live DEMO + answer live questions
41:10 Discussion around Spire + Istio + VM
45:15 wrap up

A

Welcome to today's hoots live stream interested in using style as your identity provider for istio service mesh. How does it work with your workloads running on kubernetes or maybe even on vms? In this whole live stream? Max and christian will join me to discuss all the things you need to learn about. The newly added spell integration to the upcoming is your 1.14 release sometime in may.

A

My name is lin sung, I'm the director of open source with solo.io. I am so excited today. I have max and christian join me uh so max. Can you introduce yourself please.

B

Sure hello, thank you for having me on uh I'm a senior software engineer at hpe, so I joined cycle in 2018 and started working on spiffy and spire projects and libraries become maintainer of the spf library and then some years later, cypher was acquired by hpe and started working on project mithril. This is the code name for this project whose main goal is integrating spire into eastern identity control, plane and before all this, I I've mostly been on java server developer for many, probably in too many years.

A

That's awesome a little fun fact about max, actually he's very well known in the israel community, because his most recent contribution of spell integration into israel, uh christian, doesn't hurt to reintroduce you.

C

Yeah thanks for uh joining us here max and thanks for anyone on the live stream, or anybody that watches this afterward, but I'm uh christian posta, I'm the global field cto here at solo, I've been here for uh quite quite a long time been involved with istio since the very beginning and uh recently just published a book. Actually I happen to have it right here, um istio in action that came out uh a few weeks ago about a month ago, now so yeah look. I'm super excited about this contribution max.

C

I'm super excited about this feature coming into istio 1.14, and I look forward to talking about it here.

A

Yes, totally, uh let me start with the news uh for two minutes before we get to the uh today's hood um with spell integration. So last week we had istio khan. It was a hugely successful event. We had about four thousand people registered for istio car. In the day.

A

One keynote we had around 1200 attendees, the biggest news, though from istio khan, is it still has applied to be a cncf project, so something many of us waited for a very long time, so excited to see craig box wake up at 3 a.m and submit a pr to apply to be a cncf project. So that's awesome.

A

um The second news I want to share is vmware has published an interesting report on kubernetes is here to stay, and why so they've done a state of kubernetes surveys so totally check this out. One thing really interesting in this survey, though I thought was: security- is still a top priority, as even as kubernetes getting easier and multi-cluster multi-team for security is continue to be a top uh priority for organization. Many of them still have concerns in this area.

A

I think this is exactly what we are doing in istio and what solo is driving um around the service smash ecosystem too. The third news I want to quickly share is our next uh episode has been announced, so we're going to talk to the special spectral cloud team during to talk about the declarative, kubernetes lifecycle, management across multi-cluster and multi-cloud with cluster api. So very excited. You learn more about cluster api from the spectral cloud team.

A

That's the news for this week with that. I would like to start picking on the brains of my speakers here so max. How is your experience contributing to the spell integration to the istio project.

B

Well, it was a whole learning experience not just for me, but for a whole team here at hpe. But istio is not my area of expertise. Really. um We haven't had much experience with history internals before so. For some time we explore the code, the internals of istio, to understand all the components, architecture, how everything works and is connected and started thinking about where this integration with spire could be achieved. So we did a proof of concepts.

B

Then, basically, we wrote proposal document and started joining the istio networking and security working group meetings to discuss it, so we got feedback and suggestions to rework the approach and to improve the integration. And finally, after a couple of months, we we were able to submit uh this pull request with integration to instill incogripo.

B

um It received very good stigma from the community, so we worked through many comments. uh Many useful comments to improve the implementation until we finally got the approvals we needed to to to merge the dpr, so it was finally merged and we are very happy. I think it's very important for foreign spire projects. It's something, sorry that the speedy community had been looking forward to for four years. At least it's a it's a big deal really, and I want to thank you, lynn and all the ac maintainers for your support pushing this forward.

B

So I really appreciate very much.

A

Well, thank you so much for your contribution. You know, like you said it's something. We've waited for a long time and hopefully you do agree like the new design. The community work together is better than where you get started. When you first presented the idea to the community.

B

A

B

Yes, the interaction. The discussion with the community is really useful. The feedback makes you things harder and better, and you come up with a better solution, always.

A

Yeah totally now, uh let's dive into what problem does spell integration solve for what users uh christian you want to start talk about that.

C

Yeah I can, I can start, um I would say the number one biggest thing that I'm particularly interested in um and that's it's probably a reflection of the use cases that we see from uh some of our largest deployments of of istio in service mesh, and that's that's around the area of um you know extending the capabilities of identity uh outside of what istio already has, and this becomes extremely important when you're using vm workloads right so istio today use or even before this.

C

This uh you know this integration with spire is still used, the spiffy uh spec or this you know spiffy identities inside of of of issue when it issued workload identity. Now it.

A

C

It happened to be very tied to kubernetes, and um you know a lot of assumptions that's baked into kubernetes and when you extend out to vms, you know that identity model had to you know, live within those constraints with uh with spire.

C

Now we have, uh you know a a separate and more complete way of looking at uh identity that is not tied to kubernetes but includes and can use coup and be a part of kubernetes that can extend out to all of these different ways for uh integrating vms attesting their identity and testing those workloads and uh and continuing to play nicely within the uh the istio model of security. So I would say: that's the leading: that's that's one of the most important and most exciting parts, and most uh you know widely.

C

A lot of people were waiting for this. Now I would say that you know using istio and vms is, is still very much possible. um We have a lot of people that are that are doing that. They just kind of contorted, I would say their automation and uh and the way they they bootstrapped the vms and integrated them with the with the with the service mesh to these constraints and assumptions that were already there. But now this this opens up. uh The things are far more flexible in this. This way.

A

Yeah totally, I I remember going through the vm documentation. We have in istio io right. It's just a lot of manual steps. Users have concern to transfer the keys and search over the wire to the vm, um so I totally agree on that perspective max. What's your perspective on what problem? This is solving forward user.

B

Yeah, from my perspective, I I think inspire enables a vast array of features. Some flexibility so aspire is the completing implementation of the spf specifications so specific spec this that defines the spfid that was already using by used by by istio, and it also defines the vocal api, the spifferation, the truss bundle usb this spf verifiable identity documents.

B

um Inspire is implemented using a plugin oriented architecture, so it's very extensible and on what spire I think brings to the table is strongly attested identities. So spire leverage is a powerful multi-factor station engine to determine with certainty the issuance of identities and this multi-factor station. So it's extensible. You can combine different station mechanisms to determine if our global calling the conspire it should be given an identity, and I think that the main tenet here is that a certain trust through multiple independent mechanisms provides a greater assertion of trust.

B

So you get robust, attested identities, so you can define a policy to assign an individual service based on selectors that may come from kubernetes docker unix kernel, so you can combine different station mechanisms different a tester, and there are many that spire comes and spy provides out of the box. You can and you can create your own testos based on your needs, your platform, your configurations and I would say that strong identities through multi-factor extensive attestation is the main capability aspire brings to the table.

B

There are there are, as we mentioned. Sometimes um you may need to integrate with system pki with existing certificate authorities, so you can use a plugin to integrate with an existing certificate authority or maybe you you may have um specific requirements for storing your private keys. So you can use, for example, spire key management plugin to store your product keys in aws, or you can implement your own key management system or sometimes you want to, or you may be required to use hardware based at the station.

B

So you can use tpm tester, that's by device, or you can also implement your your own um at the station model um and yeah. There are spy, for example, can help improve uh the observability of your whole infrastructure, as you can have logs and events um like critical events like identity issuance identity, registration view, registrations, uh certificate rotations, and I think all this can help provide a more complete view of your infrastructure.

B

So in in general, I think spire provides an um uniform service identity control plane that is available across multiple platforms um through a consistent api. So it can help build a common infrastructure to provide identity in a generic way. I think that's very powerful.

A

Yeah totally for services running regardless, whether it's on kubernetes or on vm, uh which christian was uh mostly interested in this work um and then observe. That's an interesting perspective too, to kind of have in that login tracing system for all the things related to identity.

A

So, let's dive into, can you describe the architecture of this integration max.

D

B

um Sure I can share my my skin just show us some slides.

A

B

So can you see my screen.

A

Yeah, we can see it well.

B

Great, um so the the integration was solved in a pretty simple way, so we define this well-known, fixed path for the sds socket. This is the socket where the the invoice secret discovery service api starts.

B

So this is a socket that the invoice process and the cycles in the data plane connect you to to get their identities, and this new, well-known, socket path can be used to integrate with either with spire or any other ci provider that supports my sdis api.

B

So how does it work institution at this startup and will now check whether this socket in this path exists and if the socket doesn't exist, asiation will be responsible for serving the mysds api, so it will start its own um sds server and provide certificates to angular proxy and in this diagram it's the spiration who will mount the socket on this path and institution will detect that the socket already exists.

B

So envoy proxies will connect and we'll do a mvo sds request to fetch the secrets um and we connect to them to the sds api api provided by inspiration, and at this point can.

A

You talk about that fetch secrets where the the secrets come from the number two sorry.

B

Yeah yeah, so this fetch secrets is the request that the envoy proxy use um to fetch the certificate, the ex finance certificate and its trust, bundles and all the configuration and how to to use uh those certificates in the for mutual dls communications.

A

Oh, I got you so basically, you used to ask: is your agent as the sds provider for that now it's asking through the socket to the spell agent.

B

A

B

In a default ec configuration ico connects to the installation for the sds communication to get there, it's secretive as a certificate to get this identity. Now it will connect to this socket on this fixed path. So we have changed with our implementation. The envoy bootstrap configuration file to always connect to this fixed path.

B

This fixed socket path, so it will be either installation or other ca or could be aspire and mount in this socket for providing the sds api in this case in this diagram envoy is connecting to this socket is asking expiration through the sds api for its identity, and you will get um all the certificates, the trust, bundle, the ex-finance certificates and private key. That envoy needs to do secure communication in a data plane, and here it takes place a very important process, that is the local attack station.

B

So, when envoy proceed calls the sds api inspiration, the process is attested using all the workload attestation mechanisms configured inspire, checking, possibly multiple selectors, like um docker images running in the pod docker image, calling the spiration the unix user id or group id of the process running the the move proxy. Also kubernetes, nine space or service account can be used. So all this information can be used combined to decide whether to give an identity to a particular android proxy calling the sds api inspiration.

B

C

Does the sds? How does the spire agent know that so, let's use a concrete example? Let's say I have workload, a we'll call. It workload a workload, a has the steelside car and when work the work, the sidecar in workload, a spins up and starts to go through this process.

C

What does it pass to the agent over this api that says, hey, I'm workload, a.

B

No, that's a matching of all these through workload station expiration with other evidence, selectors of that workload a and with the registration entries these policy identity, mappings will define which identity to give to that and we proxy that's calling the expiration. I will show that in the in the demo sure how these identity policies are defined uh with different selectors, so aspire can know which identity to give to a specific uh android proxy.

B

um So and, and then once the envoy, god its secrets from inspiration, uh it will connect to a situation for all the dynamic configuration for the clusters, listeners and endpoints that are generated by history and are provided uh by uh by institution um starting the xds server, as is usually done by by issue. We. We didn't change that.

B

D

B

Will show a demo, I will deploy the booking for application, and so it's the the application, this microservices application provided by istio, so we'll define identities for each one of the envoy proxies of each of them of the workloads. For this this application.

B

I have- I have here a kubernetes.

B

Cluster running kind, I will deploy spire.

A

Can you also describe the experience to istio user? Do they have to install anything to enable this any changes on control, plane and data playing as far as when they install seo if they want to leverage this feature.

B

So, um to enable this feature, uh the only thing that they need to do is mount a allow. The I will show here.

B

A configuration of.

B

The side cycle injector I'm mounting um through using a csi driver, I'm mounting the the socket the spiration socket on this path, and that's the only thing that I'm doing with this integration, that in the descent and the integration, so if the socket is mounted uh on the part, the issue is situation when detected, and you will let anybody process connect to that socket and that's the magic of all this integration that you don't need. Any special configuration.

B

You only choose to tell the the data plane or control plane to do anything because uh initiation will detect the presence of the socket and void proxy will connect to that socket to get their identities.

A

Okay, cool. uh We have a question from our audience: uh hi expositor. How did you install spire and well, did you get the entire yaml file in a single go and can you put the link to the folder.

B

Yeah, I will, I will put the link to the spiffy documentation. There are some yarns to install spire, so there is a tutorial to start with uh spire and stars fire and kubernetes inspired server expiration, so expiration is running as a diamond set.

B

So it's running one inspiration for for now and I can share all the documentation how to uh achieve that.

E

C

And just and just for completeness uh max, you know that there is document there will be documentation on the istio.I website on this specifier integration. That will walk you step by step. How to how to do this even using an installing spire. You know all bundled in together.

B

Yeah, along with implementation, pr we submitted to issue, we created other prs for updating the documentation with a step by step guides uh how to enable this integration, how to install spire and how to define the the identity policies, create registration entries to to make um to enable the workloads in a mesh be given their identities.

A

That's a great point: yeah, I'm sending the issues pr to our chat. So if you are interested, please help us review and even test the steps the pr is currently under active review. Now we have another question from our friend faye: do you need special linux capability, privilege, long route uh etc? To run spell agent.

B

D

You don't any special privilege.

C

So the aspire agent has access to all of the things that it needs an operating system to do at the station.

B

um Yeah I'm looking.

D

Something I'm not sure about it.

B

A

I think in the samples you guys provide, there are some type of like rows and row binding in the kubernetes resources you deploy.

B

Yeah, I I think one it requires one one privilege.

B

I can't double check later. I I'm not sure.

A

Sounds good yeah they will get back to you. Let's see, we have one more question from exposito. How did you deploy fail in a single goal? I think you might ask that earlier. Yes using the documentation, you have to run each cubecardo command.

A

Oh, he was just asking: how did you get this in a single command? I guess it's just a script.

B

Yes, yes, it deploys and applies all the jobs for deploying the spire server and then the spiration, the csi driver to inject the the socket into.

D

The pods, so it's this script and does all the magic so.

C

I'm looking at the default installation real quick about uh about the spire agent, it looks like it is deployed as a privileged pod.

C

Oh, do you have one deployed already? Yes, let me just look at the the ammo of that that pod for that payment set.

A

Can yeah just search for privilege in a game.

B

Yeah security council privilege, yes,.

A

Yes, good question: faye.

D

B

So it needs a security uh privilege security context to get the attestors for the um to get information about the process calling the api. Thank.

D

You for asking.

B

So I have here and.

B

Spire running so now, I'm I will create the registration entries to map the identities to be given to all the workloads and they will be for looking for application. So.

D

B

Let's take a look at one and, for example, for the workload booking for details. I'm saying that you will be. You will get this spfd.

B

um You will be assigned to this parent id this the identity of the spiration itself. When aspirations starts, it will um connect pi server and through a process. That's called a name and note at the station.

B

You will get the its own identity, its own, as with its own explanation, certificate to establish a mutual tls communication with aspire server. So this will be the spf id of the spyration itself.

B

It's using a kubernetes psat, a tester to establish trust between inspiration and spy server, and I'm here, I'm defining five selector, a six selectors for um mapping this identity to a to a to a process. So, for example, the docker image do calling the expiration sds api should have this docker image id with this sha 2 56, um the name space quantizing space will be default and there should be a second um docker container docker image container running.

B

This will be the actual local image of the book info details service and then there should be a label on that port. With details, a service account should be this and the pros unix user id of the process calling the the expiration sds api should be this one. So expiration will be combining all these six selectors to um that that match that should match the process. Calling the envoy proxy calling the the api um it's just one selector of this six is that it doesn't match. You will not get the the identity so.

A

Christian does that answer your question early.

C

It makes sense about the the selection right and um you know the attributes that you're looking for to verify.

C

But then the question is: how does how does the agent know that the person calling you know the person, the the workload calling the agent is the one that has those attributes.

B

um I think it the trust, relies on on the on the platform, so expiration will in suspects uh the process using the kernel, unix kernel or we'll ask a kubernetes couplet ask him for some specific process to get information to these selectors.

B

I don't know without asking us.

D

Answer your question: yeah.

A

I think that also goes back to what faye was asking around running as privileged right. We also have a user from the audience alexander hai also said the host pid. True, I assume that's a privilege. We give to a spouse agent too, to be able to do the attestation.

B

Okay, I will, I will deploy histo.

B

This is the the control plane and I will deploy.

D

It takes a while.

B

Everything is fine. I will deploy the book info.

D

So it's starting.

D

um If I go now to.

B

I will forward the.

B

Traffic into the cluster to be able to access.

D

To access the product page.

B

And it's working um so what they want to show now is what happens if I change one of the selectors in this identity mappings and show that, in that case, the workload that should be given the identity will not be able to to get that identity if one of the selectors doesn't match.

B

So I will recreate one of the identities with another selector. I.

D

Will delete this, I will create a different selector.

B

In this case, I changed the unix id and that it was 1336. I changed it for 1000 and.

B

If I restarted this, this part, I will just.

D

Delete it to make it restart.

D

B

Now, if I reload the page, see that that workload is not working because it's not getting its identity.

D

So I will regulate it.

A

By the way, do you have mutual dr street enabled oh you're just familiar at the moment? Yes,.

B

I I'm using visual dls strict, enable yes, okay, so if one of the workloads doesn't get this identity doesn't have its certificate, uh you will not be able to establish mutual delays with their.

A

We have another question from faye: um if you have, if I have a deployment with uh 100 replicas, I assume spell agent is smart enough not to call spell server 100 times to issue search. Is that correct.

B

Yes, the spiration.

B

Will ask for all the certificate that is isn't titled is allowed to provide to to the workloads and will cash the certificates for the period and all the the certificate shot. It has half a ttl.

B

So for that period it will inspiration, will cash the certificates and will provide a certificates to workloads from the from the cache and before the certificate expire aspire. It will ask the spy server for the new ones, so the the certificate will be provided in a very fast way, very quickly and very efficient way.

A

Okay, that makes sense, does.

C

Is are the private keys delivered over this way too?.

D

D

C

Them and caching, those as well yes.

B

They're attaching a custom memory. Yes,.

A

But it's only delivered to the local node right, because every single node you have this agent running as a demonstration. So the way you deliver it, you don't deliver it over the network. I guess that's what I'm saying.

B

Exactly yes and the despiration exposes serve the local node api is local api or that support as a support sds api is now local is not served forward than ever over. The network.

C

And and private keys are never exchanged over the network right exactly.

A

Awesome anything else that you want to show in the demo uh max.

B

No, the the thing I think.

A

B

It I I recreated the registration entry with the correct selector and now the application is working, because you know they will could could get there. It's identity and it's that's all. I wanted to to show you.

A

That's great, uh this is so exciting. um I I guess I want to pick christians foreign next to you know to ask you christian uh what if I have workloads running on vm and I want to use spell as my identity provider. Do you see that as a common user cases, I know you recently, you've done a lot of work in their area or anything you want to share with our audience.

C

Yeah- and I was hoping to have a demo for this, but I ran into some dns issues. I think, with the with with what I was setting up, but um I'll be able to knock that out and I'll try to record a video, and we can attach it to to this later but um yeah. I I think the uh you know we can.

C

We can discuss various uh approaches to doing this, but uh the approach that I was gonna take was installing a spire agent onto the vm itself and using the uh the various plugins that exist for the agent to do node attestation, and this node happened to be running in gcp, and so you know connecting that agent up with the spire server that in my environment would have been running in istio, but maybe in a real environment.

C

Fire server would be running outside of istio somewhere, but either way having the agent connect up to the aspire server, and then that vm agent. That's where you know you have to mount the unix domain socket in the correct path and have the envoy proxy pick that up um and.

D

E

C

Looked looked to me like it was working, just fine, uh a caveat to that, and maybe this is you know we can dig into this more in a separate.

C

uh Hoot is the way that I was going to use the envoy proxy.

C

So envoy, in kubernetes, as you probably know, relies on some automatic redirection of traffic to the sidecar, and you know between the application and the way that I was deploying the sidecar onto the vm was in what we call white box mode or istio even calls it kind of buried in the code in the documentation.

C

But um white box mode means the proxy isn't relying on any redirection in the vm, for example, iptables, because you know maybe there's a vm workload that has already has workloads running it and you can't start messing around with ip tables.

C

So you know running the istio proxy in a white box mode just means hey, everyone can see it and if you're going to use the service mesh, the workloads running locally have to communicate directly with the proxy right. Instead of talking to example, you know servicea.example.com they'll talk to localhost colon 7001 or something right. They are specifically talking to the proxy or maybe you're using the hdb proxy, but you know you're forcing the app to do it that way, but nevertheless, redirection happens explicitly to the white box mode proxy.

C

The proxy is getting its certificates from the uh spire agent and you know there's some setup that has to go in. You know registering the the agent a little bit ahead of time and that kind of stuff, but but then you get the spire issued spiffy identities uh that are then compatible with the istio running in the in the cluster, the kubernetes clusters, as well as the vms. So, like I said I'm this close, uh I didn't have the time to to figure out that dns thing that I was seeing.

C

But um you know I'll post the I'll post, the the demo afterwards.

A

Well, thanks for the update sounds like you are extremely close. Maybe we can have another hood, uh just debugging. You know connectivity with vm in general in.

C

Service mesh and specifically.

A

Users, well as identity provider. I think that's super interesting um yeah. Well, that's a fantastic demo max! You showed, I feel, like I've learned quite a lot today, like there's no uh change to the control plane of istio right. The only change is in the data plane to enable this. uh The change to issue is really simple. You just change the cycle, template to add the the new socket and then deploy the spell agent and spell server and then you're pretty much ready to go, and then you can craft your attestation registration rules.

A

So that's interesting to see how that works. So thank you so much for explaining that to us um any other things you guys want to share before we wrap up.

B

Yeah, um I would like to mention that when you have a cluster with a lot of identities to map, there is an extension of spire server. This is fire kubernetes, vocal register that implements um a validation and mission record, and this facilitates automatic workload. Registration within kubernetes, so entries is the registration entry. This identity policies will be automatically created for pots, and so you, you will have more or less flexibility how to define these identities uh based on the on the mode you you use with this registrar.

B

That is really very powerful and really helpful when you have a cluster with maybe hundreds of even thousands, and that is to define.

A

That's a great, as you know, thanks for showing us the hard way to do it manually, though we appreciate that yeah anything you want to show christian.

C

um Not specifically, just that you know, keep an eye out for istio 1.14, that's coming.

D

C

Getting people to actually use this would be will be awesome. I'll get feedback from from the community and.

E

D

C

I'll try to post that demo integrating vms. As soon as I can.

A

That's awesome, so uh how do folks reach out to you uh max? I assume it's still slack or twitter.

B

Yeah you can reach me out on spfislac, I'm on speaker, slag or and you're enjoying also speeches luck, and there is, it's also a great place to ask questions, and there are a lot of amazing people who are willing to help you out. So please don't hesitate to ask questions or ask for help on this if you like a special slag or just uh send me a dm on speedy vloggers with the initial slack, I really more than willing to to help you and also let me get in on twitter.

A

That's awesome. Thank you max uh christian. How do you folks reach out to you.

C

Both slack uh twitter, twitter, I'm christian posta, all one word and then on slack various cncf kubernetes istio solo slack as uh c c e, posta, p-o-s-t-a, so yeah just reach out and I'm happy to uh to switch out as well.

A

That's awesome: uh we got um exposure comment about interested in the deploy spell file you made so max. I will work with you. Offline, maybe publish the demo instructions you have today. Certainly we got a lot of interest from our audience today with that. I want to thank our wonderful speakers, especially max for pulling out a demo explaining us the architecture and everything I feel personally. I've learned quite a lot today too. So thank you so much and thank you for christian uh for discussing this work and we look forward to your vm demo too.

A

So folks, I want to you know, thank you, everyone for joining and I'm super grateful for everyone who liked our past uh hood live stream, who subscribed to our channel so really really appreciate that please hit the subscribe button. If you don't want me any of our future episodes and happy learning, we'll see you next tuesday thanks everyone.

A

D

A

Thanks, alright bye, bye.