Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Bridging the Great Divide: SPIFFE/SPIRE for Cross-Cluster Authentication - Andrew Harding, VMware

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Bridging the Great Divide: SPIFFE/SPIRE for Cross-Cluster Authentication - Andrew Harding, VMware

Cross-cluster authentication got you down? Losing your hair trying to get mutually authenticated TLS inside, outside, and everywhere in-between? Fret no more! In this talk, Andrew Harding, a maintainer on the SPIFFE and SPIRE projects, will dig deep into a turnkey SPIRE deployment within Kubernetes that provides workloads and proxies with X.509 certificate-based SPIFFE identities. Andrew will demonstrate how to use these identities for cross-cluster authentication by declaring federation relationships between clusters using familiar Kubernetes primitives. On top of that, a live coding and demo session will show just how easy it is to leverage SPIFFE from within Kubernetes workloads with just a few lines of code.

A

It's good to see you all it's good to be here uh together in person with our virtual attendees as well. um My name is andrew harding. I'm a spiffy, spire maintainer. I've been involved in those projects for the last uh three and a half years, or so um I'm a staff engineer at vmware, where I'm allowed to continue that work and work on it full time which is which is fantastic. It's awesome um real, quick before we get in today.

A

All these celebrations and and and uh good times, everybody I threw my back out, so I won't be doing any break dancing today and if you see me like tilting to one side, maybe some can run up and kind of, like you know, tilt me back up into the upright position um anyway. Moving on, so we have a real, quick like we got a lot to get through today. So our agenda is fairly short, but each of these sections is going to be meaty.

A

We're going to go over our problem statement, make sure we're all on the same page, we're going to do a very quick, hopefully just a few minute, refresher of spiffy inspire to kind of set the stage for the kubernetes controller work that we're doing and to demonstrate accomplishing cross-cluster authentication using spiffy inspire in this kubernetes controller work. So real, quick, our problem statement, you know, we've we're all very familiar. I think, with with this this sort of setup, we've got multiple kubernetes clusters.

A

We got workloads running within those clusters, the uh intra communication store within kubernetes. You know, there's there are things that work there and that people have done to communicate between workloads within the same cluster. When you start talking about workloads between clusters, then the authentication story gets a little more complicated and there are solutions out there with various pros and cons. Today, we're going to talk about solving it with spiffy and spire.

A

So what is spiffy? It's the secure production identity framework for everyone. It is essentially a set of specifications that are all geared around. How do you get a workload, a cryptographic, identity that it can use to authenticate with other workloads?

A

And how do you get the public key material around so that you know receiving parties of these verifiable identity documents? Can you know, um verify the signatures and verify the authenticity of those documents and get your authentication, so at the heart of spiffy is something called the spiffy id? This is your username for your service, your workload. This is your your identity.

A

uh It's a uri, very basic! uh It's got basically two parts to it. You've got a trust domain part and you've got this like entity, part within that trust domain and the trust domain is essentially acting as a namespace uh for these entities right um it represents a security boundary or maybe a different administrative domain. uh This could be like you know. You could have a company has a single trust domain. Maybe it has a trust domain for prod and staging.

A

Maybe it has a trust me between business units, the banking business unit and the hr business unit or whatever. uh The point is that these these trust remains form this cryptographic boundary the security boundary and uh trust me typically has its own pki or its own set of cryptographic, keys that it uses to sign identities within itself. Okay- and that is what we'll talk about next, which is you've got this id.

A

You want to be able to assert this id cryptographically, and so we put it into a document called a nesvid and we've got both x509 and jot documents in the within the specifications for svids, and this is just a signed document over that id and it's signed again by an authority within that trust domain, and this is what workloads present as their proof of identity. Okay, receiving parties take something that is called the spiffy bundle and use this to verify those identities.

A

Now the spiffy bundle is just a collection of the public keys for the signing authorities within a trust domain. Okay, we we call it like. We called a bunch of different things: it's a trust, bundle or a trust domain bundle. But in you know the specification correct standard is or correct. Naming is the spiffy bundle, so we've got these signed identity documents. We've got these bundles. We can use to verify them.

A

How do we get those documents to the workload itself so that it can use them and the answer there is something called the workload api. So this is an unauthenticated api workloads, don't have to bring anything to the table in order to talk to this thing and obtain credentials over it.

A

They just stand up wherever they are find the workload api say: hey give me my identity, the workload api through all the spiffy magic identifies the workload applies, whatever policy is necessary and then ships down across this workload, api, the svid and the bundles as those secrets or materials change and are rotated, because with spiffy you can get. You can get rotation all the way from the you know, the leaf certificates all the way up through your certificate authorities as those materials change and rotate over time, sometimes very frequently.

A

Those updates are streamed down to the workload they're pushed down to the workload, so the workload always has a you know very quick, snappy idea of like what the what it the identity universe looks like how it's able to authenticate other identities and present its own.

A

Now. The very last thing we'll talk about for spiffy is you know: we've talked about how the the workload receives these documents. The question is: how do other trust domains receive these documents? This is where we're going to talk about federation right if I have two trust domains and they want to be able to authenticate the asvids that are produced from each other, they need to get their hands on the spiffy bundle material for the foreign trust domain, and so this is, uh it's not really called the federation api spec. It's called the bundle endpoint.

A

It's called the federation specification inside that there's something called the bundle endpoint, which is. um You know it's very simple, https endpoint that you talk to to obtain the spiffy bundle and there's rules in there on how you authenticate that endpoint a couple different profiles. You can use to do that, but this is the federation api. It's the way that a foreign trust domain is able to obtain the bundle for your trust domain, so they can verify the identities. The svid documents that come out of your trust me.

A

So in a nutshell, what you get with spiffy. Is these cryptographic verifiable secret zero, like federatable, um you know frequently rotated namespace, uniform identity and that's kind of like the the crux: that's what spiffy is giving you and that's what we're going to build on today or use today leveraged today to do our cross-cluster authentication and we're going to do that with spire.

A

So spire is basically an implementation of all those spiffy standards that we just talked about, and the goal of spire is to take all the functionality you get through those apis, all the interoperability etc and just shove that as many places we can get it light.

A

This thing up everywhere, whether that's you know inside different, hosted cloud services, azure aws, gcp, whatever, whether it's on bare metal servers, whether it's running inside kubernetes or outside kubernetes, like we want to stick this thing everywhere, because the more places you have the workload api and the related you know, spiffy specifications like out there, the more powerful it becomes, the easier it is to just say: hey, I don't I don't care where my workload's running, I'm just going to kick that thing up.

A

It's going to be able to talk to workload, api, get credentials and authenticate, and that's really what we're striving for is kind of like this. This very easy way to to provide workloads this identity in a way that that isn't complicated for them. They just and we'll show that you later we're going to do a little bit of live coding. We'll see how easy it is to pick up credentials off the workload api and do this sort of thing.

A

So, let's talk real quick about the components of spire we're going to start with the workload. That's not spire, but it's. The whole reason spire exists is to like light up this workload and give it identity.

A

uh The workload api we talked about is uh the thing that workloads talk to to get their credentials and that is hosted by a component called the spire agent. Okay, the spire agent is able to identify the workload and take the policy that it has and use that policy to figure out which identity is supposed to go to that workload and or not, and and deliver those materials.

A

The spire agent itself is not the signing authority for the trust domain that is delegated and also the policy is delegated to a component called the spire server. This has a management api that you know, operators can use to configure and register workloads and federation relationships and all that jazz. It's also the component that hosts the spiffy federation api. So, in our example, we'll have these spire deployments running each cluster. The inspire servers are going to be talking to each other to obtain the spiffy bundle for for the other trust domains.

A

So we can do our our federation across cluster authentication. So how do we typically deploy spire inside kubernetes ta-da? We did it there. It is.

A

We've got uh spire server running in either in a deployment or in a stateful set depending on you know, a few different ways that you might want to decide to deploy that the important bit is that on each node, we've got a daemon set, that's going to be running the spire agent, so that no our uh you know, workloads that come up on that node can talk to the workload api hosted by that spire agent to get their credentials all right.

A

So the next bit we're going to talk about is this new work that we're doing in kubernetes? Now I say new, because this is this is a new body of work, but it's not as if spire hasn't had kubernetes integration.

A

Before this point, we've actually had support for a couple years in a component called the the kubernetes workload registrar. um The kubernetes workflow registrar, is something that the community heavily contributed to their people are using in production. It's got a couple different operating modes, and it solves mainly the problem of like how do you automate the registration of workloads within your kubernetes cluster so that they can talk to the workload api and get their credential.

A

We've learned a bunch over the past couple of years. Seeing this thing uh happen in production, seeing the type of features that it has grown over time, and we think we can take those learnings and and go even further and make it even more turn key to light up spiffy inside of inside of kubernetes with spire. So right now we currently have two crds defined with this work. We've got one that'll allow us to declare workload identity.

A

One that'll allow us to declare these federation relationships and we'll show how that's gonna, how that's gonna break down, and the idea is that these controllers are gonna. Look at these. You know declared resources inside kubernetes and when they reconcile they'll, take a look at fire server management apis and decide what has to change in order to reflect the declared state of the world back into spire servers.

A

um Specifically, we've got. This is the the workload identity crd, it's called the cluster spiffy idc rd. It allows you to define the shape of the spiffy id that your workload's going to get. It allows you to declare what sort of trust domains that workload should federate with.

A

It also allows you to kind of sharp shoot and target the specific nodes in the specific name spaces where this identity should be available.

A

The second crd is called the cluster federated trust domain. Crd super basic has the the name of the trust domain that you want to federate with, and also we talked about that bundle, endpoint specification and the ways you authenticate and connect to that bundle. Endpoint.

A

uh It has all the materials and all the the information that you need to talk to, that endpoint authenticate it in order to receive the bundle and that's it so far, we might have other control or sorry other resources that we added in later points maybe go down the operator route, we're still all this is still in flux under uh design and development right now. So, um let's get into our demo architecture.

A

So we will start today with our typical spire deployment site kubernetes and then we're going to toss in this new controller into the mix. Okay and the controller is going to look at the kubernetes api watch that get reconciliation events for these different resources and then it's going to reconcile the state of the world with aspire, server, programming, the workload identity and the federation relationships that are needed and then of course, spire server is going to be applying policy and what not? And it's going to end up.

A

You know at some point uh shipping these s and bundles down to the spire agent and on to the workload and that's that's our whole picture within the single cluster now between clusters. You can see here on the left. We got cluster one on the right. We've got cluster two and we've got we've kind of lumped all that spire infrastructure into into one little box. There named spire on each side and then we've got our workloads. We have a greeter server running in cluster one and we have a greeter client running in cluster two.

A

The spire servers are gonna, be programmed using that cluster federated trust domain, crd to federate, with each other, so they're, going to they're, going to start talking to each other and pulling spiffy bundle material into each other's trust domains and, of course, the workloads are going to connect up through the workload api and obtain those s fit and spiffy bundles over the workload api and then they're going to talk to each other over mts and do mutual authentication all right.

A

Let's do the demo.

B

B

Let me fix my display up here, real quick, so I can also see my terminal all right.

A

You guys see that okay do. I need to bump up the font size.

A

I'll go a little bit bigger, never be too careful. Okay, all right! So, uh first thing I'll show you is this workload that we've developed? Okay, uh we're not going to write it all by by uh scratch. That would take way too long, so I've got most of the code in place.

A

Let me show you what it does first, so the greeter server I'll just uh build that real, quick and the greater server is just going to be sitting there listening on port 8080 over plain text, because we haven't changed it to take advantage of the workload api yet and on this side we will start up the greater client and this thing's going to connect to the server it's going to issue a request and say: hey, say hello to joe and the server is going to respond with a message that said: hey someone wants me to say hello to joe hello, joe type of a thing right, but if you notice on each side like the server doesn't know who the client is that connected to, it has no idea that connected over you know this plain text connection issued the request.

A

It doesn't know the identity of the client. Likewise, the client doesn't really know who satisfied the request right it. Just it's some server, okay, so we're gonna fix that up, so that each side can not only authenticate but but look at the identity of the the calling party. Okay.

A

So let's make those code changes. Real, quick and really. The point of this is to demonstrate how easy it is to wire up the workload api into your into your applications. So um we'll start with the server all right. Here's the server code very easy. uh We have a little flag to to configure our listening address. We create our listener, bind or report right then down here, we're going to create our grpc server and register our implementation and then we're just going to serve that grpc server forever right.

A

um Our implementation is very trivial.

A

uh It essentially receives the request, does a little bit of logging and then responds with a little message to say hello to whoever it was asked to say hello to we've got a couple of placeholders inside our our implementation. We've got a a to do up here to wire up the workload api to get some some spiffy credentials to power, this grpc server and then down here. We've got a little to-do uh to extract the identity of the client from the request context, so that we can log it and send back the appropriate message.

A

So we'll start here with this first one uh we're going to be using a library that is provided in the spiffy org.

A

It's a go: spiffy library, we've got a version 2 that we've been working on for quite some time and that's what we're going to use to do this work? Okay, so the first thing we're going to create is something called an x509 source. This is a source of x509 materials that we're going to obtain from the workload api so we'll create that source. Here, I'm going to borrow some error handling from the thing above and when we're done, we should close this source down.

A

Okay, and so now we have a source for x59 materials. This is both for the svid and the bundles. So the next step here is we're going to create some grpc credentials. You notice that when we created our grpc server, we didn't configure it with any credentials.

A

So, let's create those credentials now and for this we're going to use a little helper package.

A

Provided by the ghostbiffy library to give us mtls server credentials and we're going to give it the source twice here- and this is a very low level and flexible library- the first parameter here is the source of the svid. The second parameter is the source of the bundles, we're going from the same source for both materials, so we'll pass the same source twice and then we're gonna.

A

The third parameter here is a little authorization hook that you can do for the server we're just gonna authorize any id as long as it is authenticated against a bundle that we that is provided by the source, we'll allow the connection in.

A

uh But you could imagine if you wanted hook in your own custom authorization. Libraries if you want to do rego and opel or whatever or something else, a flat file like whatever you want to do. This is this: is the parameter that you'd pass in order to like influence the authorization of the connections?

A

So now that we've got our credentials, we can come down here to the grpc server.

A

Type things right pass the creds into the server okay, and that's it that's all. We need to do to to provide spiffy credentials to our grpc server down here. To obtain the client, we are going to again use that grpc credentials package as a convenient little helper.

A

We're in we can give it the context of the request and it will extract the peer id from that context and if we're able to successfully do that we'll overwrite the value of this client id variable with the identity of the peer, let's make sure that builds.

A

No. It did not.

B

Miss the c there we go now. It should build.

A

Okay, so server's done, let's go to the client.

A

Client again very basic, you can configure the address through a flag. It dials. The server, creates a little client over the grpc connection and then sits here in the loops and every 10 seconds just issues in a request to the server and then down here to issue the request.

A

It just invokes this rpc on the client and then logs out the response.

A

We have the same set of to-do's inside the client that we did on the server instead of here to wire up the workload api and a to-do down here in the implementation of the rpc or on the response of the rpc to extract somehow the the id of the server so just like in the server we'll start here at the top to wire up the workload api. We're going to do the exact same thing here that we did on the server create a next 509 source.

A

B

Borrow some error handling there.

A

And this time, when we create our credentials, we are going to use the mts client credentials and we're going to give it the same source for svid and bundle, and this time we'll do something cute we're going to authorize a specific id. We if it doesn't have this id we're going to reject the the the connection and the id we're going to give it.

A

Here's the spiffy id of the greater server and the cluster one trust domain.

A

Let's hope I didn't typo any of this, otherwise this is going to crash and burn all right. So that's our first to do. We've wired! Oh, no, we're not done yet again we're not providing any transport credentials to this dialing operation. So let's fix that.

A

Okay, that should be it for our first to do all right, we're wired up to the workload api to get our client creds and the bundle we need to verify the server and then down here we're going to do the same sort of thing.

A

Now this time we don't have a request context, so we need to get our identity from appear object and that I went a little backwards here.

A

Where does this pure object come from well to do that, we need to pass something into the rpc call.

A

And grpc will helpfully populate this peer structure with the uh peer information that for the server that fulfilled this request so down here, we'll extract that peer id out and if we're able to we'll go ahead and overwrite server id with the identity of of the server.

A

All right we built okay, so let's go ahead and just make file it's going to build the docker container. It's going to push the server into the cluster into the cluster 1 cluster and the client into the cluster 2 cluster, which I have running, hopefully in a pristine state.

A

Let's make sure this finishes great, so I've got a tmx window here for cluster one. I've got a similar window for cluster two, let's go ahead and deploy all of that spire infrastructure.

A

uh Including spire server, spire agent, the controller, etc.

A

Okay, that's all deployed. Let's do the same thing in cluster two.

A

All right and like we talked about next we're going to federate these two trust domains and to do that, we have that cluster federated trust domain. Crd, real, quick.

A

If you look at the services provided in this first cluster, we can see the spire server bundle, endpoint service, and this thing is- has an external ip. This is what we're going to use for the second cluster, to talk to the bundle endpoint in the first cluster.

A

Creating the crd by hand is a pain, so I got a little script here that will create it for me and if we scroll up.

A

There we go, we've got a cluster federated trust, main crd. It's got the trust domain that we're going to be sorry, we're going to be federating with it's got the bundle endpoint url. uh It's got the profile information. So how do I authenticate to this thing to pull down the bundle? um This is the crd that for cluster one that we need to apply into cluster two, so I'm gonna go ahead and just put that in my clipboard we're gonna jump over to cluster two here.

A

And apply that thing.

A

And now we're going to do the same thing for uh cluster 2.

A

Copy the cluster 2 crd and apply that inside cluster 1.

A

Just to see what we got here, oh sorry, wrong wrong. Crd.

A

We've got this: the crd, that's programmed with the right endpoint url and has the profile information it needs to connect to it. Okay, perfect, all right! So now we should have um if we, if we, if we peek into the spire infrastructure here a little bit um inside cluster, one we're gonna, we're gonna peek into the spire server and say: hey give me all the federated bundles you have, and we can see that it has indeed the bundle for cluster two okay, and we can do the same thing in cluster two.

A

If we want to show that it has the the bundle for cluster one okay. So now now that these things are federated, let's go ahead and deploy that greeter workload, uh server and client into each trust domain.

A

A

In cluster, one we'll deploy the greater server that should be getting.

B

Up and running, and now we will do the same.

A

For the greater client, and now we wanted using the cluster spiffy id crd, describe the identity for these two different workloads, so in cluster one I've got a little crd prepared here for the greeter server. uh You can see here the spiffy id template. Now I don't have any like template parameters on this, we're just doing a fixed id, but there's a lot that you can like fold into this template to take properties from the pot or the node or whatever, and reflect that in the id.

A

We also have a pod selector because we're going to sharp shoot the greater server running in this cluster as the one to whom this identity belongs to, and then we've got a little federates with clause here and this this tells spire to allow this workload to obtain the cluster. 2.Demo trust bundle over the workload api all right. So let's apply that.

B

And we'll go to cluster 2.

A

And apply the same sort of thing for the greater client. I can show you that it's very similar got this fixed id, we're matching on the greater client pod and it confederate with cluster one all right. So now we can look at the logs before we probably should have for this workload. But if we look at the logs now for these workloads, they should be talking to each other. That client should be sitting in that loop, every 10 seconds talking to the server. So, let's dump the logs from that greeter client.

A

A

Logs uh and there we go, this is much better experience, no more some server or some client. We have the identity of uh the server right here right. The greeter server.

A

That the client was able to obtain over that mtls connection and the server was able to pull out the client id from from its request and reflect that in the message that are returned to the server, and so this is uh there's a lot going on underneath the covers. But this is it. This is cross-cluster authentication uh using spiffy inspire inside kubernetes um there's a lot of details. We can dig into.

A

We don't have time in the presentation, but I will be posting up this demo online, so folks can run through it themselves if they want at a later point so.

B

A

What do we do today? Well, we used crds to program our kubernetes clusters with the workload identity that we wanted.

A

We also declared our federation relationships and using those we were able to get two clusters to perform federation with each other and exchange spiffy bundles, so that we could authenticate s vids from from each of the trust domains, and then we using the workload identity, crd, we're able to configure spires so that workloads could connect to the workload api and obtain their credentials and then do mutual tls between each other and get that strong authentication layer.

A

If you'd like to learn more there's a website at the top there's a couple of our github repositories, there's more than this, this specific controller itself has its own repository I'll, make sure that gets published here in the next week or so and then last but not least, we've got our spiffy slack, and this is a place where not only maintainers on the projects, but users of the projects and other interesting parties are we're there. Every day it's it's a very active um spiffy, it's a very active slack.

A

You can come there to get help or just shoot the breeze or to like you know, figure out where spiffy and spark can maybe help in your deployments um and that's it do we have.