Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Throw Away Your Passwords: Trusting Workload Identity - Ric Featherstone, ControlPlane

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Throw Away Your Passwords: Trusting Workload Identity - Ric Featherstone, ControlPlane

Trust is required to secure our systems: we need it to bootstrap infrastructure, to run workloads, and to reassure our customers of their privacy. But how do we establish and secure this "trust" in a dynamic cloud native system?

Historically we relied upon identifiers such as IP addresses, passwords, and certificates, but can we do better than these antiquated authentication mechanisms? In this talk we:

Demystify machine identity and its relationship to secrets management and access control
Discuss the issues with historical approaches in a cloud native environment
Solve the "bottom turtle" trust bootstrap quandary
Appraise the open source implementations and technologies available to you
Demonstrate practical examples of how to acquire a workload identity or secret zero
Strive for a world in which passwords and static keys are replaced by dynamic credentials and hardware roots of trust

A

All right good afternoon, everybody welcome to throw away your passwords, trusting workload, identity.

A

My name's rick featherstone, I'm a engineer at control plane for the last couple of years. I've mostly been working with secrets, management and machine identity.

A

I work for control plane, we're a cloud native security consultancy. We do security, audits, consultancy, pen, testing things like that. Hopefully, you've had a chance to come to our stand and talk to people. If you haven't pop over before you go home.

A

So on the left are the things that I've claimed I'm going to talk about this afternoon and so we're talking about trust uh in the context of secrets, management and access control, but focusing primarily on machine identity. What it is, how do you get one.

A

We're up against the post lunchtime dip at the end of a long week, so you're, probably all as tired, as as I am, I'm going to attempt to cover a lot of ground in sort of 30 minutes, or so I'm going to focus on concepts rather than going into too much depth on some of these things. The idea is to give you options that you can kind of take away and apply to the things that you're doing when you get back to your day, jobs.

A

Just talking about some of the things that are available to you, some of the things you might already know. Hopefully some of them- you don't yet. Otherwise it's not going to be very interesting talk for you.

A

So historically, infrastructure was reasonably static. Iop addresses were statically allocated, so you could reasonably use these to represent your identity, firewalls used ip and port combinations, ip addresses were used in x509 certificates, secrets and certificates were manually deployed to machines by admins.

A

You might use protocols like kerberos to initiate trust between machines. If you didn't really like yourself very much.

A

But then cloud computing fundamentally changed the way that we do infrastructure.

A

Ip addresses are no longer a useful identifier. Workloads come and go. Your workload will get a different ip address as it gets restarted. You could have multiple instances of your workload, so they've got multiple ip addresses. At the same time, secrets distribution becomes a much harder problem with these dynamic workloads.

A

How do you get the secrets on there with more services wanting to communicate with each other? The old ip and boundary-based approaches that they no longer work.

A

Sequence: management, solutions like hashicorp, vault cloud secrets, managers, cloud kms, they help with the sequence distribution, but in order to access the secrets, we need a secret to get access to the secrets management solution.

A

This is commonly referred to as a secret zero and that's synonymous with workload identity. This thing that workloads can use to prove who they are.

A

So what do we need? We need a trusted third party issuing workloads with their identities.

A

We need a way for these subjects to retrieve their identities in a secure manner, without the need to have a secret in the first place and possibly without any knowledge of the identity that they're being issued.

A

We want these identity documents to be short-lived and we want the acquisition to be seamless um and and simple for these workloads.

A

um We need a way for the relying party to be able to ver verify uh the correctness and the valet the validity of these identity documents, so, ideally, the basis for that verifiability is cryptography and the format of choice for those identity documents will be jwts or jots and x509 certificates.

A

So that all sounds great, but how do you get one? How do you get this secret? Zero, this workload identity so we'll start simple, something with something that you're probably uh already familiar with, and then we'll sort of build on that so who's running workloads on kubernetes at the moment, quick show help me out a little show of hands yeah loads of you excellent who's heard of the token review api.

A

Oh that's surprising, I thought I thought we would have more hands for that one.

A

So, basically, we've got two two service accounts, the subject and the relying party they've both got um service account tokens and uh we can use the token review api to verify that token and obtain some information about the subject.

A

It's a simple api call, the um the subjects token in in the payload there and then the relying parties token uh in the authorization bearer field there, and we can just make this api call if you're passing jots around when you're making api calls and things like that, you want to make sure you're using a secure channel to protect yourself against interception and replay attacks and once you've made that api call from the response. You can get information back about the the subject.

A

The username, the fully qualified name of the service account there. You might use the groups and now you can use this information to start to make authorization decisions, and this is the most common way of integrating uh kubernetes workloads with hashicorp, vault and kubernetes authentication method.

A

So now we can get dynamic secrets, passwords api keys certificates from vault, that's pretty cool, so the trust domain here is rooted in your kubernetes cluster kubernetes is your identity provider and if you're, building something that's going to run in a single kubernetes cluster, you can build on this api yourself.

A

It's it's not it's not complex, and you can use this to then start to make authorization decisions.

A

If you are passing, these service account tokens around you, you need to be careful about um about this. The tokens themselves they exist for the lifetime of that service account they're replaced if you delete the token but they're not rotated, so there's some things. We can do to kind of restrict the usefulness of a compromised token.

A

Everybody heard of bounce quick show of hands. You heard of the bound service account tokens.

A

They're sometimes called projected service account tokens. They use the projected volume functionality they're bound by audience and time. So in this example. Here, if our demo service is malicious and we pass it, our service account token, it can't use our token to impersonate us to another service because it's bound uh by that audience there um and this this works kind of within a single kubernetes cluster.

A

But if we're let's say, we've got vault on outside a cluster and a service account in the cluster talking to vault.

A

Vault needs to verify this token, but it can't use a bound service account token, because it's not in that kubernetes cluster. So now we need to create a long-lived service, account token and share that with vault in order to be able to validate our token.

A

So now we've got a sequence management problem. Again we can reduce the risk around that token by just giving it permissions to create token reviews.

A

But can we do something so that vault or any other relying party doesn't need a token to validate our tokens?

A

Well, unsurprisingly, we can so since version 1.2, uh 1.12 of kubernetes and stable since 1.20 the service account issue. Discovery feature was added quick show of hands if anybody's heard of this before uh quite a few, not, but in my experience this isn't quite as well known as the token review api, um but it's based on the oidc discovery protocol, which gives you the means to dynamically retrieve the details.

A

You need to validate these tokens and you can have it set up so that you don't need any uh authorization in order to get this information, so you can verify the identity from the token and you can get basic profile information uh stored in the claims in in your job.

A

Your relying parties should maintain a a white list of trusted issuers, otherwise anybody with an oidc compatible issuer can can authenticate. um So now you can start to extend your your trust domain and in order to kind of really understand how this works, we need to understand. What's in a job.

A

So it's it's a it's a simple serialization format: we've got a header, a payload and a signature, base64 url, encoded and separated with a period. So we can pipe that through jq. If we want to look at the claims uh in the in the payload of a token, you can't decode the signature, because it's not utf-8 encoded but you're generally interested in the payload. You want to see what claims are when you're messing up with tokens.

A

You might be interested in the header, um so there's no, no need for you to copy your service, account tokens and paste them into jwtio just to see. What's what claims are in there, you're welcome.

A

Okay, so here's the decoded token it's in the header there we've got the algorithm and the id for the signing key and we'll come back to that in a sec and then in the payload we've got various claims.

A

For the token there's, we've got some around the validity of the token the expiry time, not before the issue that time and then we've got information about the subject and also the audience. So so, who is this token valid for but key to the oidc discovery? Is this issuer field so we'll go back to the previous slide.

A

So with that issuer field, you can append this well-known suffix onto there and that will give you the discovery document and from that we can extract the url to get the json web key set, which allows us to validate the signature. And then we know it's a valid token, and then we can trust the claims that are encoded. In that token, it's it's a fixed url out of the box with vanilla kubernetes, but it is customizable um and so and and it's also protected with uh vanilla kubernetes.

A

So if you tweak that url, you need to tweak your your r back so that you can access it or you make or you make that um unauthenticated.

A

But it's also now supported by your major cloud providers, but they're managed kubernetes offerings it's previewing as in azure, because they kind of went off in a different direction initially, but they've now course corrected and are going down their idc route.

A

So what does it look like just to kind of reinforce this simple protocol we'll go through an example and look at the data structures for each of the major cloud providers.

A

So that's that's the payload for uh gk. You can kind of see the issuer. You can tell from that that it's a gke cluster. We append a well-known url on the end of that, and then we get the discovery document.

A

We can see some metadata in there and the key one is that jwks uri and if we do a get on that we get the key set. So you can see there the key id. We talked about a little bit earlier, the algorithm and then we've got the the modulus and the exponent for the rsa public key, and then we can reconstitute that public key and verify the signature.

A

Eks looks the same. Take that issue afield. Stick the well known url on the end, get the discovery document slightly different metadata, but the same kind of thing. The key field again is that jwks uri and we get the key set and for aks same sort of thing notice the trailing forward, slash on the issuer field there. If you're building something yourself. It's I mean it's a simple protocol.

A

It's get, extract, get and you can validate these um and the oidc spec tells you to check for that trailing slash um so that you can uh so you can get it so you can build. You can build yourself, a simple codes to get the keys that you need to verify these tokens and extract the claims.

A

So there's the discovery document and the key set so in the key sets that we've seen so far. You've only got one key in there, but periodically these will be rotated and so you'll see more keys in that list there and that's why the key id is important. You need to make sure you're getting the right key.

A

Otherwise your signature's not going to not going to validate, even if your token is valid, so that key ideas is very important.

A

So that's all well and good.

A

What can we do with it? Well, you can use it with anything that supports our odc discovery. It's a it's a widely used protocol. All the major cloud providers allow you to hook that into their web identity federation. So you can take this token and swap it for temporary cloud credentials.

A

So you can access cloud secret stores, cloud kms, basically all of the cloud apis.

A

So now you can start to run your infrastructure's code pipelines on your kubernetes clusters with temporary cloud credentials, you're not having to put long-lived cloud credentials into secrets in your kubernetes cluster. You can exchange your token uh for these temporary credentials so and you can go across cloud providers as well. So, for example, you could run in gke and you can you can access aws, so I am roles for service accounts, the um from aws that builds on this odc discovery um and this web identity federation.

A

So there's uh there's a mutating web hook that aws provides. You can install this into your gke cluster and then you annotate your service account with the. I am role that you want to get credentials for and then, when you do a deployment, it will mutate your pod and it will automatically um it will automatically uh mutate and add the bound service account token for you, and also the environment variables required to use the aws to authenticate for the aws apis.

A

So this is all seamless to your workload, just with just with that that web hook in there, your workload can just use the aws apis without having to worry about it. It's pretty cool right.

A

You can use opa to enforce access control based on bearer tokens in your api calls, so you can verify the token and make policy decisions based on the subject, the audience and the issuer.

A

You should probably not just as I mentioned earlier, not just trust any issuer. You probably want to maintain a list of trusted issuers and check against that.

A

But that's yes, we can get cloud credentials if we're passing tokens between services, we can do author authorization using open.

A

But now we're now in the oidc space we can start looking outside of kubernetes to identity pride. It doesn't have to be kubernetes, it could be github.

A

So we can run our infrastructure's code, pipelines in github and get temporary cloud credentials and uh and create infrastructure. So in this example uh we're configuring uh gcp to get temporary credentials. You need to create a service account, a workload, identity pool an oidc provider, and then you do some mapping between the claims in the github token and then bind that to the gcp service, account that you want to impersonate.

A

And then google provided github auth action that allows you to exchange that token for credentials. So you can get an access token to use in authorization bearer field for making api calls. You can get a credentials file and in this case, we're just getting an id token, which is the same as the the kubernetes jots that we've been looking at just so that it kind of looks the same.

A

And if we pipe that through jq, we can see a similar set of claims that we were getting from the kubernetes cluster that subject field there, that that long, numeric string, that's the oauth 2 client id for your gcp service account and you can look in the console and verify that. That's that's correct.

A

What, if you need what, if you need more than this, what if you want a meaningful name in that subject field? Unlike that numeric value we just looked at or the fully qualified name of your kubernetes service account.

A

You might have multiple kubernetes clusters and you want to simplify validation of those tokens between workloads.

A

You might want to use the same workload identity across multiple clusters. Your clusters might be repaved regularly and that issue of fields might change. You don't want to have to automatically keep changing that that configuration for the idc provider.

A

You might want to extend your trust domain over multiple kubernetes clusters or you might want to have separate just domains for each kubernetes cluster, but you want to federate them so that you can simplify verifying those tokens across those kubernetes clusters. You might not even be running in kubernetes, you might be running on vms, you might have iot devices, so we need something a little bit more. We can't just rely on that that kubernetes is the identity provider and that's where spiffy and spire come in, so spiffy defines the standards for an identity framework.

A

The spiffy id is the basic, the format, the representation of that that identifier, it's a it's a uri and it's and it's made up of uh the spiffy protocol. The trust domain in this example acme.com and then an arbitrary string on the end as your workload, identifier, billing.payments here.

A

Those svids, the spiffy, verifiable identity documents, they're short-lived and rotated frequently the trust domain itself provides you with a bundle which is basically the root keys that allow you to verify that s vid and it could be a jot, but it could also be an x509 certificate.

A

The workload api- that's that's how your workloads get get these identity documents um and other workloads can get the trust, bundles and verify these identity documents and spire is the implementation of these spiffy standards.

A

Aspire has aspire server, then you've got agents that run on nodes and they attest themselves to the server, and then the workloads will attest themselves to the agent running on the node that they're running on and use the registration api to map selectors, basically metadata, depending on how they test themselves to the workload you map that to spiffy ids and when you're, when you're, creating your spiffy ids for your workloads. You map them to the spiffy id of the agent.

A

So now you're tying your workload identities to the nodes that they can run on, and then the workloads just use the api to get their their s-vids and they don't need um they don't need any secrets up front they. Just by virtue of the attestation, they can get these identity documents.

A

So we've got a list of the uh of the plugins listed in the documentation there, so the node ones agent to server top three. There are our pre-shed secrets. The next two are for kubernetes. um The sat is a regular service count token. The psat is a bound service, account token and then the next three are cloud identity documents.

A

So you can use uh kind of information that this node already has or that you've pre-populated and stored on there and then for workload at station three key ones there unix docker and kubernetes, and the um the selectors. So for the unix, for example, the selectors might be user id username group id group name for the process. That's running that workload. uh It could be the path to the binary on the file system that the digest of that binary itself for docker it'll, be things like the image id labels uh environment variables for kubernetes.

A

It's your service account your namespace, the image things like that kind of just just metadata about this, these running workloads and then, when the workloads come up and talk to the node they're just attesting themselves, the node's checking for this. This metadata mapping it knowing which identity to assign.

A

So in this example, deployment we're running on a on a single kubernetes cluster using the projected service account token. The bound service account token for the no data station. Now under the covers. That's the token review api that we saw at the start of the talk um same as the vault authentication, the uh the workloads. Are you because we're running on a kubernetes cluster we're just using the kubernetes workload data station, the oidc discovery provider, that's an optional component, and that exposes the endpoints to retrieve the discovery document and the jw ks url.

A

Let's have a look and see what that looks like, unsurprisingly, the tokens look, the same. The discovery document looks very similar.

A

And the key set looks very similar.

A

So we talked about x59 s vids as well. We focus kind of primarily on jots before and and the reasoning for that was because you can get these from your kubernetes cluster. You don't need to install anything else. If you're running kubernetes, you've got uh you've got some administration of that kubernetes cluster, but without with the service account issuer discovery stuff.

A

You can start to use the tokens that your kubernetes cluster is assigned to to authenticate to things outside of your cluster and that's kind of it's not free, because you've got to manage a kubernetes cluster, but it's freer than installing aspire server and an agent somewhere else.

A

So that's just some of the information in the x 509 s vids! You can see in the uri sand field. There we've got the spiffy id so just domain arbitrary, workday identifier, um and if we're running uh some, um that's that's kind of different to service account names. It's a nice, meaningful uh identifier, so that's kind of useful. We can see that we can use the certificate for server and client authentication. So we can now start to use this for mutual tls.

A

um We can log into volt with this x509 certificate or anything else that supports certificate-based authentication.

A

Spyro itself implements envoy's secret discovery, service sds, so envoy, it's if you're running envoy kind of as a lightweight service mesh that can retrieve these x509s vids and use those for your mutual tls and now you're, using the same identity for your um for your certificate for your tls that you'll be using.

A

If you're going to pass your your jots around, whereas if you're running a service mesh like istio istio is issuing different identifiers for your certificates, then kubernetes would be issuing for your jots and again you can plug envoy into oppa for policy based authorization decisions.

A

So I did say something about hardware route of trust, um so the rest of the rest of this deck gives an indication of where, where I think uh kind of workload, identity is heading kind of trying to tie that trust down to the to the hardware level. There's been a couple of interesting talks already this week uh wednesday, there was talk about the confidential containers project using running kubernetes workloads in trusted execution environments.

A

There was a talk this morning about running kubernetes in interested execution environments really kind of interesting stuff like right down to to hardware level uh at station security. um So this is kind of this is something I'm just starting to look at.

A

So please don't ask me any difficult questions at the end about this, but I wanted to just uh give you some information about some of the things you might want to go away and look at and keep an eye out and say it's it's an evolving field, um and so some of this stuff isn't quite production ready. So don't think you can go home and and start to run this stuff, but it's definitely worth keeping an eye on so trusted.

A

Executive environments, we'll leave that guys who know what they're talking about much more than I do I've already spoken about that this week. um Actually I've I've just scuffled myself there, because I'm making up some of this stuff as I go along uh so tpm's quick show of hands. Who knows what what a trusted platform module is.

A

Oh, that's interesting. More people know about that than know about the kubernetes token review api. That's so you can't make assumptions when you come and do things like this. So tpm is a is a cryptographic device. um Does things like secure generation and storage of of keys and these keys can't leave that tpm in a in an unencrypted form? um So that's really that's really useful property for for tying things down to a specific machine.

A

So we earlier, we saw that you could use an x509 certificate to attest to node to spire, but if I get hold of that certificate and walk off with it and put it on another another machine, I can impersonate that node with the keys from a tpm they're they're encrypted outside of that tpm, and you have to load them into the tpm to decrypt them and use them. So I can't I can't steal your certificate and pretend to be you. I have to be you with that tpm.

A

um So that's! That's! Quite a nice property uh pcrs they're special registers, you you can read from them, but you cut you, don't directly write to them um the you can you basically extend them so um the new they store a hash of of something.

A

So the new hash is the old hash, plus the new measurement um and key lime really interesting to look at so this uses it with the linux ima subsystem, and so it's doing remote attestation. So you can.

A

You can use that to kind of scan the files on on your node. Stick that uh the hash of that into the uh the pcr and then keylime can remotely check that. So that gives you the ability to continuously verify the integrity of your remote machines and on the keylime site. They've got a nice little demo of removing a compromised lcd node from a cluster.

A

So we can use this with with spire. Now I only came across this last week, frantically playing around with it trying to get it to work. So I could, I could get it in here, but it's it's! It's pretty cool, um so you've got a a local device id which is a a public and a private key and a certificate that are provisioned out of band, um and then they come out of the tp, the the tpm encrypted um and then in the attestation process, the agent loads them into the tpm.

A

The the server will then go through a challenge process um that can only succeed if, uh if you've got the keys in the tpm decrypted, so now we can.

A

We can provide harder guarantees that that node is than the node that we thought it is. um The spiffy id uses the fingerprint of the certificate. So it's it's very! It's very predictable, um just like the x509 um and the ssh certificate um attestation methods.

A

So that's it's not. I only found it last week because it's not on the main documentation site for the uh for the spy stuff, but it it looks, looks pretty interesting.

A

And that's it so key takeaways from my rambling this afternoon you might already have a workload identity if you're running on kubernetes, you have you've, got your service account token and, depending on on what you're trying to do. Your kubernetes cluster might work as your identity provider. If you're running on kubernetes and just talking to volt um on the same kubernetes cluster, then the token review api might be fine.

A

If you're just running kubernetes workloads- and you want to get temporary cloud credentials, then maybe the oidc stuff works to just swap your token.

A

For for that, if you would kind of go into the wider fields where you want to use the same job and the same identity in your job and your x 509 certificate, then you might want to think about uh spliffy inspire. You kind of you've got more complexity um and an operational and administrative overhead to run spire, but if you need the extra uh capabilities that gives you um it, it's certainly worth thinking about it.

A

So yeah consider spiffinspire see if your. If your needs really require that and investigate things like tpm's, key lime and trusted execution environments.

A

And that's me done.

A