Cloud Native Computing Foundation Production Identity Day: SPIFFE + SPIRE North America 2021, 30 Oct 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: AWS OIDC Authentication with SPIFFE at Square - Roy Xu, Square Inc.

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

AWS OIDC Authentication with SPIFFE at Square - Roy Xu, Square Inc.

As part of Square’s migration to the cloud, we found that we needed an easy way for our datacenter applications to communicate with services in AWS. We were able to use the existing SPIFFE infrastructure that we’re already using for our service-to-service authentication and extend that to AWS with federated identity through OpenID Connect (OIDC). Our AWS OIDC infrastructure provides automated AWS credentials for applications in our datacenter to assume roles in AWS. We leveraged the SPIRE OIDC Discovery Provider provided by SPIRE paired with spiffe-aws-assume-role, a custom open source tool we wrote to exchange SPIFFE JWTs for AWS credentials. Our new process means that app owners only need to make a simple config change to allow their applications to connect to AWS.

A

Hey this is roy. I work at square on the cryptographic, identity and secret management team today I'll be talking about our aws odc authentication infrastructure using spiffy at square, so we're going to be going over what is odc and why are we using aws the architecture, our solutions and the current status of our dcf square start with the overview some background about awsome square, we're in the process of transitioning to the cloud with a focus on aws? It's been a slow process, there's lots of apps still in the square dc.

A

These dc apps will often use services in aws, such as s3 and sqs.

A

Apps in aws live on separate accounts, so we have a isolation there and then we also have isolation in terms of the environment that the apps are on, so staging and production will be on different accounts.

A

So what is oidc oidc is no open standard and decentralized authorization protocol. It allows third parties to verify the identity of end users does so by sending data about users in jots, which are signed by an authorization server which in our case would be expired. So, there's no need to use a separate identity and separate authentication for every third party you try to connect to so we'll be talking about a small subset. This oedc spec, which is the discovery provider the discovery provider exposes json webkey sets or jocks for jot validation.

A

So there's two endpoints that the provider has it has the well-known openid configuration and the keys. The keys is pretty straightforward. It just serves jocks.

A

The discovery document contains metadata about the provider, including information about the issuer of the jots, the uri, where the keys are served from and the algorithms used to sign the jots, as well as the response types.

A

So this is an example of a jot. We would have at square for oidc authentication to aws uh in the subject. We have the spiffy id of the app and the audience we have the intended. Aws account. This is like the aws account that is supposed to receive this jot, and then we have the issue that time.

A

So we had three options. When we were choosing what to put in the audience field, we could have had it be the intended app. So if my app was trying to talk to like test app or something that the audience would even test at the problem with, that is that it doesn't differentiate between environments, it wouldn't differentiate between a staging and production version of test app or the dependency, and so we went with the database account and you know at square. We separate uh each app and their environments into a separate account.

A

So this works for us.

A

A third option would have been to isolate isolated further down into the role that we try to we're trying to assume in that account, but there's a limited number of audiences that you can have so we found we chose the middle ground of just the aws account. So this prevents this job from being taken or stolen and used to impersonate the app or my app. So this limits the scope of where the jot can be used so quick overview of how idc and aws works.

A

Aws im support supports using rdc identity providers, so these are tied to provider urls and then the roles can have a trust relationship with these providers. So it can check that the provider is coming from the correct url. It checks metadata in their jaunt and we check for whether the subject has the spiffy id that we expect and also the audience is the correct one.

A

The aws uh s so aws sts allows us to use these jots to assume into a role. You know the role through iem checks, all the information that we need.

A

So this is kind of how uh we were connecting to aws from the dc prior to idc.

A

uh App owners would create another aws user, attach a policy to that and then in the aws console request the user's access keys record, those keys in an aws credentials, file used typically on their laptop and then upload that credentials, file to our secret, store, keywiz and then dc apps would be able to fetch those secrets out of keywiz and then use them to run as native as user.

A

So this process, you know, is pretty manual. There's a lot of steps involved. There's a lot of places to go. It could go wrong, there's a lot of places where the keys could leak and it's very unlikely that a app owner would go and refresh those user access keys.

A

So we've had, like keys that you know have been been beat, haven't been changed in like years, so um the new method, odc method, involves a lot less work from app owners, so we've created a terraform module that automatically creates this role and sets up the proper configurations for the identity provider and attaches an aws policy to it, and all the app owner needs to do is, of course, implement that terraform module use that terrible module and then add a configuration to their app.

A

uh We call that the p2 manifest, so all you need to do is add a a couple lines to their manifest. In this case, you can see here's an example where we specify the role that we're trying to assume into and then a name for that role. So in this case extra role- and this is like basically the have- you worked- the potential files before it's like the profile name- yeah. That's all that the app needs to do the rest of it is automatically done by our odc architecture.

A

So let's go over how that works, so the broad overview of all the moving parts and then I'm going to break this down into smaller pieces, so we're going to start with the p2 hook. So p2 is like our container or orchestrations software.

A

So at startup we uh hook we have a hook that reads the credentials file from their manifest or the configuration and generates a credentials file, an aws credentials, file from that that's stored in the app's home directory and then, when a dc ad needs to talk to aws, they read that credentials file and then inside the credentials file. We specify a credential process, which is one of the ways that you can fetch credentials when trying to talk to aws and the credential process is actually a open source tool.

A

We've written called spiffy aw system rule that tool requests a jot from your spy agent, which is signed by this fire server and then sent off to aws, where it is verified against uh the cached jocks that we have in s3 with a cloudfront domain in front of it. So how do we get those jocks into s3? We have a cron that syncs the results of um the spiffy oidc discovery provider, which is provided by the spy implementation to s3.

A

We'll talk a little bit more about that later, but yeah. That's, basically how the architecture works. The spiffy oidc provider fetches the information from the spire agent, and we sync that to s3 so the spyro rdc scope provider and it's provided by spire. I've included a link for you to check out uh it serves the jocks in order to discover document. In our case, we serve the endpoints through an odd voice socket.

A

Now I'm going to talk about the custom tools that we've made to enable our setup so a little more detail about the p2 hook and how that works. So again, this is a sample configuration that would be in their p2 manifest. So we have the name of the role the spiffy odc test and then the role itself they're trying to assume so and then below.

A

We have what the line would be, what the block would be in their credentials file, so the profile name would be specular idc test and then the credential process calls out to spit their consumer role with a few options. um See here that you know we're assuming to this role in speech, test role with the spf id of the app speak through itc test, and we specify the audience and then the socket for the spire agent.

A

We also have options for specifying the region of sts and the endpoint of sts that you're trying to use this is in case you're, using a ppc.

A

Cool, so we also wrote a cron job to sync: the discovery provider to s3 buckets, and so we did this because we had issues with exposing our cgm points to the public and aws needs to be able to access those endpoints in staging.

A

So instead of trying to expose our staging environment, we just uploaded everything to s3 and then used a platform domain. We also use custom square domains which is possible using acm certificates, um and we did this so we had more control over the domains.

A

This also provides us with, like caching, of the odc endpoints and more availability, since cloudfront can serve it from more edge servers. So we pull the jocks every 10 seconds just in case they are refreshed by spire. We want to keep them up to date and then the discovery document doesn't really change. So we just change it. We just pull it every 24 hours, cool and then our open source tool spit the adapts, assume. I've included a link for you to check out uh the tool, uses spiffy jots to assume into aws roles.

A

So, as you saw before, it's used with a credential process option and it calls sts assume role with web identity with jots that we retrieve from your local spire agent. So it supports retries sports logging. We have metrics, there's uh support for vpc endpoints and you can even configure the sds session duration, which I didn't show before so that defaults to an hour and it's customizable with this tool, so cool.

A

So conclusion: the current status of spiffyrtc, so we rolled out to general availability. We just needed to make some changes to shared libraries to use the potential spells in app's home directories. Instead of fetching it from kiwis. Our secret store we're currently migrating amsterdam idc, but it's pretty been. It's been a pretty hands-off process. Teams have really been able to do it on their own. I haven't really encountered any major issues.

A

Those minor blips with you know we have issues with um connecting to cloudfront and platform fetching from s3 which we're uh working through, but it hasn't really been a blocker cool and you want to learn more about the uh security infrastructure at square uh in aws. You can check out our developer blog on the corner blog, which include a link to there's a more in-depth article about aws or idc authentication, some articles about how we manage secrets in lambda and how we provide identities in lambda.

A