Cloud Native Computing Foundation Production Identity Day: SPIFFE + SPIRE North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Changing the SPIFFE ID of Every SPIRE-Enabled Workload at Uber - Challenges and Lea... Prasad Borole

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Changing the SPIFFE ID of Every SPIRE-Enabled Workload at Uber - Challenges and Learnings - Prasad Borole, Uber

This is a story of migration of SPIFFE IDs of workloads deployed across thousands of nodes in Uber. As our identity platform and core constructs in the underlying infrastructure model evolved over time, we had to undertake an initiative to change the format of SPIFFE IDs for varying classes of consumers. In order to roll out a new SPIFFE ID convention across our microservices deployment, we had to understand the landscape of service-to-service authorization mechanisms in use at Uber in order to update all authorization policies referencing SPIFFE IDs. We also had to plan for the potential effects that creating many new registrations could have on the reliability of the SPIRE control plane. There were few challenges we encountered along the way like hard-coded SPIFFE IDs, lack of ways to choose preferred identity from multiple identities etc. Listeners could benefit from this presentation by knowing more about the SPIFFE ID format and selectors we have chosen from learnings and problems we faced during migration and avoid one in future.

A

Hey everyone. Thank you for joining. My name is prasad and I'm an engineer at uh workload: identity key uh in uber.

A

So this is the story of changing the spf id of uh all spire enabled workloads and the challenges and learning came out of that exercise. So, let's get into it. So this is a brief agenda.

A

We'll go over the background which will involve the current state of spire that runs in uber fraud. Some details about the infra, then we'll move to the problem statement.

A

What problem we were trying to solve and what approaches we took those problems. uh We can go over the challenges and then learnings of this exercise and we can end it with the q.

A

So, let's look at it. The scale uh so uber has thousands of hosts which have a spiral agent running on them and we operate in dozens of data centers. Our intra is, is in still evolving. uh It has different orchestrators, which schedules different types of workloads, and this is a important detail because I'll get back to it later. This also means these workloads are going to have different nazi requirements and if moving on to the identities, we have around million plus unique identities across uh different zones and uh odzi is like a house solution.

A

uh It's like any policy based solution where you list a policy for your workload, which has you know a lot of list of bunch of 50 ids and then it's all wrapped in our library which we built on top of uh ghost buffy or gojava.

A

And let's look at the sample spy registration, especially what what it looks like. So this is a session which has fields like spiffy id, uh that's a identifier of overcloud, then it's parent, it's where it rolls out to and selectors. uh In this example, the docker labels, which service name foo and the partition prod.

A

So how it works is when the docker container matches with the service name and partition. These two values that docker container will receive this identity and it will contain the spiffy id of uh example.org, which is the first domain, the one variable that we have defined and then slash prod.

A

So the problem here was, as I mentioned, our intra is evolving and they plan to change some environment variables and their values, and, like I mentioned in previous example, like our spf id, looks like trust domain and it contains some environment variables and then the workload identifier.

A

Now, if the internet changes the values of some random variables, then spiffy id will change and which means the same workload is going to get a new stiffy id now, and so this means all the oddsy policies associated with that workload. You need to change and get updated to the newspaper.

A

So what are the problem approaches? We took to address this situation right? One of the basic approaches is: keep the old spiffy id there's no need to change, even if underlying environment variable changes, you just have a custom hardcore logic to you know, assign it to the previous value uh we feel like. This is not the right solution. It's like pointing the problems, not really solving it, and it will involve changes to spy registration flow with some complex logic which may work, but not necessarily all the time.

A

So another approach was uh getting rid of this environment uh variable in this from spiffy id, especially, this variable was really not needed to uniquely identify workload, so it makes sense to get rid of it uh because this can change in future as well.

A

So when we decide take this approach of getting rid of this unwanted variable from id, which essentially means all 50 ids, which follow this format needs to change and exactly sounds like migrations and all the parties involved in migrations.

A

They know it's not fun.

A

uh So, as we dig into this approach and decided to get into details, then we realized that we have a lot of artsy strategies uh in place for different type of workloads and the the reason for that is.

A

uh There was some workloads that are onboarded to us before we even created a custom home based aussie solution, and some of them had different requirements, which uh uh you know like circular dependency ones where they could not deploy can depend on the solution that we have developed so and some of the weird cases we saw as the spiffy id was hard coded into the code.

A

This part of config, and this essentially translated into us. You know chasing all these different stakeholders which are directly consuming spiffy id and uh updating them. uh uh You know deploying those changes making sure uh all the policy policies are updated and another. uh As we looked into this more, we saw another opportunity where essentially, what we can do is create a new registration right, so the same workload can receive two identities uh inspire upstream. You can get multiple identities uh same with our rt solution, but there's no really way to choose a preferred identity.

A

So in the previous example that has shown foo who can get two identities but there's no way to choose a preferred one, and the same was the case in our uh rt solution, where there wasn't a way to define a preferred identity.

A

So what this essentially meant was as soon as we would have created, a new registration uh workload could have gotten the new identity and could have caused uh failures if the odds had not been updated before.

A

So as the migration steps, the that you know limited us to update all artsy pulses before we even change the format- and this was like I mentioned- is the time consuming process.

A

uh We had to chase uh various stakeholders to update their conflicts, so we changed some one fix for them waited for the deployments and there were some of the snowflake cases where uh so the workloads you know, especially you- can think of a platform services. You know which run on every single host.

A

They could take a days to receive a new build, so essentially we were blocked uh until uh all of these uh workloads are have received. This new build, which contains both the old and new spiffy ids, and so you know some of the snowflake cases we have to create white white lists. You know to move ahead with uh other workloads which are ready to consume for new format, so that was the first step and then we just went and updated.

A

The preferred format created all these different new registrations, and another thing we had to look for is. As I mentioned, we have a you know, a lot of registrations. We have to look for scalability uh in terms of how many registrations uh sparser can handle uh recently. We are also seeing a agent which caches uh different identities can cause out of memory uh traps, so that was uh one.

A

There was one issue where you cannot just go ahead and update all the identities. You know we have to take it into batches small batches, update the new ones, introduce the new ones and remove the old ones so that you don't cross the total number of registrations or a increase in registrations per agent count.

A

And then the last step was just you know, remain gold, spiffy id from authies.

A

So from all this uh entire effort, there were obviously fewer learnings and one of them was we need to advocate for a uniform rc solution across different workloads, and if we had that and then avoid you know, the software case is like directly using spf ids, especially in code, so that would have essentially helped us our introduction to just a limited number of folks and then could have uh prioritize and do the deployments- and you know save some time that we lost in the first step of a migration, and second point is uh around handling of multiple identities, and this one is still an under discussion uh like we are not sure like.

A

Where uh was the right place to handle multiple identities like should spy registration has a field for a default identity or preferred preference number, or our option should limit on some time based or preferred way of choosing identity, and this is something that we can actually use the help of our open source community and get ideas on how we can handle this and another learning we took from the this exercise is our spiffy.

A

Id format needs to be as concise as possible, and it should have a very limited number of dynamic variables, which are only associated with the uniquely identifying workload.

A

Adding static fields are fine, but if uh your dynamic variables in the spf id format depend on a lot of stakeholders, then there are chances that they may get. They evolve and then require you to do uh spiffy id migration.

A

So that's something that we thought thorough and decided to came with this new speciality format, which obviously contains trust domain, and we introduced this orchestrator as a field.

A

So orchestrator is it's not even a schedule of a workload, and the reason for introducing this into the format was: uh there's usually no guarantees the fee of uniqueness between two workloads. So, for example, the foo may be scheduled by scheduler, a and also by by scheduler b. So it might cause confusion if the scheduler is not field. uh Stereo field is not present in a specified format and someone needs to define separate lc policies for those and another advantage of adding the orchestrator we felt is when we enabled our back.

A

So if registrants are the ones who are doing, the registrations or orchestrators are the ones who are doing registrations, and we can simply put this as a prefix, so the orchestrator a can only work on the spf ids, like a creation deletion on of the trust domain plus orchestrator a prefix second part uh last part of this specified format is unique workload. Identifier and this one actually depends on the type of workload and the odds requirements uh so typical service.

A

You know, stateless services, uh we could have something like service a and the partition it could be production or staging as a identifier. Some other low-level services may not even care about this partition field. So it's it really depends on their audit requirements, but our goal is, as we decided, could be: keep the identifiers concise as small as possible.

A

So this was our experience and I'd be happy to take any questions. Thank you for listening.