Continuous Delivery Foundation Spinnaker Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Developer Enablement through a GitOps Driven IaC Control Plane... Jesse Sanford & Manabu McCloskey

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Developer Enablement through a GitOps Driven IaC Control Plane at Autodesk - Jesse Sanford, Autodesk & Manabu McCloskey, AWS

At this point GitOps is a well understood concept amongst DevOps practitioners. Talks and posts promising magnifications of developer enablement through GitOps abound. However, they rarely focus on the intersection of dev enablement and security/compliance. In this talk Jesse Sanford and Manabu McCloskey will walk through a recent PoC done at Autodesk that flexes GitOps for the delivery of compliant infrastructure as code using Managed Delivery and Crossplane. The PoC focused on using Crossplane to split the responsibilities over the ownership of the IaC and separating the concerns to the appropriate stakeholders in the organization. Security team members can develop security controls alongside platform and product engineers who develop capabilities and functionality on top of them. Additionally, using Spinnaker Keel, all parties can work through a unified git driven workflow where PRs serve as our change requests and merges trigger the updates to secure infrastructure across the organization. Finally, novel approaches for wiring up Keel with Crossplane through the use of the K8s plugin, and sharing of secrets between Clouddriver and Flux through a custom sidecar will be discussed.

A

All right, I, uh I'm Jesse um and, as you saw earlier in the keynote I'm uh senior principal software architect at Autodesk and I'm, currently working the juncture of our security compliance teams and our developer enablement team. When I'm not in front of my computer, I'm, absolutely dependent upon uh to be a continuously delivering parent.

B

Hey guys, my name is McCloskey I'm a.

C

B

Architect for AWS I work in open source team I am focusing on infrastructure tooling, um outside work I like to play video games mostly like Dark Souls series.

A

All right so, uh first, a quick agenda. We're gonna look at uh acdp, uh which is our Autodesk cloud deployment platform, which we call Cloud OS and then go into some of our IAC pain points specifically, which drove some of the work that you were about to show you guys and then we're going to look at the architecture of the solution that we came up with and then manabu is going to give a deep dive on how everything works.

A

So the Autodesk cloud deployment platform or what we call Cloud OS, is our current attempt at consolidating our continuous delivery Primitives used to ship Autodesk software. About four years ago, we produced an internal strategy at Autodesk about how we should start doing. Modern delivery of our software to the cloud quicker was core to that strategy, as were the basic tenets of get Ops. Although get Ops is still a little bit of an experiment for us still not truly commit to production.

A

We've been using Spinnaker for about three and three and a half years now in much of the early ux for this platform was informed by our previous homegrown CD Cloud orchestration tool and to ease our users transition onto our Spinnaker based solution. An abstraction layer was built to match that prior experience.

A

Central to that abstraction is something that we call the ADF our application definition file, which you saw also might have seen in the keynote that Cole gave with me earlier.

A

um It's essentially a declarative, yaml, manifest of infra and app artifacts to be deployed. That definition has piped through our translation layer, which, which sure is the the project. That's that's critical to that and translate those from these.

A

You know abstractions into Spinnaker pipelines and and terraform uh values, files and I want to stop here and make a brief plug for sure, because uh if you would like to know more about it and how we're doing those translations and the wrangling that does some of my fellow autodeskers are going to be doing that talk later today, you should check it out.

A

So those IAC pain points that I was talking about right.

A

um You know the the imperative pipelines that we're producing are a combination of Spinnaker native stages and and terraform, and also uh ad hoc containers mixed in the terraform stages, use arm-rays terraformer to invoke a curated set of terraform modules, but because of the diversity that our product teams need to to to to be able to do the products they're building. We must also allow for a bring your own terraform path as well, and the unfortunate side effect of that is.

A

It allows for a lot of diversity in the module of design as well, which is an issue for our security and compliance teams. How can we know that the infrastructure and the modules that they're producing meet our regulatory requirements? How can we alleviate our developers from having to think about how to implement those controls into all of their modules right? There's a developer enablement play there too, and our security and compliance team want to help.

A

They want to be partners so they've, even go so far as to produce a reference set of hard modules, uh for teams to use like like Cole, also showed a little bit earlier. However, those hard modules are not over, not not a requirement right now. They're, not it's not compulsory to use them and adoption is still a little bit best effort.

A

How can we Empower this security team to take ownership over these concerns?

A

And then, finally, you know lazy reconciliation causes its own host of problems, because we've taught our infrastructure lifecycle management to the delivery of our applications built on top of it. We end up with varying gaps of time between our terraform runs.

A

This will not really cause this drift to occur in these Dynamic Cloud environments, but also limits how quickly we can respond to mandated changes. Company-Wide enter Crosslink right, a kubernetes operator based control plane for apis, be the cloud provider apis like AWS, Azure or gcp, or your own internal apis.

A

Cross-Plane can map them to declarative crds that Define your desired state, but no time to go into cross-plane in in great detail, but just know that cross plane has first class support for major Cloud providers, which gives us that infra in for cycle and infrastructure, lifecycle management that we were looking for, Crossman also models its code reuse around the idea of composition.

A

It's similar to how terraform modules uh terraform sub modules work in using this, we can build compositions that teams can combine and extend to meet their needs like you're, seeing with this diagram, and because this is all done in context of the Kube API, we can make use of Kate's rbac and enforce that separation of concerns right some are under lock and key.

A

This means that we can build a library of curated Primitives and nurture an ecosystem of higher level abstractions like Autodesk compliant database that mask away a lot of the toilsum Nuance that comes from working with High Fidelity tooling. This is what developer enablement starts to look at look like.

A

Finally, since it's designed as an operator, we can rely on the battle. Harbor battle harp battle, hardened kubernetes reconciliation, Loop to continuously make sure that our desired state is abided and we get or we get notified immediately.

A

So considering the problems we were experiencing and the desire to continue to push our developer enablement bar upward, we decided on some fundamental design goals right. The first was to truly move to fully get Ops driven workflows.

A

We wanted to tear off our imperative Band-Aids and lean into that forced multiplication that declarative workflows give us, and next we wanted to enforce that separation of concerns between our product platform and security teams, enabling each of them to focus on their own domains and not color outside of the lines and then finally, we wanted to explore topology that allowed us for more secure, actuate actuation of these changes in product environments, limiting the blast radiuses.

A

So, in order to meet our goals, we developed a hub and spoke model of cross pen instances running on a network of kubernetes clusters inside of our client app accounts here, you'll see Spinnaker kind of it's a little small sorry, guys Spinnaker's logos there in the center in that Central kubernetes cluster, which is the entry point of the system.

A

It's running all the usual microservices, as well as Keel, which is monitoring our Enterprise GitHub down there in the bottom left uh for for managed delivery manifests it's also running the Kate's plugin for managed delivery so that it continuously it can continuously deploy cross-plane crds into those cross-plane instances that are in the Clusters around the outside.

A

An important aspect of this design is that we can continue to use the same tooling, to build and maintain that Network of Kids clusters than underpin it as, and so you know, as as we onboard new client accounts, the cross plane in the central cluster can be used to build those case clusters in the new accounts and install Crosspoint on those clusters kind of like turtles.

A

All the way down during this bootstrapping phase of the new client account Spinnaker writes those cross-plane crds, representing the desired state of those new client clusters directly into the cross plane. That's co-tenented on it in that Central cluster, see that little lollipop there in the middle I want to point out that this crossbite instance will need AWS API permissions on these client accounts client accounts to allow it to build those new case clusters right.

A

It's got to be able to reach out to Amazon and say, give me a new ecast cluster, and then it also needs Kate's API permissions on those those new clusters to be able to install those subsequent cross planes on them.

A

However, once those new clusters are up and operational right, those permissions can be reduced just allowing the management of those resources or revoked altogether kind of severing the tie, removing that that Central Pivot Point.

A

Spinnaker, however, in the middle, there will retain those permissions to be able to write objects out to those case clusters, so that keto can continuously write manifests to them. This allows for managed delivery to continuously deliver the crossband crd artifacts from our client GitHub repos that represent our client apps to those client clusters in those client accounts.

A

The client cross-plane instances will then build and continuously reconcile the infrastructure required by our client. Apps. Remember that that centralized Spinnaker does not need to have Cloud driver permissions to those AWS accounts anymore. It just needs Cloud driver Kate's credentials to be able to write to those Kate's API servers in those those accounts around the edges.

A

The cross main instances in those client accounts are now the ones who communicate with the Amazon apis on behalf of our infos Cloud platform, again, the reduction of blast radius there.

A

So now that we have a bit of a high level understanding I'm going to hand it over to manabu to dig into the details.

B

Thank you, Jesse.

B

All right, so, let's dig right into the um the workflow that we wanted to enable for this uh for this work.

B

um So first you have the infrastructure repository on the right, uh so this defines other infrastructure, that's needed for the application, so you might have like you know, kids, cluster definition and IM roles.

B

um You know easy, to instance configuration any of that right and then um in the middle you have a that's the Hub cluster that Jesse was talking about. So it's the um hyper cluster that has Spinnaker and crossbane installed.

B

So the first order of business is to uh for Spinnaker to be able to talk to infrastructure repository right, so the infrastructure repository manifest has to go into Spinnaker and Spinnaker needs to be able to continuously reconcile the differences between the what is actually defined in the repository and the actual state of the infrastructure.

B

I know you get all that manifests to the um cluster close bank and the cross brain is going to do its magic and then uh create the uh you know the infraction that is needed for client account to function and as a result of the reconciliation process. It's going to create a brand new case cluster with cross brain installed, and then finally, you have an application repository, that's uh you know, has a code and you know artifacts and everything that's defined and you have to be.

C

B

That between Spinnaker and application, Repository and then finally Spinnaker needs to be reconciled. The differences between these um the differences right.

B

um So you know that this application repository might contain some. You know information about infrastructures as well. So in that case you just submit that to crossbank and the crossbank is going to create S3 packets, easy to instance, whatever right you name it, and then you can also deploy all these applications into the KH cluster in itself as well.

B

Okay. So this is a little complicated right. It's got a lot of workflow going, so I am just going to simplify that into just four steps, so the first step just create a cluster right. You just ask that cross plane in the hub cluster to to create a brand new cluster and then cross brain is going to reach out to the client account and then create the eks cluster with crossband installed and then cross brain comes back to Spinnaker and just say: hey this class is now ready.

B

Here's the credentials, here's the name of the password um you know so that you can deploy whatever you want into this cluster and then finally, Spinnaker is going to take the information that's stored in the client application repository and deploy that into the brand new account that I just created okay.

B

So the first step is to be able to sync between the infrastructure repository and the Spinnaker itself right. It's gonna be some way of you know. Syncing between these repositories.

B

um Like Jesse mentioned earlier, uh Q is a optional microservice within Spinnaker that enables get Ops driven workflows, as you can see on the left and kill has its own specification for its its manifest.

B

um So you have like you, know, you're, defining artifacts and you're defining environments and then attending kill to deploy all these artifacts into these environments.

B

So um then you, a kill, is going to continuously reconcile with Cloud driver to deploy all these resources into AWS.

B

um Typically, you you deploy into like ec2 instances or ALB, or anything like that right, um but for case we needed the arcades plugin. So there is a kubernetes plugin Fork here that allows you to deploy into um the kubernetes Clusters instead of using ec2 instances.

B

Okay, so we now know how to um correct or connect to the infection repository and then deploy, and then you know corresponding is going to provision right, so that workflow would look something like this. You'd have a um GitHub or you have a git repository that has a q manifest stored in this case, we're just using.

C

B

And then uh kills plugging is going to take that and then finally, it's going to ask cross pane to um multiply all that information into the brand new AWS account.

B

um There is a problem, though: um at Autodesk They Don't Really allow um Road, chaining, Road training means you know, you have a Airbase role and then you are trying to um you know assume another role from that role.

B

um So typically, when cross pane deploys its own infrastructure or it deploy anything in there, uh you'd use portal right so IR say so: it's pretty much a um identity assigned to rule and they use that identity to assume a role, and then, from that role you assume another role. That's usually how a crossbanding do its own. You know when you're deploying things that's how they do it problem is that you know all of us didn't allow it. So how can we solve this right?

B

um So thanks to Jesse's effort and we he actually um implemented this in the provider? Aws.

B

That's mostly. Yes, um let's see uh yeah. So uh in this new model, um you have an eks oidc. uh So whenever you create an eks cluster, you have a uh option to create an oidc provider right, and what is happening in here is that client account is trusting that Principle as an oidc um yeah, the oedc within cross brains and namespace.

B

That is, and then um finally you have that cross stream pod- that has its own token that's injected by the eks admission controller and then use that token um to assume a role in the clients account. So this way there's no. um You know Road chaining to worry about.

B

Okay, so with this we will actually be able to. um You know, create the cluster and then come back to Spinnaker and then say: hey it's ready right, but that's a problem. How can we deploy it to the brand new account? You know coming back, that's like really no information. That's given to you right like we need the information like what what server are we going to be able to do what credentials are we going to use to deploy to right all that information is still not there? So how do we do this?

B

um Thankfully, the people from close Spain thought of all this right, so Crossbay has this concept called connection details and collection details allows you to describe certain details about these managed resources, so managed resources could be like you know: S3 packet, dynamodb table easy instance. You know.

C

Any anything like.

B

That right, so what these connection details hold is just information about these managed resources. So when you're talking about you know S3 bucket, it could be like you know, name of the packet which region it's in. You know Secrets our encryption settings. You know anything like that, but in case of an eks cluster you are actually storing and it could config file. So you, as you most of you probably know, could config is used to connect to the uh these customers right, and so these uh could configures are available as a secret within crossbones namespace.

B

So that's good right. We can somehow use this. um If you can connect the you know Spinnaker to cross plane, then we can probably do it so to solve this.

B

We uh we created a sidecar that runs along with Cloud driver and what this sidecar is going to do is just go into the cross, pins, namespace and then list all the circuits, that's contained within the um close pane's namespace and see if these are group config files and then, if you find quick question file, just going to write them to IMO file on its own file system um and then have a cloud driver.

B

I'll read that your mobile, um you know, as you know, the uh containers share their file system right within about, and this Cloud driver uh has its own uh plugin. It's the external account plugin I think this was developed by Omri uh people. uh What this plugin is going to do it's just going to read that file the yaml file and then read and add these accounts into Spinnaker's memory.

B

um And then you know, if you remove a account from the ml file, it's also be going to be removed from the Spinnaker's account as well. One thing to note is that these secrets that's maintained by um uh crossbank. uh They actually refresh so this. You know the tokens included in Google compute file. They expire.

B

You know about 60 minutes or so I think um so, every time it expires and then gets refreshed, it's actually written by the society guard as well, and it's written to the yaml file so that you know you Spinnaker has the latest and valid uh token to use and every time it needs to deploy something okay. So what that means is that it actually now allows you to use the Spinnaker's pipelines right. So you.

C

B

On the left, we have a Spinnaker pipelines, just an example: one just defined, and it's just going to do, Orca and then okay just go. Do.

C

Its own thing and.

B

Then and finally, just you know, deploy it to Cloud driver and when you're trying to do Cloud driver, uh you know which account you are supposed to use um just because you know we just made them available uh from the crossbase namespace.

B

Okay, um but there's a problem right like we don't we wanted to do git, Ops right, we don't want I mean Spinners, you know imperative pipeline, it's good right, it's great, it works, but we wanted to do a get Ops driven workflow.

B

So to do that, we had to leverage, kill, um heels uh kubernetes plugin um under the hood. It actually uses Flex. uh For those of you who do not know what flux is flux is a toolkit just allows you to have the enable get of a workflow within a kubernetes cluster. So it's just going to reach out to the git Ops uh get gear repository fetch all the information, that's necessary and then finally apply that to the brand new kubernetes account, but there's another problem again right, flux and spinning guard.

B

They don't get to talk to each other. You know they don't really know how to talk to each other, and you know explain, exchange information about these accounts so flux. So it knows our University that doesn't know how to connect to the client cluster right. It was just created, it doesn't know anything about. You know uh crossbones, you know Secrets model or Spinnaker's account model. So um to solve this, uh we just expanded that Sidecar to just read from the secrets in the closet.

B

Namespace write it within its own file system and write that into the fax namespace as well. That way, Flex knows what I could configure to use, and um you know use that quick config to finally deploy all the infrastructure and application. That's needed for the client account.

B

All right, so the githubs um you know, driven workflow, now looks something like this. You have a um a customized, that's defined on the left, uh just it's defining the git repository as well as a where, within that repository, you are supposed to use to create this customized overlay.

B

So the first order of business is for crowd driver to publish the um config file as a flux secret. So Flex has its own. You know a way of handling its good coffee file, so it's uh formatted in that way, so that Flex understands how to use these files.

B

Okay and then um so this manifest gets submitted to kill and kill, is going to do its own thing and then um finally ask flux to deploy uh using Code, configure one right and in fact it's just going to reach out to the um the secrets in within its own file, get the code config file. Just you know, read it and okay now, I know where I'm supposed to be deploying and then finally, it's going to be able to deploy into the client's AWS account.

B

So uh with all that in place. We now have this uh nice and you know easy to read our model right, so you have an infrastructure repository that defines all the you know. Infractions are just needed for the client account and then it's just going to ask cross brain to deploy it and then finally, the cross brain just comes back and then say: hey the cluster is ready.

B

Here is a um you know, secrets that you're supposed to use, and then Spinnaker is going to use that secrets and then deploy to the application into the brand new account that was just created, and then you can use imperative pipelines or you can use the decorative way of doing it by queue.

B

Okay, so that's it. um I have for you to. If you have any questions, uh please let us know.

D

Hi uh I guess I'm curious of the details about how number three Works, how the feedback gets back to Spinnaker that the cluster has been provisioned.

A

Yeah I'm gonna, let you take it, but I want to just review the question so so you're you're interested in how Spinnaker becomes informed that the new eks cluster is ready and how it can speak to it.

D

Well, really just the Readiness part I think he covered how you speak to it, um but it seems like there's an opportunity to like do number four before number: two, oh.

A

Okay, yeah, so there's a timing, a timing, question: okay, right, um yeah, so I I can I can explain it a little bit so it really. It really depends upon, like the workflow around this tool, right like um at Autodesk, we we still have uh uh kind of like an onboarding of of clients to our system. It's not completely self-service yet, however, if it was self-service, there would be some sort of out-of-band notification. That would say like look. Your tenant has been been produced. You can now start writing.

A

Your delivery manifests in this repo and that repo might be provisioned for them as well.

A

I I can also Imagine a situation where someone comes to us with a manifest a declarative, manifest in keel's world that they want to deploy. However, they're still going to need to kind of onboard somehow like it's not it's not the case that they can just say: hey I have this repo just deploy for me. They still need to have an account and VPC that's been provisioned, and all of that right now, actually is is kind of done out of band process at Autodesk. Is that is that sound familiar like?

A

Does that make it make sense?.

D

uh Kind of I, I, yeah I, think what you're saying is that three is a manual process that it's not automated.

A

Well, so so let me let me I, don't want to ambiguate here three happens automatically by virtue of the secret and the coop config getting written back to Spinnaker and the dynamic accounts. Plugin then sees it and creates a new account. A net new case account inside of Spinnaker. Only then can Spinnaker actually Target that account. However, if someone was to create a a client account delivery manifest and try to Target it before that was written back, Spinnaker would just say: I, don't know which account you're trying to use I can't write that.

D

C

So um looks like in the architecture that you eventually settled in you have the flux operator that is, in the same cluster, with your Spinnaker doing the deployments to the client clusters right.

C

Alternatively, you could have flux deployed also to the client clusters and get Keel to just throw vanilla, customized resources or whatever to that client cluster and have the individual flux. Instances in the client cluster take charge right, any particular reason you chose one over the other.

A

C

A

Of them it'll be really quick about this, so that has a lot to do with about with with Autodesk security posture right. They don't want to necessarily give all these accounts. All these client clusters access to be able to actuate all their changes all the time by themselves, entirely right, and so we we are able to centralize that pipeline of the delivery of this. This application, manifest the you know. Ultimately, the the declarative, yaml.

A

That's been boiled down through our CD system and it's like a centralized choke point right like we can say this is not allowed. Don't do that, but but you're right, you know um you know, flux could be distributed and then just rely upon the localized credentials on those case clusters to be able to talk to the Amazon apis. For you right.

A

Okay, all right thanks, everybody.