Continuous Delivery Foundation Spinnaker Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Faster, Safer, Trusted - The Evolution of Autodesk’s Cloud D... Jesse Sanford & Cole Duclos

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Keynote: Faster, Safer, Trusted - The Evolution of Autodesk’s Cloud Deployment Platform - Jesse Sanford & Cole Duclos, Senior Principal Engineers, Autodesk

Autodesk’s Developer Enablement organization has a goal to increase developer productivity 100X. To do so, Autodesk has invested heavily in building their next-generation cloud deployment platform powered by open-source tools like Spinnaker.

Autodesk began their Spinnaker journey 4 years ago. Over that time, they have built a user-friendly abstraction overlaying Spinnaker and AWS. From imperative pipeline patterns to declarative GitOps models, their offering has matured, allowing hundreds of developers to deliver software easily. By reducing the toil that comes with deploying and managing secure and compliant applications, Autodesk engineering teams can now focus on innovative features, providing immediate value to their customers.

In this talk Cole Duclos and Jesse Sanford will talk about Autodesk’s past, present, and future CD platform, how Spinnaker has been central to that vision and hopefully pass along a few learnings along the way.

A

Good morning everybody I'm Cole deklow I am a senior principal engineer at Autodesk working on our Cloud deployment platform, and this is my cohort Jesse yeah.

B

Thank you. Thank you. I'm Jesse and I'm also a senior principal engineer here at Autodesk and I'm working at the juncture of our security and compliance and our developer, enablement teams and I'm really glad to be here today to talk to you guys. It's uh it's been a long time right, it's two years since the last time we were all together in the same room um but I'm a little worried that that that's starting to lose its meaning, like that, you know the more times we have these conferences now and speakers get up on stage.

B

It's almost compulsory for them to say. Oh, it's! It's been two years so good to see everybody you know, but um you know: when can we move on from that?

B

When can we get to the point where you know we kind of don't have to have to say that when we get up here, it's kind of like you know when you go away for the holidays and you come back home and you see your delivery guy for the first time after a while you're like hey man, happy New, Year and he's like it gives you a screwball face and you're like oh yeah, it's uh it's damn near the end of February right.

B

Oh excuse me thank you yeah. That would help, um but yeah. So I am really excited to be here now. I really am I really am I might be the most excited person in this room. Yeah I got a little click happy with the old call for proposal, submission button. I did uh I'm doing three talks and this keynote now not even getting paid for this. It's for all for free somebody call my agent right but um yeah.

B

It's like before I know it I'm gonna, be doing a Netflix special up here, right, which I would do by the way if they would just give me access to some of those sweet, unreleased managed delivery features that they've been holding in their breast pocket right, but they actually, they actually produced a uh they actually produced a teaser site. For my special already I, don't know if you guys have seen this. It's amazing it's incredible. This actually is the actual managed delivery uh brochure site. If you haven't seen it manage.s delivery.

B

Incredible I was really impressed when I saw that, if only all of the software that I worked on had a Hollywood budget right, imagine the things I could do being incredible so, but seriously. Cole and I do have a great talk for you guys today we're going to be going through some of the features and functionality that we've been building into our Cloud OS Cloud deployment platform and how Spinnaker has been helping us enable that so faster, safer trusted right.

B

If this slide looks familiar, it might be because you saw Dave Hans keynote from a couple years, back called faster, safer, continuous, which is a great talk by the way. If you haven't seen it, you should check it out, he's a funny guy, but when I was developing this deck I thought you know these things are not in competition. These are actually pretty complementary.

B

This is not a value triangle exercise like in his Kino. You know it's not a time for, or it's a time for and faster and safer and trusted exactly what I'm trying where we're trying to achieve. As we build out our Cloud deployment platform, it's also what we're trying to achieve what I'm trying to achieve when I bring my kids to school in the morning, but there it's rarely the fact that I get all three I try I do I, try so faster, safer trusted.

B

What do we mean when we say that right well before we get there? I should give you a little context about Autodesk, because these definitions are a little Autodesk Centric.

B

You see, we are software engineering company building software for engineers. You could say that we make software for people who make things we're a leader in the design and make tooling for across a variety of industries, from entertainment to architecture and engineering to manufacturing and construction.

B

Our customers workloads are diverse, but they do have one thing in common: they depend on our software to build and make those things, and, as such, our software Engineers depend on our platforms to build and make that software.

B

And that diversity comes at a price right. We must support a broad spectrum of solutions. Autodesk has to be creative, not to mention, but so our our products span, continents and industries, not to mention devices operating systems and runtimes, and through these unique workloads you know. Historically, our product teams have had to had to Silo on things from UI libraries to deployment tooling, but there have been attempts to converge and accelerate through common Tooling in the past.

B

Unfortunately, that tooling just wasn't very mature.

B

But in the past few years we had a strong push towards a platform mindset and we've created a team that we call developer and developer enablement internally, whose core mission is to provide developers, both internal and external, with a force multiplier that we're calling the one Autodesk platform.

B

So, given that context, our definitions when viewed through the lens of developer enablement might start to make a little bit more sense start with faster. This one was easy because we were working with a mandate.

B

We've been charged with moving our developers from one times, productivity to 100 times productivity, when measured by our deployments to production sounds pretty aggressive, but we're talking this through a variety of ways, from small changes to our deployment patterns and convergence on some of our runtimes to large initiatives like our multi-region Expansion, Project and now safer.

B

It's a bit harder to pin down, but clearly no less important, safer to our developer. Enablement team means less change. Induced incidents through immutable deployments in canaries, faster recovery through automated rollbacks resiliency, baked into our platform by default, easier to use, drop-in, observability, tooling smaller chain sets push the production more frequently and so much more and finally trusted which toss actually has two meanings.

B

The first is enabling our teams to be secure and securing compliant by default when they deploy what they deploy should be known to be meet all of our regulatory requirements and, second, that our product teams actually trust our platform. Second part's, pretty loaded. The developer enablement spirit is clear. We want our teams to feel at home on our platform. In fact, we want them to be delighted to be there so faster, safer trusted. These are the three pillars that we're trying to build our Cloud OS deployment platform on.

B

So what is cloud OS? Well, this is where I passed the mic to my co-patriot Cole duclow.

A

Thank you Jesse. uh So, as Jesse mentioned earlier, Autodesk has a diverse set of requirements that we have to support. Cloud OS at its core supports a wide variety of capabilities from deploying containerized applications to deploying applications running on Virtual machines, hosted on ec2 instances to deploying front-end applications and Lambda and serverless based applications, as well as deploying databases and managing infrastructure like S3 topics, SMS S3 buckets, SMS topics and cubes.

A

So let's take a look under the hood.

A

Cloud OS has multiple layers to it. The first layer which is exposed to our customers or our application teams is the application definition file or ADF.

A

For short here our teams can specify their architecture and their application configuration through code, and this will ultimately get transformed through our transformation layer into Spinnaker pipelines which will deploy the application into the various environments and regions specified in the ADF and each stage in those pipelines are executing Spinnaker native constructs, like Cloud driver deployments or terraform-based stages, as well, so I'm going to walk you through each one of these layers, but before I do so I felt it important to talk about our multi-region account strategy.

A

So early in Autodesk cloud Journey. Many years ago, we used to shove just about everything into a single AWS account I. Imagine some folks in the room or online have experienced something similar in the past and they know firsthand the nightmare that it is to manage that at scale, you've got noisy neighbors a bit of a billing tragedy, the security scaries behind that.

A

So before moving forward with Cloud OS, we felt that it was highly important that we aligned on an AWS account strategy to enable the success of Autodesk, and this is what we landed on. So for every service. It gets an AWS account per environment and per region, and what that grants us is a much improved security posture.

A

It's it's easier to automate through our deployments. It's highly scalable and the billing becomes much more simple and all of those noisy neighbors they're permanently on vacation.

A

So here you can see a preview of our application definition file again the ADF.

A

So in this file we Define our deployment requirements. You can see things like the service ID, which is the unique identifier for that particular application. The type of pipeline that is needed to deploy this application into the cloud, whether it's ec2 ECS lambdas front, ends and the deployment requirements again, the environments and the regions that we need to deploy into. So this is our pseudo declarative model that allows for us to embrace git Ops.

A

It's essentially our managed delivery, but this actually predates that fancy website that Jesse had shown earlier, and this will get fed into our system and result in those pipelines through our transformation layer, so Cloud, os's transformation layer is the main engine for our pipelines as code solution.

A

When you submit the ADF to the transformation layer, its API will first validate that the ADF meets the desired specs.

A

It will then parse the ADF and transform it into something that our pipeline framework can use and combining that information with our modularized pipeline libraries It, ultimately, outputs a Json blob to our Spinnaker's API and that Json blob defines the pipeline architecture. So the pipeline is actually created in that Json object and will dictate the environments, the regions which pipeline type, which Spinnaker application it needs to get deployed to, and the pipeline framework here is actually something that we built in-house.

A

We call it Shore and our colleagues who are in the room here today will be doing a presentation on this later in the Spinnaker Summit, so I highly recommend that you check it out.

A

So now that we've got pipelines rendered in Spinnaker, let's talk about the basic architecture here, regardless of the pipeline type, the architecture is roughly the same. We expect an artifact to come into the pipeline and we have to validate that. We trust this artifact before deploying it once we've validated that we do in fact trust this art artifact. We can then move forward with the individual environment and region deployment stages.

A

Now each one of these stages is actually a reference to an independent operator pipeline that is built and maintained by our Cloud OS platform team and again, if we take a closer look at these pipelines, this operator pipeline follows a similar pattern across the various types we first need to promote the artifacts and the secrets so that they become available for use at runtime. In the various environments, we need to validate any sort of infrastructure that are as being deployed once validated.

A

We then go ahead and deploy that infrastructure, and then we can lay the compute layer. On top of that, once things are healthy and running, we can then execute our testing stages to ensure that everything had been deployed successfully and we can move on to the next step.

A

So everything you see in green here is actually in some ways how we bake in trust and security into our platform. Another way is through our hardening image strategy.

A

So here we have a baseline image and then our security team actually builds and maintains a layer of images on top of that base, image, which then the organizations can use to build in specific, tooling and configurations on top of that, and then you ultimately get out to the application image layer, which is where the application artifacts are stored and get deployed at runtime as part of the platform.

A

So in addition to that, we also have this idea of infrastructure as code hardened modules. Again, these are built and maintained by our security organization and our application teams can pick and choose which ones they want to use and combine them together to make a suite of infrastructure that they deploy through Cloud OS.

A

What's interesting about our platform, though, is that before making these deployments, it uses infrastructure as code validation, to ensure that anything that is getting created does not stray beyond our security requirements in our security, posture is upheld at the end of the day.

A

So what happens when our opinionated abstraction layer doesn't meet the needs of our application teams? What happens when these hardened modules don't fulfill all of the little requirements that application teams have well Cloud OS has built-in platform extensions through the way of two different Pipelines. We have a terraform pipeline in a serverless pipeline.

A

The terraform pipeline essentially allows for application teams to bring their own terraform to Cloud OS and deploy it through with the serverless pipeline. Application teams can actually bring applications that have been built using the serverless framework again through Cloud OS.

A

In both cases, we use that infrastructure as code validation, idea again to ensure that anything that's being deployed upholds our security posture.

A

So our journey and accomplishments with Cloud OS, thus far did not happen by working alone. Autodesk Armory AWS collaborated together on two very critical features. For this to be a successful project.

A

The first is the dynamic account registration plug-in so I had mentioned before about our multi-region AWS account strategy. Autodesk is creating many many AWS accounts on a daily basis, and it was a absolute requirement that we could automatically and continuously onboard AWS accounts to Spinnaker in an automatic fashion.

A

Additionally, we created the Lambda plugin together so that we can manage and deploy Lambda functions with Spinnaker's native Cloud driver. Both of these features and plugins are available in the open source community. So we hope that the larger group can can benefit from it.

A

So, let's tie it all together with the end-to-end flow, our Engineers will make changes to their application code or to their ADF and push it to GitHub from there. Github will trigger the build process in Jenkins and Jenkins will ultimately create or trigger the the deployment processes, and this is where our Cloud deployment platform takes over.

A

So, by submitting that ADF and any uh any artifacts that got built our transformation layer will combine that information with information that it pulls from our service registry regarding deployment coordinates so deployment coordinates, meaning the AWS account IDs, the VPC IDs and the subnet IDs, where these deployments need to occur once all that's combined. The transformation layer will render out that pipeline as I had mentioned and execute the deployment. According to our ADF.

A

So before we wrap up, I wanted to mention one more critical piece and that's our ability to measure platform metrics so flying a platform. Blind is incredibly scary when we're interacting with our customers or our leadership team, or we're building out road maps or trying to make business decisions.

A

It's very critical that we have at least a basic understanding of our usage on the platform who's using it. How are they using it? How healthy is the overall platform and are we making progress to those developer, enablement goals that Jesse had mentioned by tracking these success? Kpis we can make those data-driven decisions and ensure our platform is healthy, which compounds in trust within our teams and ultimately drives adoption.

A

So here we are faster, safer trusted. These are the pillars of our Cloud OS platform and our development enablement organization.

A

Thank you very much and a big shout out to Greg Eric, Michael and the rest of the Netflix Team for allowing us to make fun of the management. Thank you.