Continuous Delivery Foundation cdCon 2021, 24 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sponsored Keynote: Delivery Engineering @ Netflix - Creating a Seamless Experience... Amy Smidutz

Description

Sponsored Keynote Session: Delivery Engineering @ Netflix - Creating a Seamless Experience to lower Cognitive Load during Software Delivery - Amy Smidutz, Netflix

Continuous Delivery enables organizations to respond to customer needs in today’s fast-paced world. Join this session to learn how Netflix is continually improving this process by creating a seamless experience that lowers cognitive load by reducing complexity to make continuous integration, delivery of code, and configuration easy, repeatable, and predictable.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Hey everyone: my name is amy smith, I'm the director for delivery engineering at netflix, and I'm here today to talk to you about large-scale, continuous delivery. I want to talk to you a little bit about my teams and what they're thinking about and what they're building right now.

A

So we have multiple teams that span three domains. The first one is continuous integration or ci. They are the team that manages all the jenkins infrastructure and anything else. Ci related. The second domain is continuous delivery. These are cd teams that build and manage spinnaker our delivery platform, as well as manage delivery and meme two products that I'll talk about more later. The last domain is what I like to call continuous experimentation.

A

We have an internal platform called chap that was built to facilitate infrastructure experiments which allow our engineers to run tests in production to better understand how their applications will work.

A

So together, our mission in delivery engineering is to provide a seamless delivery experience that simplifies the complex. This is what I obsess over. This is so important to us, and this is what shows up in our work and the reason why it's so important is because unnecessary complexity equals wasted time any amount of time that a product engineer spends functioning around their delivery or operating their service, debugging issues with delivery, trying to figure out their security groups or whatever. It might be.

A

It's time that they're not providing better tools to our studios that bring new content to our users and time that they're not building the application, features that bring joy to our customers and all that wasted time cuts into competitive advantage.

A

We want to move quickly and stand out from our competitors and to do that, we need to be able to give our engineers the most amount of time that they can on their domain, so they can continue to bring member joy and while our ultimate goal is always to bring member joy, we also want the engineers that use our tools to love working at netflix, because they get to solve really hard problems that they were hired for, rather than spending time on delivery toil.

A

So how do we give our engineers more time back without sacrificing the ability to deploy their apps the way they want to? I strongly believe this is the answer. That's why this is our goal. That's why I obsess over this. That's why the team's obsessed over that.

A

So, let's get started, I'm going to talk a little bit about each of the domains of what they're working on so the boundary between development activities and delivery activities is when an engineer commits to the their source code manager. So at that point, ci is automatically triggered starting the seamless delivery process. So this is the point in which I want to make sure like as soon as this happens, everything that follows that is delivered related is easy, simple and seamless.

A

So um how does ci provide a seamless experience? One goal for the ci team is to make it so engineers don't have to deal with the jenkins ui directly ever in service to that they created jenkins and stash plugins that combined a suite of tools for continuous integration. That is easy for customers to configure. They call it rocket ci.

A

So a bunch of different benefits to rocket reliable, build triggers, minimal configuration, traceability and more and the cool thing is. Is you only need a couple of files to get this going? You need a rocket eml file with some simple properties that is checked into your repo and can be versioned and the second one is just your build script. With these two things and a lot of work under the covers. The team has provided minimal configuration ci through rocket if you're feeling familiar with other ci tools like circle, ci or github auctions.

A

You'll find this paradigm very simple.

A

Ci wants to provide a seamless experience that simplifies the complex. They don't want people messing around with the complexity of continuous integration. Instead, they wanted engineers to focus on getting from development to delivery, with as little user pain as possible.

A

So, let's talk a little bit about what continuous delivery is doing and I'm going to focus a lot on managed delivery for this piece, because I only have 15 minutes so, let's go through it really quickly, so manage luring is a way for users to interact with spinnaker, rather than the imperative way people deliver using pipelines. Vintage delivery focuses on intent and gives users a declarative delivery workflow.

A

So we simplify the complexity by providing codified best practices using the data that we already have to make saying defaults and save the decision making for what it really matters.

A

So why manage delivery? uh Well, we wanted people to be able to focus on their changes and not pipelines. um There was a strong desire from folks that to store infrastructure in deployment as code and uh simplified workflows for common tasks, there's so many different things that people are doing that are repeated throughout our entire environment.

A

We want to make those workflows very easy, so the premise of managed library is that we switch from you telling us how you want to do something to what you want to do. In other words, what is your intent with managed delivery? Engineers can spend more time, developing and less time on delivery and infrastructure. Work and manage delivery is a huge topic and there are a ton of talks and papers that are completely dedicated to the topic for today.

A

I just want to give you a quick overview of some of the things that I think are most helpful to reach our goal. So normally, when you have a problem with a deploy, you have to look at many different applications to try and track down the code that caused the problem right.

A

This is this is a common issue with that people encounter when they're trying to deploy what code was the problem deploy so with managed delivery notifications, you get a link to the exact commit that failed to deploy, and it pings you directly in slack, because you were the one that committed the code.

A

This means that you no longer need to be watching your deploy, because if something goes wrong, you'll get ping directly. The notification also tells you what change broke your app, because it's linked in the notification.

A

So um if you, if you get this you, you can also navigate to your app in spinnaker from the link in the notification and mark it as bad. A no notification is then sent to your team slack channel, so everyone is in the loop without you having to do anything, you pin production to the current version, while leaving test on pin, because you don't want a bad commit to accidentally make it to prod, while you're fixing the bug.

A

Spinnaker automatically starts rolling back the latest good version and notifies you when it's successfully done that, even if it takes you a few times we'll tr we'll keep track of that um and keep notifying the team until your fix and features um get a successful deploy once you unpin. This is what it looks like when you are finally successful.

A

Yay manage delivery is a huge part of what will make delivery seamless and how it will simplify the complex there's, a ton of features that do this, but the workflow that I just showed you exemplifies how we're going to make things less complex and more simple, so that our users have a better experience when delivering software all right.

A

I want to talk for a few minutes about continuous experimentation, so the next step in the process is to give our users a seamless experience when they're trying to figure out how we can give our users confidence in the safety of their delivery at netflix. We use a canary strategy to gain confidence that the new version of our software is safe to deploy.

A

So in this diagram shows you a basic idea of how a canary strategy works. Basically, what we do is we set up a couple of instances alongside the production cluster, which remains unchanged during this process. Those two instances represent a baseline and a canary cluster, in other words, version one and version two. The baseline cluster runs the same version of code and configuration as the production cluster and the canary is running the 2.0 code, there's typically three instances that are created in a cluster.

A

The canary cluster runs the proposed changes of code or configuration, and the baseline cluster uses the same and again three instances are typical, um and so basically after as these are running, we can compare them using our canary analysis platform called kayenta to make sure that the new version is working just as well as the old version.

A

This process was a huge improvement from the old days when we had to validate success manually, but over time our customers were not fully satisfied. Much like domino's circa 2009. We heard the message loud and clear. Our pizza sauce was terrible. I mean our canary experience is terrible. To sum it up, our engineers felt like the canary experience, was complex and flaky. So after much thought and debate, we decided to migrate the canary experience to our infrastructure experiment platform. Chap chap does many of the same things it. It sets up.

A

Canaries, it uses the same canary strategy, and so in doing this, we figured that we would be able to take over that process. Combine it with all the other infrastructure experiments that we do and provide better pizza, sauce or a better canary experience.

A

um The goal is to improve the experience to give users greater confidence and canary results and make it easier to configure and maintain the canary, just like ci and cd we'd like to make the configuration simple and as hands-off as possible, without removing the user's ability to add custom configuration.

A

So once your app goes through our validation steps, we want to have a clean handoff from delivery activities to operational activities. If we can achieve this, the user does not have to concern themselves with the deployment so from continuous integration all the way to continuous experimentation. That should be seamless.

A

However. So far, I've talked a lot about the three domains in delivery engineering and how they are independently working to create a seamless experience that simplifies the complex. The next piece can actually help fill in the gaps and make the process seamless throughout the process and also scale it from single applications moving from ci to ce, to multiple applications being able to be managed throughout the entire process. To do that, we use meme. Meme stands for managed experience for most of engineering. It is a tool for managing, build and delivery infrastructure from families of apps.

A

It simplifies app owner's life by providing a high level experience, as we have migrated to microservices and have built more and more of them. Our engineering team started having a new pattern of often owning more applications than the number of people on their team. Apps keep getting smaller and in many cases they are built very similarly to each other using the same best practices even outside of a team. An organization that has many similar teams working on similar products likely have many similar apps that are using the same practices.

A

A common frustration from teams like this is that, despite the fact that they have many applications that were are very similar, we don't really have a way to manage these similar applications in bulk. If we have to change one thing, a best practice changes for example, then we have to update all applications manually.

A

This is where mean comes in. You can create a template or an application profile and apply it to many applications. Allowing you to manage your jenkins jobs, your spinnaker pipelines manage delivery definitions, canary configs and even observability resources.

A

Meme takes all of the delivery products and pulls them together, so they can easily be managed as a family of applications. Additionally, using neem we've simplified the configuration rather than manually, setting all this up. You instead answer a few human readable questions, and customers can enroll into meme and get multiple resources created for them, after that they can reconfigure those resources by changing the answers to spinnaker, with meme taking care of all the implementation details all said and done. This amounts to orchestrated jenkins jobs.

A

Spinnaker pipelines manage delivery definitions and more and that's all without having to worry about implementation details without having to craft and maintain the configuration all meme resources are templates and managed by application type owners who can codify the best practices as mean team templates and update those for as best practices change, which one last time helps to provide a seamless delivery experience that simplifies the complex well for joining me today. I hope I sparked some ideas for how you can give your developers more time back. Thank you. So much.