Continuous Delivery Foundation cdCon 2022, 24 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Journey to Automated Releases - Kranthi Challa & Deepa Padmanabhan, Netflix

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Journey to Automated Releases - Kranthi Challa & Deepa Padmanabhan, Netflix

How do we go from dreading a release to embracing the sense of accomplishment that a release is meant to bring? The solution needs to provide high confidence to bring about a culture shift, where humans are no longer gatekeepers of the release process. Our methodology meets this high bar with ideas like - application partitioning for independent rollouts - build-confidence scores for sign-off - promotion checks that backtrack a releasable build and many more. In this talk, you will learn the Netflix UI team’s journey in building a “Release ROBOT” for continuously delivering a large monolithic codebase while seamlessly entertaining the world.

A

All thank you for joining us today to learn about our journey to automated releases, I'm deepa and I'm a senior software engineer at netflix outside of work. I mostly spend my time with my daughter. Along with me. I have kranti chella. She is a senior site, reliability engineer here at netflix.

A

So let's start with the story. We all like stories right, especially the ones we can relate to it, has been a great day. Everything is peaceful. Things are great up until the phone starts passing.

A

It is the dreaded 2am pager, where a slew of alerts were going off side errors, latency three cpu thresholds. Many more so the incident turned out to be a ddos attack on our landing page, although it was just our landing page. This was shared by all traffic member or not.

A

The arab was wide from both the ip range and as well as the volume making it so hard for our abuse detection service. To react, though it is just the landing page that is getting hit. The sheer volume was spinning the instances of its course.

A

This incident made us start thinking about how unfair it is for anyone. You know, especially our paid members, to be impacted by something on the landing page, adding on another story. So this is. We were hoping to take some rest after our ddos attack, but then it was the day of our release.

A

Our weekly release, especially and every team exchange, was ready for prod, so we wanted to be very safe, so we got all the team sign off and then we were ready to start testing production through canaries kennedy found a bug.

A

It was a huge change list. We were not able to narrow down the change list, but we were able to narrow down the app. This is because a lot of different teams are pushing into a huge code base.

A

At the same time, we also had a very date sensitive change that had to land on production from one of the teams, but unfortunately we had to cancel the release because we wanted to be safe first and then we started bisecting and then running cannery on each of the bisect, which extended for such a long time that this team was not able to land the change to prob that they had was supposed to do.

A

Unfortunately, they missed the release. Why is this even an issue? So, let's first understand our team structure right. We have one huge code base. Multiple teams are committing to this code base, so the individual teams are responsible for developing and testing their changes, and our ops team is responsible for the ci and the cd process, the tooling involved for it, the release and the on-call.

A

So if one team introduced a breaking change, every other team was blocked. This is because we had one centralized release process owned by rob's team. So if a candidate found an issue, then the op team was responsible for narrowing down the issue figuring out which app could have caused it, but they had no idea of what the product changes went into this release.

A

So we were seeing this week over week and then we were missing releases deadlines, getting missed and then changes were. You know adding on top of each other. These two incidences were the major start for our journey.

A

We wanted to relieve the pain of the push team and the on-call team and as well as the individual, app teams, because they were also not able to land things on prod when they wanted to so a long term plan to remove the bottleneck for us was to split and deploy them individually, but that goal was very long-term.

A

It was a multi-year project and we cannot live in this unstable state for a really long time, so we had to make some small innovative steps that will relieve us of the pain that we are facing today and improve and increase our release cadence.

A

So this gave birth to the first step, which is split deployment, as the name suggests. This is splitting the deployment process and the infrastructure for deployment and not splitting the code itself. That was an ultimate goal, but in a first step let's dig dive. uh You know deeper into what split deployment is on the left.

A

It is how we were before split deployment, one app deployed to different regions using the same infrastructure, but with split deployment. We have each app having its own infrastructure, but the same code base deployed to each of them.

A

So when a request comes in, we then route the traffic to the right app or the cluster, depending on which one should be serving it.

A

So, let's take an example to understand this: let's say we have onboarding app right, onboarding will have its own cluster and then we have accounts app that will have its own cluster and then now today, onboarding decides to deploy a version, so onboarding will deploy only to its cluster, whereas accounts app could be living on its old version because they didn't do a deployment today.

A

So when a request comes in, we now have to know how we will determine where to send that request, and this is done using a keymaster rule and we will route it to the right place, which means we had to now support multiple versions of the app find where the you know routing has to go, and additionally, we also had to support the operational models for doing this, so this project basically involved building the separate cluster or the web for each of the apps, and also defining and building the operational models for running multiple apps.

A

You know when it comes to operational models. What I mean is, like you know, rollback, so roll forward story. How will each app do it separately, fast property changes across multiple shards or splitting the alerts and all on calls, um and how are we going to monitor metrics separately for each apps?

A

So this was the main the project as part of split deployment and at the end of split deployment.

A

What did we look like? We still were the same code base teams continue to merge code to one place, but each team can deploy at their own time without being blocked by other teams, so every app had its own cluster providing insulation from the first attack.

A

The ddos attack that we were talking about other than that we gained a lot more from this split deployment led to individual teams taking over release, while the ops team solely just maintained the tools to enable deployment, so teams owning releases came with a lot of power for them, the teams knew exactly what they were releasing and where a problem would lie. If something fails, they significantly also improved the kennedy, metrics and tweaked it to stabilize them.

A

You know adding to the metrics or modifying it depending on what changes to the product, so the on-call dashboards also were made specific and targeted exactly well based on the failure. So you know, other teams were not disturbed if one of the team was causing a failure, the biggest advantage that opened up was the desire for individual teams to release faster, so teams now have the ability to release at their own pace, and so they wanted to release every day.

A

Well, we could have done it. We could have just pushed every day, but then this is how it would have landed. It was not that easy. We couldn't just turn the switch on and say hey. Let's do these every day we had to figure out why we were actually releasing once a week previously.

A

So there were a few bottlenecks that we had to address before we could go to everyday database. Two, the major two, uh our questions that we had to answer was confidence and productivity with weekly releases. Our changes had a lot of time to bake in the test environment, so we had the time to identify bugs.

A

So by the time the changes were released. Multiple integration tests would have ran, however, with frequent releases. How do we know we are confident for a change to get out? The other issue was productivity.

A

If we released every day, are we adding a lot of work in terms of on-call or having someone to manual release to push the button verify if the sign-offs are there and so on, and so for us to be able to get there?

A

We had to automate all of this and additionally, to improve productivity. We had to automate confidence.

A

So let's see here um here is this is the ultimate place where we wanted to get to most of the ci and cd process existed, but then they were all you know. A lot of steps were manual.

A

There are three major phases for which, through which a build has to go through to determine if we can release it or not. So the first phase is from commit to promotion, which is our release branch, then, from the release branch we have to identify if the canary was successful and if the canary was successful, are we ready to you know to move forward all of this decision?

A

Making steps had to be automated, so let's break it down further starting out with our first phase, which is from you know, a commit to getting into the release branch. The first step for us is primo checks. Premier checks existed, you know, like unit tests, static, analysis, linting, but then the changes were merged with failures, because, most often the belief was my change did not cause this issue. It's common right: it's not my issue, so we address this with mandatory code checks on prs, without which a change cannot be merged.

A

This significantly improved bill confidence and prevented broken pre-merged bills which improved productivity. This did, however, increase our time to merge for every change, so we had to address that as well. So we added auto merge on prs. After a mandatory code check passes, we also added labels which can instruct our auto merger to tell hey additionally, if only a reviewer completes review then go ahead and merge it. So this improved our confidence and also it was a productivity boost.

A

The next improvement to pre-merge was pre-checking for string localization. One major blocker for daily releases was getting back localized strings for the ui to render netflix is available in so many different countries and languages.

A

So we had an external team dependency to get localized strings, and this had a 48 hour turnaround time any code that referenced the change cannot be released until we have the localized string available. So we solve this problem as part of our pre-merge improvements.

A

Where a change that uses the string will be blocked from being merged until the localized strings are available, so once a pr is created, primer checks. If the new string is used in the code and if so it look, it looks for the localized string to be present.

A

If it's not, we block the pr and keep checking for its availability.

A

So once it comes back, we auto merge the pr for the engineer, so this prevented other changes that does not need localized strings to be merged and released without holding innovation. You know just waiting for one string to come back.

A

The next step post, merge automation, so daily releases meant no time for manual testing.

A

Automation is our single source of truth, so this was a mindset shift, so we focused on improving coverage and reliability by moving tests closer to code. End-To-End tests is always a very unstable given partner service dependencies right, unreliable tests meant no automated decision.

A

You know to check on the build head, so we focused on strengthening our test pyramid, more unit tests and integration tests. We remove dependencies through mocks, so we built a framework to fetch the data needed and validated the contact contract that we had with our partner services.

A

If this contract passed, then we saved the response and used the exact same response to load the ui and validate the ui. This drastically increase the reliability of our tests and improve the speed of execution.

A

Also, this provided bug isolation from partner teams, a common story where ui teens always end up finding issues for our partner teams, issues.

A

Next step was evaluating the build status after automation was completed. We wanted to measure the confidence based on the automation results, so we connected with the test reporting system and built on top of it to get a reliability test score.

A

This included predicting the build status based on three factors: the state of the test after it was rerun, you know to account for any environment issues.

A

Does a known issue exist and what is the threshold passing rate for this app so in cases where an external service changes and causes us to be blocked? The health system interacted with the common known issue system and made a decision whether we can proceed with the release or not.

A

So this was a combination system which would come up with a score, and then we already had a threshold, and then this makes a decision whether we can go forward.

A

Finally, if the build is successful, we tag the build with its status. The tagging would help us to go back in time to see if which build is in a successful state. So we always had a bill to move forward with. So much of the release phase um is the same so considering team.

A

I'm going to time, I'm going to skip on the release phase, because most of the systems were reused from what changes we made as part of the first phase, the one additional thing that we added here was: how do we make space for hot fixes, and that was an addition that was added to the release phase so jumping on to our third phase of the release cycle.

A

Once we identified a release candidate, you know, which is automation, ready. We tagged the build, so we saw about tagging right, for which we had to build our tagging system.

A

The tagging system tags the build as release ready or not.

A

We could have multiple releases coming that are released ready, but at any point we should be able to pick the correct release that needs to be selected for a push. This is where our targeting system comes in, which takes care of identifying the correct release that needs to be pushed forward.

A

So now that we are ready for testing in production, which is our calorie, we already had our kennedy and everything set up, but we validated canary manually. Now we had to automate this.

A

Hence the reliability of the canary metrics had to be very high. We mentioned earlier when we split apps each team owned kennedy metrics, so they had better knowledge of the metrics to be tracked and to be measured. So the kennedy metrics were revamped to be reliable and moved to measuring by percentage variation from baseline than absolute numbers.

A

So we spent a lot of time strengthening it. So we don't have unreliable failures from galleries, so we can automate this process.

A

um Kennedy thresholds were then established so that we can sign off at automatically. So once a build passes, the threshold. We are ready for pushing the build automatically.

A

So we made all these changes in different. You know parts of the pipeline. If everything went well, we tagged the build, promoted it and pushed it out. This is an ideal world where everything went well. What about? Otherwise, you know, which is most often the case.

A

This is where we went with the model of push versus pull. What this means is to only pull engineers when things go wrong. Otherwise we assume everything is fine and we go ahead with automated release.

A

This concept was to make sure we do not impact productivity by having someone manning a release every day, because previously we had only once a week release so pulling someone once is fine, but because we're pushing every day now we don't want to keep pulling people every single day.

A

So how did we achieve this? We started constantly monitoring each step of the pipeline and set up thresholds wherever necessary. We identified groups who will be responsible to be pulled in when needed. So here is an example of our kennedy. Failure the canary is monitored constantly.

A

You know at frequent intervals and if it goes below the threshold, the appropriate person is pulled in prior to this mode. We had a release team who was always present, prepared the release, monitored the failures and so on, but with this model we eliminated a dedicated release person. Rather we have someone in the rotation who will be pulled in if something goes wrong other. Otherwise, this monitoring system will constantly be checking for the scores and then automatically sign off. If everything meets the threshold.

A

We had similar kind of monitoring for automation as well. We started monitoring reliability status on each build.

A

If the bill does not meet the criteria, we will post to the team responsible for the failure they can sign off through an external system or add a known issue and then our build starters evaluator considers this to make a decision.

A

Remember in this case it is the product team who is aware of the comments who will be notified, so this happens asynchronously starting from the post, merge phase itself. So at you know, as part of the first month. Automation, if there is a failure, will immediately intimate the team, so they can take a look, and at this point they can either sign off through a known issue or decide to fix the issue. So when we are making the release decision, we already have it accounted for.

A

Okay, so did we get to daily releases? You know: is this how things are getting done now in our team? Let's see, we did not get there on day, one we slowly ramped up. We increased release to twice a week.

A

We revamped our documentation detailing how to use each step each system, what it meant when some someone is pulled in right, they need to understand why they are pulled in in the first place. We also continuously gathered feedback.

A

We set up monitoring to see if we are increasing prod errors, we measured if things are getting better or getting worse. We learned that twice a week feedback was very positive. We actually had a lot of success.

A

The number of releases that were getting missed, reduced, drastically product changes landed every week for sure and twice in fact, rollbacks and hot fixes reduced. So this gave us confidence, so we were ready to increase one more push, but before we did that we added another step, we went to promoting changes to release branch every day and also running canaries every day.

A

How did this help? So this helped us by narrowing down issues with smaller subset of changes, which means we had more than one approved release candidate prior to the release date that can be pushed even if the latest failed.

A

So at this point we had more confidence to increase one more release and we got to three releases a week and at this point we also got 90 percent of our tasks automated.

A

So we went from one release to three releases with ninety percent of the automation tasks automated.

A

We are at this point right now, where we are gathering feedback, we conduct regular sinks to learn about what went wrong and what we can improve to see. How, if are we negatively impacting it? How are our you know, crowd releases going and are we increasing more issues in production with this productivity?

A

So what are the takeaways that we have here? It was a long journey, but a very satisfying journey that we will still continue to improve as a takeaway for others.

A

Breaking it down, helped us a lot and evaluating our current process and thinking from a different perspective, helped us a lot.

A

Products need is different, so finding the bottlenecks help automating confidence pays off going slow, collecting feedback along the way, helped us as well unmanned release is great as long as the product is reliable, with less rollbacks and production issues.

A

So monitoring and clearing and collecting metrics is very important and iterating on the solution to make it better was also very important. Our end goal, which is our ultimate goal, is to split every app is still in progress.

A

However, we made things so much better for us by taking smaller steps and achieving smaller milestones, so that is where we are and we are still making progress, but that's our journey to automated releases. Thank you so much for listening. We are here to answer all the questions. Looking forward to the questions. Thank you.