Continuous Delivery Foundation CD Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keeping builds green at scale - Alec Stewart, Aviator

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Keeping builds green at scale - Alec Stewart, Aviator

Most large scale companies struggle with Continuous delivery due to unstable build infrastructure.

Often, teams that work on a monorepo struggle with keeping their main branch stable, especially as the number of engineers merging changes (and consequently, the number of code-submissions per day) grows. This happens because incompatibilities emerge when multiple changes are combined, causing builds to break frequently.

Poly-repo setups present their own challenges: synchronizing merges when changes span multiple repositories, rolling back related changes across repos, and testing across multiple build/test pipelines can become coordination time-sinks for developers.

This talk will feature a distillation of various merge strategies that help teams scale, keeps builds stable and be production ready.

A

Try to Quick move quick because we're a little behind schedule, um but yeah so focusing today on keeping your builds green at scale uh or specifically merge strategies um so quickly about me. um I've been working at avader for the last four months or so um I love for engineering and open source and I'm a sailor. I make that point because I'm gonna happen dumb sailing pictures in my but uh yeah anyway. Moving on so merging. How hard can it really be?

A

I mean just go to GitHub and press your little green button there and everything should work out, uh jumping ahead a little bit. The problem ends up being uh well, the problem. Sorry, the problem ends up being um oh shoot. uh I am just blank on the word: uh lots of people all at once, but anyway moving forward. So the important thing uh here is how you set up your repository. So if you have a mono repo um stuff like easier dependency management, it's simple or refactor your code across projects.

A

If there's a vulnerability, you don't have to fix a vulnerability across multiple repositories and you can standardize your tooling for that one repo. um Obviously there are some disadvantages. If you have a poly repo, you have inherently independent pipelines, because all of your builds are separated out um yeah and moving forward so uh spoken specifically on monorepose. uh Why is merging hard? Well, in this case, you can have stale dependencies. You can have implicit conflicts in the PRS that you're merging um stuff in your CID CD pipeline, like timeouts various infrastructure issues.

A

All this can cause. Basically, your builds to fail to not be green, and so particularly today, I'm going to be focusing on the stale dependencies and implicit conflicts and how we can resolve those to make sure that our builds Stay, Green jumping forward all right. So.

A

Typically, we kind of with the way that we usually do and handle CI CD. You know you have all of your PR's running through your continuous Integrations test, Suite your pipeline, and you know this is a vaguely innocuous situation where you just have two PRS that you've pulled from your main line.

A

Both of them are passing STI, but then you go and merge the pr number two in and all of a sudden, your your build fails, and it's it's really confusing as to why the problem is, is that sometimes you can have implicit conflicts between your PRS? So this is a pretty simple example of you know: pr1 they updated your function, such that you know. The function now takes two parameters and PR2 you know, makes a call to that function.

A

All of a sudden, you know it's not using the right number of parameters when it goes in um in and it breaks your build. uh So really, what's the what's the solution?

A

um Actually, oh sorry, so as team size go up um in general, you time to find that these sort of breaks- and this sort of issue happens more because of uh you know, congestion, basically, and actually that was the word I was trying to remember earlier sorry uh because of congestion. uh You know you have a lot more people pushing code, it's a lot more easier for these sort of uh implicit conflicts to come up or really just any of the problems that we listed earlier, um but uh yeah.

A

Just that you know, as your team goes up sign team size goes up. You generally run into these sort of problems more frequently. um So how often does this really happen? uh So I? Don't know we joke that. uh You know the new uh the new. My code's compiling is well, it's not merging and I just have to you're waiting there for your your build to be green um but yeah. So how do we? How do we fix this problem?

A

um So there's a lot of uh solutions that exist um or GitHub is coming out with a merge queue. Although I feel like it's they've been saying that for a while, but supposedly in like November or December, it's actually going to be widely available.

A

Gitlab has options, merge, train, boers is open, source I think also mergify is open source I'm, not sure if any of you have heard of them, but we also Aviator also has our own similar solution. So, looking at a simple merge queue, what does it do so?

A

Essentially it is a cue in the traditional sense. uh You know. Basically, you have your PR. It runs through your normal CI Pipeline and instead of you know, pressing the green button and just merging it, you basically notify the merge queue. Whichever implementation you decide to go with, and you just basically tell it that it's ready to merge your PR.

A

um The queue then usually in the symbol case, will just pull the latest commit from your main branch and basically uh put the pr onto that, commit um basically to confirm that in your latest version, that you are in fact uh good with the latest version of the code, and then you know moves ahead so now when in this case, when PR2 comes along, where you had the issue previously with the implicit conflict.

A

In the same way, it checks that it's ready, it checks the latest Branch, where now, instead of having that implicit conflict with pr1 we're checking not just with the CI pipeline, we're checking against the latest version of your main branch, and then you can go forward so a little bit of performance um if you're on a small team- and you have a CI pipeline or CI time of maybe 30 minutes, which I think is pretty industry standard I, don't know if you guys might be faster or slower, um but you know it's for a small team.

A

It's it's still pretty doable. If you want to have a merge, if you want to guarantee uh that your burden, your builds Stay Green, um you know it'll take five hours about to to merge all of your stuff sequentially with all the uh CI runs that you have to do um for a large team. You sort of now can see that you run out of time in a day to be able to merge your PRS. So obviously this doesn't really scale.

A

So what can a large team do to get the sort of benefit that a merge queue provides well, and this is coming back to my picture earlier. This is the sort of the congestion problem you have too many people all trying to merge at once. So so, how can we? How can we solve this all right? So one easy solution about your changes rather than uh uh you know just do one PR at a time. Let's say we'll do a batch size of four and just when we hit four PR's that are ready to merge.

A

We group them all together and uh pull the latest commit and apply those four PR's on top of the latest commit and if everything's good you're all set. uh If something breaks just bisect the batches, and we can do something like essentially binary search for the actual.

A

um Basically, the bisection would continue if the next thing failed, and so we binary we base essentially binary search for the pr that is causing the problem in the batch um and how so? How does this dozen performance? um Well, we we're doing. 100 PR runs so we have four and four in the batch and so we're doing slightly better. We've cut down our time significantly and we've cut down a number of CI runs significantly and with batching.

A

You know on a you know, they were actually sorry so we're focusing here on the large team case, not the uh small team case, because I think in small teams the simple merge queue sort of solves it.

A

um So you know we've cut our in the best case, we've cut our CIS runs by uh by a fourth and we've cut our total merge Time by a fourth. um But obviously, if there are failures- and we assume here- 10 failure- rate um you're sort of just barely hitting all of our PRS merging in a day, um so can we do better?

A

So this is where we want to think about merges a little bit differently so, rather than uh just a PR that you're trying to merge in um kind of think about, like the space of possibilities or I guess to use my little sailing example. What if you had a everybody, could round their own Mark now, there's no more congestion.

A

um So uh here what we would do is uh optimistic queuing basically, so when pr1 is ready, rather than pulling the main branch, what we'll do is apply pr1 off of the commit the main branch and basically make a new main when the pr passes main gets moved to that new Branch, um and we can just continue along the line as we, you know, everything passes uh what happens if the other one fails well just make a new branch.

A

Now. The problem here is uh in the best case, we're still doing. 100 CI runs because we're basically doing a CI run per branch that you're pushing up uh in the worst case. As you fail, you know we're going much much faster, but your CI runs go up significantly, which can obviously you know, cost you more, um but can we still do better?

A

So, if we're doing an optimistic queue, we can also still batch it. So you know similarly similar process we had before. um If you, you know, in this case batch size of three with the same process where we bisect the batches and we move ahead as we.

A

You know sorry um so yeah so looking at the uh again. So if we're, if we're doing the the batch size here, uh we again improved our CI runs. Really. The biggest impact here um is cutting down your CI runs um because you are batching your changes. uh The time it takes is is about the same, um but can we still do better so in this case, looking at potentially predictive modeling, so you could look at your PRS that you have and try to figure out.

A

uh You know: is this just a change to the readme, or is this a change to something else, and do you use some kind of code? uh You know code analysis, basically to figure out which changes are most likely to break your build. So in this case uh yeah here's, maybe some optimizations, you could use of like lines of code changed the types of files modified. Like I said. If it was a readme file, it's probably not going to break anything.

A

um Are you adding tests or moving tests and dependencies?

A

In this case, we can maybe uh assign a score essentially to each PR and each chain of PRS and then basically pick the best path as it were, to merge your changes.

A

So I think that's about as good as you can do uh just relying on a normal system, but we can still do better. uh So in this case it's a multi-q, um especially in mono repos you'll, often find that uh when you make a code change in one section of the monorepo, it really doesn't impact or should not have no impact on another section. So in this case we look at something like affected targets.

A

So I don't know if any of you have used a system like bazel before, but that would be a good example here. So in this case, what we're doing is basically we're. Looking at what Targets in your code is the change affecting in this case you kind of build a chart and you would look and see like okay, pr1 is only affecting a PR2 is affecting B and C Etc, and we are create four cues that we can process through, and so you know now we're paralyzing their processing.

A

So we can, you know, run a little bit more efficiently. um A pretty simple use case for this might be. If you have your back end and front end, all in um you know all in your code base, you can sort of push changes to the back end without it affecting the front ends. Well, we'll we'll see later, but you know it's it's not always that simple, but you know if you can make that assumption.

A

uh You know you can just happily uh do optimistic queuing um in both cases uh and the two cues can run simultaneously um and and not impact each other.

A

But uh can we still do better, so this is kind of more further optimizations, less stuff that we've actually made. But uh you know you could reorder your changes to help get stuff merged faster.

A

um Failing fast, like you test order execution, a lot of people will do um a lot of times. People will you'll see that people will optimize for having a uh tests that run on every build and then sort of, like uh the full battery of tests run on their main branch, um but obviously, at that point, you're no longer really guaranteeing green builds, but it helps uh it gets PR's merged, faster um and then okay, yeah, yeah, pre and post merge testing all right.

A

So that was kind of most of the use case for merge, cues and monorepose poly repos also still have their problems with merging.

A

um So let's say we have. uh You know four repositories um in this case. It's your API and then your the various different uh front-ends for that you have a PR going into each of them that are all sort of related to each other. uh uh In this case, maybe the pr number one in each of those repos is relying on the same API change um in this case. uh Android fails. Everything else goes through and so now you've. Maybe your continuous deployment has pushed out uh the Android front end.

A

That's relying on a back end that or has pushed out or the Android front end hasn't been updated yet or if I guess, if the API failed, then maybe you pushed out a bunch of front end changes that the back end is not supporting yet. So how can you really solve this problem?

A

Basically, you would want to build. You need to have stuff, like depend, essentially dependency Management in your PRS systems. That kind of know like okay, these four PRS are implicitly tied together. um There's a bunch of different ways that you can do this, um but essentially you want some have some guarantees on merging all four of these PRS as like one Atomic unit, some other stuff, that's uh around merging, so stack PRS. One of the big problems with also with keeping your builds green is code. Reviews are hard or they're annoying.

A

If you got 10 lines of code, people are going to see a bunch of issues, and you know if you have 500 lines of code, I I, don't I, don't want to read that. So it looks good to me. Those sort of code reviews can cause a lot more problems, so Shameless plug. We also do have an open source library for managing stack PRS. Basically, what a stack PR is essentially is you create like a dependent like stack dependencies of PRS, where one PR depends on the next.

A

Pr depends on XPR, which allows you to break your code into smaller chunks and get the effect of you know better code reviews and similarly, on the back end, you know you can manage your stack PRS to merge, because if you can imagine trying to merge, stack PRS manually and then rebasing everything after every single merge would potentially get annoying yeah. Another I think was brought up earlier. Is flaky tests.

A

As you can, as you can imagine, every time you have a flaky test fail, it's really hard for the merge queue to like. If you go back to the emergency solution, it's really hard to know if the merge queue uh should reset, because the test was flaky um or uh really what went wrong there. um So managing your flaky tests when you have something like a merge queue setup is, is also uh really important to avoid a lot of extraneous CI runs um yeah. So hopefully that went pretty fast.

A

um Does anybody have any questions.

A

Yeah I mean that makes sense uh so, like a lot of that is around um just talking back to the user, uh so in this case they would be yeah, no 100, I. Think in this case um the the the nice thing about something like a merge queue is that hopefully it's a bit of a set in and forget it process where, if there's a problem with your PR, you can, for example, integrate with slack to get a notification that hey my PR broke.

A

um Rather, so it's really. It really comes down to.

A

Yeah, so you it hits Jenkins and that's where the failure happens in this case um I I guess so. On Aviator side we do have a lot of Integrations, with slack, for example, to notify you that, like hey, your specific PR, failed.

A

um So the system would do that automatically. It's not just going to like when. So. If, in this case, uh the way a batching merge queue would work, is you know it would bisect the batch such that the two good PRS would continue to try to merge and the one bad PR would eventually just get uh quarantined.

A

A

A

I'm not sure I, understand the question, uh but.

A

A

Yeah they go in the queue is.

A

A

Yeah so in this case uh they would get run as one one build um and then, if that build failed, you would get some sort of bisecting where eventually the the bad PR would be pulled out.

A

Yeah, in this case, it should do that automatically, for you is, is really the the main, the main benefit yeah or one of the benefits.

A

Yeah um I, don't know if there's I, don't know how long that took yeah I think you did pretty well okay. Thank you very much.