Home
Contribute Contact Us Browse all meetings
Home Contribute Contact Us Browse all meetings
Continuous Delivery Foundation / cdCon 2022 / 24 Jun 2022
Continuous Delivery Foundation / cdCon 2022 / 24 Jun 2022
Previous Meeting Next Meeting
⏯
Add meeting Rate page Subscribe
youtube image
►
From YouTube: Merge Strategies at Scale: Developer Happiness Through Better Code-submission Workflows - Ankit Jain

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Merge Strategies at Scale: Developer Happiness Through Better Code-submission Workflows - Ankit Jain, Aviator Technologies
JW Grand Ballroom 3

Speakers: Ankit Jain
Code-submission processes can highly impact developer productivity, especially as engineering teams scale and codebase complexity grows. Often, teams that work on a monorepo struggle with keeping their main branch stable, especially as the number of engineers merging changes (and consequently, the number of code-submissions per day) grows. This happens because incompatibilities emerge when multiple changes are combined, causing builds to break frequently. Poly-repo setups present their own challenges: synchronizing merges when changes span multiple repositories, rolling back related changes across repos, and testing across multiple build/test pipelines can become coordination time-sinks for developers. This talk will feature a distillation of various merge strategies that help teams scale, and their associated developer-productivity trade offs.

A

All right, thank you, everyone for joining here. So today my name is ankit. Today we will be talking about merge strategies at scale and uh thanks for joining. I know I'm only I'm the only one who's standing between you and the karaoke and evening today so bear with me we'll try to finish on time. um Let me just start here so a little bit about my background, I'm a co-founder of startup, a new startup. We call aviator, it's aviator.com. You can check it out.

A

We are building developer, workflow automation platform, so essentially helping developers automate some of their workflows.

A

Previously, I've worked in uh as an engineer in companies like google and adobe, where I've built a lot of infrastructure, and then I've worked in startups, including shipper, home, joy and sunshine, where I've like essentially helped build a lot of like internal tools and help scale the engineering in those companies.

A

So uh you can also find me on twitter at uncut xg on top right. Don't have a lot of followers, so don't care as much uh but but yeah, like uh generally, I'm very passionate about, like helping developers be more productive, so this conversation is going to be just like talking about some of the strategies.

A

Some other companies use as well as some things that we have also built ourselves to help some of the large engineering teams, but I think the first question many people may be asking is merging like how hard can it be right like there's this big green button, you press that like? Why do you need like to even have a talk about that? So I think one way to actually like reframe this or before. Even we reframe.

A

This, like one thing we need to understand, is like differences between different repositories and even before we go a lot into those details. Like I mean, like I'll, probably not spend a lot of time here. There was a talk in the morning by jeff was talking about monorepo's.

A

I know it's still a debated topic but like at least it's worth understanding the differences before we go into a lot of details about like what problems we are trying to solve. So to give you some overview, mono repos they're, easier to manage dependencies, because all of the things are in one place, it's easier to refactor changes like kind of like cross project. Everyone can collaborate together. If you have vulnerabilities, everyone is kind of like you know, it's easier to manage those vulnerabilities as well, because kind of like everyone is using the same libraries.

A

It's standardized tooling, across your entire company. It's not like just every team has their own tooling infrastructure and, like code sharing, becomes a lot easier but like there are advantages of using poly repos as well so kind of like some of the moments I've mentioned is like your ci.

A

Cd systems are very independent and like easier to manage easier to build um pipelines, you can very easily build as well and if there are build failures, typically they're like very much localized and kind of, like the last point, we'll give you a little bit more sense of where I'm going with it. So uh actually, I may have skipped one slide.

A

uh Actually, we'll come to it. So I think yeah. This is the one right. So we talk about.

A

One way to think about like why merging is a problem is to think about, like how often your mainline bill fails like how many people here like have never have no mainline bills fail ever right, so you have your ci systems, but the question here is like is that ci system enough? Is that, like full proof that, like your main line, is never failing? There are many different reasons why these these change, like your main line, can fail. Even though you have like your ci systems.

A

Some of the reasons I've explained here are like still dependencies. For instance, you are having you're running ci on like actually a piece of code which is using a very stable version of your main line and then you're modifying changes on it.

A

You can have like implicit conflicts where two people are working on the same code base and then trying to merge at the same time where you can kind of like have conflicts with each other, where your main, like your ci, may pass, but your build may still fail, and this is something we'll discuss more in detail and then there are like other issues like infrastructure issues where, like your infrastructure, is unreliable, you may get like timeouts. You have like dependencies on third party, uh you have like dependencies internally or or like.

A

There are race conditions within your system, um and then there could be like maybe test or written like like not efficiently, where there are like shared state uh flaky tests, because of which your bills can fail too. So for this conversation we will focus more on still dependencies and implicit conflicts and explain more why merging can be challenging at scale so to see a little bit like more understand the problem. Like imagine, there are like two different pull requests.

A

One was branched out of like, like both of them have branched out of two different points, and both of them are actually passing your ci independently.

A

But eventually, when you merge those changes together, it's possible that they actually will cause a bill failure, so this happens very commonly, especially if you're working in a monorepo, where there are like many people working on the same code base, you may modify one part of your one part of that change and, like some, somebody else will modify something else and that can cause bill failures eventually, when both of them are trying to merge at the same time or like around the same time.

A

So why is bill failure, something we should be worried about, so impact of bill failures can cause significant impact on developer productivity. If you look at it like, as your team sizes increase, the number of bill failures also keep on increasing kind of. If you have like over 50 60 developers in your organization, you will start seeing a lot more bill failures happening in your main line and then eventually what will happen is people will set up like bill police?

A

Oh bill's, failed now kind of like somebody needs to figure out like who's going to debug. This and they'll be like somebody needs to roll back like do we roll back like to the last green build? Do we need to like roll forward? So there's always all of those conversations, and, and now it's like until the bills are green. Now you cannot even deploy right.

A

So, if you're relying heavily on continuous deployments now, you're kind of like your pipeline gets stuck and there could be like chain reactions where now people have possibly changed branched off of a failing main line. That means like their changes and their pr's are also going to fail. So all of those things can cascade and eventually like similarly like compiling.

A

There could be like scenarios where developers are just waiting for the bills to be green before they can do anything else right, so they might be just like stuck playing playing games. So I think that's one perspective and like we want, I wanted to explain you. The problem around like why build failures and merging is a problem even with current systems that exist.

A

So then, what's the solution, so there are like some uh solutions that exist today for merge automation, so there's actually a concept called merge queue. That github is also recently announced. It's in beta git lab has a solution called merge, train and there's open source products like boards, which actually also provide some of these capabilities. We'll go over some of these details and explain you kind of like how some of these things work for simplicity. All of these merge automations for rest of our conversation.

A

We will consider them as kind of like I'll use them in the same way as like merge queues, so we'll talk about like different emerging capabilities at scale. So, let's talk about first, a very simple version of merge queue, so what happens? Is let's say you have a pull request which was branch out of this point and it's ready to merge so instead of like merging manually, like kind of like pressing that green button, you can actually like just tell a merge system that it's ready to merge.

A

What that system should do is actually it should pull the latest main line from your uh from the main line and then run the ci again. What we are doing at this point is: we are validating your changes with the most recent main and we are ensuring that you don't have like still dependencies or dependencies on something else. So once this passes, you will eventually merge the change. So essentially the merge automation is doing pulling these changes from the master main line and then merging it back after validating the ci.

A

Now during that time, if you get another pull request to be to be merged, you will do the same thing, but now you're going to wait. The first for the first pull request to actually complete and merge before picking up the second one, because then it's going to take the changes of the second one. So now you're seriously merging these changes and that's why it's called merge queue which essentially ensures that you're always validating on the most recent changes.

A

So that's, essentially the basic concept of simple merge queue and it's going to just validate and merge as well the same way right so now, if you have a very small team, let's say you have a team which merges like 10 pr's a day. Your cia roughly takes about 30 minutes, which is not ideal, but it seems like a little bit more industry standard these days.

A

uh We're talking about a merge time of about five hours and your total number of ci runs will be 50., but I mean this sounds still reasonable, but if you have a team of let's say a hundred developers or like you're, emerging 100 changes a day you're looking at like every day, we need 50 hours to be able to merge changes to guarantee that your bills are always green. So obviously this is not realistic.

A

Then you would ask: how can we do this better? So do this better one of the things that boards like system the open source solution I mentioned what they do is you can actually like combine the changes together.

A

So, instead of like running validation on all changes, you essentially say I'm going to wait for, let's say in this case for prs: once there are four pr's, I'm going to consider them as like one chunk of change, pull the latest main line and run the validation once the validation passes, I'm going to merge that change again now what happens in case of there's a failure so in case of failure, now you need to bisect.

A

You need to kind of like break these changes down into two parts to essentially like uh to def identify which of those pr's actually cause a failure. Eventually, you'll be able to merge one of them and the second one will fail if in this case the second one fails, you will bisect it further depending on your batch sizes, it may have like almost like a binary search to identify the failing change now what that means for for the results.

A

We saw earlier right for simplicity in this case, I'm going to just go over like you're merging 100 pr's a day, because we understand at least if you're, only merging, 10 pr's a day. It's like you can solve with simple solution. So in this case, if you're doing a batch size of four, in best case scenario, we're still looking at 12 and a half hours to merge. All of those changes consistently with 25 ci runs so we've reduced the ci's by by a fourth, and a median case we are considering is like.

A

Let's say there are 10 percent failure rates, in which case the total merge time roughly depending on how the failures are distributed, will be about 24 hours. So it's getting there like. If you have exactly 100 changes, you may be able to merge all of them in the entire day, but, like it's still unrealistic, to expect that you'll be able to like you would want to merge changes that takes 24 hours to hit the production um and the total ci runs are about 48, which is about half of what you were running before so then.

A

Can we do this better one way, so here's kind of like one where it gets interesting. I want you to reimagine merges like the way we think about mergers is there's a main line, there's a trunk where you're merging changes so instead of thinking of like you know, there are pr's that you're merging in mainline.

A

I want you to think about this as like, parallel universes, so parallel universes is as if there is not like a single main line, but there's like possibility of very many different future main lines and you're, essentially now defining that one of those main lines will become eventual future.

A

So what that looks like is essentially instead of having a single main line. Now you think of mainline, as essentially like a series of commits and you're, essentially defining what the future like you're, essentially trying to predict. What the next feature looks like so kind of like it already gives you a little bit of sense of where I'm going with it.

A

So, let's say your main line is here which is like this commit and you get like a new pr which is now ready to merge so instead of merging it in main, you actually merge it in another branch. We call it just main alt as an alternate reality and once and let's say, there's another period that comes in now you're running ci. Both of them are running in parallel. We call it the optimistic mode in which case now you're expecting if your first pr validation passes.

A

What I can do is, I can just say, mainline points to the new comment. So now we are saying this is the new main line and anything that goes on top of it essentially becomes part of it and again, like the second one passes, you do the same thing now. The advantage here is, if your first one actually fails so, instead of uh instead of like kind of having your brick and broken main now, you can just like create a new commit and essentially say like now.

A

This can become the future made right, so in some ways, you're validating these changes before these changes hit mainline and there, thereby keeping your builds green before it actually uh like yeah, because you're already validating it before it becomes main.

A

So this is a very interesting concept right like now in your best case scenario, given that you're not waiting for any ci or sorry any kind of like ci to complete before queuing anything else. Technically now you can merge all your 100 changes in less than an hour, assuming all the 100 changes came in less than an hour, because you're kind of like just running it in parallel now. Obviously that is the best case scenario where there is no failures in case there are about 10 failures.

A

This is again a little bit of estimation, but now it's six hours, so six hours is still realistic. If you think about it like it's only for 100 changes, which we are assuming, let's say, if they're 100 people more each merging one change now, you can still finish it in during your work hours right.

A

So it only takes like six hours to merge and your the main thing here is like the ci runs, are going to increase, so instead of like 100 cr runs that you were running sequentially now, you'll probably run like 150, because now you're kind of like doing more optimistic, builds.

A

So can we do something better, so one other thing which I explained before was batching. We were doing batching in a main line concept where, if your main like, if your batch fails, you don't merge it, but can we combine both of those concepts together? Can we actually do optimistic cues as well as batching in the same flow? So essentially we are saying in this case is: if a batch of three you essentially combine them and create a commit where you're running the the main, uh the runs and then in the second one.

A

You do the same thing. You combine all of those changes together and you run another batch. The advantage here is now you're running significantly lower number of ci's than you were running before so once it passes. You kind of like just continue fast forwarding your main line to to the next garment.

A

So in this case, number of, like total number of ci, runs, have significantly reduced. So instead of running 100, you're running 25, your time to merge in worst case scenario, also reduces because you're kind of, like you know, breaking like kind of like doing it. In batches and and the total total number of ci runs are also like 48 instead of uh 50.. Well, I think total number of ci in batching is actually about the same okay.

A

So now can we still do it better so kind of, like you understand the concept a little bit about like how I was thinking about like parallel universes right. So now, let's kind of like take it one step forward right, we don't need to think about a single alternate reality. Can we think about like multiple alternate realities? Can we think about, like all these different commits prs as creating separate comments and creating multiple different branches?

A

So in this particular case, if there are three pull requests we have created? Essentially uh these are seven different comments, which represent seven different possibilities of what main line would look like, and these are all different combinations of when a particular change passes or fails.

A

So obviously like this is this is a some of the one of the most optimal way to reduce your uh reduce your time to merge the risk it comes with is obviously that now you're running significant number of ci's than you were running before right, you're running essentially, seven ci runs just to merge three pull requests, so what we can do there is.

A

We can actually like use predictive modeling to figure out like which of those chains we can actually like skip, because we have lesser uh expectation that they would uh pass, or we can kind of, like you know, say like some of the some of those possible futures may have higher predictability of passing, and we can use essentially a scoring mechanism to identify which of those we would want to pick. So. In this case, for instance, we are doing lines of code is a good metric to identify whether a particular change would pass or not.

A

So, at this point we are using linear regression to predict well based on history like how often do bills fail, you can use like types of files modified. There may be certain files which actually like cause, build failures more often if whether a test is added or removed, that can also be used to to know whether a particular bills will pass and number of dependencies. If there's like a lot of dependencies being modified, it's likely that it'll it can cause a bill.

A

Failure versus not so based on that you can kind of like for every chain. You can define a particular score, and that way you can eliminate the ones which have a like lower score. In this particular case, let's say that if the cut out was 0.5, you drop anything which was below 0.5 and only try to go with the optimistic approaches, with the possibility of kind of like using uh the same fallbacks that we were using in other systems.

A

So it's sort of like a mix of what I explained before with optimistic cues, while in this case it kind of like also goes with like multiple possibilities.

A

All right so can we still do better.

A

So now we are kind of like getting a lot more into beads of like how other possible scenarios we can run and like what called optimizations. We can do to reduce your build times or like to reduce your time to merge.

A

One way to think about is multi-queue, especially if you're working in a monorepo, not every change, depends on every other change. So, let's say: if you have like a system like bazel which helps you identify what all targets were affected by a particular change.

A

Can we actually like almost build a table structure like this, where every change, every pr has a particular target that has been affected. So if we know that, like there are like in a mono repo that, like essentially have a front-end change, you have a back-end change, they may not depend on each other. Can we actually like run them in parallel as multi queues? So if you have a structure like this, let's say there are targets a b c and d, and there are like these nine pull requests.

A

How do you think about like defining the queue, so we can dynamically define queues as like? First, queue will run like 1, 6, 8 and 9, because that's affecting target a second cube will run 2 and 4, which is affecting target b and c.

A

Third q will run 2, 5 6, because two fives now, if you see like pr two, is actually part of two different queues and that's because, like pr2 actually depends both on b and c and uh second fourth q actually is only d, because there's only one uh pr there, so what you're doing here essentially is now avoiding having to kind of like run them sequentially and kind of like doing them in parallel, thereby also reducing the number of queues and number of bills. You have to run.

A

To kind of like just give you a little bit more example. Let's say if you have like back end and front and as two different dependencies, we can actually like define these optimistic cues as two different cues, which are completely independent but merging in the same code base. This is still guaranteeing that you're actually merging the code and keeping your bills green, even though they are actually like representing two different queues and two different states.

A

So now can we do something better, so I'm going to just leave these as like some further optimizations we can do. This is something which we have like thought about like read about, but like we haven't implemented it, but like some things we can do is we can reorder certain changes, essentially, the ones which we know are higher priority or possibility of like merging.

A

Having built successful, we will merge them first before we merge the other ones. We can fail fast, where essentially, we can reorder the test itself. We run the tests which are more likely to fail up early so that, like your bills, essentially uh as soon as we know, their bills are failed. You can kind of like switch over and, like start a fresh, you can split the test execution.

A

This is something which many companies do where you can run some tests, which fail often before it merges, and then you run the smoke test after the emerge. This doesn't guarantee bill. Failures like this doesn't guarantee like green bills, but it at least like reduces your time to merge. Okay.

A

So so far we talked about monorepo, because this is kind of like where a lot of the challenges I explained come in, but it's at least worth like talking a little bit about like how these systems look like in poly repo. So the challenges for merging are very different when we look in poly ripple. Let's take this example. A poly repo is essentially like every project. Every dependencies in some ways is different. So if you have like four different repositories, one is api.

A

One is web, one is ios and android and you're kind of like merging, let's say you're modifying an api. This api essentially now also has to modify your web, because this web is going to call the same api. You have to modify ios because io's also going to call this api and so on in android.

A

Now, if you have like this is just a random example but like if you have this kind of a scenario where, in this case it's not like, um even though they're independent changes, they sort of like still depend on each other in the way that I would ideally want to actually merge. All of them together right.

A

So if you have a scenario like this, where let's say all three of the three of these changes pass, but the fourth one failed now we have actually merged three changes which, technically all four are dependent on each other, that they should go together right. So how do you solve this?

A

So we have like built the system where we understand dependencies of different pull requests when I say understand it's like explicitly defined, but if you're able to explicitly define that, oh, like my pr in ios, android and web, actually depends on the pr and api now we can actually run the system where we validate like whether all the prs can merge together and they're considered as an atomic unit.

A

Now, in this case like either, you merge everything together, while keeping all your bills consistent or you don't merge anything at all right. So this is how you're kind of like building this consistency in a poly reaper system.

A

So now you can use the same concept also in a mono repo system. You can actually like use the same dependency model and say like actually like pr. Two and three, although they are two different pull requests, they actually are dependent on each other.

A

If the first pull request doesn't merge in, I don't want to merge the second one as well, so this is kind of like also a system you can use in stack prk, so you can also use it like independently, where you're essentially like saying multiple different changes are like atomically same or like they're, together so uh kind of like this was some of the examples I wanted to share about like how we do merging a little bit glance at, like other things that we do at aviator.

A

um So obviously like there were a lot of conversations around flaky test. It's not just like. We help uh identify flaky tests. Other things we do is like we help automatically like. If you use, for instance, pi test, you can install a plug-in which helps like automatically rerun those flaky tests, and you can kind of like from a central place, configure like how often you read on those things.

A

We also generate these reports, where we actually like show you a suppressed uh like essentially suppress the flaky test that you can use to validate whether your changes are passing, and this ties in really well with, like our merge validations and we have like a api or sorry a cli that you can install to to manage your stack pr. So if you have like multiple pr's running small changes, you can actually like sync, all of them together.

A

uh So if you have like, for instance, you introduce a comment in your base, stack pr, you can actually like just use av stack sync, this syncs, all those comments to all your future. All your subsequent stacked prs as well as you can kind of like identify how you like merge all of those things together, so you can use av stack, merge which will actually like help. You merge all of those things together and that's about it. So uh these are some of the references do check out.

A

uh Obviously, if you are not familiar with merge queues, I would encourage you to even like check out github merch queue or like if you use gitlab, there's more strain and if you are interested check out aviator.com.

A

uh This is a product you can sign up for free as well, and this is my email and if you have any questions, let me know drop me. An email drop me a note on twitter or I'm happy to take any questions here. So thank you. Thanks for coming.

A

Any questions here or I can also dig it later. Yes,.

A

So if you're saying you can do it as an atomic unit, but isn't one of the benefits of having polyrepo is that you can move independently.

A

So you correct, so I think the example here is like essentially scenarios where so one of the reasons uh like it kind of like boils down to like poly reapers generally have certain library like let's say, if you're modifying a certain library and that library has backward incompatibility and now, if you're, using that library method in a different repository, you will have to kind of like modify both of those things together.

A

So in some ways, what I'm trying to explain here is there are cases in poly repels, where you may have a dependency on some other repository right. So in this case, like the example I was giving with api is like these are the apis. You have to also call from ios. These are the apis. You have to call from android. These are the same apis. You have to call from front end.

A

So if you're modifying the api itself, that essentially means you'll have to modify it everywhere else, so our system is guaranteeing at this point that these are atomic units and either we are merging all of them together, while keeping it consistent or we are not merging any of them at all right. So it only it's not going to be useful in every scenario. It's only going to be useful when there are dependencies across different repositories right so like.

A

Essentially it's a place where somewhat like somewhat some of the limitations in party repos, where it may actually break if you're modifying a version in a poly repo, like kind of like it or if you have like a dependency in the body elsewhere,.

A

Cool all right, all right, yes,.

A

Yeah I mean, I think in some ways like most of these systems are able to capture that. So the question was like: how do you identify merge conflicts in two different pull requests that may not have merged conflict with the main line right, that's kind of like some of the main like problems. You're trying to solve here is like we call these implicit conflicts where you may or like your bills, may pass independently, but they fill together.

A

Your in your case, you're, explaining kind of like your prs independently, are uh like not having conflicts but have conflicts together. So any kind of this time, if you, let's, let's say if you try to do optimistic queue, you're kind of like pulling all the changes from your previous commits or like previous prs and then trying to add the new pr to it. So at that point, we'll automatically identify uh the merge conflicts, so it will, it will flag it and it will remove it from the queue automatically.

A

Right so generally, the way it works is like every pull request when it's ready uh people can use like either a slash command or they can use like a github table or something else to identify that it's ready, so they assign the pr to this merge queue system and in the order, when they're assigning we're essentially automatically automatically adding it to either a batch or we're adding it to a particular queue.

A

So, at the moment when the batch is created or when that, like you know, this um optimistic uh branch is created at that point itself, they're able to identify uh a merge conflict so typically like it's within like a couple of seconds of actually like tagging, a pr is ready that we will identify this as a failure and report it back like our goal.

A

Eventually is like also at helping identify like where we can actually optimize these things and even be able to identify, merge conflicts before you even assign it to be uh to be merged right, because we already know what changes are there. We can actually even look at your open changes, even while it's being reviewed that there are like other changes in the queue that make conflict with this right. So generally, like our philosophy with aviator has been around like.

A

How do we identify and like automate some of these developer workflows so that you don't have to like worry about many of these build frustrations.

A

Cool all right, yes,.

A

So how do we know whether the pr is ready, so it's kind of like similar workflow right? So if you go to, uh if I just go to our first example, which I was saying in this case right, so I was saying, like generally people will do- is like press this button that it's merged right merge directly. Instead of that, you can kind of like either add a slash command or kind of like it's manually triggered.

A

So you can kind of like do slash available ready or you can kind of like attach a github label if you're using github and uh similarly like there are other ways to do it in bitbucket or gitlab, but essentially it's as a developer.

A

Adding it to queue is same behavior as like it's kind of like the same mindset as like merging it directly. So, instead of merging it directly, you're essentially saying is like run these pre com, pre-merge commits or like pre-merge validations before you merge it right. So in some ways, they're saying like just assign it to this merge gear system.

A

I mean so yeah I mean there are like different ways through which you can so. The question is whether, if somebody merges any of these pr's manually, what happens so there are like definitely like resist like configurations you can set up like, which is what we have helped many teams with, so that they can actually um like restrict people from directly merging.

A

So one way to do that is like we can actually like define a special status check which is like if the status check is not validated, you will not merge it so that status check is like a separate status check that we only validated before we merge it ourselves. So in some ways it's kind of like this can very easily be done with using branch protection rules in github or like or any other system right. So as long as you're using a git based system like all of these workflows should be able to work.

A

All right and thank you very much.