GitHub Git Merge 2022, 14 Sep 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 500,000 files and counting - Git tooling for monorepos at Canva - Git Merge 2022

Description

Presented by Alex Sadleir

At Canva, we made the conscious decision to adopt the monorepo pattern with its benefits and drawbacks. Every week, hundreds of engineers work across half a million files generating close to a million lines of change (including generated files), tens of thousands of commits, and merging thousands of pull requests. We have developed some techniques and tooling to help us deliver Git at scale including a caching proxy for CI use and local git execution tracing via OpenTelemetry.

About Git Merge:
Git Merge is dedicated to amplifying new voices in the Git community and showcasing the most thought-provoking projects from developers, maintainers, and teams around the world. Git Merge 2022 took place at Morgan Manufacturing in Chicago, IL on September 14th and 15th.

A

Cool all right, uh hi, I'm, Alex uh I lead the source control team at canva, um I'd like to share with you today a bit about the journey we've been on with mono repos um and also a bunch of tools that we've used along the way.

A

Cool uh yeah, so we're going to look a bit at the kind of problems we saw about looking analyzing, the actual repo and then talking a bit about the server side, the CI system and the local environment, which I think you've heard a lot about now cool. So at canva every week you know hundreds of Engineers have thousands of pull requests. Tens of thousands of files are changing these days. You can see there. It's almost 80 000 files change a week um during Peak working hours.

A

Trunk moves every two minutes, and this started people started to notice that things were taking a lot longer. In fact, they were doubling year on year in terms of using git, and this wasn't great in terms of pushing, but some people were really waiting quite a while for their pushers.

A

So uh the first step we looked at is: is there something wrong with our repository uh I mean as you've heard about half a million files is pretty standard.

A

What's going on here, the first problem we found was uh we initially considered stale commits and branches to be six months old, but things were actually moving so quickly that two months was a better place to put it, and by the time we realized this about half the branches we had would have been automatically deleted at stale.

A

um We then used uh the git Sizer tool, and you know most things were okay, but the thing that gave it the most concern was the number of directories. uh We have a lot of microservices um and it turned out that uh canva relies a lot on microservices and as more and more became developed and they became even more complicated. The folder structures were you deeper and deeper. So, for example, this is a 16 level, folder structure that might be familiar to Java developers.

A

uh We also looked at the content. uh You know, uh there's a bunch of file types there, but there's this weird one at the top XLF, um so canva is translated into over 100 languages, uh and every component has a corresponding translation file for per language. So you can see this just skyrockets.

A

um The other uh interesting file was feature Flags. uh We had a feature flag system where you had to declare the flags in the source code. So, as there were more features, there were more feature Flags um for these features. uh These ways that we're using git as a database we're now considering like other approaches, but you know having a large code base already it's a bit hard to make changes like that.

A

um The other thing we looked at was the CI system. um We store ICI pipelines in git, so uh every time a task has to happen on CI, even if it doesn't really need the whole content of the mono repo. It does need a fetch in a checkout uh and more concerningly. This maximum checkout time started to go up and up and up uh you know the absolute difference between 16 seconds and 21 seconds for a human, probably isn't that different.

A

But when you're talking about millions of times it really starts to add up- uh and in fact uh the numbers are a bit small here, but it's about 80 000 requests. We were seeing uh every 20 minutes, whoop.

A

Yeah yeah cool, uh the initial idea we had was okay. We can store a copy of the repository on S3, uh pull it down. The problem again is that, throughout the day that snapshot was already well out of date, so we had a bit of a think and we googled around and we found this project uh from an open source project from Google uh called goblet uh goblet is a git proxy server. There was this little warning note at the bottom. It's always good to have a warning note.

A

You will need to write some glue code, so we wrote some glue code. We linked it up with the way the GitHub authentication works. We added some extra monitoring so that we could kind of see what was going on. We deployed this out to a very small percentage and all the case servers instantly stopped responding. uh 100 memory, usage, 100, CPU usage.

A

We had failed to realize what goblet was warning us, which is that it was designed for Google open source code where there's one commit that lots of people download. While we have lots of people wanting to like download lots of different commits all at exactly the same time, so we had a look around in that previous uh screenshot. It said pack object, so we googled pack object and we found this note in the uh git source code commit.

A

That said, you might want to install the insert a caching layer around pack objects, it's an ideal place to consolidate identical requests. So that's what we did. uh We uh there was lots of really great um enhancements for mono repos uh coming out. You know, multi uh index packs all that kind of stuff. Unfortunately, we couldn't use that with go get because it was completely separate to the libgit we added a cued pool. So if one request came in for one commit, we'd wait until we had that ready.

A

But, most importantly, we added a pack object case hook, and this was uh we found it really helpful to look at the way that gitlab getaway. They had some really great notes about implementing this that helped us design it. If you're interested I've put up our Fork on the canva public repository, it probably isn't what it isn't upstreamable, because it's a completely different use case, but uh have a look.

A

So when we did that we actually managed to process more fetches Yeah by a whole third So Not only was it, you know more reliable, but it was actually more efficient.

A

There were some other interesting things we found with that analytics we put in um you know the worst case was now less than 20 seconds, which sounds all right, but the really interesting one was when we looked at the server side of that analytics, we were actually making the git server. We were pulling from a lot happier because we were pacing. Our pulled worker queue was actually kind of providing a rate limit which makes everybody happy, because that's the same uh git server that uh Engineers use cool. uh This will sound very familiar to you.

A

uh We had implemented a bazel, sparse, checkout uh engine. uh This was pretty good. We also switched from nothing to Watchmen to Native FS, monitor which our spice checkout is 7 to 32 percent of files, um so maybe not the best. We need to optimize the build graph, but you know decent stuff, but to really get to the next Frontier. We needed a bit more data, so in looking at previous talks at git merge there was this mention. You should really measure first, and we did that.

A

We looked at the trace two uh stuff and we hooked it into our existing traceability solution, which was based on open Telemetry. So we built this glue tool called Ollie, which translates Trace 2 into open, Telemetry traces out. So here's you know, for example, fetches uh in Jaeger, which is the interface um you can drill into a operation and see all the sub operations. uh Probably the most interesting for us was. We were augmenting the data with extra tags. So, for example, we found once we implemented this on Engineers laptop.

A

We found 13 different versions of git in our environment, including some that we didn't know about inside, say a atom or xcode or GitHub desktop. The other thing was we were able to actually quantify turning off some of those features or turning on like many files or you know, untracked cache. So this was really good, because uh you know we actually knew what we were doing was actually beneficial.

A

Again, we something we didn't realize was we could actually take that data from Jaeger and put it into Gabbana, which meant that we actually now had visibility over all the errors and all the timings which helps find these issues, and this works so well that we decided we'd, implement it on co2. So you can see you know, there's 155, 000 traces is is no problem, so that was really good um as an example of something we found later.

A

uh We disabled enumerate on tracked on CI, because CI didn't really need to know that information in status. It just wanted to know you know like what branch am I in. um We also discovered some bottlenecks about git lfs. There were some features we weren't using like uh being able to lock files, there's also a pure SSH protocol that didn't seem to be doing a lot. It was just time outing for four seconds, so we turned those off our and I actually saved us 15 seconds on checkout and commit which made Engineers very happy cool.

A

uh So that's my talk. uh If you want to hit read more there's a post on the engineering blog or I'm at maxius on GitHub and Twitter cool thanks.