GitHub GitHub Universe 2020: Enterprise - Developer Experience, 11 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Optimize your monorepo experience - GitHub Universe 2020

Description

Presented by Derrick Stolee, Staff Software Engineer, GitHub

Stolee is a core Git contributor and a member of the Git Client team at GitHub. This team supports Git at the largest scales by maintaining Git for Windows, Git Credential Manager Core, Scalar, and VFS for Git. These tools are critical pieces of the Git strategy for the Microsoft Windows and Microsoft Office monorepos.

For more from GitHub Universe 2020, visit https://githubuniverse.com

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

Thank you so much for that introduction and hello universe. I'm excited to talk to you about how to optimize your monorepo experience. I have a bunch of git tips that will help you structure your repositories.

A

These will be especially valuable if you're a system, architect or otherwise running your engineering system and are already focused on making your developers and ci process more efficient.

A

Before I get too far into the technical details, I want to acknowledge that everything I talk about today is a community effort myself, I'm a part of the get client team at github, where we work to scale git to the largest repositories ever there are also contributions from the git contrib and git systems teams and contributions from the wider open source git community.

A

Let's start by talking about your development ecosystem. Typically, we think about developers producing and consuming source code as the primary actors in the development process.

A

However, we need to think about how that code is delivered to the build and test infrastructure, as well as to production or to customers via releases, also keep in mind that leadership and other stakeholders are involved in planning and measuring developer outcomes.

A

All these different actors are connected via your git repository. This includes your local clients, your hosting solution and the protocol to communicate between them.

A

It's important to note that no one starts with millions of lines of code. Everything starts small. It's as your repository evolves that it grows and continues growing at a faster and faster rate as your team grows.

A

If you started your project in git, then you likely started small enough that you had no growing pains.

A

You could grow quickly without noticing any changes.

A

However, as you grow large, you start to push the boundaries of what git can do or grow in ways. Git is not designed for you start to feel some pushback in terms of slow commands.

A

Then what happens if you continue growing without control, you'll burst the limits of what git is capable of doing now. My team's role is to grow what it can handle and we've already done a decent job of handling many scale factors that have historically been blockers.

A

There are still some remaining pain points that we want to address, but we can only do so much when repositories grow too large in some dimensions.

A

Let's also keep in mind that if your repository grew in a different version control system, then perhaps your repository grew in ways that fit that container, but do not fit well into a git repository.

A

It's important to recognize why you want to migrate to git, but also which parts of your old version control system are not equivalent in git.

A

The most important thing I want to communicate today is that your choices matter, you have agency in how your repository is structured and how it grows.

A

Let me drop some universal advice for optimizing. Your code code never gets faster. It can only do fewer things, whether you're talking about taking an algorithm from order n squared to order n or just reducing that value of n. Everything is about executing fewer instructions or transferring less data.

A

It'll be a reoccurring theme in this talk that I will ask you to reduce the size of your git repositories or reduce what you're expecting your git repositories to do.

A

First, you need to learn more about what git is doing in the first place in order to help you on your journey, I'd like to dig into some git internals that will help us understand some advanced get features.

A

First, let's talk about git's object model. That is how does git store your data? First, you have object called commits, which I will denote with circles represent snapshots and time as time moves forward left to right, commits are added, and they point back in time to previous snapshots.

A

Each commit has a single root tree, which is a snapshot of the files and directories directly within the root of your working directory. I will use triangles to represent trees.

A

That root tree contains pointers to other objects which are trees for the subdirectories and blobs for the files I will use boxes to represent blobs notice, the alliteration circles for commits triangles for trees and boxes for blobs.

A

As we recurse down subdirectories, we expand more trees, which point to other trees and blobs.

A

Finally, we have enough objects to completely describe the working directory expected at this commit as we walk back in time, we can see. Other snapshots share some objects with this first snapshot. These shared objects correspond to paths that have the same content between the two versions.

A

This allows git to efficiently store all of the snapshots without repeating identical content. This is called a merkle tree representation for all of you, blockchain nerds out there.

A

As we continue walking back in history, we can see that every commit has a root tree and these trees form an interesting directed graph. I will come back to this picture a few times, so, let's recall that circles represent commits. These are points in time. Triangles represent trees. These are directories and boxes represent blobs. These are file contents.

A

When we examine a typical git repository by object type, we see on the left a typical split by object count. We typically see more trees than any other type and more blobs than commits, because on average, more than one file is changed per commit in the middle. We have rectangles representing a good distribution of the size these objects take. Within the repository I used the linux kernel repository as a good example, because git is designed for the kinds of changes done in the kernel.

A

We say that blobs take up a little bit more by proportion than their count.

A

This really relies on most files being plain text source code that change only a few lines at a time that allows git to delta compress the blobs very efficiently on the right, I have what is, unfortunately, very typical when blobs contain binary assets or generally have large diffs and do not compress well. In fact, this ratio is not even as skewed as would happen.

A

If you regularly check in a binary executable, to find out more about how large your repository may be check out the command line tool getsizer this tool will measure many different dimensions of size in your repository and also will give an opinion on how problematic that dimension may be.

A

Give it a try today to see if you can quickly identify what's wrong. With your repository.

A

Looking back at this object model diagram, it's important to notice that there are arrows from your recent commits, following through your older commits and their trees to all of the blobs in your history. This means that those old blobs are still causing you pain today.

A

Think of sisyphus pushing the rock up the hill, except occasionally he gets a new rock, but all the older rocks are connected by a long chain. He has to pull them all up the hill, making progress even harder as he goes. This is kind of what happens when you regularly check in large binaries into your repository.

A

So naturally, my first recommendation is to remove large blobs from your repository. One way to do this is with git. Lfs stands for large file storage, which removes the large blobs from your git repository and places them in a secondary storage layer. This allows your git object database to be more efficient, but you still need to download those large blobs when you do a git checkout. This means you have hidden the problem somewhat, but is still there causing you pain.

A

The other way is to take any large binary dependencies and put them into a package management system such as ghetto packages.

A

If you need a binary to build or test your software, then only download that binary when that build, actually needs it cache these binaries locally, but also in a way that a user can easily remove old packages when not needed. Generally, my advice is to use package management whenever possible.

A

However, that might be a difficult ask, so that leads to my next hot take. If you can't review it then delete it. I mean that if you cannot review the changes to a file within a pull request, then that file does not belong in your git repository. It can go somewhere else.

A

Remember: git is designed to hold source code. That means it's really good at plain text documents that change a few lines at a time you will never be able to review the change in a test data stream or in a binary executable needed for your build.

A

So, let's think again about our friend sisyphus. We want to lighten the load so pushing these rocks isn't so hard.

A

If we manage to remove the large blobs at the tip commit, then sisyphus can pick up the rock and carry it more easily. However, the git history still has all those big blobs, slowing them down.

A

A natural thing to consider is to rewrite the git commit history to not have any of those large blobs anymore. This makes the history much lighter.

A

While this is the best option for optimizing your git data, it is not always feasible because you will need to transfer all ongoing work to the new history and your rewritten commits in the new history will probably not build since they depend on those large binary assets.

A

What if we could lighten the weight by keeping the connections to those large blobs, but not downloading those large blobs right away? This is kind of a halfway point between rewriting history and keeping the large files forever.

A

The way to do this in the core git client is partial. Clone by adding the flag dash filter equals blob colon none into your git clone command. You enable partial clone, the remote may not understand partial clone and it might default to a full clone, but the good news is partial. Clone is enabled on every github repository and in github enterprise, server, 2.22 and later so go give it a try on your favorite repositories.

A

Let's recall our object model graph in a full clone. We need every reachable object. That is, if there's an arrow from a needed object to another, then we also need that second object, partial clone says, don't give me any blobs. I don't need so. The initial clone downloads all reachable commits and trees, but skips blobs that are not necessary when checking out the tip commit git makes a second request to find the blobs it needs for that checkout.

A

In the case, where blobs make up a huge portion of the git data, this can reduce your clone size by 10x or more.

A

I would be remiss if I didn't mention an older clone option called shallow clone here. Not all reachable commits are downloaded instead, only the commit at tip and the trees and blobs reachable from the root tree are downloaded. Now this is less data than a partial clone, but there's a catch. You don't have access to the commit history, so you'll have trouble using things like git log or git. Merge.

A

In fact, even commands like get fetch, will become much slower when run from a shallow clone.

A

This leads me to another hot. Take. Shallow clones should be thrown away.

A

I'm not saying that shallow clones don't have a purpose. Only that shallow clone should exist only for ephemeral builds, for example, if you're using a github actions public runner, then you can't keep a repository across multiple builds instead you're going to throw that way that data anyway, so you might as well use a shallow clone.

A

If you do have your own build machines that keep a repository across multiple builds, then partial clone is an excellent option to optimize your build times in terms of the data they download after we focused on reducing the necessary data transferred due to your git history. Let's be mindful of the present.

A

Let's focus on the working directory at your tip, commit before a build your working directory matches exactly what your commit says. It should contain the file contents in your working directory match the blobs it knows about, but after you build, you might have many new files appear that were generated by the build.

A

If these are located inside your working directory, then git needs to inspect them and determine that they are not actually interesting. This is typically done via dot get ignore files, the more files your build generates, the more work git needs to do in order to run git status or get add.

A

Git needs to start walking directories to discover which files are new or modified as it walks it discovers all of these build files and compares them against the docket ignore patterns.

A

This leads to a lot of extra work that git needs to do even to report a clean, git status.

A

This leads to my next hot. Take, get ignore files should be tiny. The get ignore feature uses an extremely flexible pattern. Recognition format, and the issue here is that the paths must be checked against these patterns, one by one. In order, this can lead to quadratic growth. In the pattern recognition algorithm, we recommend that to improve the situation, you should push all of your build time: outputs out of your working directory, my projects, typically clone into a repo src directory.

A

The src directory, contains the dot get directory which indicates it is the root of my git repository.

A

Anything outside this directory is not considered by git to be part of the repo, so they don't need to be part of the getting nor files your build system might have an out directory for files it generates, while a different directory contains packages downloaded from external sources such as npm or nuget.

A

Speaking of builds having developers wait on builds is an age age-old problem. You really want developers to be able to stay focused and not get distracted while waiting for an automated task for really large repositories. It is critical to have a way for a developer working on a small slice of the repository to not need to build the entire cone.

A

One pattern I've seen and recommend is to use a flywheel pattern where multiple modules or services within your monorepo depend on some common code, but don't need each other to build a way to make this possible is to have clear implementation contract boundaries to build each service. We only need to know the contract that the other services provide, not their implementation.

A

This can drastically speed up, builds for individual developers while allowing build machines to handle the integration scenario in ci and pr builds.

A

So if you have such a structure to your build system, then why would a developer need this full working directory?

A

Instead, a developer could focus their get repository on only the files they need in order to do their own work and their own builds in sparse checkout. The get sparse checkout feature allows us to focus the working directory only to the files necessary for each role in your engineering team.

A

Just some quick pointers to using the sparse checkout feature. You can initialize a sparse checkout in an existing repository with git, sparse, checkout, init dash cone. You can set the sparse checkout definition using git, sparse, checkout, set and listing the directories you care about. You can also add directories in batches and you can disable sparse checkout quickly to get back to a full working directory for more information. I recommend my blog post about the sparse checkout feature.

A

Now, let's combine our earlier recommendation of partial clones with a sparse checkout. If we run the clone command here at the bottom, it contains the partial clone flag, the sparse flag and we'll clone into an src directory. The end result is we get. This object database where we have all reachable commits and trees, and the only blobs are those immediately in the root of the working directory by adding a subdirectory. We care about to the sparse checkout definition.

A

We see that git adds those blobs within those directories, but not the directories outside this allows users to have a much smaller footprint when working with their repositories.

A

However, I must now lean upon you. We built the sparse checkout feature to help users who are in this situation of not needing the full working directory if they have it set up to do so. Someone needs to tell git which directories are important. This means that there's a connection missing between your build system and sparse checkout.

A

How can we integrate sparse checkout into existing build systems? I'm calling on you to build this and contribute it to your community.

A

Hopefully, you've learned a trick or two about get today and recognize how problems you are already solving to speed up. Your builds have parallels to speed up your get operations.

A

If you use all of these features, then you are truly at the forefront of git at scale. You are leading the way as you lead the way you will be well poised to take advantage of new git features as they come out, keep up to date with the latest by following the github blog. We regularly update the community on new git features that come out with every release. We do deep dives on important features and have some other important content as well example.

A

I wanted to leave you with a teaser of an upcoming, get feature background maintenance. We are working to deliver the ability to easily schedule maintenance on your git repositories in the background, so you never get blocked on a git gc auto again. This will keep your repositories running smoothly without blocking user facing commands.

A

So look forward to this coming soon and now you might be thinking that was a lot of new information. Well, I personally think everyone who takes the time to become a get expert will reap rewards in their productivity.

A

There's not always enough time in the day to learn all the things you want to learn add on top the burden of trying to keep up the latest, get features and evaluating them for yourself and there's even less time what if there was an easier way well, the good news is that the scalar project might be able to help.

A

Scaler is a thin wrapper around git that helps you set up repositories that have already opted in to the latest and greatest features. You can go to the github page to see the source code and installation instructions for windows, mac, os and now linux.

A

Most users would be interested in using the scalar clone command, which creates your repo in a dedicated src directory. It defaults to partial clones, so you don't have to remember the special syntax.

A

It defaults to a sparse checkout, so you keep your initial download as small as possible and it sets up background maintenance for you. It already has that feature. Finally, it initializes some other recommended advanced git config settings. Some of these are really deep cuts.

A

Further, as the new versions of git come out, we update scalar to take advantage of the latest, get features and update your repositories automatically with that upgrade so go and give it a try today.

A

Thank you so much for your time now get out there and optimize thanks.