GitHub Git Merge 2022, 18 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monorepo Spring Cleaning [Workshop] - Git Merge 2022

Description

Presented by Preston Martin
In this workshop, participants will work through exercises and get tips and tricks to help learn how to manage their monorepos.

About Git Merge:
Git Merge is dedicated to amplifying new voices in the Git community and showcasing the most thought-provoking projects from developers, maintainers, and teams around the world. Git Merge 2022 took place at Morgan Manufacturing in Chicago, IL on September 14th and 15th.

A

Howdy, my name is Preston Martin I'm, a devops engineer at GitHub and in my side time I'm also a moderator of the devops stack exchange. So if you guys see me around, those are two places to find me so today, I'm going to lead a workshop on monorepo spring cleaning. This is kind of a subject close to my heart. It's something I see in Professional Services. Quite often, people come to us with mono repos. They have questions about it. They want to know how to make them size down, trim them down.

A

So yeah we're just going to kind of walk through some of the basic tools that can help you declutter your mono repo and then we're going to talk about some few examples of those.

A

um So the main goals of this Workshop that we're going to do is we're going to learn how to gather information. So we're going to use a couple open source tools to be able to gather statistics on our mono repos to figure out what's kind of going on and then second we're going to interpret those results. So we're going to take all the results we have from getsizer get filter repo and we're going to learn. How can we can finally remediate upon those which is important because we can gather all the information we want?

A

But if we don't have a plan of action, then doesn't do us much good, um so yeah, some of the reasons you might want to perform a monorepo cleanup with some of these tools for one long git operations, our clones, our fetches- are becoming really extensive. They're slowing down developer lead time remediation time, so we want to downsize our monorepo so that our gibbery boat and operations are more performant. Second, a repository is just too big. It's taking up too much storage size.

A

Again, it's affecting our git operations, it's affecting the height, the server we're hosting it on. So that's not good. We need a. We need to learn how to fix that consolidation and migrations. If we're moving from different Source providers, they may have different requirements for our repos and also we may be taking multiple repos consolidating them together. So we need to be able to downsize too and and clean up. Some of that unwanted clutter and finally, just removing unwanted history. There can be compliance reasons.

A

Maybe maybe somebody's accidentally put an SSH key in that they don't that you don't want in their bad, commit history, bad commit authors, um you name it. So those are kind of the the main reasons. I think that we would we would go towards the cleanup so to get started. If you all can navigate to the GitHub the get merge, Workshop a reposit organization repository, you can kind of see the link there I'm just going to have 10 minutes, so we're going to roughly kind of get all the tools installed and set up.

A

um So the the tools we'll be installing are get sizer, git, filter, repo and then, if you don't already have git installed, you'll want to make sure that's, updated and installed to to the latest version. So I'm, just going to give 10 minutes here and I will jump to the repo link here and then yeah.

A

Then we can kind of get started with the walkthrough and, if I want to add, if anybody has any questions on the setup or installation, if you just raise your hand, we have a couple staff walking by that can kind of help. You get anything installed. You need so.

A

A

A

A

A

A

Okay, if everybody's got those installed, um I think we'll walk through the demo. Now, if, if you didn't get a chance to install those or having issues again, you can raise your hand. The outputs for all these commands will also be in the repo itself. So if you want to follow along and not not have to walk through it yourself, that's an option as well yeah and again, once the tools are installed, we're also going to clone an example of a bad mono repo that we have.

A

That has some some binaries and large files in it. But if you want to use for this Workshop your own repository, your own local clone of a mono repo, that's perfectly acceptable too, so it in fact that's a great opportunity to to learn more about your repositories. So, okay, so in the first example, we're going to cover git sizer, so git Sizer is an open source tool that will help us to learn more about our repository.

A

It's going to give us some some useful statistics about the blob sizes, the number of commits the commit sizes, a ton of helpful information, so I'm going to go over here to another window. So this the workspace is already set up. We have everything installed here.

A

So I'm going to go to our example, mono repo here and I'm, going to navigate to the instructions at the main readme page. If you want to follow along here, it's you could follow along on the git repo itself posted on GitHub or you can follow along in the readme in a local cloned version of it as well.

A

So the first step here uh we're just going to follow this first activity: Gathering overall stats with Git sizer.

A

And we're just going to hop into this repo and just take a look at the contents of it um just to see what we're working with.

A

And I paste there we go there, we go. We got plenty of info there, okay, so we can kind of see here um from from a base perspective of the repository, a fresh clone of it. uh We we don't have very much info to work with. We only have three files. It looks relatively simple, they're, all relatively small in size. You know they're all just a few bytes nothing. You know a few hundred bytes, nothing crazy.

A

Here we have it's an example: python Repository, so there's nothing too astounding going on here and we can even see here in the output matches. You can even see here if we go list that to nothing out of the ordinary, but when we run a tool like get sizer we're going to be getting information and Gathering it on the entire history of our repo, the entire commit history, and if we open the results here.

A

um It's a good Sizer outputs, a table that will tell us things some relevant information here. We can see the total number of commits in our Repository.

A

We can also see things like the number of trees, the number of blobs you can even see here the total size of all the blobs in our git repo and we'll also get this level of concern. So this is kind of what git Sizer determines our high highly probable areas or areas that that you should investigate and kind of a warning in a sense and down here under our blobs, we can kind of see some levels of concern.

A

Here we have some blobs or Max blob size is 50 gigs, which, when we looked at the base of our Repository, uh we could see that it wasn't there weren't any large files present in that low in that fresh clone in that local checkout. So this is indicative, that is in the history of our our Repository.

A

um So we can see that here from this input we also see down below. In the section, uh an annotation that'll actually tell us the particular blobs that it sees, and it says in the history at uh for this git object at this particular Shaw, that we have a backup file, a database backup file that was presumably accidentally committed to the Repository.

A

um So yeah kind of the base relevant info that gets Sizer tells us and we can actually look at that file, so we can further that input. We can take this object, Shaw actually printed here and we get a bunch of binary data. So that's pretty indicative that this is 50 meg back database, backup file, binary that somebody accidentally committed that we need to to enact on. We need to clean up.

A

um Another thing we can do is we can take that particular commit Shaw.

A

So that would be this this one here for that binary and we can see what other files are introduced in there and we can see that our base at our base level from a fresh clone. We still see those hello world files, those python files we had, and now we also see that database backup file. So at this particular point in time um we can kind of see what that repository looked like.

A

A

So yeah that's kind of that's the output that we can gather from git sizer um at a base level. So now we want to go back and we want to see what other information can We Gather about.

A

Just our blobs in general that are stored in our Repository.

A

A

With that, we can use another tool called git, filter, repo, so git filter repo will give us more evidence about the particular blobs in our repository. So things like blob sizing directories. How many blobs are in our directories what the sizes are of those, so we can operate within the same Repository and can see what information is output.

A

um So, working in that same mono, repo directory, we're going to run this git filter repo analyze command, and this is going to generate a handful of blobs or some information on our blobs and the commits and process those.

A

Then we can actually go into the.

A

Filter repo directory, so this will store within your dot, get directory it'll, create a filter, repo directory and then within there there's an analysis directory. So so, if we list the contents of that um we kind of see a handful of files here, they all contain different pieces of information.

A

If we actually I'm just gonna open one file at a time.

A

A

So um yeah the most relevant file to us, for this example, is going to be the blobs Shaws and paths directory or text file. So when we open this up, we're going to see a couple things here, the first thing we notice is that at the very top of our file, this is categorized by overall blob size.

A

So we see here a couple of interesting things. We see this database backup file and it's multiple listed multiple times in our output, which is indicative that it was committed, multiple times, committed upon erased. um Who knows what? But there are multiple commits where it exists.

A

We also see here a couple things with the pack size so this shot. This is the object Shaw of the backup file, but we also see here the unpacked and packed size and we can kind of notice here, there's not a lot of difference here. One of the reasons being is that that's very indicative that this is a binary file.

A

It's not compressible, it's not difficult, it's just it's consuming a lot of storage and we don't have an easy way to compress this, um so even in its raw State, when we compress it it's just it's it's 50 meg file, that's being not compressed at all, and then we see our other files here, all these everything normal that we expect those python files, um so nothing out of the ordinary here.

A

So yeah one of the things we can do is we can actually look at these files. So now that we know that this database backup file is our problem file, we want to kind of Explore More and see you know. Where did this get introduced? What happened? You know we kind of want to build the story so that we can go back to our developers later on and say hey this is this is a bad practice. We need to figure out how to prevent this in the future.

A

um So yeah one things we can do is we can run um this get log command and we can actually follow and Trace back all the way back to the original point of using the diff filter flag of when it was originally added to the Repository, um and we can see Preston Martin, the Troublemaker was the one who committed this source file back in July, but this this would be a way to go back to the developers and say Hey. You know we. We found this file. This is a problem. Clearly in our history.

A

It takes up nearly the entire size of our repo, um and we need to figure out a way to be able to prevent this in the future.

A

We can also go um down below and then we can also see what other files we're introducing that commit. So by running diff tree, we can actually see taking this particular mitcha. That was output here, so you can take the commit from the above command and then down here. We can see that this was the only file introduced in that backup, but this would also be a good command to be able to run to see our developers accidentally committing files. Is this maybe an automated process? That's going through and and pushing the same.

A

You know backup file the same archive file database file. You can kind of narrow that down the scope. So again, these are all useful commands to just gather more information about. Why are there larger repository or large binaries in our Repository.

A

So yeah again, if you're following along in the lesson, all the outputs are displayed in these little drop downs, um so feel free to follow along that way as well.

A

So now we've kind of gone through the the first activity and being able to analyze and figure out. Okay, we've gathered all this information about repository we've gathered information about the blobs. We know that there's some seems like some Rogue commits of people just committing the backups themselves, but now we need to be able to do the second part, which is to remediate upon this.

A

So we want to remove these backup files from our Repository and the tool we'll be using, for that is also get filter repo, while the analyze functionality is one component of it, there's also the main functionality of it, which is to remove historical commits from your Repository.

A

So what we're going to do is we're going to go into one of the copies of our repo.

A

So if you're, if you're following along and you didn't get a chance, I would make a copy of this source because we're going to be removing history and we'll be wanting to use this repository later. So I've gone ahead and done that, but again you'll want to copy the bad monorepo or whatever mono repo you want to use and then creating creating this secondary copy because we are going to be just making local changes to it. So the First Command we're going to run, um and this will actually be be.

A

The cleanup command is we're going to run, give filter repo and we're going to pass in the path to our binary and then we're also. We want to make sure we have the invert pass flag so get filter repo by default will remove the history of everything, but the pass you specify. So it also takes regular Expressions. So you can say I want to remove everything outside of a certain directory, for instance, but in our case we only want to remove one file.

A

So we're going to take this this command here, to remove the single instance of the backup file, but we want to remove it from every commit in history.

A

A

Okay, yeah- and we can see here nothing in the present, but it is still showing the history so we're going to run.

A

The git filter repo to remove the the backup files.

A

Seer ran pretty quick. If you have a sizable repository, a sizable monorepo with a deep history, this command can take very long to run. It can take hours, you might want to run it overnight. It also can take. The analyze command in particular, can take quite a bit of storage. Each of these text files can be over 100 Meg in size in and of themselves, just because there's information on every single object in your Repository.

A

um So you can see here. This ran pretty quick. This is just a sample. Repository and it says, says it ran successfully, but we want to be able to check that because we don't trust, running arbitrary commands from somebody's slide deck. So um let's check that out.

A

So, let's run some of the commands that we ran before on a repo before we started cleaning up so the first one is we're going to run this git log command with the diff filter to see when it was added to find the original commitment was added and we can see we, we get a null output here. There's nothing there, meaning that looking through our history, there was no there's, not an instance of the file being added.

A

um So that's good! That's a good sign! That means right now we don't have any trace of that file being added and then next we're just going to run a account of the number of commits in our repo because we saw before there were 10 commits in our repo. It looked like that backup file had been committed multiple times, so, let's kind of look through and let's see what's going on, so we can see now our accounts down to six.

A

So there have been four commits that were removed, which is good because when we look through that that output- that's kind of what we saw.

A

um So then, finally, we're going to run git Sizer again so git size are running. At this time we should see a completely different output. Obviously our main goal is to not see those backup files within the history.

A

So when you run it again, you get this huge output.

A

And we can see here a media thing we notice commits are down to six the total size of our repo as shot down. We were at 150 Meg before those those multiple commits of the blobs. They were taking up mostly all of our storage space, but now we're down to 190k we'll keep scrolling down, and this is the main data entry point we were looking for is that Max blob size is now 690 bytes. So clearly our rewrite worked.

A

We can also see down here, there's no mention of the largest blob, no link to that object. Shark mitsha. So we've achieved our goals of being able to rewrite that history, so that is a huge win.

A

Yeah- and this just covers that the the other important part to note um so if we look within this git filter repo directory, so we'll go here.

A

um uh There's so there are some important secondary files that whenever git filter repo runs, it also outputs as well. One of those is a commit map, so this is relevant because if you do have any references to Shaws in your com, in your git repo or in any CI CD pipelines, any other third-party tools, whatever this will provide a mapping of any shots that were Rewritten from your old ones to your new ones.

A

So you can see here all these are the ones that changed and then any ones that are annotated with just string of zeros. Those are ones that were completely removed from the history.

A

So again, if you have any references in your pipelines or any other tooling, that mentions specific Shaws. This is a great output, that's that's vital for those systems, and then the other thing to mention is that every time you run git filter repo, it will rewrite all of these files and it will generate new outputs. It will do another rewrite of your history, so all of these Shaws will change again.

A

So if you want to, let's say we wanted to not only rewrite the history of the backup file remove that, but we also wanted to maybe remove another database file. It's best to run all those commands together to concatenate them be into one one: I guess one parameter when you're when you're running the command, because then I only do one rewrite. You only do one rewrite of the history and it'll only output, all the files once so definitely important when you're doing your own implementations of the tool.

A

So now we learned we learned how to gather statistics from our repository using git Sizer at a at a high level, and we've also learned how to gather statistics at a more granular level for each blob with Git filter repo um and we've also learned how to rewrite with Git filter repo. So not only take all of those statistics, We Gather, but actually go through get into the repository and rewrite the history. So our final kind of activity we want to go over is is the scenario where, let's say we can't rewrite our history.

A

This happens a lot where maybe we're so dependent upon those Shaws we're so dependent upon the history. Maybe we have compliance reasons where we can't go back and do a rewrite. It's just not feasible.

A

um There is kind of a third option. That's involves using git, replace git replace, is a built-in function of the git tooling, and we call it repository. Grafting is kind of the term that I've seen float around, but essentially what we'll be be doing is taking the history of our mono, repo and grafting it to a new repository which will contain the the head commit of the historical Repository and then combining them together.

A

So it seems like one seamless repository, but any future development would be done on the new Repository, um so we'll actually go to.

A

This example here, so we have some great diagrams here: that'll that'll, show kind of these pictures, but um to start we're going to have one Repository, that's just going to be our our main mono repo. So we're going to want to make a copy of this.

A

So if we go back to our main location, where our repositories are stored, we're just going to copy the mono repo whenever we're using tools like git, filter, repo and rewriting history, I, definitely recommend just making a ton of local copies just because we're obviously making destructive actions in our repository, so I always want to start with a fresh clone make copies. We can always delete them later, move around, it just makes things go smoother, and then we have a point to restart later.

A

We don't have to do a fresh clone of the monorepo, especially if it takes 15 to 20 minutes like some do, which it might, if you're intending on cleaning it up. So we're just going to make a copy here, we're going to name it something historical, not very creative,.

A

So we're just going to copy all the contents of one of the repository into a new one.

A

And then we're going to actually go into the historical one.

A

We're just going to make sure all our contents are there. It's there do git log yep.

A

We have some output here. This Preston Martin user thought he removed all the binaries, but he was wrong.

A

Yeah, so we can see there, it's a it's a direct mirror of that repository. So now, what we're going to do is going to take the uh the main repository and we're going to delete everything the entire git history, because our goal is to have a state where it is just the files at the head commit of what the historical one was.

A

So if we go down to this diagram here, this kind of shows a better example. Sorry for the bad font size, um it shows up better in the repo or the bad font colors. But we can see here that our goal is to have the main repo.

A

This will be the new one that we're going to have all future development on as just one single commit and then all new commits will be put on top of that, and then the historical repo will have everything from that commit and the entire history, and that will be by historical. We mean it's going to be a read-only or an archived State, it's not going to be manipulated ever again, that's just going to be for historical purposes when necessary.

A

So our next step here we're going to go into.

A

We're going to go into the monorepo Repository.

A

And we're going to remove the entire git directory, um so you can see here not a valid git repository and then we're going to re-initialize it as a git repo.

A

For this reason, this will again generate this kind of single commit going forward. New repository new history.

A

And then next we're going to add all those files we're going to generate a new commit.

A

And then we want to put in right here. We want to put in a message, because this will be when new developers clone this repository they're going to want to have a point of reference say what was the first commit in the repository and then um you know where do I go to if I have, if I want to refer to the rest of the history, so I'm in this commit message.

A

I recommend putting in kind of something along the lines of this is this was the start of where the history was kind of moved over and then this is where you'll go. To and I have the remote destination there so that developers know that when they do the grafting later on, they know where to go to.

A

So right now this is kind of what yeah. This is the state we're in. We have the main mono repo with just the head, commit from the historical one, and then we also have the historical one in its entirety.

A

Excuse me so now, finally um we're going to in in a real world example, you would take the remote of your Source destination if you're hosted on GitHub bitbucket gitlab, and you would add that and then fetch the repository straight straight from the source. But in this case we have a local copy. We're just going to add that as our remote so now I'm going to in the the new mono repo the base model, repo I'm, going to add the historical one as the remote and then I'm going to fetch.

A

All the commits in the history into this as a as a remote destination.

A

As you can see here, we have multiple branches. That's that's now in there as a new remote.

A

Yeah, we can go, run, git, remote, Dash B, and we can see here yet we have the fetch repository as our remote destination or as one of our remote destinations.

A

And then next we should also see here that we don't want at the present time. We don't want any other commits in our repo. We just want to have that one single commit and we can verify that here, initial commit of the monorepo. This is the one we just added in the previous step.

A

um You can find the information, so everything is is looking as it should right now.

A

We can also see here that we're running if we run ahead, so this is for the head commit we're getting the files that we expected without the um without the binaries before and then, if we do, if we fetch the head commit the fetch head as well, then we'll see the same file, so we need those to match up, because when we graph them together, we want to have that similarity in history.

A

So, for the final step, this is the actual grafting phase, so we're going to be taking the head and using git replace to graft those two commits together. So it's one seamless history.

A

So for the Final Phase we're going to run the graft command, we can see that it doesn't take time at all.

A

We again, we have both copies of the repository locally, so it won't take any time to run, um and it's important to note so while we can use the shortcuts for head and fetch head, if you have developers doing this in their own environments, they're going to want to actually reference the particular Shaw so oftentimes, what I'll recommend um to customers to anybody looking for this type of solution is that they have like a startup script like a shell script, A bootstrap script that contains all of this grafting commands within it, including that shot of the initial commit of the main monorepo or the the the one going forward so yeah.

A

So at this time, ideally, this is this: is the state our mono repo is in right? Now we can see here. The black portion is our history, and then now we have the head commit grafted onto the fetch head. So one reason or one way we can kind of.

A

Demonstrate this is this is kind of the output of what it looks like. So we can see here we have the head commit and then we also it also annotates to us, get those it's really clear that we've specifically replaced this commit with the history of the historical repo um but yeah. In from our perspective, it's a it's a pretty seamless history too. We can pull up and show any any history of those commits.

A

um You could do here like a if you run the get log command you did earlier.

A

With that particular Shaw.

A

I'm, just gonna actually do this.

A

We can see here um it's as if the history hadn't changed, but we actually do have two separate repositories.

A

So, along with this, we can also freely commit on top of this, so if I, without without affecting the history of the original historical repo, so for instance, here I can make a new commit and then run git log again we can see here. My new empty commit is now in the history. It does it's not affected by the historical repo. It didn't change anything there, but it only affected the base repository or the the new model, repo, the new source. So.

A

So yeah, um just as kind of a final wrap up I, wanted to cover all the takeaways we can have from this demonstration again if you want to follow along and do this on your own monorepo I highly recommend it it's kind of cool to see some of these statistics and gain insights into how you can clean up your own model, repos your and improve the performance, but the main takeaways I want to get from this.

A

This exercise this Workshop is we learned how to analyze a monorepo with Git Sizer and get filter repo using open source tools. We learned how to safely remove large files from the mono repo, um and we also learned how to in the event that we can't safely remove those files can't remove history. We can't do any rewrites. We learned of a way to preserve history, while still being able to make do commits, make new changes to our repo through get grafting uh with Git, replace.

A

And again, there are some great if you want to learn more about Mono repos want to learn more about these tools. Tomorrow, there's going to be three great talks on monoreepos that I highly recommend going to and they'll be covering some of these tools that we're using and it'll be real life scenarios of how that works. So, thank you again. Thank you. Everyone and yeah. If you have any questions, uh feel free to reach out.