GitLab Source Code Group, 1 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Investigation into reducing repo size

Description

https://gitlab.com/gitlab-org/gitlab/-/issues/220104

A

Hi I'm James Ramsay group product manager here at gitlab for the crack stage it develops and I wanted to do a quick demo of how to remove large files from gitlab some of the little trick, tips and tricks and and.

B

Also talk about some of the improvements that we probably need to make to make this simpler and more straightforward. So.

A

I've got a little test, repo setup on a copy of gitlab and.

B

So that I can log in and poke it and do some stuff as you'll see later on in the demo. Hopefully we can expose these improvements directly through the interface in a later iteration, but yeah. Let's start by creating some large files.

B

I've got a little command here that I can run to generate some video files.

A

Just of some random noise, very unexcited that they're pretty quick to generate and they're reasonably big to make it a realistic demo.

A

3, so that's probably enough.

A

Yeah they're very boring, yeah, just absolute noise. There you go, but that's a cool party generating a video on demand.

B

Okay, so let's create a branch and also create a second branch with a merge request and then we'll see see how there's slight differences if you're using merge, requests and will have other kinds of keep around rifts that might exist.

A

So create a branch.

A

A

A

A

This is the second branch that will use all the image requests to compare.

A

Also create a text file.

A

B

That's going to take a little while to upload, because it's a large file and.

A

Then I better push branch, one patch one as well, because I didn't push that.

A

B

So we can see patch two: let's create a merge request.

A

And we'll merge that back into master.

A

Looks good to me.

B

And also we can see that the file size has been updated now it does see that the file size updates are not particularly immediate, so just bear that in mind that sometimes they don't propagate straight through. We took a take a look at patch. One branch we'll see, there's a different large file.

B

Okay, so we've got 80 megabytes of video files in.

A

B

Repository we want to remove them. How do we do that? Get lab has some documentation on how to do that. There's two sort of approaches and you should always follow them in order addressing the first approach first, which cleans up I. Guess there, your public branches, like your main branch or any of your feature branches, removes any large files from those.

B

That's really important that you do that first, so everyone's working on a clean copy, it's going to cause some disruption when you start changing those branches and rewriting history, but it's much easier to get that updated and then clean up all the other history.

A

That might be lying around other places.

B

We'll get to that in a second but super important that you start at this step. Do this one first before moving on to the deeper cleaning if it's needed, so the best way to do this is to clone a fresh copy of your repository.

B

A

B

Never know if it's going to include that HTTP or not and.

A

I never get it right. Okay,.

B

So once we've cloned it, what we're going to do is we're going to use git, filter, repo, gif filter repo has lots.

A

Of different tools for reducing the repo size, I'm going to use the strip.

B

Blobs command to clean things out and you can use whatever commands that you think are appropriate to meet your needs because you may have different.

A

Requirements around which files you want to remove, which files you don't want.

B

To remove so please refer to the documentation so get repo cleaner.

A

Okay, so let's take a look.

B

All right we've got the main branch. I've got the patch branch patch two is deleted when we merged it and now we'll run filter repo I'm going.

A

To delete all files larger than a megabyte.

B

And now the next stage is to push all those branches back that we've just modified so that there's now clean copies of those branches on the server, so that everyone's looking at cleaned.

A

Up versions that don't have those large files so.

B

We need to check if there's a remote there is so we can just push.

A

Straight away and yeah so I think I can possibly remove this command, which just seems wrong and get push origin force. All.

A

All right, great, that's all done.

B

If you have tags, it's important to also clean up your tags, because they will also be connected to those large.

A

Files but there's nothing for me to change.

B

A

B

To come back and take a look at this, we should see the assets directory disappear because we got rid of all.

A

The large files that were in the assets directory and similarly in the patch one branch we've removed that.

A

One of the things to point out here is that.

B

The repository size hasn't decreased immediately and that's because git objects aren't cleaned up automatically. So when you push to get and add things to get, we don't clean up on every push and we only do optimization and cleanup tasks on a semi-regular basis after a certain level of activity.

B

What we can do is we can manually trigger garbage collection by running this housekeeping button, but it's worth pointing out that the get GC command only prunes objects older than two weeks and so I've added all these files in the last five minutes, or so so, none of these are going to be garbage collected.

B

One improvement, we're working on is making it so that garbage collection is more aggressive when you request manual garbage collection, so you get more immediate feedback, and so what I'll do is I will run that on the server to show you the impact of doing so so I believe here, I'm a cessation to the server, and we can see now that this folder is 82 megabytes and what I'm going to do is change into that directory.

A

And get GC burn equals now.

B

So that should have had a more significant impact. So if we go up and run that command again see that it's now dropped to ha 41 megabytes, and that's because we were only one of those videos has been pruned. Even though it looks like the prune, and let me show you why, with.

A

Remember we had a merge request. Let's look at that. Merge request.

B

In the changes tab, there's.

A

The video I can still download it that.

B

Means the video is still stored in the git repository.

B

This is actually a feature of gitlab. We create.

A

Really stupid do.

B

We create keep around refs and merge request refs to prevent your commits from being deleted that are associated with merge, requests are associated with comments, and this is so that if you come back to look at an old request and you're like how did this thing get into my repository, you can find out. However,.

A

If you're trying to reduce the size of your repository or remove some kind.

B

Of sensitive information, this makes it more difficult. So what can we do about that?

B

That's what this second step is all about, so let's go and follow these instructions. This.

A

Explains a little bit about what I was talking about with the emergent requests reps. If you want to read that we're.

B

Going to export the project and use that to get a look at everything inside of it so go to general and then under the Advanced Options we'll go to project export and that's going.

A

To generate a complete copy of everything, that's on the server we.

B

Can use that to clean out the stuff that we want the server to delete and then use this special feature.

A

We have and to tell get to your lab, to delete it. Okay,.

A

Alright, here's the clicker explore.

B

And OC project up bundle, that's the output of the get bundle command and what we can do is.

A

My temp directory what.

B

We can do as the documents tell us is. We can do a clone from that bundle. File.

A

And we use the mirror argument.

A

I've been doing this too many times today, 321.

A

Ok, project bundle now. So what do we call? This repo clean up.

B

Alright, so last time I ran for each ref, and you can see here there's now a lockable refs keep around refs and not advertised by the server. So you can't normally see these. These are whatever part of what I was mentioning about. How git.

A

Lab actively creates these to prevent data being deleted.

B

As a feature, similarly, the merge request: refs you can access those if you want, on a read-only basis, to check things out and your CI I might use them or you might use them for a bunch of automation, because they're really handy to have those except they make it harder to delete.

A

Data so we're going to run the same command that we did before for filter repo going.

B

To delete rewrite the repo and now, instead of pushing these changes up, what we're going to do is use a file that was created by filter repo. That's because we won't accept a push to this location. Gitlab won't allow you to overwrite this keep around ref all these modes request refs, it's just not permitted, but what we do provide is this repository cleanup.

A

Feature which is very similar so.

B

I'm going to open this directory, so this was this is the repository that we just ran. The cleanup on and you'll notice that there's a filter, repo folder.

A

And this is the output of the filter, repo.

B

Command and the commit map is very useful in that it tells us what commits used to exist and what commits now exists to replace. It's like old, commits new commits, whether it got rewritten or just removed, and we can use the output of that to.

A

Clean up your repo on the server.

B

So we'll come down here into the report into the project settings click repository.

A

Go to the repository cleanup feature upload.

A

This file, repo cleanup bundle, ok, filter, repo folder I'll play the commute. Matt stop the clay.

B

You'll receive an email, I work because I didn't configure my server because I was lazy. But if we go back here because I'm an association to the server, we can be lazy and.

A

See when it's finished running.

B

Actually, it's not going to change and here's. Why so I'll prove to you that it's run by going back into that repo.

A

B

Alright, so I get for.

A

B

See lots of briefs have been deleted.

A

What hasn't happened is the garbage collection, garbage.

B

Collection still runs with the prune policy of two weeks, so if we run git GC Kern equals now, we should see this has reduced in size significantly to what we expected to me so filter repo working as expected, it was cleaning up the reps. The only thing that we've been having trouble with is the fact that crooning is not immediate, and so, if there's things that reference, that that mean that it's not a very old object, older.

A

Than two weeks, it just won't be pruned. You'll have to wait two weeks and then garbage collection or permanent working.

B

To change that, the other thing is that these things are not updated. Super immediately.

A

That's because I didn't configure my server properly.

A

B

We go so let me click Edit and maybe this will trigger a file size up, taste.

B

Well, forty megabytes, okay, so it just seems like this I think this is probably cached, so it takes a little while to propagate so I think we need to investigate caching, because that could be causing confusion normally. This is fine, because your repository won't change in size very much with each push, and so that is something that we do need to investigate. So.

A

That's a quick demo of how to reduce repository size, a couple of the challenges that make it.

B

Kind of a tricky problem, but once this file size updates consistently consistently and.

A

B

Options are changed for when you're running these cleanup tasks. It should make cleaning up your repository much easier. Hopefully this was helpful in.

A

Explaining some of the things that are going on catch you later bye.