GitLab Source Code Group, 1 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Investigation into reducing repo size with Git LFS objects

Description

I got stumped by how to use `filter-repo`, but I worked it out afterwards using the `--blob-callback` option. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35707

A

Hi I'm James Ramsay group product manager, collab mrs., to follow up to my video about reducing repository file size in this video I'm, going to take a look at LFS and how to clean out large LFS files that you don't want any more than aren't referenced by the repository and look at some of the challenges associated with that. So I've got an LMS file. I'm headed to this repo and I'll. Just push that up into the server.

B

A

Hopefully, we'll see that turn that, on the file utilization.

B

B

A

Repository it does, there is occasion layer, so file. Size updates, aren't immediate and so bear that in mind.

A

Alright, we can see it being uploaded. I.

A

Think is forty megabytes, so just keep.

B

A

Alright, so once we've uploaded this, what we're going to do is we're going to follow the standard repository size reduction, steps.

A

And we'll just follow this first step. That should probably be enough because I'm not going to put it in a birch request. Reading up anything else, so there shouldn't be any keep around ref summer dress that reference this file, so that means I won't have to do step two. If you've watched the previous video you'll know what I'm talking about that I'm just trying to save myself a little bit of time.

A

Okay, here we are, we finished pushing that up to.

B

The repository, let's take a look.

A

Refresh this great, we can see. There's now, 40 megabytes used that's coming from this large video file.

A

Forty point six megabytes we'll see so that all makes sense. Whoo you even see that it's in LFS, so let's work on removing it following the instructions we're going to clone this repository, so I'll grab the clone.

B

Url I'll come out here.

B

And we'll put this into repo cleanup LFS.

B

A

So now we'll use get filter repo to reduce the size of it, I'm going to remove all files larger than a megabyte, and then what we need to do is then push those changes back to the server and we'll check that we've got a remote to push to. Yes,.

B

I've got origin, so what we'll do is force push.

A

B

To the origin, remote.

A

Could just copy and paste of that, and she also.

B

Let me see if I need to do anything tricky for LFS, because this probably.

B

mmm Because this is a bear repository, this won't work. Alex.

A

Well, that's good to note for the documentation.

B

hmm So maybe cloning with the bear is not the best idea.

B

Let's see what happens if we don't clone bear, and then we can see how this guy's all right get cloned.

B

Aside copied something else.

B

B

Hopefully, this is downloading those LFS files for us.

B

Let's have a look. What we ended up with no no wrong directory I.

B

A

James mistake: oh yes, so the LFS file has been downloaded. We can see our beautiful random noise.

A

So now, let's run filter repo, see what happens. Okay,.

B

A

Alright, so we can see, remotes have been removed. That seems to be the difference between cloning, with a bare repository and not cloning, so probably need to update the documentation so we'll get.

B

Remote add a version and then we'll get push force all to Odin same as the documentation.

B

Everything up to date, interesting.

B

B

A

Guess this isn't straightforward, but to be honest, if you were using LFS for large files and you're probably intending to keep those. So it's probably why there's no easy way to do this, so what we'll do go back to.

B

We can keep working on this one which we haven't changed and.

A

What I can do is.

B

A

Then I'm gonna amend that to the previous commit.

B

And then I'm gonna do a force push so.

A

This is a trivial example, because it was easy for me to clean it out.

B

Seems to have this.

A

Is a trivial example of removing it.

A

But the interesting thing here is that it probably hasn't been removed by default, because there's a special break task for cleaning out that unreferenced LFS file. So there should be nothing that references LFS anymore. In the repository at all.

A

So the next question would be: how do we? How do we clean that up? If we look in so this is on the server? If we look at the repository size, you can see that the actual repo size is still very small, because there's no binary files because they were in LFS. So we want to work out how we get rid of those LFS files, and since here that we can run this task.

B

We've unreferenced LFS files from system I, don't know what the difference between these two are.

B

I think this might be for.

B

Cleanup often LFS file references, cleanup, often LFS flats I, don't know what the differences are. Only a music reference.

B

A

So this actually removes the reference from the database.

B

Even though it says file system.

A

And then this removes it from the file system sounds like some doc updates needed. Here we go. Let's run.

B

Punching paths- okay project path- is root, 3 Pro, cleaner, but to avoid some copypasta do that.

A

Okay, so I think this goes through checks if the every LFS object is actually connected to the repository.

A

So it's found one.

B

A

It turns to actually run it again with dry run, equals false.

B

Okay and then we'll actually want to prune those from the file system be interesting to see if pride.

A

So ones been removed will be interesting to see if that impacts the actual repository size, even though it hasn't been.

A

B

What does this one say? This is a manual task.

A

It doesn't get run automatically, however, this one does get run on a daily basis, automatically interesting.

B

Let me try modifying readme, because editing.

A

The file should trigger a new, updated file statistics.

B

Rocket the Poggi.

B

Okay, so we'll just give this a few minutes to update and I'm going to pause the recording make.

A

A cup of tea come back see if the Katia's expire.

B

A

And back with my cup of tea and you'll see that the file sizes are now gone down, so it looks like this. This tasks up here remove unrest for in still FS files, orphan LFS file, references that has an impact on repo size calculation and then in theory, the file will actually be removed from the hard drive or object storage at some time in the future, with some scheduled job, or it can be run immediately so that cleanup task will happen.

A

That's good looks like the cache expires after like a couple of minutes, but that's all it took for us to reduce the file size. It is to be probably this is that only administrators, direct access to the server can run this tasks. So there is an issue to add this to the repository.

A

Cleanup job probably also makes sense to run this as part of housekeeping generally, so we'll be looking into that and I think we'll also look at the case whether we can speed up so when you run garbage collection housekeeping. It updates straightaway so looks like reducing the file size of LFS objects is possible with administrator intervention, but I think it's an overcome of all obstacle. So there you go have a great afternoon bye.