GitHub Universe 2015 - Build, 4 Feb 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Overview of Git LFS - GitHub Universe 2015

Description

Rick Olsen provides a high level overview of Git Large File Storage (LFS), including design goals and methods of implementation.

About GitHub Universe:
Great software is more than code. GitHub Universe serves as a showcase for how people work together to solve the hard problems of developing software.

For more information on GitHub Universe, check the website:
http://githubuniverse.com

A

First, we're gonna go ahead and bring up Rick Olson. He is an engineer at github. He also has a favorite video game of destiny. I'm gonna give you some random facts of information about our speakers today and he actually lives in Boulder Colorado and plays a little bit of bass. So welcome Rick.

B

Hi how's everyone doing I'm, Rick, Olson and I'm gonna be talking about get large file storage I'll, see if I can figure this out.

B

There we go yeah.

A

B

uh As a high-level overview, uh get large file, storage is an extension. To get for working with large files. Get by itself will happily store large files for you, but not very efficiently. What happens as you store versions of files, your repository grows too big in size and becomes unwieldy to use.

B

So the get large file storage extension helps that long. It adds some new sub commands to get, and but it integrates with get through internal hooks and filters so that it doesn't disrupt your your current get workflow. It also has support for git hosts like like github, so the whole idea for git LFS came from this. This prototype called get media that one of the cofounders of github, Scott Chacon, wrote so, basically what it did.

B

It used internal filters called clean and smudge that when you added a big file to your git repository, it moves the file somewhere else and actually puts like a little text file pointer inside your git repository and then, as you check out the big file, it replaces the pointer with actual big file, and does this for you all transparently.

B

But at the time like, we couldn't build the support in to github.com like the companies, and this was like 2009, so the company is way too young and we had to focus on get itself so get media had custom like storage adapters. So you could push the files to s3. That was a big one at the time, but as as we got feedback from people that really wanted to use big files and get we realized that we had to solve this. So so I went back to first principles like how.

B

How do we take this get media prototype and make it into like a github product? So one of the key features is a easy installation get media is written in Ruby and you have to have like the right version of Ruby and the right Ruby gems to pay on what storage adapter you want to use and all that that's way too much to ask of people, especially if you don't use Ruby I mean I use, Ruby and I still wouldn't want to deal with all that. It also required users to configure like which adapter.

B

So you know you would have to clone a repository and then put in like your s3 credentials in the git config. So that's an extra step. I really didn't want also, instead of even having storage, adapters, I thought about just basically building on building an API on top of the get remote URL. So it knows intuitively like where to store and fetch the big files, and this means it can use get authentication.

B

So if you can authenticate into github through you know your username password or SSH, then you can also pull and store your your big files.

B

So this this illustrates what my goal was like.

B

I should be able to push a file to github and it should look like just if I stored it with LFS or or just regular get so so why open source like this is a the some people saw this as some secret sauce and the competitive advantage, but really like Ivy today, one as part of the get eCos system like one of the really cool things about get and github is you can you can work with stuff on github, but you can push to other other get servers like like bitbucket or Visual Studio online, and it would you know it's just the same.

B

The same tools work everywhere, also by being open source, we can work with a very diverse community of developers that have very different needs than I. Do like like Tracy mentioned I'm a big video game nerd and their their development cycles really fascinating to me and I really wanted to work with them. But I, don't I, don't know anything about making video games. I know how to make you know. I'm a back-end developer.

B

I can make servers so, but this way I can release a tool and work with people very, very closely I'm able to work with our competitors, which are gonna, be speaking right after me.

B

Also so something that really surprised me too, is just having the open-source tool and documented API meant people were wanting to use this for new use cases, not even for like storing large files and get necessarily so I was talking. I spoke to a data scientist about using it to store, like big data sets and some massive cluster which again I don't know anything about. So that's really cool also spoke was so on about making a new package manager for for a language that really needs one.

B

So so today we're announcing get LFS 1.0. This has been months of work since we announced the open source project in april. So since then we completely rethought how the API works. It's a lot more efficient now and we can also there. We have ideas on how to extend it and make it. You know better in the future as well. Another big thing that that Steve will be touching on more is a local storage management. That's where the git LFS tool stores the files.

B

Also with 1.00, we have some really good installers so especially like for Windows. The original Windows installer script was not very good, and a lot of people had problems, but now it's a graphical installer and also there are packages for Linux users, there's a rpm and aft packages and we're also playing around this idea of LFS extensions, which is another thing. People want to change how the LFS objects are stored, for instance, they want to compress them before pushing them up to to github.

B

So that's the idea is still in an experimental phase, but really excited about, and that's it. Thank you.