GitHub GitHub Universe 2020: Developer - Open Source Communities, 11 Dec 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Project Skara: migrating OpenJDK to Git & GitHub - GitHub Universe 2020

Description

Presented by
Erik Duveblad, Principal Member of Technical Staff, Oracle
Robin Westberg, Consulting Member of Technical Staff, Oracle

Erik and Robin are the two developers who wrote over 50,000 lines of Java to enable the transition of OpenJDK—which is relied upon by millions of developers around the world—to Git and GitHub. This session will cover the challenges they discovered over the two-year migration, as they transitioned an established open source community from using a self-hosted Mercurial server to using Git and GitHub. You'll learn how Erik and Robin converted over 65,000 Mercurial commits to Git, created GitHub apps to customize contribution workflow, and how they scaled pull requests to work well for an open source project with over 250 active contributors.

For more from GitHub Universe 2020, visit https://githubuniverse.com

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

All right, thank you, and we're really excited to be here today and I'm robin.

B

And I'm eric, and I would like to give everyone a really warm welcome to our session today on project scorer migrating open, indicate to git and github, and our goal with today's session is to share with you all how we use github for a large open source project. You will see a little bit on how we actually got to git in the first place, how we ended up on github, but we also want to talk a little bit about the customizations we've done to the default github workflow to suit a very large open source project.

B

But before we begin who are we.

B

We are coming to you live from stockholm, sweden today and I'm eric and I've worked in open source for more than eight years. Usually my day, job is to hack on the yola virtual machine, but lately I've been more focused on git and github and if you swing by my desk, you will often see me living in a terminal.

A

All right and I'm robin and I've also worked with and contributed to open source in various capacities, since at least 2005, and my background is mostly an operating system and platform. Specific development and I've been working on runtime aspects of the java virtual machine for a few years now, but, unlike eric, I willingly admit to preferring a good ide to perform my daily development activities.

A

But enough about us, so let's get started so I'm gonna start with a brief introduction on what the open jdk product actually is. So simply put it's the product that develops the reference. Implementation of the java platform and the java platform includes the java class library, the java compiler and the java virtual machine called hotspot, and while java itself celebrated its 25th birthday earlier this year, it was made open source by sun microsystems back in may of 2007.

A

So that's a little bit over 13 years ago. So this is when the open jdk product was created and the from the open jdk source code, oracle and many other vendors like red hat sap, amazon azul, just to mention. If you go on to create distributions of java in various forms,.

A

And what do I mean with a large open source project? Well, the open jdk contains over 7.5 million lines of code, which form over a hundred thousand distinct comments, and over the years the product has seen over a thousand unique contributors and of those there are over 300 that have remained active over longer periods of time and have attained the roles of reviewers and committers in the products.

A

So I'll talk a little bit about what exactly those rules might mean later.

B

But I think it's.

A

Fair to say that we're not exaggerating when we say that open edk can be considered a fairly large open source.

B

Project we've been on quite a journey for the past two years. Almost uh we started out with moving from the mercurial version control system to git and then later on, once we were on git, we moved to github, and now we want to share the first steps of this dna with you all.

B

So if you are converting something from another version system to the git version chrome system, you are going to find yourself staring at this manual page quite a bit. This is the man page for git, fast import, which is an excellent tool whenever you need to get data into a git repository.

B

As the description says, this program is usually not what an end user runs directly. This is more for people like me and robin who want to transition from an existing version. Control system like mercurial subversion, cvs, whatever you have, and the git fast import, is really the right tool for that work. It's like a swiss army knife for getting data into git.

B

You write an exporter which outputs data on standard out and then you pipe that to get fast, the import which will take care of all the little details of getting your data into a correct git repository million robin wrote, a custom exporter in java.

B

The git fast import protocol is simple and when you're doing a custom exporter, you can take the opportunity to also adjust commit messages or do other kind of modifications to your commits before they end up in the git repo, we took the chance, for example, to adjust the commit messages a bit to better suit the way the git community formats commit messages, and, with this we've been very happy to convert more than 100 000 commits.

A

So after the mercurial to get conversion was taken care of uh it's time to talk a little bit about the transition we did from using a self-hosted mercurial repository to starting to use the github platform and to get a little bit of context on why this matters I want to start with describing the existing open, jdk development practices a bit and before the move to github, and so these practices they have had over 13 years to stabilize and they can be considered quite mature by now and first of all, all discussions over and reviews.

A

So what goes into the open? Jdk is handled using various mailing lists and a quick look at the index reveals that there are over 150 active ones which can certainly be feel a bit intimidating, especially if you are a new contributor and secondly, the open jdk product defines multiple contributor roles.

A

So, after your have a first few small patches accepted, you can attain the author role and that makes it a little bit easier to contribute. Further and after you have contributed a number of reasonably sized changes, you can obtain the committer role and finally, the reviewer role. That requires a good amount of significant changes and also participating in reviewing changes.

A

However, as soon as you obtain the committee role, you also obtained direct push access to the mercurial repositories and this, however, it did not mean that you could just start pushing whatever you felt like now. Every push must be done with the correctly formatted commit message. It's a reference, a proper issue in the issue tracker and at least one capital r reviewer- must have approved it and, as I said, all these reviews are made over email before anything can be pushed.

A

So this is a little bit different from the usual github workflow, where you open up a pull request and eventually a maintainer presses. The green merge button, perhaps after a bit of discussion, discussion in the in comments in the pull request.

A

So one of the main questions for us was: how can we make both this and open jdk contributors as well as new ones, comfortable with working on github and using pull requests?

A

So, in order to support this existing open, jdk development practices, where possible, there were three key areas where we have made customizations to the standard, github workflow and those would be checks, notifications and commands.

A

So we can start with a quick look at how we are using github checks to enforce open jdk, specific requirements, and one goal we had here was to automate as much as possible because there's no need to waste valuable reviewer time on manually. Checking that lines don't end with the trailing white space, for example.

A

So instead we really want to enable reviewers to focus on the important aspects like design and correctness of an implementation, and these checks are performed as soon as a pull request is opened up, because when a pull request is created, you're, usually at a stage where you're looking for human reviewer feedback and are ready to make changes based on that feedback.

A

So it makes a lot of sense to also provide this automated feedback at the same time, and some examples of the checks we have implemented is role-based access controls. So this handles our notion of reviewers and committers and gives them slightly different capabilities.

A

We have our own bug, tracker integration and we make sure that every pull request, references, an open issue and that the issue itself has not been used in a different commit already, and we also look a bit at what's actually in the in the commits like we look for trading white space, incorrect file permissions and we don't want binary files or symbolic links in the repository either, and I guess we can also mention that we have also started using github actions to perform initial building and testing of changes in a pull request.

B

And the second area that we've been focusing on is around notifications, so, like many other open source project, there are many more contributors in open dk than there are experienced reviewers, so reviewer time is scarce and is very valuable.

B

This means that we need to provide the right notifications at the right time for reviewers.

B

Also remember that, as robin said, opiate decay is more than a seven and a half million lines of code, it's impossible for a single person to be able to review a patch to any of the source files, so instead reviewers have different areas of expertise where they can engage with contributors and provide feedback. This means that we need to reach reviewers at the right time and also need to tell reviewers about changes they care about and skip the ones they don't care about.

B

To enable this, we implemented automatic labeling of pull requests with the help of a github application. This means that, for example, let's say you make a pull request and you change one of the make files used for building opendk.

B

Our program will then automatically look at the files in the pull request that you changed realize your only change to make files and then automatically apply the build label.

B

The build label will also mean that an email will be sent to the corresponding, build dev mailing list at opendk, where the experienced reviewers for the build system for opendk hang out experienced reviewers of sorry experience. Contributors can definitely override this automation, but for newcomers. This is great when you're coming into a project as big as open ak, and you don't really know who to ping for looking at your pull request.

B

So again, reviewer time very variable, we don't want to waste that resource uh now, in addition to automatically labeling the pull request and sending it to the correct mailing list. When your viewers hang out, we also delay a little bit when an email is sent. So, as we mentioned, we have several checks that automatically kicks in. When you create a pull request, we will not notify the corresponding mailing lists until those automated checks have passed.

B

This really isn't no need for a viewer to look at a change that hasn't passed. These basic checks like trailing whitebase or having a symbolic link in the pr and again we use these labels uh to map requests to the correct main list, and I want to point out there that, even though many reviewers prefer perhaps the more a little bit old-school way of interacting over main list, there are also reviewers who can choose to use github notifications, natively and those though, and that, of course, is fully supported.

B

They will interact, as many of you do, on github with the default notification system, but for those a bit more perhaps experience the viewers hanging out on the main list more used to that way of working. We also implement that bi-directional thing with regards to the main list, so once the notification hits the mainly if the reviewer has comments, then he or she can post a reply to the main list, and that reply will become a comment on the pull request on github.

B

This way, uh people used to the default github workflow can interact with open education. The regular github workflow and experienced sloppy mdk reviewers, who prefer using main list, can still opt to choose to use main list and they can both communicate with each other.

B

The third and the last area we focused on customization and tailoring to open the case needs is around pull request, commands a pull request. Command, might sound fancy, but it's really just a regular comment on a pull request, starting with a forward slash, you might have seen this from other applications like slack or irc, where, if you start a message with a forward slash, it usually means that you are invoking a command for opening case needs. We have implemented several such commands that both contributors and reviewers can use when working with pull requests.

B

We use these commands to automate the open, ldk workflow, to streamline things and make everyone a bit more efficient.

B

For example, opendk committers do not have direct write access to the option repositories, but with the help of one of our pull request commands, they can easily integrate a reviewed, pull request into the option repository.

B

uh We also have support for automatically formatting a commit message for the result of a pull request, and these commit messages can be adjusted with the help of these pull request, commands and, finally, some pull request needs more attention, so reviewers can then flag them using commands to say that they need more review from either more reviewers or a different kind of review, and on the next slide we will see a couple of examples: for example the main command that almost every opinion k contributor uses is slash, integrate.

B

This will rebase and squash the commits in a public quest on top of the target branch. It will also automatically format the final commit message for the result of the pull request using opennd case conventions and format.

B

This means that the reviewers don't have to spend time merging already reviewed, pull requests. Instead, the water of the pull request can, once all the criterias are met, integrate easily the pull request without resorting to git rebase or anything similar.

B

An example of a command that is for reviewers is the slash csr command, which means that this pull request requires a compatibility and specification review. uh Opendk and the java programming language and platform takes backwards, compatibility extremely seriously, so for many changes to the language and to the core libraries, there needs to be additional reviews from very, very experienced reviewers who take backwards, compatibility into consideration when looking at those pull requests.

B

Finally, to help out newcomers into the community, we have the sponsor command, which existing open, uk commuters and reviewers can use to easily integrate changes from newcomers who are not yet part of our community.

A

All right so, finally, I just wanted to talk a little bit about our product itself, so the project scara is an open, jdk project and it was created to at first investigate and then to actually perform the transition of open jdk to git and github, and the product score itself is fully open source.

A

It is written in java and it obviously makes heavy use of the github rest and graphql apis and also, in addition to implementing the services and tools we have touched upon briefly today. It also provides a suite of cli tools for open jdk developers that prefer to work in the terminal so for open. Jdk contributors like eric there's, actually no need to open up a web browser or any other graphical tool at any stage of the process of getting a change integrated into the open, jdk.

A

And that was pretty much what we wanted to talk about today and if you want to learn more about the openjdk product itself feel free to visit us at openjdk.java.net, and you are also very welcome to follow us at twitter.com, openadk and obviously, all the source for open adk and its various projects, such as project scara, is available at github.com openjdk.

B

So, thank you so much for tuning in today to our session. We are happy to take answers that you can questions that you can tell to our hosts and we will happily answer questions about the githubs, the github rest, apis, graphql, apis, um optimization and github, and also, if you want to transition from mercurial to git. We are happy to answer those questions as well. So thank you so much.