GitHub GitHub Universe 2020: Enterprise - Developer Experience, 11 Dec 2020

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: CodeSnippetSearch - search GitHub repositories using neural networks - GitHub Universe 2020

Description

Presented by
Rok Novosel, Software Engineer, CodeSnippetSearch.net
Hamel Husain, Staff Machine Learning Engineer, GitHub

For more from GitHub Universe 2020, visit https://githubuniverse.com

In this talk, Rok Novosel and Hamel Husain will present CodeSnippetSearch, a web application extension that allows you to search GitHub repositories using natural language queries and code itself. It uses PyTorch to power deep neural networks which embed natural language queries and code snippets into vectors. Using nearest neighbor search, it can determine which code snippets are most similar to a given query or another code snippet. What's more, a tool like CodeSnippetSearch can allow you to easily explore unfamiliar code focusing on the semantics without getting bogged down in the syntax, which is especially useful when onboarding a new developer onto a project.

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

Hi, I'm hamil hussein, I'm a staff machine learning, engineer at github and I'm joined by rock, who is a engineer at source graph and today we're going to be talking about co-snippet search. But first I want to talk about code search net, which is what powers code, snippets search and a number of other projects as a way of introduction.

A

So, first I want to talk about what code search net is so code. Searchnet is a project released by github, a set of data and models that are meant meant for researchers to use for various artificial intelligence, machine learning, research tasks now in popular culture.

A

You might think of artificial intelligence in in these ways, so things that people are familiar with are self-driving cars.

A

Winning games like alphago, um mundane things like text message, completion when you're on your phone and even other things such as winning jeopardy. These are all things that have been popularized in in the media as a lately. uh As far as things that artificial intelligence can do. However, there's another side to artificial intelligence, there's a side of it that can help everyday people, do their jobs and especially skilled labor. So what do I mean by that?

A

So, for example, radiologists? We see artificial intelligence, helping radiologists zone in on potential areas where there might be a disease or a diagnosis that needs to be made oops. I went to the wrong side and you see here that there's a heat map where the artificial intelligence algorithm has highlighted for the radiologists points of interest that they might want to pay more attention to and what this does. It doesn't replace the radiologist it augments the radiologists and it helps make them more effective at what they're doing.

A

Similarly, we see artificial intelligence augmenting writers, so this is a very popular tool called grammarly. You may have used it, and not only does it help you with spell checking, but it helps you with more nuanced type of things like engagement and uh grammar and things that are a little bit harder to do with non-ai approaches.

A

We also see artificial intelligence augmenting lawyers. So there's a lot of tools. There's a growing set of tools out there that oops. This is the wrong slide.

A

um Sorry we have artificial intelligence augmenting lawyers, so uh we see that harvard intelligence is allowing lawyers to go through lots of documents and cut out the mundane work of sifting through lots and lots of data manually and helping lawyers focus on more high value tasks that they are trained for, and we also see that artificial intelligence is helping in basic science, so helping with things like drug discovery, helping researchers, scientists find molecules and compounds that might help in things like having creating therapeutics or medicines for various different diseases and being able to find those a lot faster than ever before.

A

We also see ai augmenting things like artists, so even creative professions are being aided by ai, which is something that folks may have not anticipated ai helps with, but it does, and here there's a technique called uh generative adversarial networks that help uh artists even create interesting pieces that some of which have turned out to be very popular.

A

But this begs the question: what about programmers and that can artificial intelligence aid programmers and how and that's a question that we have been interested in at github as researchers.

A

Now one key facet of artificial intelligence in machine learning is it's very data hungry? You need lots of data. You need lots of examples to show an algorithm or training algorithm on how to understand a domain like programming.

A

So this is why github released code search net code search net is a large corpus of of data and it's essentially code and comment pairs. There's two million code in comment, pairs pulled from various github repositories, cleaned and curated, and um what that what that is, is a treasure trove of data for researchers to then use for various artificial intelligence and machine learning tests.

A

So well, not only is it the data, but there's also some other things. I should talk about in a moment, but this gives you a breakdown. Various languages have been represented in this data, go java, javascript, python, ruby, so on and so forth, and we can see there's a lot of diversity in here for researchers to dive into.

A

So, in addition to the data, we also provided benchmarks and reference models, and one task that we propose is search. So given a natural language description of some kind of code, the task is that we propose is: can you find code that does that and to do that in a very semantic way, so not keyword search. So you can imagine this case. That is illustrated on this slide. Let's say you're trying to find code that pings a rest, api and returns results, but suppose the code doesn't contain any of those keywords at all.

A

The code doesn't contain the words ping or rest or api. So how would you find that code, if you didn't know if you're not familiar with the syntax you're just new to a project? So what this is meant to illustrate is ai can help with tasks like this semantic search of code, and this is one of the references provided by codes code search net.

A

So code search net is a competition where people entered and competed on this task, but it's also a data set and that data set. You can find more information on github at github codesearchnet, where the reference models and the data set are hosted.

A

But I don't want to spend too much time on that, um but I want to I really want to hand it off to rock who is one of our top participants in codesearchnet who, in my opinion, made the most interesting thing out of searching it, and I want to head it over to him. So he can tell you more.

A

um Thank you I'll give it to you.

B

Thank you, hamil um hi, my name is rook, and today I would like to present code snippet search. It is a web application and a web extension that allows you to search github repositories and uh guitar repositories using natural language, queries and code itself. So next slide.

B

Hamill next slide um so code search, for me at least, is one of the most important tools during coding, I noticed that searching and navigating through code heavily outweighs the actual coding. So when I'm already familiar with the code base, I find that exact search is what they need, for example, regular expressions and case and sensitive search. That is because I'm already familiar with the naming scheme, and I can reasonably predict how to formulate a search query and every id and code editor is already equipped with it.

B

But when I'm not familiar with the code base or even a new folder within an existing large project, it makes searching more difficult because it's more trial and error, a tool like code snippet search, would allow me to easily explore unfamiliar code, focusing on the semantics without getting bogged down in the syntax. For example. This is especially useful when onboarding a new developer onto a project, because it can be a significant boost to their productivity outside of a work environment. We encounter unfamiliar code in the form of github repositories.

B

Semantic search tools would provide a faster way for users to find answers to their issues directly in the code. Consequently, it would lessen the burden on maintainers to provide these answers. Next slide, please.

B

To give a short example, let's say I'm a new developer at a company and I get a ticket to implement an api endpoint that fetches products and their shipping information. Additionally, I want to implement as much as possible on my own. Traditionally I would start searching and going through all classes and functions that mention products or shipping, depending on the size of the code base. That may be a lot of code to go through.

B

There really is no good way to automatically narrow down the search results until you get more familiar with the code base and you only get more familiar if you work with it for long enough, so this leads to bootstrap problem that all developers eventually overcome to shorten the bootstrap time. I'd rather enter a few general queries that automatically surface the best possible results. So, in our case here I will enter get all products get shipping info for product and paginate api response.

B

I would expect to get product and shipping info classes, for example, if we're programming in an object-oriented language and some functions dealing with sql and retrieving the entities from the database. I may be a bit stretching with this example, but hopefully you get the overall point next slide, please.

B

So, let's move on to the demo, or rather some screenshots of the demo. First we'll check out the web application located at codesnippetsearch.net. On the first page, we see a list of supported repositories with their descriptions and these supported languages. Now we need to select the repository to search and since django powers the back end of this project. It seems like a good place to start next next slide. Please um so, let's say we wanted to know how to get a database table name in django.

B

Well, we get back a list of code snippets each one containing the link to the github file match. Reading similar code, snippets and syntax highlighted code, everything except the match rating and the similar code. Snippets link should be pretty self-explanatory. We'll go over the similar code snippets later on. The match rating is calculated from the vector distance between the query and the snippet, so higher the match rating, the closer the query and the snippet. This is a little foreshadowing into how code snippet search works under the hood.

B

In this case, we didn't get the exact answer we were searching for, but looking at the first snippet, I think we could make it work with a bit of tweaking next slide. Please to showcase the similar code snippets option. Let's say you wanted to know how to convert a time zone to a local date time by clicking on the similar code, snippets link code, snippet search, will look for other code snippets that best match the selected one, as you can anticipate we'll get a lot of time zone converting functions.

B

This is great for exploring the code base and similar functionality next slide, please, okay! So, let's move on to the web extension web extension can be used directly on the github repository I'll, be using the python repository as an example, because I'm using it to implement the neural networks within code, snippet search, the web extension is implemented in the form of a sidebar that is opened by pressing alt shift and s there. You can search by entering a query or a code. Snippet next slide.

B

So if I was interested in one dimensional convolution in pytorch, I can enter the query in the sidebar. The search results have the same format as an as on the web application, but they are more accessible through the sidebar next slide. Please, a convenient way of searching by code is selecting a code snippet right, clicking and selecting the search by code option with the search by code option you can search using any arbitrary code. Snippet and the most similar snippets from the repository will be returned.

B

This could be repurposed into detecting duplicate code in pull requests as an automated check or even as an offline tool to help us while developing next slide. Please the main data source and inspiration for code snippet search is, of course, the code search net project. It provides various baseline implementations of neural code search and tensorflow.

B

They range from simpler ones like back of words to the ones using state-of-the-art techniques like self-attention. My implementation was inspired by their neural bag, awards baseline implementation because it was performing the best overall. Initially I had written code snippets, searching keras and was able to search through the code search at corpus.

B

The cairo's models became really unwieldy to work with due to the specific structure of the neural networks. They also did not place the play nice when developing the app together with django. So I had to look for alternatives. I decided to switch to pytorch when I wanted to add support for searching github repositories, because the model code became really simplified and there were no issues when developing and deploying the django app next slide. Please so code snippet search works by using joint embeddings of code and queries to implement a neural search system.

B

The training objective is to map code and corresponding queries onto vectors that are similar to each other. There are multiple ways to measure similarity between vectors, but I am using cosine similarity throughout with a trained model. I can embed a natural language query and then yield and then use nearest neighbor search to return a set of closest code snippets during training. I use function, dot strings as approximations of natural language. Queries next slide, please.

B

So here here is just a simple visual representations of these visual of these um embeddings that we're we're trying to learn by training the models next slide. Please, and in this image I have the entire model split up into layers. First is the input layer, one input is four queries and one input each for languages inputs are forwarded into embedding layers with jump out applied, afterward the magic happens in the encoding and pulling layer.

B

Here I take the simplest, but in this case the most effective way of encoding the tokens using a weighted mean the weights I use for weighting are learned in a separate, strainable layer. Since we split up the languages at the input, we have to concatenate them back in the same order as they appear in the queries.

B

Finally, I normalize the rows in both matrices and multiply them to get cosine similarities. The last function can be, alternatively, explained as maximizing the similarities between the corresponding coded query pairs, while minimizing the similarities between non-corresponding pairs.

B

Okay and how is the actual searching performed? I take all of the code snippets. I encode them using the trained model and store them in a nearest neighbor's search index. When I want to look up the nearest code snippets using a query, I first encode the query and then look for the nearest code. Snippet and here we have the training procedure, which is split into two major parts: training the base language models and then training the repository language models, with the help of the base models both contain roughly the same steps.

B

First, the code and query sequences are pre-processed. For example, the tokens are converted to lower case and invalid tokens are filtered out. Then we prepare the vocabularies for code and query tokens and we only keep the most frequent ones with the help of the vocabularies we encode the tokens into integers that can then be used in the neural network. Embeddings. All that is left is to train the model and save it. I train the base language models using the code search net corpus.

B

First, the main goal of the base language model is to train the wording code, token embeddings. That can then be transferred to repository models. I'm hoping that the base language model learns general language features that can then be fine-tuned by the repository data with repositories. I was on my own to extract the corpus, I'm using github's treeseater to find functions and then tokenize them repository models.

A

Have the exact.

B

Same structure, I just initialized all embedding layers with base language embeddings. I also want to mention that there are, of course plenty of warnings attached to the code snippet search model. You need.

A

A reasonably large.

B

Well-Documented repository to train the model, and even then it only works on functions, so the first next step would be to figure out how to work on smaller chunks of code, for example, a couple of lines of code. With a comment above from a model perspective, I would like to experiment with the tree based models that could potentially capture the information hidden in the exchange syntax tree. But for that I would need to gather a lot more samples next slide.

B

Please, and I saved a small anecdote for the end, which highlights why reinventing the wheel is sometimes good and perhaps even necessary. As software engineers, we have thought that we're thought that we should avoid reinventing as much as possible and just use the already proven solution. This is generally good advice that prevents us spending too much time developing unnecessary things, but if you're looking to learn, deepen your knowledge and have fun while doing it, then rewinding the wheel is a great place to start.

B

That was the main source of motivation for me and why I wanted to join the codesearchnet challenge. Of course, I could have taken the already implemented baseline models and tinker with them. My goal was to learn newer networks in depth and actually understand the mechanics behind them, and my re-implementation efforts led me to discover a small bug in the region in the original baseline pipeline that was causing a big problem with the evaluation.

B

As I've mentioned, I re-implemented the code search and baseline models, including the evaluation part. I used scikit's nearest neighbor search class instead of hanoi index, which is a nearest neighbor library used by codeserchnet. I wasn't expecting a good result on the challenge, since I wasn't doing anything special, but it turned out. I was at the top of the leaderboard, almost doubling the previous best evaluation score.

B

This puzzled me and the organizers as well, so I did some digging since I re-implemented the models twice once in keras and once in pi torch, I was fairly confident that my models were close enough to the baseline models. The only thing different in my pipeline was the evaluation I swapped out the sidekick version I had been using for searching with annoy index and my score instantly dropped.

B

It turned out the baseline code, search at the variation code, didn't update the size of the annoy index and the final search index was too small to capture the necessary information that meant it was returning terrible results. So the lesson is always benchmark the index size if you're using annuity index and don't be afraid to reinvent the wheel, you just might find yourself giving a talk at github universe a couple of months later, and that would be all from me. Thank you.