GitLab Create Code Review Group, 5 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Merge request size and performance research synthesis (April 2021)

Description

Pedro Moreira da Silva, Staff Product Designer in the Create:Code Review area of GitLab shares the insights from the merge request size and performance research synthesis done in April 2021.

Issue: https://gitlab.com/gitlab-org/ux-research/-/issues/1374
Slides: https://docs.google.com/presentation/d/13i_aagzPLusuVCx_jOVYPCyT1SSx__0OAt6cSHmWRdQ/edit#slide=id.g29a70c6c35_0_68

“Dealing with large merge requests” was ranked the number one problem that users have today with our code review features, in internal and external surveys. It's comes up frequently in issues, social media, and customer conversations.

Before we dive in to this problem, we need to build a complete picture of what we know today. By synthesizing the existing research and user feedback we will be able to spot high confidence and low confidence areas. Ultimately, this synthesis will allow us to know what to do next.

A

Hi, my name is pedro moreiro de silva and I'm the staff product designer working in the code review area of gitlab, uh and today I'm going to walk you through the research synthesis that we did on the merge request, size and performance, and this was looking at all of the existing research that we've done the past and also the user feedback from multiple sources to really dig into and find out what this means, what the merge request size is what the average size is and what does merge request performance really mean to users.

A

So, let's first cover what we would go through today, so first we'll have the key takeaways. Then a quick background on what this uh entails. uh What product versus user performance is? What's a large, mr, if we were able to define what that means, the pain points that are currently uh recurrently uh happen with large mrs tactics and workarounds that users employ and finally recommendations and open questions that we weren't able to answer with this synthesis.

A

So the key takeaways are five first dealing with large, mrs is reported as the most severe problem that merge requests have on people's efficiency, and this is not only was not only covered through the existing research, but also in two recent surveys that we did.

A

One internal and another external survey dealing with large merge requests was the number one problem um we will go and try to understand what this means. Next, we also discovered that product and user performance are two aspects that are related, but in the end they are separate and we can divide, have a dividing line between them and from these two product performance. Just pure response times is the most visible and impactful issue.

A

We also understood that merge request size definitions depend on the team's coding practice, so if a team usually has small merge requests, their perception of what is a large word. Merge request will be different from a team that is used to code and create merge requests that are very big in nature and finally, we've discovered that users often employ tactics and different workarounds to try to mitigate this problem of large merge requests, so not only the product performance aspect, but also the user performance, so they can work faster.

A

So, as I said, dealing with large mrs was ranked the number one problem in recent internal and external surveys, and it's something that has come up often in issues: social media and customer conversations. It's one of the main themes in user feedback anytime that someone talks about gitlab, um okay. So what? What do I mean by product versus user performance?

A

So, looking at all of the research, we discovered that there's an aspect just about pure performance, um that is the product performance and that is can be explained by just slow response times, and then we also have the other half I like to think of this as a coin, that has two sides, so one is product performance and on the other side of the coin, is user performance, and that is just how quickly can users, with their actions, just go and achieve their goal?

A

And so, of course, product performance is a big part of user performance. So if the product is slow by nature, it would also affect the user's performance and the the way that they can achieve their goals, how fast they can achieve their goals, but they're not just exclusively impacted by this. There are other aspects that affect users performance and we will go and look at some of the top issues that users have um and also what. What is a large merge request.

A

Sorry, when we're talking about this, how how clearly can users uh define what large means to them, and it really varies? uh What we've found is that, like a good rule of thumb, uh way to to look at this, is that this definition of merge request size depends on the team's codic practice. That's the most helpful way to look at it um on the right side of this slide. Here. You will see that the way that internal saver, server respondents and external survey respondents perceive large, mrs, is different as well.

A

For some of them, the large merge request threshold is over 100 changed lines for others. It's over 1 000 change lines, so it really varies a lot and but overall, what users tend to perceive as large has to do with the number of changed files and the number of changed lines. uh That's the uh where most of the consensus lies, so pain, points with large merge requests, um product performance issues in general and again this is just product, pure product performance and response times, and it was mentioned multiple times in multiple sources.

A

As we can see here, the sus surveys, the nps survey in hacker news by customers and in eternal external surveys here- are some quotes here that I'll get back to um in in a bit and then just about user performance. What are the pain points and things that affect how quickly users can get their job done?

A

um One of them is finding changes that affect multiple files, and so for that what users tend to do is use the browser, find function to find specific words or specific methods or functions or whatever it is they're looking for across files, and that tends to work a little bit well and then the other thing that they try to do is resort to their local ide when they find that the merge request is very large and that it's affecting them.

A

So that's a pain point and they end up not using the gitlab ui to review uh those large merge requests.

A

The other problem that they have today is that sometimes files are hidden and they are hidden because we have from their perception the limits that we impose on diffs on the changed files are too aggressive and they immediately collapse and hide those files so that then users can expand them.

A

One thing that we have done recently was change how these files were collapsed. So this was the before view. As you can see here, we would only show this small white box saying this. Diff is collapsed, click to expand it and because it is critical that users review all of the files before accepting a merge request.

A

We've now changed it to have this orange background that you can see here in this video that clearly highlights that these files need to be expanded and that users cannot miss them. But still the problem exists that we have very aggressive.

A

Diff limits- and this is all related to the product performance in the end. So if we can't have really good product performance, we need to impose certain limits and that would make the files or some of them the larger ones, be collapsed by default.

A

Another thing related to large merge requests is that often the activity feed ends up having many comments, many threads many entries, and that makes it very difficult to parse and follow large merge requests. One aspect in specific is the difference between what threats and comments, and let me quickly show you what that means. So here we have a merge request that I have open and it has many entries here in the activity feed uh saying that andrea changed the milestone added these labels etc.

A

We then have this comment and then what happens that if we have a series of threads um in in succession, it's very difficult to understand where a comment ends. Where does the other comments start? Where does one thread end? Where does the other thread start, and that is specially visible here when I have, for example, these threads open?

A

It's it's difficult to make the distinction between or when you're scanning the the activity feed to make a distinction between this thread and this thread and the comments. And then this is the final comment, and then this is another thread. So when you're looking at all of this, it all has the same weight and visual appearance.

A

So you it's difficult to understand where one starts and where the other ends. So this is another aspect that users are affected by with large merge requests and that affects their performance, um trying to achieve their goals, so tactics and workarounds that they employ so one of them that we have shipped recently is the single file mode and, as of february of 2021, 44 of our sas users have viewed merge requests using this mode, so this mode is changing how files are displayed in the changes tab here by default.

A

All of the files are stacked, one on top of the other and you scroll through them. You can see the changes here so here I'm in the diffs scss file, and this one is the notes, css and so on, and you can see all of them by scrolling, but we've allowed users to change their user preference here to only show one file at a time. So when I enable this, I now only see one file and I can then proceed and change between files.

A

One by one- and this helps alleviate some of the performance, sorry, product performance issues and also user performance issues, and this is something that users have been employing.

A

Another one which I mentioned before is reviewing in local ide and then also the browser find function as a way to find a specific line of codes that you want so recommendations. The number one as we've clearly highlighted, is that we need to prioritize improving the merge requests product performance. Just pure loading changes loading comments, navigating the ui.

A

How fast does the ui respond to the user's intentions, the other one, which is related to that workaround about users, viewing things in their local ide or trying to find certain words or functions or methods in their in their code is trying to extend our existing code intelligence feature that we have for the repository view when you're viewing one file uh in the repository try to extend that code. Intelligence feature to also support, merge requests and this code intelligence feature allows us to quickly jump from one reference of code to another.

A

Let me see if I can quickly show you how that looks. So this is how it looks like today. You see the definitions of functions. You can see the references and jump between the definitions and references very quickly and if we could extend this to also exist in the merge requests tab, it would alleviate some of the pain points that users have and avoid having them. Leave the gitlab user interface to go to their local ide um and third.

A

Another thing that we can start doing today is exploring ways of viewing changes that can scale for many files and lines so trying to inspire ourselves by how the local ids behave and look like to have a similar experience or even considering an automatic single file mode, so instead of users having to choose between showing one file at a time or showing all files stacked on top of one another. What if we could do this automatically for them, depending on the number of files, or even the number of changed lines in a merge request?

A

So that's an idea, not something that we're set on doing, but it's a possible thing and in terms of open questions for future research. So what are the answers that we could not get from this synthesis? One of them is how do users grade the product performance?

A

How does it compare to our competitors, so we have been able to um start comparing quantitatively how, on the merge request page how fast it loads compared to our competitors, but we don't have that same grading that same comparison for more subjective perception of users, and this is, if you look back at one of the quotes that we have here for product performance issues in general, this user says the merge request. Page is extremely slow to scroll through on large, merge requests.

A

The experience is much worse in comparison with other hosted git services, so that is one user's perception. But how clearly can we define this user perception and if we could have, for example, users grade our performance versus competitors?

A

If we had that baseline, maybe it would allow us to then start making changes and see how that perception evolves over time over time, so not just the product performance, because if we make changes there, there might be some lagging until users perceive that increased performance.

A

The other open question is about the single file mode. So as we I shared before we today, we only have four percent of our sas users viewing the single file mode. um So there are two things that we could do and try to answer. So is it first of all: is it a viable solution to improve the user's performance and also the perception of product performance?

A

Do users enjoy that single file mode, and if it's really a viable solution, how can we introduce more users to that feature? uh How can we better make that introduction better so that more users experience it and finally, is it reasonable for us to consider an automatic switch between file modes? So, as I showed you and tried to exemplify before so these are the recommendations and open questions that came out of this research synthesis.

A

uh Thank you so much for your time feel free to leave any comments or questions in the issue that I'll link from the video description below. Thank you so much and have a good day.