GitLab Create Code Review Group, 28 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: UX Showcase - Measure and improve merge requests perceived performance

Description

Pedro Moreira da Silva, Staff Product Designer in the Create:Code Review area of GitLab, shares the work the team has been doing to measure and improve the perceived performance of merge requests (MRs).

• Measure MRs perceived performance: https://gitlab.com/groups/gitlab-org/-/epics/6141
• FY22-Q2 Perceived performance audit - Create:Code Review - Review changes: https://gitlab.com/gitlab-org/gitlab-design/-/issues/1608
• Collect 3 MR performance inputs in the handbook: https://gitlab.com/gitlab-org/ux-research/-/issues/1473
• Code Review Direction - Secondary metrics of success: https://about.gitlab.com/direction/create/code_review/#secondary-metrics
• In-product survey on MRs performance: https://gitlab.com/gitlab-org/ux-research/-/issues/1475
• July 2021 task completion estimates: https://gitlab.com/gitlab-org/ux-research/-/issues/1474#results
• Other references: https://gitlab.com/groups/gitlab-org/-/epics/6141#relevant-links-script-prototype-notes-etc

A

All right, hello, everyone, my name is pedro murray de silva and I'm a staff product designer in the create code review area of gitlab. um Thank you so much for joining me and today I'm gonna share the work that we've been doing on, improving and measuring the merge requests, perceived performance and I'll try to keep this as brief as possible, but I'm very passionate about this topic and especially when you can link academic research to the work that we're doing here at kit lab and what people use every day. It's fascinating.

A

um I hope you find it fascinating as well. If not, you can watch the recording later if you want, but anyway let me share my screen and I'll. Try also to share some examples. So it's a bit more practical.

A

So this started out four months ago when we were investing heavily in improving the technical performance of merge requests the response times the loading, the memory usage and it was incredibly successful. I don't know if you have uh as users of gitlab if you have recently experienced those improvements, but they were quite massive on the orders of 40 or 60 percent of improvements.

A

But anyway we created this epic to measure the merge request, perceived performance um and the questions that we were trying to answer were: how do users grade the product performance and how does it compare to our competitors and the idea here and the business decisions to be made from this information is not only to set a baseline in benchmark but ultimately to allow us to measure the return on investment of the improvements that affect this perception of performance over time, and this might be a little hard to understand so I'll.

A

Try to break this down and first, let's look at the. What perceived performance is so this is a short and sweet description um or definition from the mosilla, docs and perceived performance is defined as the measure of how fast, responsive and reliable a website feels to its users, and the perception of how well a site is performing can have more impact on the user.

A

Experience that the actual load and response times so think about that a little bit the perception of how well a site is performing can be more impactful than the actual loading response times, and this is incredibly interesting because it meant that maybe we didn't have to invest a lot or everything in the actual load and response times. Maybe there are other things that we can do to affect the user's experience and how they perceive it. There are two basic things related to this that I think, are visible in this example.

A

um So if you look at this animation, you have the cursor here, it's clicking on the start, loading button and when it's clicked it um tries to fetch new content right, and there are two different animations, the spinner on the left side and on the right side. You have this masking of the initial content and then it reveals the new content, and so what research has shown is that you can significantly reduce the feeling of waiting by keeping users attention occupied.

A

So I'm not going to ask you which one you feel is faster than the other, but research has shown that the one on the right feels faster right, even though it's exactly the same objective uh clock time. Okay, so those are now. What I'm going to explain to. You is the two concepts of objective time um and uh psychological time, so we can also call it and drill down a little bit in the psychological time into active phase or active weight and passive phase or passive weight. So imagine you know the same time.

A

10 minutes is what is marked on the clock. It's the amount of time. You have to wait for something if you're sprinting on a bicycle, it will feel like five minutes, but if you're waiting boards with nothing to do you're not entertaining you're, not your attention is not active.

A

It feels like it's 15 minutes, so I imagine that many of you relate with this feeling of time, feeling like it has gone faster or slower than it actually has, and um there are two interesting stories that I'd like to share with you, so one of them is related to elevators.

A

So, in the beginning, the first half of the 20th century, when elevators were introduced into very tall buildings, uh the building managers were getting more and more and more complaints about people that were not comfortable with the amount of time they had to wait for the elevators. They were feeling stressed out bored. It was too much time to wait. So what did they do?

A

They decided to place mirrors in the lobby and inside of the elevators, and uh it basically replaced the passive wait time of those elevator passengers with active wait time, because now people were looking at themselves in the mirror were looking at others, perhaps, and so they were more tolerant of the weight because they were in some way entertained.

A

Right and now something more specific to our domain is the difference between the copy, the file copy action when users upgraded from windows, xp to windows, vista and in vista's file, copy behavior was actually much better than xp. It was an improved algorithm that performed better and in clock time. Objectively, you could copy the same file faster in windows, vista, but the perceived performance of the copy was significantly worse than in windows xp, and that was very strange right.

A

So it was faster in clock time, but people felt it was slower so why there are two uh specific things, so the windows explorer which for people that don't don't know it's the finder in mac.

A

It waited 12 seconds before providing a copy duration estimate, which certainly provides no sense of smooth progress and also the copy dialog is not dismissed until the right behind thread has committed to the data to disk, which means the copy is slowest at the end. So in in contrast, windows xp did not provide a accurate duration estimate, so it was faster to provide a duration estimate, even though it was not accurate and it dismissed the copy dialogue very quickly.

A

Even though the data was not written to this yet and so vista thought they would do better with a more accurate duration estimate, they would do better by only dismissing the dialogue when the data was actually written to disk, but it felt slower to users. So they did a lot of research on progress bars, for example, and they felt they realized that, by changing the animation of progress bars, some users felt that the same duration was 11 faster.

A

So here's a small example of how that looks like so they showed these progress bars with different animations separately to participants and the result was that's a specific, some or a specific variation of these felt much faster than the others by 11, and that is just by changing the animation of it right.

A

So uh why does this matter? And how can we actually make something out of this? So there are two main things that we can do and that are informing how we move um towards improving the perceived performance of merge requests at kit lab. So two basic techniques- one is preemptive start.

A

So we have this original event that lasts this long and we're trying to cut the initial loading phase the start and moving the start closer to the end, so that it feels like the event is much shorter than what it actually is, and a good example of this is in uh safari on ios that when you're typing a url or searching for something in the browser, the first url is already being downloaded in the background and you don't notice it.

A

So when you click on it, it will load much faster because it knows that for 90 or 80, if you want to use the pareto's law uh of clicks are for the top hit and it also uses an animation to keep users entertained. While it's loading. So that's why you have this animation that maybe some people might find a bit annoying, but it feels faster in the end right.

A

The second technique is early completion, which is the opposite. We have the original event timing and we're trying to move the end of the event closer to the beginning. Closer to the start, and a great example that everyone knows is youtube and other streaming platforms immediately, starting playback, even though the we haven't loaded or downloaded the whole video, but it keeps playing and it keeps buffering with the minimally required data for the video two other nice examples. Is this optimistic card ui?

A

Where um is, instead of waiting for everything to load to show the content of the cards? It introduces an animation that you know this slides from the side. The the cards details. You have the skeleton loader you have this loading bar at the top and all while this is happening, we're entertaining the user and it feels like it's loading faster and compared to the competition.

A

This card experience is feels faster. Another common one is clicking on the like or favorite button on this on twitter. um It gives instant feedback. It does not wait for the api to respond back and say. Okay, I have received your request to favorite this tweet. No, it is instant and it's also resilient to network failures. So, even if the I don't know, the user closes their browser or closes their mobile app, it will keep trying um to call the api until it succeeds.

A

Okay, so now back to gitlab. So what did we do with all of these? This interesting knowledge, and when I say we I say I um I went ahead and did first of all a very brief, uh boring and lean performance audit of the perceived performance of specifically when you review changes in merge requests, and I went through the experience in my local gdk trying to keep things as slow as possible so that I could see how things were behaving and started documenting.

A

All of these inconsistently placed spinners that we also were not consistent and we're not giving enough uh feedback entertaining users when we were commenting and a bunch of other things, and these resulted in issues that we can now improve uh to a certain degree in small ways, the perceived performance of these actions, while people are going through this flow and related to the merger request performance. And how can we measure the perceived performance?

A

There are three main things, so machine reported. What is our machines and data that we can elect, can collect about the technical performance users task times and human reported performance, and all of this I imagine that now you're a bit confused, so I'll try to break this down.

A

This is something that we added to the create direction page that describes how perceived performance can be formulated, and you have the the formula here of technically or or theoretically how it can be um measured and we're looking at these aspects separately. So one of them is the expected performance in the ux and we're measuring this primarily by users, feedback and secondary by actual performance of competitors.

A

Sorry, the remote work at its finest anyway so expected performance in ux, we're measuring by users, feedback and also by the actual performance of competitors, and one thing that we're trying to do is measure uh get feedback from users on gitlab.com through an in-product survey on merge requests, um and this is we're following the hats framework from google, which is something that they have been doing extensively on. Google products and it stands for happiness tracking survey, and it looks something like this. I don't know if you've experienced it in google products.

A

You have this pop over. You click to show a question if you want to help improve the product- and you say how much you're satisfied or dissatisfied with the product and, in the end, we're trying to do something like this in the merge request.

A

Ui, although this is a very rough mock-up, it's not necessarily what we're going to do, but give you an idea, maybe showing a banner at the top that takes people to a survey to answer just a couple of questions, and this will allow us to get feedback about the performance in the context that people are using.

A

The product- um and I think this is I've- been working with ana on this, and we think that this can be a more granular way to get uh user satisfaction feedback than, for example, the sus, which you know, spends the whole product and it's difficult to be more granular. This way we're targeting specific features and areas in contacts um going through this very quickly.

A

Another thing that we're looking at is maybe what is the expected performance based on competitors and we have a bunch of internal things, but this is a public one run by sourcehut, one of our competitors actually, and it runs a lot of performance tests on many different kinds of pages for git, related products, sources, github, codeword, pager, gitlab and bitbucket, and this is a way that we can look and see for exactly the same content and repository and merge requests content how we can compare to one another.

A

And uh finally, we have the actual performance that we've been measuring over and over again over the years with some specific technical metrics and the task completion is how often, how quickly can users get through the task and we're doing this.

A

Through estimates, because we found these to be easier to implement and with uh possibly 80 confidence level- and I'm not- I can't show you the estimates themselves because they involve competitors, but I can show you how it looks like or how I have been doing it in practice, using a tool called calculator uh where I describe what people are going to do, and it gives me the task time it also breaks down into small operators what people will do and it's an estimate of in this case, how long people take to merge a merge request so four seconds on average um yeah.

A

There are other things that I can share about uh other laws and and how we can compare these competitors but I'll keep it short for now, so that we have time for questions. Thank you so much for your time. You can find all of the references in the google doc and I'll also attach them to the youtube video. So thank you.