GitLab Create Code Review Group, 14 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Merge Request Intercept Survey Results DEC2022

Description

Ben Leduc-Mills, Sr. UX Researcher for Create, reads through the initial results from the merge request intercept survey.

A

Hi, this is Ben Leduc, Mills uh senior researcher for create and I'm going to go over. The first results from the merch request, intercept survey that we've been running um first, a little background, so the code review team had wanted to understand sort of over time. The changes in satisfaction level, both overall and with specific regard to Performance of merge, requests uh over time and broken down by um subscription level for your paid and um experience.

A

uh So like account age, and this will give us um a nice way to track uh sentiment, changes over time and uncover any sort of differences among these different user populations as well as kind of understanding. If changes that we make um to our merge requests, experience are affecting sentiment in any way.

A

um This survey is ongoing, so this first report will be looking at data from um September through December uh of 2022 and how we did this is we implemented an intercept survey essentially like a pop-up that would appear on a merge request page after the user had been on it for 10 seconds.

A

um This would only happen once every 90 days to any particular user and you can look more into the ads framework from Google, but essentially um it's just two questions that are sort of a likert scale um from delighted, meaning like very satisfied through unhappy, meaning, very dissatisfied um and, as we mentioned, segmented, the responses by um uh activity level subscription and account page.

A

So, the most recent data set, we observed several notable changes, there's a statistically significant increase in the delighted, so the most satisfied responses from both overall and performance. Satisfaction across all of the demographics that we looked at.

A

There is a statistically significant drop in happy responses um both from overall and performance satisfaction from free users, so free users um sort of went in two directions, um mostly downward, but a little bit away from Happy in the delighted Direction, um so directional, meaning not statistically significant changes, have indicated greater satisfaction from paid and experienced users and less satisfaction from free and new users.

A

So the methodology briefly, um the survey structure, basically was two questions. Like I mentioned um the scale from delighted down through happy, Mutual, sad and unhappy.

A

um So we score unhappy as a one delighted as a five to get averages and there are two questions. The first was overall satisfaction um phrased as, overall, how satisfied are you with Mark requests? The second was about satisfaction with merge, request performance and the pop-up uh appeared, as you can see there on the right.

A

We kept track of uh Four Points in the response funnel so render meaning it was. The survey was rendered visible to the user. This missile, meaning the user, took action to dismiss the survey whether they went and answered the first question about overall satisfaction on whether they completed uh the question on performance satisfaction.

A

um We had about a five percent response rate that answered the first question on overall performance um and about half of that five percent are like two and a half percent overall answers the performance question as well, and about 60 of the renders resulted in dismissal.

A

um We don't know exactly what happened to the other 35 percent, but they probably left the page or closed the browser window. Something like that for this report. uh We're comparing two sets of data, so the most recent 30 days of data, since we pulled this so November 2nd through December second and data collected before that from September 5th through November 1st.

A

um The reason for this unequal breakdown I can go into it or you can read the slide, but essentially it's much easier to pull the last 30 days of data and segmented that against prior um and so that's what we did.

A

So as far as the total population, um you can see there's a significant change statistically um in delighted users. Oh, you know, yep skip the slide, uh so overall satisfaction moving higher, um that's a 5.3 percent increase which is statistically significant, so you can see they're moving um away from being simply happy or sad and moving towards delighted over the past 30 days or so.

A

The same holds true for performance. Satisfaction overall, um great news, an eight percent change to statistically significant in this case, and you can see the large decrease from Happy meaning.

A

uh You know people were happy before I know, they're delighted, so that's great looking at paid subscriptions versus free, so overall um there's a uh positive change, though not statistically significant um in paid users and a slight decrease in overall satisfaction from free users.

A

uh So yeah, like I, mentioned overall satisfaction from Paid users is going up, never increases in happy and delighted uh ratings and decreases from neutral, sad and happy for an average gain of about 0.14.

A

um This is a great Trend, but, like I mentioned not statistically significant. So we'll have to monitor this to see if it continues.

A

um Free users are split. So there is a positive change in delighted uh scores, but a statistically significant drop in happy scores, and so you can see that there's there's more delighted scores, but there's also a greater increase in neutral and unhappy scores. So people are moving in both directions away from writing happy and either move moving towards delighted or moving down.

A

And as far as performance, satisfaction for paid versus free paid users show a positive change over the last 30 days or so, um while free users showed a slightly negative change. Neither of these are statistically significant, but will be interesting to track over time uh again. Paid users are uh showed a very positive change in performance satisfaction. However, the end was not great enough to achieve statistical significance, so this will be something to keep an eye on um otherwise uh yeah other other things are also interesting.

A

um Free users are polarized on performance, so again free users less happy with performance um that it was a statistically significant drop um away from happy again. You can see many of them move towards delighted, but also moved down to neutral or even unhappy.

A

As for experienced versus new users, um experienced users showed a positive change new users slightly negative. Neither of these were a significant statistically, however, experienced users did have a statistically significant change in delighted, so at the top of the satisfaction scale experienced users are becoming happier or more satisfied, which is great.

A

um As for new users. Again, we see this polarization of moving away from happy and towards delighted. Some lights but um overall, that the net change is negative.

A

um For performance satisfaction, um a big sort of flip here, um it used to be the case that new users were more satisfied with performance over experience and that has actually changed. We're now experienced users are more satisfied than new users. With regards to experience, however, this is not um a statistically significant change um due to our small population size, so we'll have to keep an eye on this as well to see if this continues, um we did see a statistically significant change in delighted responses from experienced users with regards to Performance.

A

That's a large change um in just a short amount of time uh so and once again, similar sort of uh split with free users or sorry new users, um sort of moving down and up away from happy to either being delighted or unhappy.

A

So the themes and insights here, basically, um the encouraging trend is that there's increased delighted users, basically across the board. um It's all true for free versus paid and experienced versus new users across both overall and performance. Satisfaction.

A

However, for free and new users, um the percentage of delighted responses did go up, but so did the percentage of unhappy responses and as I mentioned overall, the net change is negative, so average scores dropped by about .06 points, um not statistically significant, but something to keep an eye on and performance. While you know overall performance scores were quite good. The average scores um you know compared to overall the performance scores are lower than for overall satisfaction.

A

The next steps here. Basically, we will continue this as a rolling survey, um as mentioned. Several of the numbers uh are too small to give statistical significance, but we do want to keep track of certain Trends over time, um and you know, as we take a deeper look into this, we'll have to see if there are corresponding changes to the product that that mirror or or could possibly be reasons to affect sentiment, scores here and then, obviously, we will assess any further research opportunities generated by this data.

A

Thank you. So much.