GitLab UX Team, 7 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Usability metrics to measure impact of SUS-related changes more quickly

Description

In FY22 we strive to improve the areas we identified to have a positive impact on GitLab’s usability and user experience. To allow product teams to continuously listen, test and iterate with end users during development and understand the impact of their changes more quickly, a small set of usability metrics is proposed.

This video is a brief introduction to these new usability metrics, how to use them and helpful templates.

Related Q1 KR: https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10465

Handbook page: https://about.gitlab.com/handbook/engineering/ux/ux-research-training/usability-testing/

A

Hi, this is anna from the ex research team at gitlab. I wanted to share with you a small update to the handbook. Basically, we added a page focused on usability testing, which you can see here, and you might wonder why this was needed.

A

As all of you obviously do lots of usability testing already, and it is because we have this company-wide initiative to improve sas and, more specifically, this quarter, and there was a kr focused on understanding the impact of our sas related changes more quickly and the proposal to do that to be able to understand the impact of sas related changes more quickly is to have a small set, namely three metrics, that we collect doing usability testing that will help us to measure the impact of the changes you're proposing and yeah these metrics needed a home.

A

So on this usability testing page, um you find a table which outlines the different metrics, how we're measuring them and why we're measuring them- and I briefly want to talk you through them and the first one is the first metric is task success or a completion rate per task, because we ultimately want users to succeed in reaching their goal.

A

The second one is focused on um task difficulty or task easiness, um and it is the same question that we asked during cms. It's the single east question and you might be familiar with this wording.

A

The third metric we want to collect is a adjective rating scale, which is highly correlated to sus, and basically we ask users about the user friendliness of gitlab and they respond on a you know, within with an adjective with a word that is um highly correlated to with the sus numeric score, which is great.

A

um The first two metrics are being collected on a task level and the third metric. The attractive rating scale is only collected at the very end. After all, tasks are completed so basically going forward. Whenever you evaluate a specific solution or really anything you work on, that is user-facing and you feel it might have an impact on us. We highly highly recommend to you know, collect these three metrics as part of your solution, validation.

A

In addition to that, um you actually also want to understand. You know where the current implementation is at when it comes to these three metrics. So how does gitlab today perform when like against these three metrics, because assessing this will give you a baseline and a benchmark and so to speak against which you can compare your solution against?

A

So this was a brief outlook on like what is going to happen, um or you know what we're basically proposing and what I would um love for you to try um moving forward and you can take off now if you want and just enjoy this page or you stay for one and a half more minutes and I'll show you what I did in user testing to kind of help you with um getting started there. So friends of user testing follow me along.

A

um If you want to set up a new unmoderated test to assess these usability metrics, you can go and create a test from a template and within the team templates section you'll find a usability, metrics template and I'm just clicking myself quickly through the panel set up the screening set up, because I really want to get to the test plan. Some summary which is already pre-populated here, which you just need to edit um according to your own tasks.

A

So for the purpose of this template, I used my personal blog enjoy and I put a scenario in which you of course need to edit according to your feature. um But here is basically the structure, so we always have a task here. It is we, you know you enjoyed reading the blog post and you want to leave a comment, and that is the task and then, following that, we'll ask the easiness rating the task difficulty rating.

A

So, overall, this task was followed by a verbal response asking why they gave that score, and then we have the next task again followed by the the single ease question with the verbal response and then lastly, our adjective rating scale focused on the sus and again verbal score. Why they gave this score and the same is actually also in the usability testing script template, but I'm running out of time. So take a look at yourself and shoot any questions. My way.