GitLab Runner UX, 6 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Category Maturity Scorecard: Runner Fleet to Complete in FY23

Description

Gina Doyle (Product designer: Pipeline Insights and Runner) talks through the results from the CM scorecard for Runner Fleet and recommendations from feedback and experience of the participants that were invovled.

CMS issue: https://gitlab.com/gitlab-org/gitlab-design/-/issues/1995
Category maturity scorecard documentation: https://about.gitlab.com/handbook/product/ux/category-maturity-scorecards/

A

Hey everyone I'm Gina, Doyle and I'm the product designer for pipeline insights and Runner I'm, very excited to present the results from the category maturity scorecard for Runner flight today before I, get into the scorecard process and results. I wanted to give a quick refresher of what Runners are, so that the features and jobs we were evaluating are put into a little bit better context.

A

Runners are the things that execute jobs anytime, you run a pipeline or a job. The runner is what's picking it up and then executing the code. There's different types of Runners, which are managed at different levels in the product shared Runners, are the ones that most of us get lab. Team members have probably used in our day-to-day work or folks that are using gitlab SAS. Those are administered at the instance level by git lab admins and they can be used by everyone.

A

In the instance, group Runners are registered at the group level by a group owner and are available to members of that group and then subgroups and projects in that group group owners are the only ones who can manage those Runners and edit them, and then finally, there are project Runners that are registered at the project level by product maintainers. These can be used by members of that project and then, if you share that Runner across other projects, members of those other projects can also use them.

A

The reason that I brought that up is because, when we talk about the category Runner Fleet we're targeting the features that help those platform. Engineers and devops Engineers manage the runners at the admin and group levels in eventually the project project level.

A

This is our first category maturity scorecard for this category, since it is on the newer side and the job that we're evaluating here is when I am managing the execution of many CI jobs. I want an overall understanding of the job execution agents connected to my organization, so I can make the effective decisions.

A

uh So you don't have to wait any longer to hear the results. I'll just provide them now, so spoiler alert. We did score at complete. We were finalized at 3.63 and we ended up being able to speak to five participants, three of them being self-managed users and two were SAS.

A

This split was important because of what I mentioned earlier, that the fleets of Runners can be managed in different ways, and then we had participants run through two scenarios: four tasks: total we had a 100, successful task, completion rate which was awesome and then, if you're not familiar with how we calculate the maturity score, we asked three different scaled questions around E's, ux and um ux.

A

So I'll provide a little bit more insight before I get into this scenario around the environment that we're using.

A

um So we used the ux sandbox environment, which definitely made my life of prepping for these sessions so way easier. Alford has been started at the login page, so they did not start here, but they did see a similar setup as what you're seeing here so there's a few online Runners and more Runners that are never contacted and then all of them had these upgrades that were recommended for them.

A

So I'll hop back into the scenario, so the first one, um the scenario was around checking the health of your Runners to ensure everything is running as expected, and the task that we asked was view these Runners at the instance level and verify that they're able to pick up your jobs and, if they're not, then then tell me why the average time to complete this was two minutes and four seconds which is pretty long and when I get into the recommendations. I'll explain more about why those results came out like that.

A

The second scenario here was around Fleet version management and then also Bulk Management. So for this one as an admin, we said that developers were creating a lot of Runners. Some of them were either outdated or they're, not following compliance standards.

A

So the participant had to locate the runners that have not contacted git lab in at least 30 days and delete them, delete five of them and then locate how many outdated Runners you have and then for one of those outdated Runners, add a note that the runner must be on version 15 to to follow compliance standards.

A

The average time to complete these tasks was a lot lower than the first one, but I still think that we could do a better job and, like I, said in these recommendations, I'll, let you know kind of what the feedback was and what their experience was like. So these first recommendations uh relate to the runner Fleet feature as a whole and understanding at a glance how the health of your Runners are are doing, and so this first one about making it easier to find runners in the admin area.

A

We noticed it took a very long time for SAS users to find the runners page since they had never accessed the admin area before those self-managed users had a in easier time finding it, um but I think we can do a better job at making the runner location more consistent on self-managed as it is on SAS and then the next one here is clarifying some Runner statuses. We saw that the time it took for participants to understand why a runner is not running jobs and kind of what never contacted means was longer than desired.

A

They also weren't 100, confident in their answers. For example, in that first scenario, they weren't sure, if, because that Runner was outdated, if that was the reason why it wasn't accepting jobs, we don't provide that feature yet so that wouldn't have impacted it. But what we can do is we can explain why a runner is never contacted so that they have a better idea of what that means and how it impacts their jobs.

A

And then we also have some confusion around the task, identify if a runner can run jobs, because that overlaps with, if Runners, are running a job at the moment and can't pick up other jobs, because the concurrency is maybe one or something like that.

A

um This is something we've heard in the past as well, so I wanted to connect it to these recommendations, because there needs to be a clearer indication if the runner is busy at the moment or not.

A

These next recommendations are related to the version in Bulk Management features, so it was pretty clear that we need to provide better Bulk Management for fleets of Runners. This includes deleting those Runners at once, which all the participants ask for after having to go in and click each individual Runner to delete, and then we also heard that there aren't that many Bulk Management features to be able to run on a fleet of Runners that they need in their daily jobs.

A

So that includes pausing many Runners at once and editing or adding labels for many Runners at once, and those were actually brought up specifically when we said when we asked about the umux scaled question um after the the tasks were run, so participants were confused about the difference between recommended and available upgrades as well, and so I think we can definitely do a better job at defining.

A

What the difference is between those and then like I said before they were confused if it was possible for a runner to not pick up jobs because it was outdated and that's not the case, so we definitely need to make that more clear of exactly what that means and then finally, we need to make a maintenance note more discoverable.

A

So the last task there was around adding a note that the runner doesn't follow compliance standards and needs to be updated, and so participants were easily able to add that maintenance note, and just so you can see it when you edit, a runner here. You can add the maintenance note there, but then, once you add it, it only shows up in the runner details view.

A

So if I had added it, it would have shown here- and they were expecting it for to to show up in this runner's list view, because they were um saying that if it's something that's that important, that your admin is asking, it should be discoverable elsewhere and that wraps up the recommendations that I created there. If you have any feedback or questions, please feel free to leave them on the category. Maturity scorecard issue that I've Linked In this video and thanks for listening.