GitLab Pipeline Insights Group, 30 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Verify:Testing Internal Customer Call - September 2021

Description

Today we talked about some current research around Usability JTBD and insights uncovered related to Environment Management. We also did quick overview of how GitLab is using Metrics Reports (https://docs.gitlab.com/ee/ci/metrics_reports.html) uncovering a possible degradation. Lastly we talked about the future of test reports and how filtering by pipeline is an important use case for our internal users.

A

This is the verify testing internal customer call for september september of 2021. uh Now that we're every six weeks, I think our next one will be in november. um We have joanna with us today. Thank you for joining uh looking at the agenda um a couple of quick notes and uh from the roadmap deck that is shared in the agenda.

A

There is research happening as we validate usability testing jobs to be done, gina wrapped up those interviews this week and has already posted insights, so we're working on rewriting the job to be done around usability testing. It did uncover a lot of interesting things in environment management and code review as much as it did around how we use review, apps and visual review tools, which was really interesting. So as we continue to push forward to move that category from minimal to viable and driving more internal usage.

A

We're looking at things like the doc site and handbook of how do we use review, apps and visual view tools as part of the review process with those, and then we're I'm going to be kicking off research later this afternoon with the first uh interviews around just the overall customer journey from when you start authoring that gitlab.yaml pipeline gitlab.gitlab ci yaml pipeline file, wow um through when you start uploading, artifacts and looking at them and see what that process is like where there are friction points in uploading, artifacts uh we're changing our main pi from usage of paid features to just overall usage tagging onto the stage monthly active user account of how many users are starting the pipeline to how many users are starting the pipeline that uploads a file one of those two test report files.

A

So that's what we're looking at! So those are some of the highlights from the deck and today I wanted to talk about metrics reports, because that is a topic. I don't think we have covered much in this call.

A

So I would post the question to our internal customer of tell me about the last time you used a metrics report.

B

To be completely honest, I didn't know that we had this feature, so I'm glad that you brought it up in this call. um I was looking at the docs uh before this and I think it's definitely really cool um I'd be curious, like if you know how we're using it at gitlab now like what other teams are using it yeah.

A

A

Let's all look intently at our screens, while we look for an example and then I'll share my screen.

A

So we're using it today for things like gem size, I want to say um see if I can find a non-docs.

A

So here's an mr uh foresight, you see it's one of the widgets here so we're using it to track some metrics around gem size um memory usage on boots. I don't know what all these gem size things are.

A

So these are gems we're either using or gems we're creating as part of the build process. But I'm honestly not sure it's like that's everything and I'm kind of surprised that those all say new, because you would think that those are existing gems and here's the diff between the two, because this view over here is now the size of a thing. Without any comparison is really not that useful. I I I would think, and that's the feedback that we've gotten from customers as well.

A

um So I wonder if either we had a degradation or if the way that this file is being created makes it always look like these are brand new entries into the file.

C

We'll probably be able to review it once we uh migrated over to the new merge request, widget extension, so yeah we can always see if it was a front-end degradation or um or if it's some kind of way. The report is handled yeah.

A

um And I wonder, is this: isn't one that's on the list uh that that working group is changing over initially, but it seems like this is a pretty simple list similar to uh the performance widgets that are migrating, so we should be able to do this one fairly soon after I would think.

C

Yeah, I think the only thing that was keeping us from moving that over was um polling wasn't um put into the core component. Yet so as soon as polling is in there, then we're good to go. Yeah.

A

Cool um I'm gonna try to find our example.

A

Oh, I did one the other day, so I was testing something out.

A

I wish I could sort my project list by last updated, instead of whatever order it's in.

A

I thought I did something yesterday I was testing out if multiple metrics reports files were consumed by the widget, so you would see all of the data and confirmed that that is true that that works there. It is all right. Let's share my screen again.

A

So this one has changes and then a new item as well. So we have one new one that didn't change and if there is a change, I don't think we're showing the percentage here yet because these are strings and not things that we can calculate on uh easily, um but it should show- or it shows that that there isn't a change or the diff between the two um just through simple conversion.

A

If something has changed so I would expect that it's not a degradation that it's in the way that or it's not a degradation and it's always showing things as new, but maybe something changed in the widget logic as we're looking at it, where it's just not picking those up as changes and showing them as new every time or it could be that the file doesn't exist on the target branch too. If that's not run as part of the normal default branch build.

B

When it's showing the diff is up between, like your current branch of the target branch.

A

So that's why I would think that it would be a lot more interesting to show like if we know how big the gym is uh on that branch. How how much has it changed with my change? I would think that that is the intent of that.

A

Having that there, um I I'll post back into engineering productivity and poke around and see how what the intent is for that I'll also see if I can figure out who the author was on, that part of the gitlab ci yml, um and try to uncover some motivation for having that in there.

A

Thank you for taking this journey.

A

Well, since you didn't know, we had the feature and haven't used the feature and we're finding. Some gaps exist around it's not showing us actually diffs, which might be more useful, but I can investigate further um that wraps up my questions around using metrics reports.

A

So I'll pass it back over to you. What other topics do you want to chat through today or questions? Do you have about what we're working on? How can we be more helpful to the quality team.

B

Yeah, so um I think kyle had added a comment on um like the text test execution reports like this is something that we're um trying to kind of build on our own right now, because we're trying to understand like what are the trends um for our pipelines over time right now, but if we had something um you know that was like a gitlab feature that we could use. That would be awesome, um so I guess my question is you know like um what? What is the current plan for that?

B

Like scheduling wise and um what kind of help would you all need from us to move that forward?.

A

Excellent question: um let's take a look at the road map.

A

That's funny because it's not scheduled.

A

I had left a comment in that issue, asking if there's anything we could contribute, and I think that they're using the end point that we set up the internal one to populate the test today um and beyond that, there wasn't anything that you all needed initially.

A

I think it's historic test. Data for projects is probably the right thing to look at and.

A

So yeah, where we had started, was that we wanted initially we thought we would do just the last 10 executions of a test, but I don't know that that's the right level of what you're looking for of did these. I guess, um I think, there's a gap between what this does and what you're trying to understand that uh I need some help figuring out.

A

So what we had done was put together, and this went through solution, validation, where at the project level you'd be able to see all of your tests and a history of them over the past. However, many runs to see how the test itself is trending over time, but that's all the way down at the test level, not even at the job level or overall. The suite level.

A

So yeah, I I'm not sure that this gives you the view that you're looking for. Are you looking more for this many tests or passing this many tests or failing.

B

Yes, I think so then, having the ability to drill down and understand which tests also.

B

Yeah I'm wondering like, if there's a way to do it.

B

By pipeline and see the trends that way, um rather than like the project level, yeah.

A

See we're always thinking about it at the project level it seems like and that's what like, that view would be to click in from our project summary, which is that one, that's probably very tiny,.

A

um What we learned in our solution, validation was that it was too far. um This test runs data and the test coverage data, which we think we can populate pretty quick, because we have an end point, for that was what was most important to the current customers external customers.

A

This would give you um at least the current the the latest pipeline that ran on the default branch for the project.

A

How many tests are running, um failing or being skipped, and then the click in from here on the full report would be then the that next level of here's all of those jobs and tests, then I think the iteration after that would be now.

A

Let's take these three data points and track them over time, so you can see it at least at the project level, but you want to see it at the pipeline level because of all the different environmental stuff, yeah right, yeah, okay, okay, that's always the problem that we run into is the the way that we organize our tests is my pipeline on my project.

A

Great, that's all right! Okay! um So if we were able to see this, but do a filter by the pipelines, then you'd be able to do it be able to. You would accomplish more of what you're trying to do with your own report, but if we were able to at least give you an end point for this and filter by pipeline you'd be able to populate that report. It sounds like yeah, that's good to know.

A

I will take away from this uh at least a to do to create the issue of create the endpoint uh with a pipeline filter on it, so that we can get closer to that.

A

um If we don't already have that or if the team isn't able to move forward with the endpoint, that is there and then let's create a view of filtering by pipeline within the project level as well, so that we can make this we'll create a dog food issue for that, so that we know that that's a use case for our internal customer awesome awesome thanks once we had that, I think the test summary view would be a little bit better for you, too,.

A

Anything else that we can help with anything uh you want to poke at for things that have come out of the team. Lately.

B

I think that was the only thing that was top of mind today: okay, yeah awesome.

A

Well, very fruitful call if a quick call. I appreciate it. Thank you james. Thank you, jordan. I appreciate all it's good to see you uh yeah love, seeing kyle as always thank you kyle, uh but like seeing you too all right thanks. Everybody cheers.