GitLab Pipeline Insights Group, 2 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Verify:Testing Group Think Big #8 (Code Coverage)

Description

Today we think big about code coverage reports, who uses them, getting data out of them for developer and team lead workflows and how we can provide a great experience for users leveraging multiple GitLab stages.

A

This is the think big session for verified testing uh for november 2020.. uh Today is nothing big. A portion we are do a think, big and then the next week do a thing small. um So getting back into these today we're going to talk about code coverage reports and I have a link into the issue which I don't think has too much more information than what is in the agenda that we're looking at.

A

So I'm going to go ahead and kick it off and ask the question: what's the problem that we would try to solve with a code coverage report.

A

Scott or max, can you provide us a developer perspective.

B

Well, the developer's perspective is to be able to easily see what code I have not written tests for right um and that's helpful in a few different scenarios. One the overall project like what is our percentage wise of uh how much of our repo is covered by tess.

B

The other side of that is in a code review when I'm reviewing someone's new code that they've made changes to how much of their new code is actually being tested.

A

And a code coverage report like if you were using jest or something and it spits out cobratura report and you're looking at it in an html view, it solves that problem for.

B

A

You don't sound sure other tooling can apply here, don't take just the brand names. I give you.

B

um Yeah we're just, I think, just making that easy to parse out would be helpful rather than having to understand a corporate report.

A

So it sounds like you talked about a couple different workflows there. You talked about the overall project repository. What's the coverage like and then as part of a code review um code review workflow are there what other workflows would this report be handy.

C

C

Oh yeah, maybe something we discussed previously, will be the fact that, if you introduce like new change, which decrease the the overall coverage of your test or of your code like you won't be able to achieve that piece of code like there could be another.

A

So like a blocking feature where, if coverage decreases, then it blocks the mr.

C

Yep, but to me, like the coverage report itself, like the feature as a developer, it's like more a visual tool that tells me right away like if my code is tested or not and as a reviewer. It's super helpful because I can I can see where the piece of code introduced was not tested, probably which, which one is tested, and I am now confident that we can ship that piece of code, because we we added, like this coverage right cool.

A

So it sounds a lot like the existing feature that we have the test coverage visualization of seeing it as part of the code review process. I want to circle back to looking at a project holistically, there's a couple of different bits of data that come out of the cobot report or could come out of it. You could get just raw line coverage function coverage and I don't remember what the other ones are, but there's a couple of different stats that can come out of that.

A

What is your experience with using those different numbers and how they're helpful or not helpful, when looking more at like holistically? The project and ricky feel free to jump in here too, to provide us with uh some team lead perspective.

C

D

I was just typing up my uh my thoughts here in terms of uh so yeah, so the the overall health, I think, is the important thing as an engineering lead, so you can focus on areas that you might want to schedule. Some tasks for the other thing that I did a lot in the past was kind of like okay, here's my project, I know which workflows and which files are the most risky.

D

So, for example, if I'm working on an application that takes payments, I know that that workflow, where I take the payment, is going to be the most risky part of my application, and then I want that to have really really high coverage, like probably 100 coverage, but the part where I.

D

Send a newsletter probably like don't need as high coverage as the part where I'm taking a payment, and I'm not saying that sending. That was a bad example. I was trying to think of something that was really uh not important, but newsletters can be important to the business as well. um So uh when, if I could like something that would be neat would be if I could highlight certain files or workflows as being high, risk can have like different rules that apply to those workflows that might be kind of nice, so I could say hey.

D

If anything is committed to this, it would be even cooler if you could do that. You could say if anything is committed to this file. This person needs to review it and it can and the test like as far as coverage goes, the test coverage cannot dip, but also this person is like a special reviewer for this. So if we could have like that type of annotation that'd be really neat. We kind of do that with code owners, I think um so, if we combined code owners plus special coverage rules for certain sections.

D

That might be neat, um but I think I think that's something that I was focusing on a lot when I was at my last. Job too was like hey, there's this part of the workflow, where we actually go and we create a subscription with somebody's like tokenized credit card information.

D

This needs like a lot of tests.

E

Yeah business critical right, I think I think your example was good being a little hard on yourself right, I mean there are things that are business, critical versus, not right. You think about the things that get into production yeah you want to have like, maybe some more guardrails or some more. You know some more stringent gates there um and that's cool, and it's interesting, because I was thinking about the developer individual developer. What do I gain from this versus um an aggregate level or a manager or team lead?

E

So that's a that's cool that we're delineating between those two here, both really important.

A

So the code coverage report as it stands today is kind of a snapshot in time. You see what is coverage the last time this report was generated. um Where are there gaps there potentially or how can we improve on that view?.

D

I I think the the much like a lot of our reports are kind of um either we just host an artifact that people can go and look at, but we don't have like a holistic okay. This is code coverage. Dashboard report thing right like same same with code quality. You can go into a pipeline. You can look at the code quality thing for an individual pipeline, but there's no like project level um code, quality page that has all the code quality information and same same with code, testing and coverage.

D

There's not like a project level thing where you can go, you can go and manage. You feel like you're, managing the code quality. Are you managing the code, testing and coverage of a project at that level?

D

It's more just like well, the pipeline did a thing, and then we smushed that thing into the pipeline or we smooshed that thing into the merge request, um which are great, but I think that that the next like level to build up onto that would be at the project level and then we're already kind of thinking this way with um with coverage reporting at the group level. So um it would be interesting to have like a finer grain view like I was talking about.

D

Okay, here's all the files I want to ignore this file in the ui. I want just pay special attention to this file in the ui and have that kind of like uh like that's that's really. What sonarcube does too is they kind of just give you the files, and then you have special um abilities to like say no, I don't care about this um code, quality warning specifically, and then it remembers, because that's what it is, that's what kind of application it is right.

A

Yeah, so it sounds like a lot of that is based on work that you as a developer. You as a manager, have done to set this up and get this out, and then it's on you to kind of interpret those things um I'm skipping ahead uh because we've talked about who has this problem who's doing this job? We've talked primarily about the developer and the team lead.

A

um But if the team lead, uh you just started up a new project or moved a new project into gitlab moved an existing project, maybe ran a pipeline just to get it all running and parts of the output of that would be here's the top 10 files that you should look at from a quality perspective because of low coverage look quality whatever else we have um so that you could then just go start creating issues from that.

A

Based on what gitlab knows about the project- and you may already have an inkling about that because it's you know your team's code, but it might be that reinforcing bit that you, uh you, then trust gitlab to say: okay, here's where the tech that is and what I should go.

A

A

What other ideal outcomes for developers or managers would we be looking to get to if we more tightly incorporated code coverage reports into gitlab.

D

I I think, if we're gonna do gating, it has to be really really really easy for the engineer to figure out specifically which line which file and uh sorry yes, specifically, which line in which file is the uh that's preventing this from happening. I know in previous things that I've used you kind of throw it up there and then it just gives you oh well. The code coverage went down and it's like great.

D

Where and wha what test do I need to read right in order to get that back up so as specific as possible to let the engineer go and fix the problem, maybe even like it would be super neat if it was like. You should probably add a test in this test file for this file so like that, even that extra thing, so you don't have to go okay well, this isn't covered. But where should I put this test like it'd? Be super neat?

D

If there's some way to say well, you need to write a test in this file that covers this line.

A

Can we leverage some of the mapping work, that's being done on the plane failures to do that.

D

I think like yeah, if you could do like a reverse test file- finder or I guess it would be still a forward style finder, but yeah.

A

But yeah, if you add code to a file and it's already mapped into a test file, you could say hey like as part of the the workflow it would block. It would say that code coverage is decreased and it would be would be some sort of tip or trick or hint of here is the file that you probably need to add a test into because of this mapping that you've provided yeah, and I.

D

Wonder if it could, even like you like copy the file path to your clipboard or something, and then you just go and open it on on your thing and start writing your test right away like that, would be kind of neat.

B

Scott max feedback could open up the web id for that file.

B

Are we worried about.

B

Portraying to the user like if they write a test that happens for that line, but it's happens to be in a different file, then it's gonna fail or worry about being too prescriptive here or too opinionated on that spot. I know that's getting a little nitty gritty thing, just it's a good question, pushing it down.

A

Yeah, I think we could probably uh think not think down um uh constrain uh later today is really about you know. What could the idea look like I mean. Ideally, we could just say generate the text right.

A

It looks like you need to add a test into this file and it might look like this. Do you want us to add it for you.

B

A

Our suggestion.

B

A

B

I would like that, do it for all my tests.

A

Yeah, hey was it dangerbot adds labels for you or, like suggest, labels.

E

Yeah, I think, suggestions always go a long way. You know just from research into the persona with developers, especially like they got a lot going on all the time right like context with you all these, so the more I think we we look at it as a very risky thing, um sometimes to be prescriptive and opinionated, and I think they look. They welcome that. um I think sometimes they even welcome over prescriptive approaches um because at least they know hey.

E

I don't want to do this right or thank you for the suggestion, but no thanks right um yeah, it's cool.

A

Cool, so we have some great thoughts here on ideal outcome for developer, using a code coverage report getting some additional information out of that report back into their day-to-day workflow. How about that team lead persona? What um what's an ideal outcome, look like for them, stepping back out to that here's your project?

A

What's the coverage? Look like? What's the quality look like.

B

I think the overall coverage number is nice, but it's a little uh opaque there be great for the lead to be able to drill down a little bit on sections of the app. Maybe they care about more, like in ricky's regard like hey. We care about this payment section more than other sections.

B

What is our coverage percentage on this area of our code?.

A

A

It matches up with a lot of the conversations I had doing a first round of problem validation around code coverage. Reports um of people just want to see like not just overall. What is the project coverage? I need that next step down, so that I can go, find those files and figure out where we can quickly improve coverage or where we might have problems of persistent bugs.

D

Yeah, I kind of like that idea about honing in on a certain section of the code, whether it be through setting up a config in the galaps diamond or whether it be um building a full-on thing into the projects page where you can go and set up a configuration that says: hey anything that has the word payment any file with that matches.

D

This regex string like star payment star that needs like more tests, that's high priority or that's medium priority or that's low priority in terms of in terms of code coverage and then further being able to set up what that means in in the project would be kind of neat. So, like okay, if it's high priority the you cannot decrease the coverage at all. If it's medium priority, you can decrease it by like two or three percent. That's probably fine! If it's low priority, I'm not even gonna apply uh a coverage gate to the merge request.

A

Would you do that by file or by section within files of the code.

D

See, that's the tricky part and I'd probably be pretty language dependent um like what what I would want to do in a perfect world that doesn't exist is be able to like kind of identify code paths or workflows. Like give me a trace and I'll show you all the things that that trace runs through and this trace cannot have any coverage decrease on it. So, in.

E

D

Perfect world it would be, it would look like that, but I, like I'm pretty sure, we're a little bit far away from that right now. So if we, if we had just file, then that could be. That would be a great, I think as well.

A

That's an interesting approach, though, of like hey: here's, the here's, all the paths that it goes through even potentially across projects and say this is a business, critical workflow and we're going to apply things at the workflow level or the value stream level versus just with thinking within a project and within files within the code base. That'd be very cool.

D

Yeah I mean that sounds awesome. I'd sign me up. Yeah.

A

Let's um so thinking big, that is a pretty big thing. I think, um and that's what I'm going to take away from. This is hey. Let me identify paths through my application that potentially stream or span groups projects whatever and apply a code coverage requirement to that path. um How could we differentiate what we're thinking about from some existing solutions out in the marketplace? I called out sonar.

A

Codecov coveralls are some of the code coverage reports and coverage report tools that are out there and ones that we hear about when we talk to with users of things that they're looking for.

A

How can we build on what we have a having source having ci having monitoring data? Whatever else you want to think about from that all-in-one devops tool chain that gitlab provides that we can really differentiate from these other vendors.

D

It's like you read my my comment. While I was typing it, I I was just. I was just typing in there. How specifically, I think, what about having an all-in-one gitlab solution? Can we leverage in terms of code.

D

A

I like that idea of the suggestion of hey here, you have decreased coverage. I see the new code, I see where you've got existing mapping. Let me at least tell you where uh you're likely to be able to add a test quickly, even if we don't provide you with the full-on test.

A

But if you take that a step further and say: hey you've already got 50 coverage in this project, so I kind of know what your tests look like: here's, what this test might be and suggest that we could probably build on some ml uh and some other bits there.

D

Even if you had kind of like a default, oh, we can tell that they're using um jest, so here's the default jest test. This is what it looks like.

A

D

It's just like whatever block of code is typical for a test that asserts nothing, and then they can kind of copy that click on this thing go to the web. Id put their test in there quickly write the thing they want to assert and then commit it in the web. Ide like from the example block.

A

Just providing that templated that scaffolding for the test would be pretty nice.

D

If you want to talk about ml like uh ml algorithms, that classify things so they could look through the test directory and then classify it like well. This is probably jest- or this is probably this is probably go and so they're, probably using ginkgo or something and then okay, here's what ginkgo maps do in terms of blank test.

A

What if we took that ml um a little bit further and said hey, we see that you're creating incidents in these projects and that follow-ups from those are mrs fixing tests or fixing code. Maybe your unit tests, weren't testing enough or something it's probably integration. Testing at that point, that is flubard and not the unit test, but might be something to think about.

D

I wonder if there's a simpler thing like as as you report incidents, I wonder if we tie them to specific files or something and then you can say hey. This incident was caused by this line in this file, and that way we can kind of build that into the project level view of what's important and what's not important. If this one file's causing a lot of incidents, then hey, that's, probably a really important file.

A

Yeah, I think that goes back to that that work stream too of or that code path of going down this path. We've had multiple incidents. We need to, you know, think more about resiliency of this code, and um you know apply that somehow it needs multiple code reviews, not just one, as things are going out.

D

Yeah I I just started thinking when you were talking about leveraging. The single devops platform, like like sewing a thread through a bunch of different aspects of the application about the feature, and if we can do that enough times, then it becomes more useful, so think about, um like you said, there's an incident think about tying that back to a file and think about exposing the coverage of that file. In the incident view, say: hey like there's an outage caused by this line.

D

The code coverage is 100 or it's 20 um and the other things I was thinking about were your you're browsing the code and you have a little thing hey. This is x percent covered. um Maybe you want to expose, like you said, a list of a view that shows the files by coverage um that that that would take into account maybe the ignore settings.

D

So if we're ignoring files, then we're not gonna like we're, not gonna shove it in your face that there's not very much coverage on that file, because it probably doesn't matter because you've ignored it so being able to like tool in the ignores into the application, while you're viewing stuff would be really neat, as opposed to, I think, a lot of things um when you when, when it does that- and it kind of like presents it to you, it doesn't take into account all the other touch points that you've had with the application where you've told it something about code coverage, uh especially when you have like plugins and jenkins, and all your stuff's all over the place.

D

When it's surfacing things into you, it might, it might say, like hey this file, is zero percent coverage. I'm, like I know it's a config file. It doesn't need any coverage.

A

Yeah, it doesn't does not need coverage hey. We were talking earlier about that. um The generated report of here's, the you know: here's your top 10 files that need some sort of work on them and thinking even a little bit more broadly, it can combine code coverage and code quality if there is a way to generate issues from all of those that contained all of the necessary information and linked back to some sort of dashboard.

A

I think that kind of ties from the source and the plan bits together as well back into verify to say, especially when you're going back through or when you're running pipelines to say, hey code quality issue found in here. Oh there's an issue open for that. If I resolve it with my merge request, I can also resolve that issue.

A

At the same time and clean up some code coverage or code quality problems, it's thinking from the manager perspective like it'd, be great to get that list of files, but I don't want to have to go, create 20 issues to address that tech, that, for my team, if gitlab did it for me, oh that's happy day. I think just because I know there's no csv uploading or I don't think, there's a csv upload to create issues in git lab.

A

I may have used that with other companies. Products in the past.

A

Awesome, well, we have I'm sorry, go ahead. Parker.

E

Say that give you the ability to look at trends too right, like not just a single point in time, but as a manager like yeah, a single point in time may be very, very important to you as a developer, especially if you're looking back at these snapshots, but that would be able to show you a trend hey in the last quarter.

E

You know how did our code coverage improve and then, if you're talking about like being able to segment or ignore like that kind of listed view, you were talking about ricky, then you've got oh, let's not look at the ones that we don't care about right. You said this is a config file right. Let's look at this stuff, that's that's important and how did that trend? If it's going up? That's a great thing right.

D

Cool yeah um just to quickly revisit the thing you said about automatically creating issues. What, if we're thinking big today? What if it automatically opens an mr? What if it opens nmr with the test that fixes it or it opens it, mr with like, uh because it knows it knows what file the test should be in because the test file finder.

D

So it opens an mr into that file with like the skip the boilerplate um test, and then you just open that up in the web id and the merge request and add your test and then commit you. Don't even have to open your ide, you just you fixed it. You fixed the problem in gate, lab and created the. I guess created the issue in git lab and then fixed. The problem in git lab not created.

A

So create an issue create an mr drop in the scaffold for the test, based on the mapping that you have with test file finder. All you have to do is apply a suggestion that actually adds the logic to the test, and then you can run it, merge it and be done with it yeah.

A

I think that's pretty snazzy.

C

What a wonderful or we're living.

A

Love it cool. Well, we are at time I want to be respectful of calendars. As always, if you have more thoughts, though, feel free to drop them into the doc or continue the conversation and I'll add notes to the doc over, in slack, I'm happy to continue it there and then next week, we'll go from the divergent back to the convergent thinking and think about great.

A

So here's this world, where ricky our team lead, gets a report and auto generates issues that auto-generate, mrs with tests in them and max, is following up and just closing things out, left and right. This is how we're going to increase our mr rate um and how we can build something in an upcoming milestone that takes us one step toward closer to that brave new world.

D

Thanks everybody thanks folks. This was good.

A