GitLab Pipeline Insights Group, 1 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Verify:Testing Group Think Big #8

Description

Today we think small about Code Quality, what some simple fixes to the experience might be and pros/cons to displaying an opinionated score of what "quality" means.

The Think Big session: https://youtu.be/aJPp32Dhazk

A

This is the sixth think big uh for the verify testing team. This is actually a thing small version of that uh and we'll be thinking small sorry. This is the seventh my bad um and we'll be thinking small about code quality. Today, let's read the wrong line in the agenda, um so quick recap from the think big which we'll link to in the video as well.

A

uh We talked about the persona um vps directors uh who would care about kind of is something risky to deploy and using code quality as a as a proxy for risk.

A

We talked about a data view that would help them summarize that risk as it relates to the quality of the code, and we think the ideal outcome is a totally customizable dashboard of sorts that they can use and push things around together to identify uh risky things I intended to in the agenda link to some design that one just did that actually for our long-term vision. That would show that really really well um and I'll. Add that into the agenda later. um Did I miss anything from that?

A

Super fast recap that we want to touch back on.

A

Great um so then, moving into from this really great awesome idea of here's a great way for you to categorize risk of your project, and if you should move forward with the deployment or not or invest in squashing tech debt there, let's think small about. Well. What can we do that in the next milestone?

A

That would be a step in that direction of identifying risk, reducing risk, especially when it comes to our current code, quality.

B

B

Could we historically look at code quality like per deployment like how it changed? So if somebody? So, if we identify deployment as having intro, you know we have to roll back a deployment, because it introduces some horrible thing. um If we could, if there's a simple way to correlate that with code quality to.

C

B

This happened here, we added this complexity and this bug came out of it because we didn't see it here.

A

So you would do or do something like take um a snapshot of your code quality report and attach it as release evidence almost or release artifact uh yeah through the progressive delivery, or um I think at the wrong group name or don't know the right group name, um but that group they're taking the junit report for test evidence already uh for that kind of purpose.

B

Oh, that's even less to do. Yeah.

A

So it would be, would it be the entire j or the entire code quality report? So what would that artifact look like to say a release manager? Let's talk about a release, manager, persona coming back and the use case might be hey engineering manager. We had to roll back this release. There was a bug it was in identified in this file, which is something you work on. Here's the code quality report.

A

A

How do we support that interaction? Is that is that the right interaction.

B

The file might be too specific. Overall, I was thinking that if we, if we had the code, quality artifact for a release and the a release manager could look at the improvements and degradations from the last release. To say, like you know, was this: did we add a lot of? Did we simplify a lot of things? Did we you know, make things a lot more complex?

B

You know what like which direction was this release? Moving in when this bad thing happened.

A

So, instead of a full report, it's more like the summary report that is on the mr page.

B

Yeah it's a similar idea. Okay, mrs might be kind of a granular level to do that at and releases might be too high level. But I don't know that that correlation is what I was thinking of.

A

Yeah, so I could see like if you're using feature branches or even with these branches- and you went to do that- merge it would generate that artifact at that time, and you could look at that summary report and say hey between what is our current stable branch and what we're trying to merge in here's the degradations here. The improvements in code, quality.

A

So outside of like, if you did that as an mr you're going to get that, is there another this? What is the use case? I'm not understanding or grocking. That would expose that data.

B

A

I it sounds like that's done.

B

Well, so that so that I was just thinking about that, because that happens on individual merge requests for us, and I don't know if we use that or look at that and just the volume of merge requests we have might make that difficult um like I don't.

B

I don't know that a bug is going to be traced back to a single code, quality diff in a merge request somewhere yeah between releases, like you know, day to day, if we can say oh on you know today when we released as opposed to yesterday when we released the version before you know, we saw this effect here.

A

All right, so, let's talk about it then like in the context of what happens at get lab with the night. The I think it's nightly when that release branch is cut.

B

A

I'm sorry, what did you say ricky I missed that. That's it! I.

B

Think it's every six hours now.

A

Oh wow, okay, so every six hours um is there a comparison that would happen what would be compared? How can we slice that down.

B

I think that it might be more useful, as almost like an after the fact thing, especially when you're talking about continuous delivery at gitlab like if something did go wrong. It could be one piece of evidence among many where we could look for trends or like. Where did we go wrong like? How did the process fail us?

B

um What should have we have done in order to avoid this kind of more like a retrospective or rca type tool that you could dig into to look for data to support your hypothesis when you're asking the five whys or whatever something like that.

D

I think part of it, too is we're we're trying to figure out risk to to do a release, as opposed to just doing follow-up after a release has gone bad right. So I what I wonder is: where does that fit in, because if we go to like releases, I think that's more of a this is what has happened uh in the release? It's already done, it's not what has what's leading up to this next release.

D

Yeah. I wonder if that is.

B

It's almost like if the company had embraced a continuous delivery process or continuous deployment process that they would use the report differently. So if they hadn't done that and they were still doing manual deployments, they would probably use it as like. Okay, how concerned should I be when I click this button as a release manager, but if they're already continuously releasing it's more of a oh, this blew up. Let's try and figure out why um I wonder if it's like something cool could be.

B

um Maybe it stops an auto deployment if there's too much risk associated with the deployment and that forces somebody to intervene. That might be neat, but this is supposed to be think small, not think big, so I'll uh hold on to that one.

A

So what let's? Let's take that and use that as to think big? No, what is the smallest version of that? We could test in the next.

B

Milestone, uh putting the severity and ordering by the severity in the mr widget there's already an issue for that open.

A

So this happens even before the before the merge before the deploy, because you can't deploy something that didn't merge right.

B

Yeah and it's the smallest thing we can do, I think, is, and we would need to take severity into account anyway, which I don't think we're doing right now, with the code quality widget, we're kind of just smushing everything into the uh comparison.

B

So if we brought severity in, that would be like the first step towards actually using it to calculate risk. I think I'm could be very I'm probably wrong.

A

Scott max eric zeff, juan parker, any other thoughts from the uh collection on mute.

E

And so you have to excuse my ignorance a little bit here right, I'm not as close to the code as y'all um so taking severity into account. Would that be something um like a red, yellow, green kind of low medium high severity um kind of display like how would that be surface to them, because I think there's value on that right. If you see red um you know, maybe maybe someone goes. Oh right, you know sound the alarm, that's something! That's super severe! I need to take care of.

A

The scale, I think, there's a five point scale uh from code climate. It goes like critical down to high medium low, something I don't remember. Okay,.

B

I think we could probably co-opt what the security team has already done with their security reports. That design already exists. I don't think we would need a new one. We would just copy what their widget does cool.

A

So stepping outside of things already in the backlog is there an even smaller version of that that we can implement that gives. So if we take the use case of a developer, who saw new problem in code in my mr related to code quality, how do I know if I should fix this or just let it go.

B

The only other thing is to actually make the comparison, compare the right things, but we're already working on that too.

A

We re-emphasize the outside of the backlog. Part of this.

E

You know, I guess even smaller than going to the severity, it could be as simple as needs attention right, an alert. This needs attention, but I don't know that that would motivate someone to or give them enough information to efficiently go and address the.

B

Issue we already linked to the violation in the co in the source right.

A

It's clickable from the code quality report on the pipeline page. I don't remember if it is in the mr widget or not.

B

I might make it easier to say, like you know, oh what what is this violation should I fix it. Yeah.

A

E

And we're talking about.

A

Let's, if we're talking about now, um if based on the hypothesis, that uh low quality code as identified by the scanner, would create an issue if we just block the deploy or block the merge, rather not the deploy.

A

And kind of force people into fixing new chair like new degradations.

B

I think we'd want to do it. Optionally, like they'd, have to opt into blocking the deploy in a setting or something I'm not sure how that's done right now.

E

E

Yeah, what if you know, because someone could be working in a different time zone or it could be one code on their right and then, if it gets blocked, you may that may be a bigger blocker than just blocking that um that merge.

B

Are we taking that as a given or are we also? Are we trying to validate like these code? Quality violations do represent risk.

A

Do we need to? Is there a simple version of hey you, as an organization have spent some time in the code you understand, based on severity levels, what you need to care about when you start to surface only some of.

B

B

If we buried everything except critical in the merge request, widget do you think it would increase the signal versus noise of the widget.

D

If we buried everything, where else would they see.

B

D

B

You could you could click into it. um You know, and if you looked at the full pipeline page, you could still see it, um but I'm just I'm thinking about how you know we have a lot of widgets that are like 800 improvements and 900 degradations and it's you know three line, change or whatever, that's partially or mostly, because we're not comparing the right commits right, yeah yeah, no, that problem that that is a big source of the problem.

B

But I'm just I was thinking about how when there's too much there um developers and release managers, sort of automatically check out and they're like that's, not it's it's just that! There's too much noise is, I think, a hurdle for people, but I don't think that there would be too much noise in the opposite case. If you think about like danger, bot and how reliable danger bot is when it comes up with like a robocop violation or something or like it did put stuff in the danger.

B

Cop comment like I feel, like everybody, pays a lot of attention to that, because it isn't noisy it's just as noisy as the amount of changes that you've made is, and so, if we could, if we had a proportional level of noise, because the widget was comparing the right things, then I think it would probably be fine.

C

I I think, a defect of the design that basically became more defective, as as gitlab uh has been growing, is that the the widgets are are very. I think people are becoming blind to the witness a little bit. You know like you, see them and you probably pay attention a lot to the pipeline one, but the other ones are like yeah. They are there. You know, like, I think, that's what really saying right like it's. It's part of the background noise. You know the difference with the danger body that that that's actually active noise.

C

That is like this rough team, like the thread and it's putting like a comment there. That says, like hey, look at this, you know like something fell, you know, I don't know.

B

It fails your pipeline, which is a very clear signal.

C

Yeah exactly yeah, that's yeah! That's one thing too! You know the code. Quality doesn't feel your piping right, that's unless by design it's not like that, but I guess you could set it up, that it tells your pilot, but that doesn't make sense but yeah. What I'm saying is that we we gotta think about these things as how like, if they really deserve more attention, then we gotta design around the constraints that we have right now, which are people are likely going to ignore those things.

C

If we don't do something meaningful to show them the issues.

A

So what is as we think about the widget, then, if the widget is not helpful in understanding the issue or if under even understanding, there's risk, is there a minimal change? We can make to that presentation. Just better understand. I've introduced risk to the code base or I've taken away risk from the codebase even.

B

I think that that goes back to what we were talking about before as including severity like as soon as we do that, and we kind of implement the same design that the sas and das reports have after we fix, which two reports we're comparing, then that would be very meaningful because you could see what happened as far as like instant feedback when secure implemented that change like they didn't change um the amount of violations that their widget was reporting.

B

They just made it red and when they did that they got a ton of feedback from a lot of different people around the organization and from customers about um that change, but like all they did was make it red like they didn't block the pipeline. They didn't do anything else. They just made it red. So if we make it red, but we also fix the report, comparisons and we won't get all the blowback, but we'll get all the the eyes on it. We'll get all the attention.

A

All right, so, let's take another step down the we've already implemented this path. It feels like we've already like six months ago or less uh identified what the problems are in making the code quality comparison, useful, uh the right commits and exposing severity.

A

So once we make that change and have the slightly modified ui, there's the hardest part about getting people to understand. It's no longer noise.

D

I think that's part of what the ui change accomplishes. Isn't it I mean if it's. If it's working with the assassin asked, then we might see the same thing with code quality that that might be the smallest change to make it actionable information, as opposed to noise.

A

Is just that subtle, ui change going to be enough for a developer to realize? Oh hey! I need to actually pay attention to this today versus yesterday, when I just skimmed over all of those widgets.

C

C

That's is that shaped that, like uh is that life, I I don't see it the the security affordances, the like the severe severity affordances that rikki was mentioned.

B

I can I can look back, they might have rolled it back because they did get a lot of pushback from people, because the report comparisons were wrong, so it would be like, like drew, is saying, there'd be a one-line change, that would say 16 new critical security violations, and so people were understandably a little bit got.

C

It yeah I mean if that, like really drove a lot like a lot of attention into it, I will say that that that seems like a reasonable.

C

I mean the expectation that I will have is that if we do the same, we get attention into it as well. So, but I think overall, there's uh other things that we gotta do at the merch request page level. uh The the interesting thing here is that we own most of those widgets right.

C

uh I think like it's like 70 percent us on fixing the visibility of that block and making it like more readable, uh and I don't have ideas I mean I I I could explore ideas, but I think I want to hear what people think about that one. One thing that I'm going to mention is, for instance, if you look at the pipeline, the coverage there is very clear right.

C

You see like what's the total amount of coverage that you have and if you're increasing it or decreasing it, and I wonder if we could do something along those lines for quality inside the pipeline. The merge request pipeline.

A

So that's a single score and we don't have that with code quality. Is there a simpler view.

C

Yeah, but we cool, I mean we could very much say, code quality.

C

I mean the things that we're saying about code quality on the widget. We could say those things on the on the actual could be saying those things on the merge request pipeline widget right, so we could say, like code quality, improve on 75 uh criteria and degrade it in 89 criteria, and then maybe we can compute the number on how like that's positive or negative. You know what I'm saying so, uh I'm not saying that that's the solution, I'm just kind of like suggesting that like.

C

Why is it that coverage, at least in my particular very biased case, works better in visibility than uh code quality right now you know, I don't know it's just kind of like I'm. Just formulating that question marina.

A

I think it's easier to understand a percentage and if it goes up or down, then 20 things got better. 12 things got worse and also all of the severity of them are different.

B

Because code climate uses a letter grade like they roll all your stuff up into your gpa and give you an a minus and you're like all right. Well, that's that.

C

In the for that, bishop uh three-year vision, uh issue that uh james and I have uh like created and that I'm working on it some of those designs, I'm actually suggesting that you know. I think that we should be more opinionated and say like this is an a. This is a b. This is a c. This is a d, or this is uh an f. You know like.

C

Perhaps we we could create ways for people to um adjust the sensitivity of those scores, so if they feel that they are working in a project that which doesn't have like very uh straight constraints, then they kind of like let that loose a little bit and then like the the score, could be a little bit more generous, but I definitely think that those letters uh help a lot uh to create a context around your your code base quality yeah. I think I'm ready.

A

Sorry, I'm gonna.

B

Jump this forward.

A

A little bit and say all right: let's just build a score and um let's make a super simple version of it. What's the riskiest part of introducing that or what's the hardest part of creating that simple version in our last couple of minutes,.

B

I think it would be that everybody would have a different interpretation of what that would mean, and so, if you were to introduce an opinionated score like that, I think you'd have to have it be either dismissable or extremely.

B

C

But what I'm saying is that most people like they are not gonna make sense of their improved or the great uh quality if there's no context around, if you like how how bad or good is that you know what I'm saying it's like almost like, if you tell me that, like my code, quality improve on 75 areas against what you know like that, doesn't tell me anything right now like it will be better.

C

If it tells me like you, know you your overall code quality is a c and you just made it like several points better and like you're, closer to a b. You know, I think it's more about the the idea of uh gradually improving that grade. You know it's like trying to score more and more and more when you're like in school, and you want to get better at something.

C

um The great is just like a very sub. It's it's very subjective, but it's more it's a better reference than just like mentioning that things are improving or degrading. So I agree with you. It needs to be very configurable or very well defined, uh but I think the value for the customer is that they have a reference. You know they they stop looking at something that it's vague and in a vacuum and doesn't mean anything to them. Right now,.

D

I posted a link in chat of uh just what code climate themselves provide on their free version.

D

If anybody wants to look at that, it's several different, it's maintainability, you talk about code, smells duplication, those different kinds of things, so I don't know was there one thing in particular we're thinking of trying to pull out of code climate because we can get.

A

I think it would be a non-code climate item, so we just talked about what's the quality as we've measured right now we have code climate as the underlying engine, but there's no reason that we couldn't have two engines running and scan the code twice and give you a better score or start to incorporate other signals as well in your code.

A

Coverage like um jj, had some great ideas around that around, like churn plus unit coverage or test coverage rather and then starting to think about hey your churn, plus your test coverage, plus how many of those tests are failing all the time. Well, you got low quality quote in there um collectively, because you're not actually testing it. So we don't know if it's good or not um so some some ideas there, uh and maybe there is a simple version of that. That just starts with the code climate engine um yeah.

C

So yeah that we can show, like the I could say, code, climate and sonar cube uh next to each other and say, like oh code, climate gave you an a and sonar q gave you a b. You know.

A

Yeah kind of, like you get like with your credit score, if you're in the states where you can see like here's, your experience score and here's your whatever the other company is score and see how those two things change, and you might think that you know different parts me should mean different things, but that isn't the case. Maybe we should take an opinionated approach, the first time and say listen. This is what it means to have high quality code and, if you don't like it well just turn this feature off.

D

For the controversial.

A

We've heard from plenty of folks that they want opinionated tools um and if they don't they'll, pick a different tool and they just they want to see a score and see how it moves it's more important to understand what goes into it, and then they can like weight themselves. Well, how much do we actually need to pay attention to that.

D

So make our baseline opinion as uh as what we expose and then, if they want to tweak it, they can.

A

Yeah I mean the core concept of gitlab: is make it easier to do devops and to have a tool chain if we make it easy with one tool, but then give you a million things to configure. Did we really make it any easier for you.

B

Yeah in when it's configurable, I think I still think that there's got to be some kind of like hypothesis, that once like most people, don't configure the thing yeah like the default configuration is going to be what's rolled with by most people, because they're like yeah, looks great good enough um and so art. So even if it is configurable like we should be opinionated about what the default is. I think I agree with that.

C

By the way, the configuration doesn't need to be like a crazy, like we can start with a very basic uh slider right that it says, like very strict. Not so strict. Very you know like that type of stuff, like that you move around and you can, because not every project has the same amount of like constraints. You know, like maybe you're working on some super secret, uh maybe you're working on like uh software, for like an airplane, you know, I think things that need to be very tight.

C

You know you want like the the the highest amount of strictness and everything needs to be perfect. You know, but if you're working on a website- for uh I don't know teenagers sharing something like whatever they share nowadays, uh then you don't. I don't I'm not saying that you shouldn't write good code for that. I'm just saying like maybe you don't need so much strictness.

D

C

D

Software industry right.

A

Yeah cool: well, we didn't quite get down to a single issue to write up, but I think we did pretty well validate that we already have most of those issues written in the realm of code quality um in the near term anyway. uh So, as we think about our next think, big and we'll talk about this in our group um group discussion later this week think, let's think about some categories outside of what we're working on currently um and see.

A

If we can't have a different kind of brainstorming discussion in a couple of weeks when we do our next think big, so open issues, upvote issues that are there, I will link to that in the video. Thank you everyone for your time. This was a fruitful discussion, though, invalidating that we are on the right track and in thinking about some of the small iterations that maybe we make even to those current issues of getting a cleaner view, more signal, less noise in the mr widget all right cheers. Everyone cheers thanks.