CHAOSS Risk Working Group, 10 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CHAOSS Risk Working Group 2/10/22

Description

Links to minutes from this meeting are on https://chaoss.community/participate.

A

And I will also share my screen so that you can see the agenda I'll, make it narrower. So it's easier to read on laptops.

A

When we last met, we were working on a new metric called defect resolution and by the way, sophia sends her apologies she's driving today, uh and that was that was kind of where we ended up and kind of putting off metrics reviews until we other than the one that benad was going to help us change the name of so um defect resolution time was the one we were working on.

A

I think we might want to either just spend a little bit of time working on that right now or see. If there's an additional metric, we want to try to add prior to the next release,.

A

That was my requesting people's thoughts on the matter.

B

The agenda looks fine, let's just go dive in.

A

All right so we'll just dive right in to issue resolution time. I know we had a good deal of okay. I understand google's new security settings changes. um I have to make this a little wider and I don't know why that's there all right is that visible for everybody easily on your screen.

A

Okay, so I suggest um we just read through it because and and see where our our questions are. Maybe just spend a little time editing. It.

A

I had them put prism in my glasses and it's the perfect distance for my home computer and I have to figure out the perfect distance on my laptop at work. That's why my head's bobbing back and forth.

C

So how are we going to deal with the case of I easily handle defects by rejecting all proposals and just not doing ever fixing anything which is, I think, one of my concerns on this.

A

Well, I'd say: you'd get a defect resolution time of never or some extraordinarily long period of time and.

B

Until you get it effective, you get you get zero divide by zero effectively.

A

B

Yeah, all over accepted off the resolved immediately.

A

Correctly, if I'm wrong but divide by zero, would result in a non-real number.

C

It's actually undefined, if you really want to.

D

C

Into this stuff- and I am.

D

D

C

Formal everybody needs a hobby. One of mine is formal mathematics. I apparently need a better hobby, um so.

A

I need to have you guest lecture in my string, algorithms class.

C

Oh well, we'll talk about that later. You better pay me better.

C

All right, um no, this is the this is very much the dancing on the head of uh what kind. Let's talk about different kinds of infinities kinds of stuff, um and uh do we accept the axiom of choice today, um all right so anyway, getting back to the the main point, though,.

C

Yeah, so it seems like you know, the overall is measured the median time between the accepting the one thing is: what do we do about? You know the ones that don't ever get through.

C

E

C

I I don't have a good answer for this. That's why I'm asking the question it's what I'm worried about, and maybe maybe this is an excessive concern. You just measure it and acknowledge that new metric is perfect.

A

Well, the way I'm like the way, I might explain that to my software engineering students is, I would use use case language and I would say that that's a failed end. State.

C

Okay, so don't bother to try to capture that within this metric. Just this is a metric and be aware that.

C

You know- maybe maybe it's just a note, then that note that a project could choose to just never accept a defect report.

B

Now, right now it just says accepts and then uh you know when it emerges and and they could be missing.

C

But I would find a glasser yeah. You know what I think. That's actually the wrong time between the the report of a defect, not the acceptor report, the report of a defect. What I meant when I was saying accepted, was that it was received and.

C

E

F

C

The receipt of a report of a defect.

F

I received: do you mean it? It went through some triage process with the maintainers and no no.

C

Just it was, in other words, whining within your org, does not count sending a report to the organization that develops the software counts.

C

So I would say when the issue, if it's on github, for example, when someone pushes the button that sends the issue to the project, the start because whining about it within the user community doesn't count, it's gotta actually have been set, because otherwise I mean people whine everywhere. It's gotta actually have been sent to the supplier. Who can do something about it?.

C

So maybe we should just say the meeting time between the report of a defect to the project. I don't think we need the word accepted. I may have even been the one who suggested that word.

B

Yeah, I think the word accept I take the word, accept it out.

C

Yeah yeah, but I think that's actually causing more problems. You know.

E

C

Report of a defect to the project using the project's defect, reporting mechanism- uh where's the url here, just a second- I see it.

B

How about the adjustments I just put in?

B

I just deleted the ear except since you said, if you want what does it mean of a defect to the project using the products, defect, reporting, method and the time where the project resolves the defect, I use the term resolve so resolution could be accepts and make available to users or explicitly chooses not to address, reject something like that.

C

C

I am on my way towards this document.

B

Is that actually a failed case? Sure.

A

Well, um so I think the fact that the project never gets resolved as a failed case, and I think it's up to the group music to determine the impact of that failed. Maybe failed is too um valence to word. um Maybe it's an alternate end case.

C

Now we've got a uh oh sorry go ahead.

C

I think the problem is made available to its users. Also, it doesn't have to be through a a formal release right just somehow. It's probably.

A

Able to get to it, I knew we had the discussion of this last time and I couldn't recall it so.

C

Okay, so all right, so I want to add that to a note um near after some of these other notes, note.

C

um A d effect repair can be made by by through a formal release, release or simply by publicly posting the source code change necessary.

C

Many many defects only affect specific users are, are only significant.

A

Now yeah I mean, I would say, that's a function of the size and scope of the software right.

E

A

I can see where there might be zephyr defects that only affect people using certain hardware.

A

I don't know if that's the thing that happens, kate or not, but I tried to find an example other than kubernetes.

B

C

Would say with pro right and it's in its main line- maybe it's just in it's in its main branch.

B

These things could be in yeah, I don't know if we want to go into the main branch aspect.

A

I think making it available to the users is sufficient. Yeah well,.

C

Because because there's a lot of patches that clearly won't work or won't ever get merged in, I hate to fix.

B

Well, so the question here is you're you're. Presupposing that there's going to be a fix- and this is a resolution- it's not expect fixed time. It's a defect, resolution time. So a a valid resolution, in my mind from that is it could be. We could be rejecting and choosing not to address it as a project.

B

The thing is leaving it in the ambiguous state is a fail case which just sort of sits there and lingers on forever and ever.

C

By houses, by accepting by rejecting it or by by rejecting it or by accepting a fix in its main branch, would that help.

B

No, I don't like to say, I don't think it. I don't think it's a read my words there. I thought it was being.

C

uh What words I'm.

B

Sean's moved on beyond it, so that's why you're confused you need to be in the document.

C

I am in the document where, in the document.

A

Yeah well I I was trying to sort of put some detail behind the discussion that you're having down under the objectives.

B

A

I think that's what it belongs, but yeah. It's.

B

A where the project resolves the defect period and.

A

B

Note the resolution could be to accept and make it available to users or explicitly choose to not address reject, so the what was put in brackets behind there, I think, is redundant. The first president with the note, because I thought we were going to do it as a note.

A

So I think, because I think, there's defects that represent some kind of security or vulnerability threat, but there are also defects that may pertain to performance or addressing specific formats of data that the project chooses not to address.

A

So it's a defect from the perspective of the reporter, but the the decision like if a decision not to address it, causes no harm, then it's reason it seems reasonable. I think I don't know if we need to delineate or want to go down the rabbit hole of trying to delineate between not resolving software vulnerability. Defects, probably not lots.

B

Of bugs books are bugs, but, like I say I think it's it's important to understand resolution that does not equal equal fixing. Okay,.

C

So so I've tweaked number one here and and that that yes well number three. I think that one actually, the new number three is actually already covered. I think in the text we just added earlier set. I concur.

B

Someone want to go through and accept all the changes, so we can see it as one whole thing and then do another pass literally see it holds together still.

A

Oh somebody already is like checking boxes like crazy, I'll work from the bottom up. Then.

D

Okay: let's go ahead again.

A

I think we're almost there.

A

Yeah, you can always tell because the width re-centers itself, when all the defects or all the edits are resolved.

C

Should we have a note about the uh the issue that you know if you never, if a project just rejects everything uh this this metric will not be. Perhaps what you were hoping, I guess a pr a project.

B

How often do you see that.

C

Practice may appear to have a rapid response time by rapidly rejecting all defect reports as if they are not defect. Reports.

F

Is there a way to track that uh rejection, maybe in the future factor that into this resolution time if they have a lot of rejects and that.

C

The problem of this practice, the the problem is, of course um there are there are I I see this in some of the uh supply chain stuff, like with amazon, where people create garbage. um You know comments, so just because somebody gives a comment doesn't make it mean it's true.

A

Just because somebody makes it like when somebody files an issue, it doesn't make it true. Yeah.

F

It's not working for me.

C

That's right, you know, in addition, um malicious commenters.

C

Can create a large number of spurious.

C

uh Defect reports, uh those situations, those situations.

C

Yeah, so you so basically there are ways to game this on both sides.

C

Yeah, if everybody played nice and did what they're supposed to there's no problems.

C

So I don't have a simple way to solve these problems, but at least we acknowledge them.

A

So when it comes to measuring.

A

Yeah, I think these glasses really do not work at this distance at all.

A

um In addition,.

C

I I just um somebody just made a change in point three, and I do not understand this grammar.

A

C

What what is this.

A

C

Is this if they are not defects.

A

Yeah, as a project may appear to have rapid response time by rapidly rejecting all defects reports as if they are not defects. Okay,.

C

All right, I didn't understand the recognize. I mean okay, the.

A

The redundancy of defect reports being recognized as defect reports was troubling the instance installation of grammarly. That is now in my browser. I have to learn. I have to ignore grammarly very much, but sometimes.

C

A

Not sound illiterate.

C

Yes, but I would say I'll listen but do not slavishly obey, because uh I.

E

C

Never want to be part of the elongated yellow fruit school. Are you still at that school? I'm, not! No! uh It's the school! That says you can't use the word banana twice so the second time it's an elongated, yellow fruit and I'm oh.

A

The whole banana would not work, then that's.

C

Right, that's right! So I'm a believer that if you're talking about a banana, you can use the word banana twice or even three times in the same sentence. It's allowed.

C

C

They aren't well technically. Reports aren't the defects as if the reports do not describe the defects.

A

C

So it's technically the report.

A

Really reports: it's like the reports, don't describe defects that are recognized by the maintainers as defects, so.

C

Yeah baby, I don't think we need to go through that. Just okay.

E

C

Rapidly, they reject all the defect reports as if the reports do not describe defects, I think, gets the point.

A

Does that really happen like just reflect really.

C

Absolutely you know hey if you want to make your metrics, I I have seen more than once the there are. If you have metrics, then you game them. This is the problem with metrics. You often get what you measure, not what you wanted.

A

Yeah I'm stupid, I just I'm honest. Some issues are open because I want to remember to go back to them when we have time some pull requests.

F

A

Open because I can't resolve the conflicts easily, I think.

C

Most folks want to be honest, but the problem is when you start getting paid whether or not certain things happen, then the quality.

A

Problem I've encountered yeah, okay,.

C

Yeah, obviously,.

A

C

Terrible things can happen how's this. Can I say it: can I leave it there.

A

Yeah, let's leave it there all.

C

Right, in addition, malicious reporters can create yeah. So basically, I I think this is good, because we're acknowledging that certain bad practices can happen uh and just warning users of metrics users.

E

How about the outlet like filter the outliers like as sean mentioned, he keeps a bug open for a time being so that they can be resolved later.

C

Right, I think median helps with that. If we want to talk about that, um you know they're. um Maybe we should, I think.

A

C

Fair point: uh there.

A

Are often outliers I did listen to, I did list under objectives that you would want to look at various statistical measures. I included mean median, but there are others um all right.

C

Well, maybe we should talk about it there, um but I I think I I think the point he raised about the outliers is actually sensible. um I I'm happy to put it in there. The objectives.

A

I think it's I don't know.

A

I think it is more of an objective or maybe it's a way of an implementation, but I think I think, if I'm using a defect resolution time, I'm using it two ways: one to understand how a particular project resolves defects, how quickly, but then also I want to understand within my portfolio, whatever the scope of that is which of the projects are quickly resolving defects and which of them aren't so there's I think an individual project, maintainer or contributor view and a like an ospo view or a foundation view or an enterprise view of some other kind that really only this is really most important to them in an aggregate form like they're, not going to look through 11 000 projects to see the defect resolution time, they're going to want to look for the outliers.

E

Or even like, sometimes I read a paper theory of open superposition when, where you deliberately delay the bugs so that a certain work needs to be done, then it will get resolved like you delay it so that other other modules are developed or fixed. Then you go to the bug specifically.

A

That's that's the case where you have a very large pull request that fixes the defect, but there are reasons that you can't close that pull request because it affects all these other things.

E

A

Are there any other alternate end cases just other than a project determining that they don't want to resolve or that the defect does not need to be resolved?

A

um I don't think we want to call out metrics gaming in the definition of a metric, but maybe I'm not.

C

I I think we should I. I think that the reality is that metrics can be gamed, and I think it's I mean I don't care where it is, but I think we should acknowledge it somewhere as an issue and- and we want users to look at that- I think that's entirely appropriate.

C

Now I don't care if it's under objectives, I'm not sure it's an objective. I just put it there because that's where you pull.

A

The inspiration.

A

C

A

B

Around gaming, yeah probably just give me the notes at the bottom.

A

So both the uh like just the whole alternate end case, section.

D

Yeah that works.

A

Is there a notes, no there's a notes about.

A

This would be a new section, maybe.

A

It's really not an implementation note. It's really a note about practice.

F

Something like considerations, metrics might be skewed if project these sorts of things not quite a risk.

F

A

B

F

A

Maybe it's defect resolution tied to a code change.

A

Or pull request is a filter and.

F

F

A

I don't know if that, uh if that is getting at, I was looking to try to put the text in a place where we had a place for it, and there really isn't a notes.

C

Well, we put notes at the top right underneath uh underneath the description. Note one two three.

A

Oh yeah, very good.

C

Copy cut paste, I was.

G

C

Okay, I'm now confused for different reasons.

A

I think it was me, I'm not quite sure where to put it and failing to recognize the notes section at the top.

C

I mean it's perfect, I don't think anybody will be shocked that a description section has notes.

C

Notes, one two: three, that seems perfectly reasonable.

C

A

I can certainly take an action item to put a visualization of this in.

A

I think it's fair to say that uh for more lab and uh you could calculate this with either gremore lab or auger.

A

um And then there is the the filter this this special. This.

A

Note about labels and github, which I think I think that's a filter.

B

So yeah I'm considered that way. It's not part of the you know. Okay, it's a way of classifying issues, variations.

C

Yeah, I do feel bad about not at least mentioning some other of these issue. Tracking systems.

A

I put get hug get lab bugzzilla.

C

Yeah, but if like, if we talk about like labels and gut have, then we should mention that you can.

A

Do things and other I'm doing it with I put labels in github, gitlab, bugzilla or any other issue? Let's.

C

See here, good old source.

A

C

A

That indicates the.

A

Issue represents.

A

A software defect is it presumptive? I guess it's not. I was I answered wrong question. I was thinking if, if it's connected to a pull request, for example in github, is it presumptively a defect, but no in fact, issues get added with pull requests at the same time in the same way,.

E

I think the this point in the filter and the one above it both are seen like. Maybe they can be merged any issue that is labeled as a defect or label in the github or any issues indicates.

A

Oh any issues labeled as a defect.

C

Yeah, but by the way, just I I just yanked up um good ol sourceforge, which has been around forever. They have ticket, they call them tickets and bugs are a different ticket than support, requests, patches or feature requests.

E

So is it just synonym, I'm sorry, is it a synonym.

C

Okay, a ticket is a synonym for a github issue, okay, sort of, except that pull requests are patches and they're also tickets.

C

But my point, though, is that for the point of a defect report, a bug ticket is what we're talking about.

C

And let's look up bugzilla, so some of these tools have this built in it's about github issues I think, is actually one of the weird ones. Most of these tools have a built-in mechanism to differentiate between bug, reports and feature requests, and you.

A

Know I get her, you know github doesn't because people were terrible at making appropriate classifications. I.

C

I you well, you know what I I think you I I think you can, but you can handle that by having a single issue trick system, and then you toggle that field. But the fact that there is no field means that it's harder to do the very analysis we are trying to handle.

A

Right, we have to use ultimately some bag of words. We have.

C

To use a bag of words and, more importantly, there's nothing built in that enforces a particular way which makes it hard and while back of words.

E

C

Of words often doesn't work, because if nobody, if just the humans, know the difference, but they don't market, you know great. We have to bring up a machine learning. You know you know a now. You know textual analysis tool to do our best. Good luck, I mean I bet this is probably not bad in terms of I think.

A

I think it's worth yeah. I think it's a case where usability won out over perfection.

C

I'm actually dubious about that it just that's what they chose to do. I'm not sure anybody thought that hard about it, uh because it's not clear if I'm a if I'm a developer, I'm going to want to filter out the bug reports versus the um feature requests seriously.

C

So I'm not even convinced it's an ease of use issue. It's just it is what it is.

A

Yeah one um yeah: this is okay, so data collection strategies um effectively.

C

Maybe many tools build this in some, don't.

C

Hey, I think we are at time.

A

Yes, we well five more minutes technically, I think, but I'm good with, I think we've I think, what's missing are the visualizations and some hard pointers to the software implementations, and I can take that as an action item for for our next meeting.

E

So sean what is at the very end, uh I think that's just.

A

uh Templated guidance about the the um the document formatting.

E

I don't think we keep it in the template here. We know.

A

E

A checklist in the when we create a issue or submit a pr yeah.

A

I don't want to remove it till we're ready to release it, though, because then I won't be able to find it.

E

ah Okay, that's fine.

C

E

It's not a part of the template or something.

A

Right, it's just more like instructions.

A

I'm just making a note to get the visualizations and then get the.

E

Or or if you want just put the visualization and notify me, then I can create a pr and every everything for the entire process. Yeah.

A

I mean what I was going to do is just grab one from gramor lab and grab one from auger um and point people to the specific places in each tool where, where one can find this information awesome, we got a metric done. I think that's great. We didn't bait the nature, meaning of the word defect for 45 minutes, so huge win.

A

As long as you're with us david, I know we have that chance. Oh.

D

Oh through the heart.

A

It comes from the heart david.

C

A

C

Right into mine anyway, thank you very much.

E

And I have wanted to discuss uh david like uh we are changing the name from uh ci badging to oss.

E

Is that image is still valid or not.

A

The badge itself that would go on the project.

C

ah Funny you should ask, um there's actually an issue. We have uh some draft logos.

E

C

So I I was waiting.

E

Whether we need to like, uh since the release is not ready yet so once the image is finalized, I can then take the entire thing.

C

I would say change the name now. The logo change is a separate step, but the name is already done so anytime. You use text, say the text now you're right, there's going to be a new logo, um you'll be shocked to know that people argue about logos uh story at 11. um I mean there's, it's all good, it's all good, but um frankly, that I have many other things that are going on my life and I'm. So I'm not rushing on that one. We.

D

Know it's going.

C

To need to be done, it's okay! If it gets pushed off a little bit so.

A

The time that was spent identifying the auger logo blew my mind. I'm, like oh yeah, pick a logo, but I had creatives on the team and they had to give it so much thought.

C

All right, I I I will quickly for your amusement if you want, if you want there is, um there is a uh url specifically on the on the issue of the logo. If you.