CHAOSS Common Metrics Working Group, 1 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CHAOSS.Common.Oct.1.2020

Description

CHAOSS.Common.Oct.1.2020

A

Ability, so if somebody can record that'd be great, it's already started awesome. Thank you. It is october 1st welcome to the common metrics meeting today. As usual. Please write your name in the meeting minutes and the link is here just in case. Anyone came in after we posted that um yeah. So let's hop into it. So first item on the agenda, review, action, items and notes from previous meeting.

A

So let's look down here and see if we have any action items so, oh yeah, the burstiness. Do we want to talk about that.

A

Matt g, you had the action item on that.

B

B

So we're starting to so the idea behind the burstiness metric is.

B

That certain times we may have bursts in issues or we may have bursts in pull.

C

B

Or mail list activity whatever it might be, so the idea here was to develop a common metric around that, so this is kind of as a result of conversations we've had in this meeting and some work that we've done kind of offline. So um we could spend a little bit of time, perhaps working on this metric.

B

Here, we've done this in dni before where we just ask people to spend five minutes or ten minutes kind of putting their thoughts around uh what burstiness means to them and how we might go about measuring it. If you want to do that, that would be great.

A

Yeah, I'm good with that, um because it looks like the other things that are on the list: the other two metrics, the technical force and language distribution, either sean or not, or here so.

B

Yeah sean won't make it because he's doing the grace hopper stuff.

A

Right so um yeah I mean, I think that would be great, that.

D

Maybe the feeling of the document we want to comment on, please I'll put it in there for you, armstrong, okay,.

B

I just put it in the chat: do you see that okay yeah thanks.

D

B

So this is, it's almost just like a brainstorming activity just spend a little bit of time. I think a lot of you have done this before just drop ideas in whether it's in the description or in objectives or how we might go about measuring.

B

B

B

A

Hey, I have a question. Oh maybe I'll, just wait till we're done I'll. Just wait.

A

A

A

E

E

E

E

Sorry I had to step away. Are we working on a homework assignment or.

B

We're all just super annoyed with each other. None of us are talking, that's the spirit.

B

We're working we're working on this document.

B

E

The last one there, okay.

B

So just any creative thoughts you have on this awesome.

B

F

F

F

F

F

B

Okay, we'll do this, maybe another minute wrap up your thoughts.

B

B

B

All right, why don't we stop? If you have thoughts, you can still keep adding them here. um Maybe for me, was there anything that anybody added that they're uns sure about I at least I found myself when I was adding things that seemed to be um pretty self-explanatory.

B

This metric, um so was there anything that was kind of weird or on the edge that somebody wants to to highlight here.

E

And I think weird I just I put one comment on the objectives around to help provide a point of interest when activity knowingly shows up, I I I was just trying to see just I wasn't entirely sure what we're saying is sort of the con like context around that, like I don't know, I just yep, I think that's fair.

B

B

Yeah, because I think this I think burstiness can kind of go two ways. One is we we're trying to improve something, and so we run an event or we do some sort of social media push and we want to see how that works, and then I think, but I don't think that's your point. John. I think yours is something does happen and we're not entirely sure what just happened.

E

Yeah yeah yeah. I guess maybe that's something.

E

Yeah and even just the term part of interest- I guess I'm just I I sort of get where it's getting after or maybe maybe I'm just like getting hung up on terminology. Maybe I don't know.

D

um I think elizabeth coming is very valid. I need some discussion around that the the one that I have highlighted right here at the top of the second page, uh the increase and decrease she has a concern concerning that is that in here, armstrong.

B

I think the first.

D

Highlight on on that description, oh okay, she's, confusing, burst bustiness to be increased, not decrease.

B

Yeah, I don't know that they're. Your point is that there's not a a decrease burst is that your.

D

Point well, I think, that's what she's uh she asked in her comment and.

F

D

It uh well, I know there are cases where we can have a decrease, but it's something that we really need to highlight so that people understand, even though the definition can capture that, for example, if we are doing squashy uh commit, there are conditions there are contexts in which, let's say an organization, a project joins and is contributing, and after what they try to withdraw, they have to remove most of their things. Then they will have to delete a lot of uh activities here. Okay, there.

F

D

A lot of other refactorings and other things that can happen within a project with repository that they need to do a lot of cleanup. I remember some few weeks back at evolution. We really sean and kevin. I think they were doing a lot of cleanup.

D

Okay, that would that would result to a decrease. It burst like a sudden decrease in activity. If you look at it at that moment, because things are going out, so um do you think that would.

B

Necessitate a different metric.

D

Or do you think.

B

D

Saying it's: okay to include both increase and decrease; okay yeah, so that I mean it's a valid uh comment that you raise and I think we really need to see things, because upfront people will think it's only an increase. I think that's where my mind was. It was only.

B

For an increase.

F

Yeah, so I'm not saying that that's.

B

The right thing, but I think when I was when I was working on this, it was only yeah because the essence.

D

Is that sharpness of the sudden increase with a sudden decrease? You know.

B

Okay, yeah so did um to that point. Did people make comments on this with respect to the decrease so like, for example, the what you just described armstrong?

B

Do you think that's captured in here with respect to a decrease. So I think, if I read a lot of these comments, I think everything is kind of oriented towards increase yeah.

A

For clarity's sake, it might because I think probably most people would assuming increase only um so maybe it is worth pulling it out into a separate metric. Just for clarity's sake, I don't know, I don't know what you would call it drop off or something abandoned. I don't know, I don't know, but uh thesaurus.

A

I go to the thesaurus, I love it, but yeah I mean otherwise. We will have to be very clear in our description and I just kind of caught my eye when I read that um because we don't really talk about decrease so yeah, I think either we add it and make it clear that it's part of it or we separate it out into a different metric.

C

I think burstiness is both a sudden increase and then decrease back to normal level.

D

Yeah, it captures.

B

Both so I'll make this as a suggestion, then it seems to be that the.

B

The push is to have both increase and decrease here. What I'll do is? I will take the comments that have been put into this document at this point and try to highlight- and I can bring this back in two weeks and try to highlight where I think we may need some more description with respect to the decrease component.

B

So I can, I can work on the text that always says increase and a decrease, but I'm guessing a lot of the.

B

A lot of the things that are in description or a lot of the things that are in filters are kind of increase oriented.

B

You know what I mean, and then we can come back to this in two weeks and kind of look at it from a decreased perspective.

B

D

B

I think it makes sense yeah. Okay, um great, I think that's an excellent point. um I.

A

I had one other question um that I wanted to bring up with regards to burstiness specifically- and I don't know if you all have been watching twitter, but the hacktoberfest obviously started today and there seems to be a huge influx of spammy kind of um prs right. So.

D

A

Completely screw up like metrics like this, so is there a way that we can allow for that. I I don't know what the answer is there. I just was thinking when I'm seeing all these people complaining about all of these, like bs pr's that are coming through, that, like that's, really going to script, all the all the metrics for everyone everywhere, that's where my head goes. So it's just something to think about it's just specifically with regard to this metric.

A

I think the burstiness, because that you know obviously, is like a huge spike and it could be complete bullcrap. So I don't know.

E

I I think, that's actually really valid. um I I sort of threw in somewhere in here social and media activity, which I don't know, that's probably a really broad bucket for talking about what you're going after, um because I think that actually could have a positive or negative, like you know, show up on hacker news. God knows what's going to happen, especially if it's something bad.

B

I put spam pr's, then I was going to put I.e hacktoberfest, but I don't think that would be good to put in a metric.

E

Thanks, google.

C

We get the same effect when we have google summer of code coming up and students are interested. They start to just show random activity, so.

B

Is it okay, do you think to include it as a filter like spam? Prs might not be the right word but like um like.

C

It's a filter like.

B

You might want to filter on when you're looking at burstiness. One of the filters you might want to apply is: was there this giant public event that has that pushes people, perhaps a little artificially to your project,.

C

I think that's the beauty of burstiness that you can see this and in oktoberfest they have this rule that you can label maintainers. Can they label prs as invalid or spam? So maybe a filter based on labels like invalid or spam is relevant here.

A

Yeah, I love that I think that's extremely relevant yeah.

D

But I think that also comes in when we want to see- let's say the quality of the bustiness, because then that would be a second kind of iteration, but just to leave it just to capture the events that let's say quazilan amount of pr came in. We we are focused with that number up front.

D

Now how meaningful would were they was. This response was this redundant? That's a different story, so see it's meaningfulness. Yep. I think that's well said.

B

So um maybe that's an objective, I think maybe armstrong.

A

B

To capture the meaningfulness, an objective is to try to capture the meaningfulness of increases or decreases in.

B

B

We can spell.

B

Okay, that's good any other comments from people that they want to bring forward. I had one comment: I'll go ahead: gerrig.

C

B

Okay, so my comment was: um do does this metric.

B

Should this metric lee, should we try to capture what a person should do about it?

B

So is this just a metric just to display the the burstiness right and apply filters and to the last point like we could look at the quality um and is that the end of the metric right there or do we say you know, based on based on this information here, some things you can do?

B

It was really just a comment I had with data collection strategies.

D

What's that I say I agree with what you said. It makes sense, for example, if we know that at this period of time we are observing a high volume of activity, let's say the release cycle. Is it worthy to put two kind of events that generates this kind of business at the same time that help us also to plan for traffic? And you know the project to know. We don't really like put everything at the same time where we will not have because yeah.

B

So let me I'll give you a different example. So, let's say kind of what john was talking about, where we have a burstiness and we have no idea where it came from. It just won one.

E

B

Through friday, like it was crazy, and so in this metric do we give people the tools by which they could try to investigate? Why that happened, or do we just say, burstiness is just about seeing that weird week we're done. We meaning the metric. We're done. You know so like uh if you had a week. If you had a week of crazy activity right, you could one option would be to reach out to community members and ask whether or not anybody has any insight as to why this might have occurred.

B

You could reach out to maybe any vendors that are using your tools to determine if there was some sort of push in the in the in the for-profit space that has increased the number of people who are drawn to the project, I giving them the tools to try to identify an unknown bursty period, or do we not care about that.

C

So in other metrics we usually stay away from giving advice on what to do about it. But in this case I think it's part of the collection, data collection strategy.

B

So maybe I could qualify it so I could say something like I'm on the top of page to three. So in unknown cases of.

B

D

Follow-Up questions could include a concern like to classify the bustiness or to give interpretation of it.

D

I do you suggest we give interpretation of it or to classify it.

B

No, it was more like if you have an unknown period like a week yeah like do we give advice on on just what to do next, not classifying it, but.

D

Then I think what uh georg just mentioned, we tried to separate the syntax of the metrics from the semantics, because what people actually do with metrics can be bigger than with. We think.

F

D

Yeah yeah, so we don't really go like implementing it and locking it in our understanding. Here we capture as much as we can. We can filter it. Then we try to because once we filter it, the meaning will start surfacing out, but the interpretation of that meaning. I don't think this is what the we should be. We should do under the same methods.

F

D

Because if we even take into consideration when we are capturing other matrices for let's say different reasons and purposes going into interpretation, more than one matrix now comes in to really see a bigger picture. Okay, so then.

B

Your suggestion is basically just stop to say: we can identify a.

D

Bursty period we can do all the classification could make the filtering and everything to really see. Was it really like a true positive or a false, positive? Okay, then, what really happens like decision will be something else agreed.

B

B

Okay, okay or you had something else too,.

C

The the inspiration for firstness came from a paper that I linked in the reference section and they have some mathematical formulas.

C

F

Just wanted to.

C

Ask your your opinion all whether we should maybe formalize burstiness, maybe have a formula for in the paper. They use inter-event intervals, but that might just be too technical.

C

So I I'm torn offering he you can actually calculate burstness as a number versus confusing people with that information.

B

Could you repost the link? I can't.

D

Whatever you know, the the what I just opened, the pdf right now.

C

It's at the bottom of the business.

F

C

F

But I see it's missing the f and pdf. I just posted it on the chat produce was listening.

B

F

B

Well, I'll put that on my to-do list, too kind of look through here. I know what you're saying here like how much detail you know so, like one of the things could be like.

B

Mathematically.

B

Mathematically calculate birthiness in accord with this paper, like we don't actually put it in the metric, but we point.

C

But that would be one way to that. We don't add, like we had at the beginning, sql statements in our metrics. I know we've gone away from that.

D

Well, I think essence here is just trying to see that this the kind of distribution and then to try to make meaning out of it, because each uh activity might not follow the same kind of distribution.

D

Well, like the paper, I'm just giving a a slide through, and I see like it's a good thing. They are using this mathematica tool to describe you know to the procedure.

D

However, we can also come up with a certain kind of event that let's say follows a poisson, a binomial, a normal or parametric or non-parametric kind of right.

D

So in my then, this equation might not necessarily play upfront, but it still gives a picture that okay, if it happens to person- and we analyze it this way, they want to see the peak.

D

The rest is just statistics.

D

Which I agree with, but then implementation it's. uh If we keep it simple, we with the math already like backing our activity. Then we capture.

B

So it's this paper, I know you, just I'm sure you haven't read it. I'm starting.

D

But I read the paper. I just glanced through to see what the like the key points that they were arriving driving at.

B

Okay, I'll take.

D

B

Too was I I thought I heard you saying that one of the things that maybe I misheard, but um that burstiness in one situation, what looks like burstiness in one situation may not actually be burstiness that it might be following a normal.

D

B

D

Because if you see like they are talking more about like the gaussian, the poisson, that's this more with the distribution.

D

So if you are talking about first of all, distribution in most software engineering contexts happens to be non-parametric, even though there are good cases where you have normal distributions and different kind of distributions.

D

Accepted for a conference of being published, they really need to convince the implication.

D

So this month now comes in to strengthen what the story they were telling now the math is there: the paper is there: we can just go ahead and implement because we are backed by the formula which is correct right, but in a certain context like what I was saying, we may observe- let's say burstiness during a release circuit can give us a different distribution.

D

We can observe both this in a different context. It gives us a different kind of distribution. All will not necessarily give us the same kind of distribution, because it is boston that makes sense.

A

Hey matt to your point earlier about um doing like a what's next kind of exercise or or thoughts. You know expanding on those. I think if I recall at some working group meeting, we had talked about providing some kind of guide to go along with the metrics on, like just general, interpret, not interpretation but kind of like how to how to do that. How to what like. So you see this you've taken the metrics.

A

Well, then, what do you do with them and just kind of giving some thoughts and ideas of like what what people or how people can use them to their benefit and if they do want to make changes, here's some ideas and like taking it a little bit step. You know a step further, instead of just like handing it off here, you go. Here's some numbers have at it. You know it's it's.

A

I don't want to say it's irresponsible because that's too harsh, um but I think if we are genuinely talking about improving communities, health, then it's like you, go to your doctor and your doctor will say: oh you need to exercise more. You need to do this and that and that here's some strategies you can use to improve your numbers. They don't just say here's your blood pressure. Good luck!

A

You know, so um I think that that's a bigger, obviously a bigger conversation, and I don't know if we all uh if anyone else has interest in doing that, and I know we've we've talked about not being seen as kind of the experts in community health, but at the same time I think that uh general guide or oh what's next would be super helpful for people who are just looking at reports like I don't know what these mean. I don't know what to do with them. I don't know how to make them better.

A

You know, so I don't know if it's something long-term, we want to think about or not I'm not sure, yeah.

B

I just shot it down, I mean it. It almost seems like that would be like another heading for every metric and the heading would be something like what what to do now, questions to ask next. um What to think about now that you have this information, um I think it's a nice idea, because I still think that can be value free.

B

You know, because we're trying.

C

B

Say, as you know, like good or bad, but now that you have this burstiness data, what do you? What do you do with it? Well, you could investigate a little bit further if you're uncertain why the burstiness show up showed up. You could um to armstrong's point like try to keep if you have too much going on, try to keep two events a little separated from each other. If your goal is to try attract new contributors, you know think about your spacing and your timing in your community.

B

um I like that idea. So it's probably a bigger conversation, maybe one that we could bring up in the community call, because I think it would require every metric to go back and add that section.

B

A

Okay, I'll bring it up next week. If we have time put it on there, yeah just see what people think cool.

B

All right, well, that was that was great- that took the entire time.

B

B

um So any my action night will be to to go through these and, like I said, try to try to clean them up and then think about them from a decrease perspective, because I think a lot of the conversation at least today, or at least a lot of the work was what I saw was thinking, increase wise and where we might need to um fill some gaps with respect to the decreased narrative.

A

Thank you. Everybody.

F

A

B

So I don't know what else you want to do. Elizabeth in terms of.

A

Time um so we have the only thing uh or the only other two things are just to review, prs and issues and then review progress on the metrics spreadsheet and when we have what seven minutes, so I don't know if that's enough time to do either of those things. I.

B

Had I prior, I had looked at the issues in the prs and I was looking at them actually from a hecktoberfest perspective, seeing if there's anything we could label that might be straightforward. I didn't see anything to be honest.

B

There was no like good first issue, um we've been getting our hecktoberfest comments, have been. We labeled one in the update the website for grammar and we've actually been getting quite a bit quite a bit like a couple on that which is helpful.

B

um So the only thought I had was do we want to have an issue in here. That is like um double check: grammar on common metrics.

C

I don't mind anybody.

B

So maybe we can just make an issue real fast on that.

C

Yeah and link out to the released metrics yeah that we have.

A

Yeah, that sounds good.

C

You could do that for all the working groups.

C

B

Okay, um I don't want me to make that issue.

B

Can you somebody track down the link to the latest release I'll, make the issue and then I'll paste that.

C

Link uh it's chaos.community metrics.

C

Are you looking for something else? I.

B

Was looking for the github link, where it's pulling from for common just so they can do a pr there.

C

Okay, I think we need multiple uh links, one for each metric, because they're all in master and for someone who is not familiar with how we work wouldn't know which ones are released, which ones aren't that's too much to ask.

B

Can you compile okay, just put it in the chat.

C

Or you can create the issue I'll fill that in.

B

A

Okay, so, while matt is doing that, um do we have anything else to talk about um again, we have four minutes to go through the spreadsheet, so um I don't know if that's really enough time and technical force and language distribution we'll just wait till next time to hear from bernard and sean. If I don't think anybody else has any updates on those do they.

A

Not really okay, um yeah final thoughts.

B

I can't do two things at once, so I'm gonna stop the issue. No, I have no final.

A

Thoughts, I cannot do it either. I.

B

Just literally can't do it, I can't talk and.

A

I can't either if I'm driving and someone's trying to talk to me, I'm I I like will pass the turn that I'm saying yeah it's hard right.

B

I did that actually, the other day I was on the phone and I totally missed the turn. I was like. Oh my god, I'm going down the highway and the next exit isn't for like four miles. What's going on here,.

A

Totally totally oh gosh, okay, all right! Well we'll go ahead and close the meeting then.

B

Okay, I'll stop the recording.