CHAOSS Metric Models Working Group, 5 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CHAOSS Metrics Models Working Group 3/29/22-3/30/22

Description

Links to minutes from this meeting are on https://chaoss.community/participate.

A

All right, this is the metrics models working group for march 29th for me march 28th for some some of you uh here on the meeting. So um all right march, 30th not march 28th march 30th,.

B

Yeah much larry.

A

It's been a long day, I'm sorry! How does time work? I don't know um yahoo, I'm gonna, let you kind of take this and run with it, because I think you want to talk about the community activity that you.

C

A

B

I can share my screen.

A

We are seeing your slack. Oh there you go.

B

Yeah, actually I just I sent this doc yesterday on the slide channel on the matrix matrix model channel and- and today I would like to go through the whole commun community activity. This talk quickly and then we can. uh I I I would like to demo it uh on this dashboard. I I we set up this dashboard through the open search dashboard together with grimlock, then we can see how it works to find some some interesting points through this metrics model.

B

Together with the data we collected for these two communities, I anonymous the community committee's name, because because today we are not going to to say discuss which, which community is placed better on orbit of forwards, just to say uh to find out how it works. I mean how this activity matrix modem works to base down these based on the data refreshed from these two communities.

B

B

B

How it uh why it matters so uh community community activity just describes how it, uh how how it uh active of this community. We set up uh four youth stories based on the different personalities for the community manager, so I I want to investigate what it should cause would abide to make activity increase or decline or as hospital from a company.

B

I am I'm looking for a community to set up this relationship as a partner relationship, so I want this company more active, so I can get quickly feedback if I want to fix some bug or to create some new features and also for the individual contributors, I'm looking for some active communities and which means my, but the contribution made by me is more visible and also I can get quickly feedback support if I was a freshman so and also this is united, so also as a community manager of this project.

B

I hope there was a simple way that would help me quickly identify how active of this project- and I pick up the metrics from below matrix of of the chaos.

B

I can see that's totally uh six metrics and also because uh for for something for some single chaos, matrix, there's different data collection strategy so- and I I define that and with the clear definition of the data collection strategy, to make it more clear how to how to implement it, how to implement it in the in the groom, live or in the future. In other data collection uh platform like in augur and in some other platform.

B

So here, for example, contributors is from our matrix common metrics and there is a data collection strategy id one about contributor count. We consider the commit author review participant issue officer issue commands punishment in the past 90 days they so these shows have have to be played, turned play these roles as active uh people in the past three months and also there's another uh strategy: data collection strategy from this contributor magic. It's a maintainer role, so I want to collect the average number of mentor per level.

B

If this community has you know as a multiple reports, I count this number if there are only one uh repository in this community. Of course, I can only get this one. Ripples, maintainer.

B

So similarly, like other matrix code change, commit and this data collection strategy is also in the past 90 days, every average number of commit, so I'm not going to go through all the metrics. I would like to see your comments in the following days. If you have an intro, if you have, if you have an interest on that, your comments are welcome, and so in the next chapter I create this accredited table for about data collection strategy. It's a matrix name, so it's totally a certain matrix and each matrix.

B

I give the their clear data collection strategy definition and each uh that uh each metric I gave them weight. So it's really based on the um and organism uh hp.

B

um And then an artistic hierarchical process, it's uh it's been used for almost 50 years. So it's a very popular uh analysis process and to to say: if there are more than two and uh uh indicators we will to uh to to say the relationship between them.

B

It's very it's a very popular algorithm. So we use this to to calculate the weight and also we define the threshold for the each finger metric.

B

Finally, we use this 13 matrix to count on the value it's a number between 0 and 1, and also we get this value for each week so, which means we can see this value changed based on in the time trend in the time, trending.

B

um Alpha means the weight and the isoi means the value we got for for each of the single metric and ti means the threshold.

B

Okay, I think we can go through the dashboard.

A

I have a quick question first, um I noticed that so some of these well a couple of comments before we go on. um I love that we have these new metrics and um we had the same issue in the evolution working group when we started to work on code complexity as a metrics model, and we realized we didn't have an actual metric for just lines of code, so it was just like a static count of something very simple and high level that we just didn't have a metric for.

A

So I think this is kind of identifying and surfacing some of the same kind of metrics that are just baseline, like how many contributors do we have at this point in time and how many contributors do we have. You know 90 days from now, so um I love that, and I think those are a really good metrics that we should have and we can knock those out pretty quickly like we can develop those pretty quickly. So thank you for doing that.

A

um The second thing I just want to ask about is when you're creating this formula and crunching these numbers, some of those meeting or some of those those metrics. The meeting count and the meeting attendee count like those will be something that somebody will have to get manually right. And so how does that like? Where do those numbers go, would would that person who's wanting to run this metrics model?

A

Just do that physical work of counting, how many meetings and counting how many people were at the meetings and then plopping those numbers or plugging those numbers in somewhere into a database or like how would that integrate in with the rest of the trace data.

B

Okay, actually we in the green lab, we have many different kind of background to support us to tr to retrieve those data from the different data sources, for example the male list. The melody is supported by the hybrid katie hypocrite. Is you know a soft software to to archive the mail?

B

So so we provide such background to automatically to fetch the data from the hypocrites and to do the data analysis and the story store the analysis data into the database and finally, we use some different kind of dashboard to to make the visualization, like kibana, supported by elasticsearch or open search, dashboard supported by open search. Of course, many other things like grafner yeah,.

A

I was cause- I was just thinking if we were to run this metrics models on the chaos community, like all of that, we have that data, but it's not anywhere that's accessible. So we keep track of. You know how many people come to the meetings, but you have to go back and like look at the documents and add it up manually. So I just.

D

Wondered if that would plug.

A

In so thank you this.

D

A

Fantastic, this is really really great work. Thank you. So much.

B

So, okay, right now I'm gonna go uh demo it. uh So, first of all I can I can introduce some background.

B

So we pick up two projects or we can call it two communities, and these two communities have uh the similar uh they produce the same software, and so I I can see that actually, these two communities they actually the the linux open source system, uh sorry, linux, operating system, distribution, communities, so project b, has a longer very longer history and the emotion they he has. It has more than 10 years history and the project b just created like a two or three years ago, and so and the way for each of thingo project.

B

You know for the operating system, uh distribution uh community, they have many many uh loopholes or repositories. We call it. We call the pack packages to to build up. You know to build up the whole operating system, so we pick up the 20 most important uh software repository in each of thingo uh community and these 200 repositories. They have the similar they have the each the same name and and the same functionalities from both of these two project or communities. So, and so this this this 20, this 200 loopholes is our sample data.

B

I can see that for the for the each of thingo project and first we can see the the general the final result.

B

We count this the matrix model activity value each of single week and if we, if I disabled project a and if we just.

B

You can say the project b has a two version released every in every year and almost a half years. They release the one relation release and- and you can see it's very simple- that they have that you know very regular uh peak and the valley- and it's a very we.

B

We can see that this community is very motor communities and if we look at project a I disabled project b- and they can say that um to see that this affects this community set up like two or three years uh more than two years and at the beginning, it's this community activity. Is it's low, of course, but it's it increases quickly.

B

And after that, from this uh score we can say that it's become more or less stable. But if we looking through the metric support by the uh by the by by this uh activity model, we can see some some different things from the different perspective, because because it's very new, so something is very different compared with the stable community.

B

So if we go inside into the each of single matrix, I I use seven metrics to compare these two communities for the activity matrix model. So all the seven metrics uh coming from the matrix models this table, I pick up seven metrics.

B

So the first one is country builder count, so also you can see for the project b, which is very stable.

B

We have one the number of the contributor counter is stable, but for the project a it's increased very quickly at the beginning and- and I think it's very normal for the commit frequency its average number of commits per week in the project.

B

If I disabled project b and use this table to combine this, the final result- and we can notice that they have regular peak and valley- and this peak and valley has a very um it matches with the each of the single uh release tempo.

B

So before each of thingo release the contributor uh commit frequency will increase, and after this release, after every single release, the con the commit frequency will become uh low, and so.

B

So we from this point, we can see that the project b has really a stable contributors and the commit frequency temples and they they know when to come, commit more, I mean, contribute more commit before the release and after that, of course, they will slow down the the development process, but for the project a you can see that the their commit frequency is very quick and increase very high quickly.

B

It's uh I mean after like uh just half years after it's created, set up and their commit frequency is become much longer and become much bigger than the product b. I think it's very normal for a new project, because it's trapped more people or contributors.

B

Together with this big picture, we can see that it because it tracked more contributors for this project and it will, you know, get more contributions achieve more contributions from other contributors, but it's not always good for this project. A if we look at this proof this picture. This is the updated scenes and I defined.

B

We found the the definition of data collections. It means average time per repos since ripple reports were last updated.

B

You know, as I mentioned, for, for operating system, distribution of communities each they have like 10 thousand packages or ripples for in their communities to secure linux distribution released together with the latest version or you know, then the new version of the packages they have to making sure to maintain this 10 000 packages in time.

B

So it's very important for such communities. So so, let's come back to this to this picture. I can.

B

And if I disable the project a, we can say the project b. First, you can see it's really regular stable.

B

You know, but at uh like uh every half year project b will update their factories uh almost all the packages quickly and to prepare the next release and they have really. You know the.

D

B

And normal temples of release plan to do that. That would secure this 10 000 uh factories to get maintained, to making sure that each of single release would using the new packages of the version of the software.

B

But if I, if we check the project a you, can see that at the beginning it looks like it has. This community has maintained all the uh packages because base uh used the 200 sample packages, that's our data source and, uh like this point, uh all the packages of these 200 or 200 batteries has been maintained, has been updated.

B

But after that it looks like uh they just focus on the sum of the single some some some part of the practice, but for the rest of the other packages.

B

They don't have a intention to maintain that which caused that this uh update sense value will increase with increase it's not like project b. It will be declined to which means it maintains for the each of single matrix packages.

B

B

And okay, the next one is creative things. So it's it means every uh every time per per ripples and swivels were created, of course, because project b has a longer history. So it's what we can get here and the project b can get here average time value, then the project a and the common frequency means every. You know every num uh average number of commons poor issues in the in the past in the past 90 days.

B

So um we can see this picture together with other uh rest of the two uh picture, to say that the issue uh part uh project ap, because we have more contributors and more commits, which means they have more issues, discussion and handling for the product a compared with product b, but.

D

I wonder um if you could talk about why you think that comment frequency is so flat and if there's anything that could be done like do you think that that's informative, like it really just doesn't change, or do you think that the metric could be more sensitive.

B

uh You mean the flight.

D

We are quite surprised, so we're 1.1.5 yeah, that's right. The flatness, particularly I mean really on both of them.

B

Because I I'm not, uh I use in the average number of comments in the next in the past 90 days and, and you mean the flight, I I mean for the age of things for especially for the project b.

B

If you check into the their issue tracker system, we can say that uh the con, the total number of the issue is not that money, it's not big, but uh in the age of things of issue they do. They do have some real discussions around this around the each of thingo issue, try to find out the the the final resolutions, but for the project a actually.

B

um Therefore they didn't discuss so much in in the issue.

B

uh I think, um which means project b has more motor uh and and more uh motor uh contributors, which could provide more good support for for the people who just attend the project, but the project it's similar like okay, many new peoples and new contributors come to these communities, but it looks like they couldn't get the the feedback or support in time. So the the issue comments is very short, it's very it's little.

D

You for that, yes, I can. I can feel now that it's it's informative, like I'm, genuinely learning about these projects, although I am I'm still surprised that project a has such a flat, in fact that they're, both so flat, I would expect them to be more meaningful.

B

Because, yes, I mean I, we may create this program already.

B

It's also making me interesting for that by so fly. So I checked all the uh in the many issues pop up from the project b to find out. So it looks like it's our regular sales result and we have the future of this robot.

B

You know issues commons and we just account the real people's comments, but it's true that we can get the average values and also consider that right for this project b, because the this project, this community has more than 10 years history and the contributor around this project is very stable. I mean the people.

E

B

So I always found the same people, and I mean a group of people could give the I mean the quickly support around the same uh around the related topic. So I guess that's why.

D

Thank you for giving me so much time I apologize for intruding and that's fine.

B

uh So I just mentioned this: that's the issue of comments, frequency for the rest of two metrics. It's about a closed issue count and update histogram. Actually, these two metrics they have similar. You can see the picture change the change you can. They have just to close the relationship with each other, because close uh this is the one action of the update and also but really think. I wonder I this update issue, because uh you know the issue.

B

The update issue means not just create not just the close, but also, if you have, uh you know, add any types or add new comments. It's also considered the action of the update for these issues. I treated eyes uh the one matrix to set up or to composed of these activity matrix models, but it's weight. It's not that big. We can see that it's weight, it's only yeah for 4.9 percentage,.

B

So in total we can figure out the final result of these two projects which have similar softwares.

B

We can compare with each other and also we can looking through the the each of thingo matrix composed of this matrix model to fit to figure out any uh any things we want relate to this matrix, especially I mentioned here the updated scenes which is very important for distribution communities.

B

You know for the linux kernel linux operating system, we have to maintain it in time and for for all the packages, but it looks like project a have some problem, but I think it's fun for for new communities which just set up two years, especially for the you know, operating system, distribution, communities.

B

And they can learn many things from these stable communities like project b, the control and the plan. They are released, temples it's in a very good way. Okay, that's all I I I can I can. I can demo and share today and questions.

A

Yo I'm curious, and I don't think you said this- I might have missed it um when you're looking at the chart you had with the weights um given to people or given to metrics. Oh I see. Why is that one like a minus weight to it, there's one that has if you go up a little.

B

A plato, okay, yeah.

A

Right there yeah: why is that? A minus weight.

B

Which means uh this value got more the value, it's a smaller value, so it's a better situation. It's.

A

Provided I got you.

B

C

Okay, how to use it in the end of the talk and like, like you described like you, explained how to see this video viewable.

B

Sorry, could you repeat your question june.

C

um I mean you describe a lot of things about how to see the picture, how to say how to see your results, and I think maybe we could add edit in the end of the doc.

B

Okay, yeah, I add the the some snapshot of the the thing that I shared today I mean the.

D

B

B

Things about each of single pictures, I did something about it or.

C

D

C

To know how just how to say this picture.

B

Okay, I can I can. I have something about it to introduce more about each of thingo results for matrix, because uh you know the reason why I pick up one matrix model, which contains like more than 10 matrix, which is the number, is huge.

B

The reason why I do that because I just want using you know some way to uh to to make the final result looks uh it's very simple: to find to figure out or to figure out uh to investigate, to let users who use this matrix model to get quickly inside.

B

But behind that we can use some, you know algorithm or matrix model algorithms to make it. We can hide the complexity of these calculations behind the visualization and make the final result to look smarter.

E

C

Yeah, it's it's very cool.

E

Okay, I have a question so I see you have taken six metrics, but I see more uh within each. So is it like, so we defined one metric and has a clear uh definition or clear strategy. Are these filters from those matrix I'm trying to connect those dots like okay? You have six metrics but like, for example, you took a contributor metric and within contributor you have a maintainer count. You have a contributor count, so is maintainer a metric or is it a filter within a metric? I'm just trying to understand that scenario.

B

Yeah, as I mentioned uh here, we have, I pick up six different chaos matrix we defined in the chaos and in each of single matrix, for example, in the contributor. You know I using two different uh data collection strategy to define the contributor count and the maintainer count. You know these two, all these two con data collections to actually belong to the the same matrix, but they represent the two different data collection strategy, and I collected these two uh uh two datas.

B

So one is the contributor count which, including the you know, the commit authors, review, participants, blah blah blah and the material can't only consider the the maintainer.

E

So for each of.

B

E

My question is: like is maintainer a contributor like somebody contributed and maintainer just approve it. So is maintainer a contributor. In that scenario, yeah.

B

Yeah, it's a maintainer, it's a it's, a role that uh who have tried to approve and approve the the the the pro request, and also they have many other uh responsibilities. You know yeah some meetings.

E

If you look at the definition of a country, but the contributor is one who has submitted the court, not the one who has approved like approval is there. I agree without approval in open source. It's like the work doesn't happen, but for contribute account. The metric focuses on the who is contributing and counting that.

B

Yeah, because here we have different many different kind of filters for the engagement, so uh I mean the maintainers. I I make.

E

B

One of the role in.

E

B

It's maintainers, okay, okay, I got it.

E

Yeah no thank you for clarity.

B

Yeah, it's the reason I mark it as yellow the things the the things are marked yellow because they, they didn't mentioned obvious adversely, but I think it's better to add it into the existing matrix or we can create every single, just the specific matrix for this one like like here, we have a metric called activity, dates and times in the common group working group, okay, but I think it's, uh it's vital that we pick up uh this uh um okay.

D

You know with the production.

B

Of this metric, I set up the the new ones.

E

Okay, so that means from each metric you are picking one to whatever the filters are available in the data source. You are picking those filters as a way to tell your story or okay yeah yeah. I really liked your approach really appreciated.

B

B

C

And, as you mentioned, we added some new metric, a new single matrix and do we have to add this new simple metric, first or.

B

I think in the next in the next meeting or afterwards after this meeting, I would like to add some uh important metrics. I think we can create it for the chaos and you know, following the metric template to set up to set up it for the each uh in the inch different uh working group and define that.

E

Yeah, that's what I was suggesting like any new metric. You have used over here it's better to create a issue for like metric idea and uh template or if you are good, just create a template and share it in the working group and that can be refined or worked on.

B

Is gonna release a new version? uh I mean yeah in the early next months, but after that.

E

B

Exactly yeah, okay,.

B

Okay, if there's no more questions, I will stop my sharing.

D

I really like that you provided detailed.

D

Quantitative metrics and we are able to look at two different projects and get an intuitive sense for whether um the metrics worked or didn't right, because we kind of we sorry um we kind of can we can check you know. Yeah and clients is like it's data driven.

E

It's not that this is science.

A

D

It's data driven it's about the real world and you have to check your intuitions against um against facts. So I found it really valuable and- um and I, uh if I had any misgivings, it is that the level of detail sometimes makes it hard to.

D

Understand the relevance of the metric to the model, I mean I get it because we're here and that's what we're doing, but it might be that your users who are looking at the at the at the graphs get a little bit lost and you might want to um see if there's anything that you could eliminate for the sake of having better um information prioritization.

D

You know more more clarity about what's important.

B

Yeah, that's that's why uh you know we have with such monks, monthly reports for the each of single matrix model, and we will highlight some important port point for the intro single metric e61 matrix model to highlight it. Okay, this is the problem of some regular patterns we found out and to let you to pay attention on that.

B

Like you mentioned, if one matrix has more than 10 metrics, we are not going to to to introduce every single matrix result is very boring, but we will highlight some important pre priority types, some some some key point for that, especially they have some some changes in this monitoring trend and, and also the one things I think is very neat very important thing is we have to do some comparisons between two communities.

B

You know, why is our own communities? One is how similar situations, which is maybe better than you or I will have a longer history or shorter history whatever, but they have the similar things situations we can compare with each other to say: okay, this one he's brighter than me and what what kind of city uh process I can handle to to to to change that this, the things. So I I mean the the comparisons it's very important.

B

D

I found that it did provide insights and I did understand the projects better.

D

A

I asked this question in this in slack, but um I also want to ask you here so looking at this wonderful metrics model and seeing these these really informative graphs. Do you think it's the job of chaos to um take it a step further and explain what these?

A

What these numbers mean, just like you have done here, like you kind of going through it with us, was extremely helpful, because I don't know that I would have caught the nuance of the the graphs and the way that they are the way that we're comparing these two projects. So do you think that that is something I mean? I have my own opinions about it, but I want to pose it to the group of.

A

uh Do you think that that's chaos job is to help people also understand the data that they may be seeing and and then to take it a little further. Do you think it's the job of chaos, to offer advice on how to move those numbers or influence that data so like? If you see that something's off in your data? Well, uh then, what you know like is it chaos. Is it up to chaos to provide that guidance of hey? It looks like your contributors were going really down.

A

Here's some things you can do to raise that number again, so I just want to pose that to the group and see what you all think.

B

Yeah, I think it's you know for the different, because we have many many different communities and the if the the trend become down or increase, it's meaning the difference for for the different communities. It's not not means it's increased means.

B

It's good decline means it's bad, so I I think we may could provide some background about matrix uh to take some best practice or examples to using some existing value, for maybe we can use chaos as our best practice to explain the data, so you can using after they say the chaos project, the real data and based on some matrix model results. Okay, they can figure out.

B

Okay, this matrix means this one and I can use this matrix model for my communities to explain the different thing, but I totally understand what it means already so yeah we can use communities, uh the chaos, communities or any other communities they would like to provide such data. I mean our community our case. All chaos, of course, could provide such data. I think it will be subject to that.

A

My personal feeling is: that's really the future of what we're doing here is not just to you know we're we're still going to con we're still going to develop these metrics and still going to work on different ways to measure, but then we're going to bring all the metrics together and then we're going to tell you how to interpret what you're looking at, because it might not be as obvious as you know, especially to an open source maintainer that might not. You know they may not quite understand.

A

um You know how these these numbers affect their project or what they actually mean when, when put in context with other things, you.

D

A

Like I'm sure they probably understand what contributor count means, but do they understand what it means in in conjunction with some of the other metrics and how that plays in so um yeah. I think that that's that's that. I think it would be a great rule for chaoses to provide that guidance. At least interpretation yeah. I don't I don't know about moving the numbers, maybe maybe not I don't know. That seems like a whole other can of worms, like you said a whole other subject, but um I don't know.

A

I think it's interesting and I think we can provide a lot of um a lot of help for people who are trying to make things better for their communities for sure.

B

Yeah, uh so we are gonna run time, but I'm gonna say that if you have more interest on this matrix model, uh your comments are very welcome and we welcome the further discussion around this matrix model and also in the future more matrix model in the similar way. We are going to introduce in such way to compare two communities and figure out uh how to investigate inside, provide some insight about it around this matrix yep.

A

Great job that was fantastic. Thank you so much.

D

A

D

Of you, uh I I appreciate you um doing the demo and um but also june and sean. uh Thank you so much.

A

Thank you, I guess thank you.

A

I guess we'll see you all next time. Thanks, everybody see you.

D