CHAOSS Metric Models Working Group, 7 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CHAOSS Metrics Models Working Group 12-7-21

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Thanks welcome to the chaos metrics model for.

B

A

My brain is fuzzy too yeah perfect, my my throat and my brain. um So the minutes are in the chat. If you could add yourself, that would be wonderful.

A

uh I will share my screen here um a couple things I just want to chat about today, um so this is just so huey, you know and and lucas you know too. um This is our last meeting until 2022, I'm not entirely sure. If we start this one up in the next week. You know like the week we get back or the week after you know what I'm talking about: yeah, I'm looking for chad, tenth or if it's, if it's the later one so.

A

Lucas and I'll put the blue.

C

D

There you go yep, it looks like it'll start the uh we'll have this meeting next on january, 18th, okay,.

A

um We'll make sure that we'll get this in the calendar so not a big deal. There was.

E

There another uh url for that, I'm getting the um my sharepoint error.

A

uh Yeah yeah sean can you. You always seem to help.

D

Yeah, I've got it up. This is the I thought, I'd updated the meeting, but you know, I think, a lot of things. um This is the one I have which does look slightly different than the one matt shared and that's from the meeting appointment on the chaos calendar.

D

I bet that one works.

A

That is working, okay, okay, awesome! Thank you um all right. So the first thing that I just wanted to to kind of talk through a little bit is um oh amazon, hi emma.

D

A

How's it going it's okay, it's good to see you.

A

um So the first thing that I wanted to talk about is the this just the spreadsheet, so this kind of goes to a larger issue that we're having in the chaos project, which is just ensuring that our documentation is accessible. You know, when it kind of, is consistent and makes sense for folks.

A

So for a lot of us on this call, I think we have the the luxury to talk through it in a meeting like this, but I think a lot of people that are new, sometimes look at our documentation and it's a little bit rugged, so I've been going through you're all familiar with the spreadsheet right lucas.

A

You you've all seen this, so I've been working pretty hard to kind of update the spreadsheet across all of the metrics working group, so this is different than the metrics model working group, but I'm trying to to to really kind of simplify the the um this thing, the spreadsheet, uh so that it it carries enough information that is useful, but it doesn't carry too much information it just absolutely like melts your brain.

A

When you see it for the first time, so with respect to the metrics model working group, I've tried to kind of simplify this one a little bit. So I think the metrics model working group.

A

I think we can follow a release cycle or a release. Cadence that's similar to the metric. So as we have a new model, that's being developed, we can kind of track it in this spreadsheet, similarly have remarks about that potential model and then also have links to the metrics model as we're working on it.

A

The process of work is that when something is in progress, so like row, 12 or any of these like 17, 18 so and so forth, 13 is a a liar on this one for a second, but when anything's in progress we just work on it in a google doc. So the google doc is our work in progress um platform. So we don't do work in progresses on github or or some other, or we could use google doc or we can use some other shared um document tool.

A

That's completely fine, like sharepoint or something else, um and only it's only when we have put the metric or the metric model under review, that we move it to github.

A

So when it's a work in progress, it's in a google doc and once we've kind of finalized, whatever it is, the metric or the metric model, we move it out of a google doc and into the github repository and it's under it's in the github repository that then the community review happens.

A

I just want to make sure we're all kind of on the same page, because, as I went through all of these tabs, we weren't on the same page. There was there was work being done all over the place, and so I've been spending time kind of organizing this. So everything that's a work in progress.

A

Well, let me go here. Is a google doc and anything that's been released is a github is in github or anything. That's under review is in github.

A

So does anybody have questions on that? I just need to get this process kind of clear, and this is actually in the handbook.

E

uh So it sounds like the item that I did as a guest should be transferred to a google doc.

A

Would you agree, I think it's okay, if it's a is it how editable or like shareable, edible and editable? Do you think it is, you know, is a work in progress, doc.

C

For me, I think I'm fine with it, because I I have conversation with with people in in chaos, china, community people, so I will share with them on the bi-weekly meeting about matrix model, provides and, of course, some of them.

C

I I believe just a little part of them cannot access to the google doc, but but most of them, and they have access to the google doc and- and I also discuss within that- uh if, if bomb opening uh cannot access it, I can, I can transfer uh those content, some of the google doc two to the other, uh just some other tools, yeah yeah, exactly.

A

Okay, so I'm okay with that and lucas, I'm okay with it just kind of living as a as a guest, because it's not that's not in the chaos repository, is it oh. That is right there is it just because it's living somewhere else. I just think when we, when we have it.

E

Could you correct me, I think the essence of what you're going for here is that um the repo is for work that has been kind of it's ready for github and and forget and get type flows, and you have more kind of shared editability through the.

A

Web and it's ready to be released, it's like kind of it's out for public consumption. Essentially, it doesn't really hit the github repo until it's ready to be consumed, whether for review by the public or to be to be published to the public okay.

A

So any of these like this is like that space that that namespace of github slash chaos, wgdei is kind of that's an important space that we don't. We don't work in there, that's only where our things are kind of done, so, of course, we can do a pull request. If somebody has an issue, you know what I mean. We can change things in there, of course, but generally speaking, yeah.

E

So, do you have that item in this document and is it ready for you to do what you do.

A

What item this, the fixing everything I've already fixed everything I kind of went through and did it.

E

A

uh The contributor funnel, oh sorry, oh no, I don't have it in here. I did not see that so that should be in here somewhere.

A

I see your question. No, so does anybody else have any questions on that? I just we. I think we needed to get this process squared away, because I think we were.

A

We were going in a variety of different directions, based on what I was seeing in the repository and on the spreadsheet.

C

I just have a quick question that do we have any public area on the website to to show this link, I mean about this whole sheet. We don't, I don't think it's on the website.

A

Okay, do we need, but we.

C

A

Okay, yeah. I think we do need that every every time we show this to somebody they like it quite a bit and so um yeah. That's that's a fair point.

C

I I already I I took my bookmark, but I don't think other body know where to find those metrics and matrix model.

C

B

Yes, okay, could we put it in the community handbook somewhere where we're talking about like how we develop the metrics? Maybe we just have a link there. Okay,.

C

Yeah, that's cool the place.

C

A

Okay, cool the webs, so uh the website's gonna be like this. This thing was one thing: to clean up the website's gonna be a whole nother thing to clean up correct. That's next.

A

We need we just we have it's interesting like like community growth and community like longevity, it creates this proliferation of documents and they end up. They end up just kind of ever so slightly like splitting from each other and over time. They need to be brought together. Okay, um all right cool. So then, in terms of the metrics model tab for this group, because it's different right, this is like this is different than the metrics. It's because the metrics models are slightly different. Yeah.

F

I think we can still.

A

Follow the like the in progress, considering all that kind of stuff right, I think this is still a fine and all I haven't updated it, so it says, moved and all that kind of stuff. We have the model name, which I think is completely fine as well.

A

um When it's released, I've been following a very just a model of when it's released, it's version one and if we modify it it'll become version something other than one all right, and so right now it's when anything's in progress. It's this not given a version all right and the reason we wanted to just version things I think, is because, as we do updates, we just want to be able to track that this has been updated.

A

This is like a 1.1 of this metric sean had made a recommendation that we have potentially a working group home for a metrics model. It's becoming pretty common. That say the risk working group would be developing a metrics model or the dei working group. Maybe we could talk about that elizabeth, but like the dei working group would be developing a metrics model. We actually talked about one just a couple days ago on monday a couple days ago.

A

Yesterday, right um so we may have a home for these, and the question that I would have for people is if we have a working group home so, for example, dei event. Badging we've talked about this as a metric model.

A

Like do we work on it here or do we work on it in the working groups or do we work on it in both places like? Is it okay like? Where do where do we do this work.

E

I think we want to talk about what is the line between on the output of a working group and the metrics themselves and a model. I started to work on a dei model and I found that um basically, I was just restating the original webpage and not really adding any value.

C

Gotcha yeah, I I shared the stimulant about this because uh I what we want to do to to to pick up the different metrics who have a logic connections from different uh current text working group.

C

So if we have to put it in one truth, one matrix model in one working group correctly, it make me people think there's, no, not so much difference between matrix and matrix model. Okay,.

D

Fair shaun did you have a comment on this yeah? It's a really good point that it will be hard for p individuals approaching the project to distinguish between the models and the metrics. I don't think that prevents working groups from developing metrics models and or proposing them and having this working group sort of help them refine. It.

D

So should we, I think, keeping them separate in a separate repo, probably makes the most sense from uh if I'm an outsider coming in. I think you huey's point is very important.

D

It makes it easy to navigate if they're littered throughout all the working groups it gets harder to find.

D

A

Be the metrics models are kind of released under the metrics model repository.

D

Right and probably probably reviewed by this group for consistency, you know I mean I expect to pass them back and forth as we define them and then provide examples of them which for gaba- and I have one to show tonight great decent start on it. Okay,.

A

um So maybe like home, isn't the best phrase there really. I don't know.

D

A

You see what I'm trying to like: it's, not the.

D

A

Of it, because the home would still be here.

D

Yeah- and I would say, that's optional, I mean I think, there'll be some models that we develop, that are not in a working group, especially the first year.

A

Okay, okay, right on. Thank you um remarks. Remarks is similar to remarks we have here. So it's just kind of a it's a note-taking section like just just just that and then a link to the model.

A

I removed the column.

A

That is the metrics that are in the metrics model, because if you want to see that you can click on this google doc yeah, I think that's a better approach and see the metrics that are in it when I had it in the spreadsheet it just made this. It made the cell really really big.

A

Some of the metrics models have a lot of metrics and it made the spreadsheet look really ugly so, and I was trying to make it look less ugly. So all right and that's it oh process model- uh don't worry about that!

A

I didn't even know that was there so.

F

A

Worry about that column. That surprised me, so does anybody think that there's anything else that we should be tracking in the spreadsheet, with respect to metrics models.

A

G

No, I think so.

A

Okay, sean lucas, no.

C

A

You think this is great thanks for doing it. Okay, you think this suits. This is at least suitable again, I'm just trying to reduce the amount of information that's in this spreadsheet, so that when people see the spreadsheet for the first time, they can understand.

A

Let's go here. Okay, great! Thank you all right.

A

All right so we're gonna go right to sean and regatta's overview of the welcoming model.

D

A

Here's the question is that in here.

D

Yeah, I thought I'd put it in this spreadsheet, but I don't see it there now.

A

It didn't delete anything. I promise yeah.

D

I must have maybe the welcomingness piece I don't know. Maybe I edited a copy of my a copy. I made at one point or something, but um is it just? Is it part of community engagement.

A

Probably yeah yeah all right. Let's just do this right.

D

Now, then, all right, so what's the name? What's it called? I mean we're, calling it welcoming, but um it's elizabeth's metric that she proposed um in one of our prior meetings, and I was welcomingness or welcoming this about welcoming whatever it's um the best phrase I could come up with. I don't know if that's what elizabeth intended.

B

I I think, that's fine, I don't I don't really have. I don't have strong feelings about the title, so yeah yeah.

D

I don't either I I couldn't come up with the really perfect ones yet so all right so before we before we have your demonstration. Is there a there's, not a doc? Well, there's it's in our notes and I have the metrics that elizabeth identified in the jupiter notebook that ragava and I will show okay.

A

So let me I'm trying I'm doing this, so I have this and so.

A

I'm gonna make a new.

A

D

Word I mean I, I can add a suffix to anything and make it a word.

D

A

I will I'm just gonna, because I really want to just keep up with all of these, so bear with me. That's totally cool.

D

Okay- and I can update that document to include the notes from the meeting where we outlined it.

A

Perfect and now we have that document and you can work right in there yep. Okay, so do you want to do you want to share your screenshots yeah? I think that would work elizabeth. You may have to oh, never mind.

D

I'm not at the top oops you kind of give some overview again, yeah so background.

D

There we go, so these are the metrics at the top. Here we I me ragava and I have been calling it. The welcoming metrics model or the elizabeth model and elizabeth identified activity, community culture, licensing, stability and code related, metrics and ragava, and I built a number of either leveraged assets that are in augur. Ragava built a number of things related to different things, and we have some holes, which I can talk about um a little bit, for example, inclusive leadership.

D

I think that's one that we're going to have to borrow from the dei working group, and I don't know if we I don't. We don't have a quick way to measure that so we'll have to figure out. How do we incorporate that into a model um it may? If a model is something that people run, it may just have to include instructions on how to good point deploy.

A

I think yeah, what is the point like some models could include like trace data, stuff and others.

D

Yeah, I think I think, a lot of them will many of our metrics um include trace data, but, like.

A

But they might include like 20 if it's not, and we may have to express that somehow.

D

Right right- and I think, as we talked about um so these are the five headings of the metrics and we've tried. This is a jupiter notebook, so it's a little clumsy in terms of presentation, we'll tidy it up as we finish it and make it a thing, um though we may store it as a jupiter notebook.

D

um Do you want me to just go through some of the items that we developed.

A

Yes, please, that would be great, thank you and just for emma. I don't know if you're hearing us, but basically we had spent just a few, maybe a few weeks ago or a month ago, we had spent some time in this meeting kind of just brainstorming on some different metrics models that people might be interested in like how they could draw metrics together. That would be meaningful in different contexts and what sean and ragava have been working on is is so we can.

A

We can specify the metrics models and that's great cool, no problem, um so we can specify those metrics models um like in those documents that I was showing earlier, but sean and ragava have actually been doing work on deploying the metrics models. So if people want to see this data, how do they go about doing that? So that's just a little bit of background, so.

D

Sean take it away. Okay, so we have these five areas and you know forgive the bulkiness of the jupiter notebook, but this is the um activity. Section is first and under activity is issue age.

D

I think we need to work on this a little bit because the issue, basically how old, are the issues that were opened longer ago, they're older, so it's um it's not a perfect visualization. Just yet um so I'll move past it pretty quick.

D

Then we have issue response time and so in many cases there's an existing auger endpoint that can deliver the data. In this case it's a it's kind of a new way of representing the data in order to make it this kind of visualization, so we'll we'll roll it in as a new endpoint, so that this big query isn't in the notebook and you don't need database access to get to it. But this is issue response time. So the example project is auger, and this just shows that it may get a little bigger at all.

D

I can I can oh yeah. It makes it a lot bigger too um and we'll probably organize this by month or quarter. It's organized by month, we could probably go to quarter if we're gonna show the full length of the project.

D

um That's kind of a design decision, and one of the reasons we'll make it available is a jupiter notebook with blogger. Endpoints is so that people can play with it and just apply it to their own auger instance, and ultimately, our goal in the coming year would be to have have examples from grower lab as well.

D

And so this is issue response time uh also under activity is issued time to first response, and so this is uh correct me. This is a mean correct, or is this a cumulative?

D

It's a it's a mean, it's a mean okay, so this is the average time to first response. um How long it took to close is what's on the thing, so I don't issue yep yeah, so uh we also have a first response representation a little bit. This is um kind of duration to close a little bit so.

A

I'm not asked if there's a sh, no a link to share, but it looks like it's there.

D

Is not yet, but there will be I'll make this available um soon um right now, there's not a link and that's just because there's database credentials in it. I have to convert a few things to auger end points first, so that'll happen, but you know before we meet next for sure I would guess in the next week we'll get that done cool. Thank you for community culture, there's code of conduct and really is there a code of conduct or not is kind of the indicator.

D

We weren't sure how to visually represent this, but we include a link to the code of conduct where it exists, so augur will um gather the code of conduct and then oops, apparently that didn't work. That's comforting, like you get.

A

What you're trying to do, though,.

D

Yeah, I don't know why that wouldn't because it's it's literally gathered direct from like code.

H

And does it look in like all the possible places it could go or do you expect it to be in one place.

D

So github has metadata around code of conduct files and it's part of how you make your project searchable. So if you declare a code of conduct file, then it's collected as part of the metadata, which is why possibly, this is wrong. Although this this link should work right, it was developing at code yeah, oh yeah, that's the right!

D

That's yeah! That's exactly it yeah! So it's just the way. I'm writing the query and displaying it.

D

I

Well, yeah, it's just sean. You don't code. Two times you have code code, yeah a.

D

Here there we go there, that's exact, that's what the query returns and apparently the whole thing doesn't get transformed into the jupiter notebook very cleanly. It leaves a piece out. um So if a project has one declared in the metadata it's gathered, are there other places? We should be looking emma.

H

um I mean you know more about the metadata. I just know that sometimes it's in the dot, github folder and sometimes it's right in the root and then other folks will just kind of put it in uh random places. I know sometimes I've had troubles finding it. That's why I was wondering.

D

Yeah yeah, so it's if a project has declared that they have a code of conduct and github kind of motivates the projects to do this. They they tell you what you need to do to be a project that is up to community standards, and this is something that github started. Maybe four years ago now at uh github universe, I think in 2017.

H

Doesn't github have that page where, like you, click on and it shows like an alert or like check boxes, if you have a code of conduct and a number of things exactly yeah.

D

That's helpful, that's the same sort of.

H

Thing: okay, that's the minute. Okay,.

D

That's where yeah that's the so github actually keeps that in the background and returns it via an api.

H

Okay, cool, that's useful.

D

Yeah and yeah, so that's uh inclusive leadership. That's the one I mentioned at the outset that we have to decide how to handle that um license. Coverage is one that is an auger endpoint and I just for convenience, just copied the image on the main page that hits that endpoint, instead of showing you the pure json, because I had something and license is declared. We also have a auger endpoint that delivers a bunch of data, but we need to process it in the metric model.

D

It shows up on the auger homepage, but it's not easily transferable, to a notebook, so we'll just take the data and make it pretty basically list out the different licenses that are declared on a project.

D

um But one of the interesting thing about interesting things about auger is that we actually have all of the licenses and open source software declared, because part of what we do is collect data about open source software.

D

So when the license scanner scans, auger it um it comes up that we have every single open source license available in at least one file.

C

I have a question which last, which stress and scanner are used by by by auger, it's uh implemented by augur all.

G

I'm guessing yeah, it's nomos, it's a faceology scanner; okay yeah, I'm just um you're showing.

A

Us your screen.

D

I know I am, but I'm trying to find the licensing link that I have. I mean.

E

You know, I think, that um the fact that we're having this conversation about how do you quantitatively measure uh the matrix kind of goes to show the value of the models project right, like you're sitting there putting um metrics into practice for the sake of many models and it's kind of helping guide us uh for future projects.

D

Yeah, okay- and this is um for badgings, so cii best practices badging. I should really call this um cii best practices badging status, most projects candidly- are not best practice badged and I won't. I won't run to the top to make that markdown cell play nice, but uh basically we show you that its passing status is met by this particular repo, and I was working on making a pretty green or yellow color, but I didn't get that finished.

D

Test coverage is very challenging. There's a lot of different, it's basically language specific, so we don't have any auger implementation of that. I don't know of any metrics toolkit that does it's um it's another one that we'll have to think about.

D

Bus factor can be calculated, but I think we want to give people parameters which is what's specified in the in the repo in the um metric definition that chaos has.

A

What would you allow, I thought bus factor was set parameters. Isn't it the number of something doing more than 50.

D

Yeah but but we do, we do allow filters and different values to be passed in, so you can evaluate bus factor and elephant factor using your own math. If you want.

A

D

um For computers.

A

You would lower the percentage of like what your percentage contribution threshold is.

C

Yes, something like a call contributor, a casual contributor and normal contributors to distinguish.

D

Yeah, I was even thinking a network of people who contribute the most.

D

For the way we present committers kind of starts to get at that, we also have a contributors endpoint that ragava and I thought of using, and you can see the you know- different colors of people basically, you've contributed to auger over time.

G

Uh-Huh, that's interesting.

D

So these are just the top ten. If you showed everybody, it would be harder. It also gives you a sort of a glimpse at what the bus factor might be at any point in time.

C

Let's use bar chart to simulate a hip, a hit mic right kind.

D

Of yeah it's it's basically stacking amount of contribution made by each person over a period of time, yeah and then it's just.

G

I

D

Off the um decoding, because I think I'm the blue.

D

And then, under code for new contributors, we actually have a visualization endpoint. So because I made this big, there's is actually a four panel visualization that just you can pull from auger by specifying the repo that you're looking at.

D

And whether they contribute so this is the first time contributors per quarter. um This is repeat, first time contributors per quarter. This is all second time contributors per quarter, and this is we. It should be flyby, apparently something snuck past my goalie, it's flyby in the narrative description below, but it's drive. By still in that title, I have to fix that.

D

And then, let's see here then.

E

This is just another if it's possible to get some kind of benchmark on those um drive-by um contributors. It would make the data um easier to understand. Yeah.

D

uh Bench by benchmark, you mean like compare it without comparing other projects yeah, I think um so. This is. This is always one of the tricks is people do need to compare projects to interpret them and the- and I think the point that you raise lucas is a really good one. You know do metrics models need to have some of those comparisons sort of enabled by default. You know, should we produce an example that lets you see a couple of projects side by side.

H

Have you also thought about trends within an organization like a github organization? It's like. Yes, I think that.

I

Would be interesting.

H

Too, but for a number of reasons right like we should be archiving this project, or you know this project's doing a good job as a model for others.

D

Yeah so organizational trends, that's a really good! That's a really good point we do have. A lot of the data can be is organized both by github org and by repo, and I went to the repo level for so we could show all of this data for a repo like a github organization as well.

D

But then I think what you're saying is you'd like to see something that maybe displays the first time contributors per quarter for every repo in a github organization right, so you can see the trend of which repos are ticking up and which repos are ticking off.

H

Well, I hadn't thought of it at all until the previous question so right, um but I think that would be interesting. I think that provides a lot of great storytelling, which is often what we're, after with these metrics right.

D

Yes, so regala, can you make a note about working comparisons in here? I'm putting it in the.

G

Notes too, but oh thank you. Of course. I can't see that.

D

um And then this is just another view of new contributors. This would just be cumulative new contributors in a month.

D

Using the new contributor's endpoint that we have that's just data and then we have change, request acceptance rates, which is also a visualization api endpoint and uh for the acceptance rates I'm just going to zoom this down a little bit just to try to make it fit.

D

We have all the ones that are merged and not merged, so you can see in 2020, agar merged, 360 and didn't merge, 78 out of 438 and then in 2021 we have 267 merged and 68 that weren't merged, and then we also look at the the 20 slowest to be merged um that are accepted and merged and slowest to be merged and rejected. So you can see in we had 88 accepted, slow ones in 2020 and 38 accepted, slow ones, and so more of the slow ones get rejected in 2021 than 2020.

D

And then this is just another take on pull request acceptance rate, which is actually wrong.

D

um My end point is wrong, so I have a task to fix that end: point for the just the data and a visualization that regava put together so yeah. That's it.

A

We've had some comments throughout. I guess I would just like to say thanks.

D

A

Hard to I recognize it's hard to.

D

Eat right now, um it's really hard to show and tell with the jupiter notebook, but I think that's it's the easiest way to build something. So hopefully you all found it somewhat in enlightening or helpful. I found it.

A

Super enlightening.

C

Yeah, I think it looks.

D

H

And I want to play with it. I want to know how to do that.

D

I will give you access to to play with it and if, if you have some repo like right now, it depends on auger data so, but it could easily be um so that, if there's a different way that we want to do these, you know I'm open to it. I'm not married to doing it with awkward data. It's just what I know.

C

Do we have any sample data to help us too quickly to set up the whole things?

C

uh Do we have exam an example yeah I mean the the data you mean you you're you're, using the the data from auger.

D

Yeah there is a. There is an instance set up.

D

At auger.io, I'm working on fixing some things in the front end, so.

D

That's the that's the home for the end points and you can find the go ahead. Matt.

A

No, you can finish your thought. I think emma was gonna, say something too. I saw you pulling down your microphone.

H

That's to make sure I'm muted properly, because I realize my dryer's going over there.

H

um Oh, I was just you know, yeah like how could we spin something up or fork it or because I know, and it's selfish, because I'm what I'd love to do is like I mean I have my item next, but I'd love to like take a microsoft project like dapper like dapr.

H

I don't know which one and like show it to that working group, because we have like a couple dapper folks there and, um and then um people sort of talk through how these may or may not apply to the thing the models, I'm calling it a model. I don't know really exactly that we're building.

H

I think this is exciting and you know I'd love to get people contributing to it as well, and if you know they can see it with their, I mean the the auger data is great, but you know people always like when it's they can relate to. Something is helpful.

D

Yeah: okay, vs code, the apr sir.

A

Put it in the notes.

E

Sean are you, okay with me posting that link to the uh slack uh for the work for the metrics model group? Yes, cool, okay,.

A

So sean the one question I had so as we I'm gonna share my screen. So um so this the highlighted in yellow is the work that, when we were working kind of individually about a month ago,.

G

From this, this has been our.

D

A

And um part of the way that we've been developing, the metrics models is like you can see here. This is the dei event badging and we have the metric and then we kind of explain why you care about this one with respect to event badging and then why you care about this one and so on and so forth.

A

You know what I mean to help people locate a little bit as to why the metric matters for this metric model, and we we don't have that for this, which is fine, I mean because then we can start building it out here. You know what I mean, and so at one point at some point, we're gonna have this.

A

I mean I can just copy and paste it now um and yeah.

D

A

D

All goes here: yep we'll get in more mark downey format. We can fix that yep and then we'll have some description.

B

I can help fill that that in as well, because these were all based on the research that we did over over the summer, so um I can.

A

Yeah right, okay, perfect context, yes, and so when you were showing the jupiter notebook, how difficult will it be for you to to take the context that say is written here by elizabeth and get that in the the jupiter notebook so that when somebody is looking at, you know they understand why they're looking at this with respect to welcome welcomingness yeah, I don't think.

D

It'll be hard, I think right, we probably would provide once the model is released, some kind of canonical link so that if it changes but we'll just probably copy and paste that context into a cell, so that someone can see it right as they're looking at some data.

A

Okay, because yeah just watching you go through that, I was just realizing that we need to keep this aligned with yours, like the notebook aligned with this marker. This google doc slash markdown file so that yeah yeah.

D

Cause like one.

A

Of the things we're trying to overcome with metrics models is just giving people a lot of metrics.

A

That's what, and so we have to give them that context, and I think this this markdown file or this google doc here is going to be that context document, and so just thinking about how those two link would be cool. I agree and I like your idea of having one source of truth um and not.

D

Two yeah I mean tell, will tell people where the source of truth is, but uh also probably get things into the notebook. Okay, at least initial adoption.

A

I'm wondering you know how like, when you do the um graphics, the images, then you provide the the caption I'm my my brain is like yeah. Words are slow right now, but the caption like you could put some of that in there perhaps yeah.

G

A

Okay, yeah for sure, okay, um just the why you care kind of thing: okay, cool! Thank you, sean and regava um is so what's next for, for you guys.

D

On this, I think, if you know, we've gotten some feedback here and we know what isn't done yet, and I think this adding context is really important. So I think, between now and when we meet again we'll have something that's publicly shared that people can look at right now. You know for you emma, I think.

D

Certainly I can. I can give you a. uh I give you a week we'll finish this up before the holiday, the christian holiday that shuts us all down so um I'll. Send you a link in the next week to 10 days.

H

D

With the jupiter notebook that you can run against dapper.

H

That would be amazing. Let me know if I can help too, like you know, if you want, if it would be faster for you to like work together with me online or if I can, I don't know, just yeah, so it's not.

D

All on you, I mean, I think I think the the like I'm gonna use auger, because that's what I know, I'm just gonna gather the data for dapper and create some comparisons and I'd be happy to work with you on that.

H

Yeah, whatever I can do to.

D

I think if there's like, we talked about comparisons a little bit tonight. So if there's.

I

D

That the dapper folks put themselves in context with that would be good to know.

H

Yeah I'll try and actually now you mentioned like that- was off the top of my head um I'll. Maybe just try and send you something tomorrow. Okay, do you have the main repo.

D

H

Yeah around comparison, okay, yeah, I get, I see what you're saying I'll get back to each more. My brain is also.

D

H

D

So much distraction about who do you think their competition is.

H

D

Usually so yeah.

A

The we have just a few minutes um and three items, but I'm guessing emma put this one here.

H

I just wanted to mostly talk through my newbieism around how to to what I'm doing now and how to connect it with this work, because that's right, um if that makes sense,.

D

H

Okay, so um so right now, I have to open something for myself to look at while I'm talking so, as you know like inside of microsoft and everywhere, I guess people are, you know asking about metrics and how do I tell my community is uh or my project is this- that and the other thing right, and so I got a bunch of folks together from.

H

The teamswordapper babylon.js.net a bunch from.net and then some researchers and someone who's in charge of github sponsors just to say like what. What do we as a group can commit to work on, and um so I had everyone kind of list, the things they're interested in. Why and then I synthesize that into two groupings which I'm calling uh maybe it'll actually be better. Do you mind if I just quickly share my screen, because then you can? No, you don't have.

F

H

Me talk through what I'm looking at, you can just read it hold on a second.

H

Which screen is it hold? On a second sorry, I've.

D

H

Moved all my monitors around recently, so I'm just like! Oh that's over there now hold on.

A

Can we record this to emma is that okay, yeah.

H

It's totally fine, the only reason I can't drop it is because it's an internal wiki not because there's anything like classified or whatever the word is, I'm still figuring out how to work within the confines of like you know, you know psychological safety or whatever you call it and um just sharing stuff.

H

So can you see this wiki in.

F

H

Right so um kind of jumping to the end thing here is like there's these two areas that are really clear and strong, like people want to know the very basics, and I think sean like you know- and I these are the initial ones that kind of came out that was like discovery, responsiveness, which you know like some of the stuff you're, actually talking about um something around contribution.

H

You know by type org region. These are just things that I personally made up. So I don't meet. You know, maybe there's other categories that exist. um Things like you're already talking about again so exciting, like the code of conduct, finding it um and usage. So usage comes up a lot. um I don't know if there's anything there and then the project sustainability is the other category.

H

So that's um maybe a tougher one, but definitely like a bucket right, like the github sponsors folks, want to make sure that you know that we're supporting projects that are you know um that that we're supporting projects and helping sustain them, but that there's certain things we look out for um you know that they're an inclusive project, but also you know we want to see if there's burnout risk somewhere. We want to send folks money there um yeah so anyways. This is just the things that I came up with.

H

So um what and then I also some um we actually have some existing metrics around use. As an example, I'm just gonna move my hopefully the I'm trying to be mindful of your time here hold on.

D

Most people might need to drop off.

H

So I I have my own spreadsheet, which I know matt you'll be so excited to hear. um So then I just start all I did was babylon has some some metrics they already use, which are like under discovery, usage and contribution, and um what are those like? How many open issues there are?

H

How many like git depend turns up things, so I just documented what they already had. So hopefully we can either validate what you have contribute these, what they're doing, but my next step is to kind of start to fill in the blanks of um of these, and you know I would love to do uh in our next. When I bring everyone together again is show them some of that auger work, connect it with either um open source or the 101 or the sustainability yeah, and then uh because there's the auger work.

H

This is where my nubianism is coming on like, um but then there's also other types of metrics that are more like things. We like evaluate yeah.

D

H

A quad qualitative perspective and I'm- and I don't that I also don't know where that fits in. So I have a lot of questions, but I don't want to bombard. I want to figure out how to be agile here. I.

D

Mean I think, I think, the safety things that you have there. Those are potentially reflected in some of the dei metrics that you developed with us yeah a while ago, perhaps under a different name,.

D

B

Developed, I think the challenge is going to be. How do you scale that or how.

H

Do you get exactly data because.

B

Those are not yeah.

H

Yeah- and I know that the for good folks so github and at microsoft, the github or the open source for good folks, that's one of the things they're trying to evaluate both from uh like helping identify projects that are. You know bad. I guess for those that need help, but yeah you need to scale to do that like going through each repo and going oh, is there? You know this like.

B

It's hard and most of those I know like in the case of the psychological safety metric, mostly just surveying the community members a lot of those. So I mean there is a way to get a. You know to get some data around it, but it's tricky because you're relying on the communities to survey their own members and yeah.

H

B

Might not be consistent and it might not, they might not roll it out properly. Like I don't know it um it's. It's might take some work to to make that scalable and consistent.

H

And I almost wonder if there's like iteration, like an iterator iterative approach to these in a little bit like you, would run something like auger to identify like all of those that had a code of conduct and then you know like automated as much as we could and then got to the point where we started to ask questions yeah. I.

F

H

What that looks like either, but.

E

So I think you're breaking ground here and it's really valuable. um So, for example, your project sustainability list is kind of a model in itself uh and um okay, and maybe it would be helpful just to kind of um you know, work in that with the group as a whole. So talk about the safety stuff and how do we measure burnout risk and so on and the whole community to um contribute to that and help.

H

Okay, yeah and the goal just to be clear, like I want to contribute as much as we can back if there's like auger functionality that you need engineering time on I'd, love to bring that up in the group too sean. You know there's engineers in there and there we want they want this. So you know if there's areas of work like I'd like to bring that too. So, maybe when I'm sending you my projects for comparison, you can send me the like things. That would be helpful.

D

um A lot of them are in our issues, but I don't know.

H

Are they responsive there.

D

You know sometimes like I'm generally pretty responsive, but um sometimes I keep issues open until the work is done. So some issues do stay open a while.

H

Okay, I can just point them there for now, but then also like, if you have a chunk of work but yeah.

C

H

That that's great I just wanted to like you know, that's what I'm doing and, however you want to collaborate, and I, however, can bring back stuff.

D

And after the first of the year, chaos is going to start developing a kind of common software tool group, including gremore, lab and auger, where we try to build up our contributor community for the software that we have as part of chaos as well.

H

D

And I think that would be a a great entry point for for engineers, although if they're bored over the holiday, I'm happy to help.

H

Matt, should I add something to your spreadsheet like are these useful, or these are like too specific to us or what I've.

A

Already done them down- oh you're, not in the spreadsheet, but I I'm with lucas. These are both. These are both metric models so and in fact, in the in the uh notes you have it called open source project health 101.

A

um We call it uh preliminary health report, so I I actually put a link in there that I think these two could probably come together in some way.

A

I'm like I'm with lucas that those are things that we haven't brought together before just looking at your list, but that are those are certainly candidates for for metrics models.

A

So if that's what you're talking about getting those in the spreadsheet and starting to build those out yeah, definitely.

H

Yeah, I'm trying to figure out how to work with you best and not be separate. Okay,.

A

Oh, if you want to put them in there, that'd be great. I think project sustainability is not one that we have in there. I think that health 101 one is is something that we could merge a little bit with some of the work we've done already. What.

H

Do you call this one.

A

uh We call it preliminary health report.

H

F

All right, I kind of like health 101, but whatever yeah well.

H

We can always change it: okay, well, okay, anyways um I'll check in again in the new year. Thanks for your time, I have to take someone to soccer here.

D

Thanks everybody emma.

A

Thanks everyone thanks lucas.

D

Thank you thanks. Matt elizabeth hi.