GitLab Development Metrics working Group, 1 May 2019

Previous Meeting

⏯

youtube image

►

From YouTube: Development Metrics Working Group 5/1/2019

Description

Development Metrics Working Group Discussion
Agenda: https://docs.google.com/a/gitlab.com/document/d/1Y50uhpRW0zSGWI-TzPxHnwEHyOl7uWiyCzXtpRJd1_E/edit?usp=drive_web

A

Oh so it's May Day, so I just put that at the top there happy May, Day everybody we're gonna, go ahead and get started with looking at our normal persistent links associated with it, which, um since I got the new webcam trying to figure out how best, just so that I'm actually facing forward most of the time it's supposed to turn off from the screens. But I can't always do that as well as I should so.

A

B

A

Do is just kind of go through real, quick, so numbers are in for May and for a Polish would say we had our largest month ever for total M ours associated with it 1775 it's a great year, so it's the the independence could argue that the show than the July release for anybody whose us basically 1775 it's decided, 76 I, guess so we go. We can go up one more, maybe anyway, so that was very encouraging. From that perspective.

A

On our average, mrs excluding per author, excluding community contributions, we did see a rise, not quite as much as maybe we had hoped but still arise from eight point. Five: five to eight point: nine three that one's interesting, because it's a little bit lower from the prediction perspective but did fall within the error range that I can originally set out kind of based on that one. So so I think that was encouraging from that perspective.

A

Actually both fell in that way and then the third persistent link we have is get lab time to close s, 1 and s 2 s and I, don't know Mac. If you updated this earth. This is still a last week's data. You.

C

Actually all took it over. This is the last week's data Tonya here could help me polish it up a little bit because we found out that we should be looking at customer affecting bugs as a whole, and we add another chart and I. Don't see your sidebar here, Christopher there's a malaria shard added for all customers.

C

Customer only all is three more. Yes, that's the one. So we had limited data on customer as once we expand its goal and I. Think it's it's the same trend here that the the numbers are increasing. We didn't have numbers in forward with this month, yet my goal is to actually try to automate this in a very boring and we see way because it's a lot of work trying to export this and and dice it in Tania has been doing some work. I'm, not sure she's ready to show yet, but essentially is a template.

C

Google sheet day we could just drop in the CSV and just just recalculates dynamically on the fly and we get a short refresh. Every time we do a CSV export. So that's been going on our side.

A

Are you ready, Tonya or next week, I have.

D

One that's working I right now, it's custom for the hypothesis three that I'm supposed to present today, but it would just take me a couple minutes after the meeting to update that and then I can add a note. Okay.

A

C

Pick Nix Nix Nix next.

A

C

Absolutely yeah.

A

A

Yeah, do you want me to update the link to this page by Mac.

C

Yes, that's why we is no one to look at just so that we don't okay, all right.

A

So I'm just gonna do this live.

A

Just double check this here.

A

Reason, I'm double chickens, because if I text it over of the description but not the link itself, but it did actually do the right thing there. So just double check on that cool all right. So that gets us to through the first item of persistent links, status of extra names and hypothesis. So do.

D

You go through number three I think.

A

D

Let me share my screen. Give me one second.

D

Okay, do you all see a chart? Perfect? Okay, so I wanted to just go through this really quickly and explain what these lines are. The black line here is merge. Requests per regardless of its author is an individual contributor or manager at the time. The red line is for people who are individual contributors at the time that the merger quest was merged, and you can see it really closely follows the total line, the blue line at the bottom.

D

There is for managers, specifically people who are managers at the time that the merger quest is merged and it is much lower than individual contributors. But overall, the number of managers are so low that it makes a very small difference in the overall trend and then I did get data from Britney about people who were hired as individual contributors and then were promoted to manager. So there's people like Dennis and they did start out as some of the most productive people in the company and their numbers should go down as overtime.

D

They were promoted, but this is only about a dozen people. So again it doesn't make a huge, a huge difference. In the overall numbers, so I think my takeaways and please validate these. Yes, managers are less productive. However, there are so few managers said it doesn't actually make a material difference and the overall trend.

E

Promoted managers particularly and.

D

Yeah promoted managers to their a dozen out of I mean it's less than ten percent of the total population. It's not a huge huge number and there is also a tab here for results. This is what the chart pulls from. So here's total you can see the numbers here, the counts for individual contributors. It's I mean again the managers, it's about 10 percent, less than 10 percent, for example, this month of the 6.1 percent, a lot of managers aren't contributing at all, so their numbers aren't being included here but of the managers who contribute it's.

D

It's such a small number that it doesn't make a huge difference.

A

Did a you didn't check the histogram on that you just I'm assuming got the average like it would be curious to know if anybody was like super prolific so like like. If you see Stan's Stan's contribution numbers are pretty outrageous and a good, but you know because of that you know it's it's it's significantly impacting and that's the only thing I would say is if I understand.

C

A

Justice Clark remand I was just curious more if, if we had any significant outlier from that perspective, I didn't.

D

Look at outliers I tried not to look.

A

D

People for this and then said, look at trend. I didn't want to call people out. Okay,.

A

D

Inspired to call people out yeah.

A

The reason I asked us because, like if you had a ten to one, if you had to tend to ones like that, if the numbers make sense to me from an averages perspective, if he had say to ten to ones that happened, then the runs alive 2018 that they got promoted. That would that would explain a shift from that perspective.

D

Yeah and to make sure that I'm, clear the the red line and the blue line, so the individual contributor line and the manager line, it does include people who were promoted, but at the time they were one or the other and they're sorted accordingly. But the green line is regardless of if they were in I, see our manager at the time. The merger quest as Marissa's. Just all the requests.

A

D

Think it's validated but I don't know that. There's any action item that can be taken. I mean. Are you just not gonna promote people? You know.

A

Well, definitely, we're gonna promote people when it's appropriate to promote them. I I was thinking more around the fact of the statistical you know. How much does it statistically matter right like and it's less about that and more to be kind of aware of it. From that perspective, I guess is maybe the right way to put it trying to find the right level of grading here.

A

To from the bottom.

E

This area jumped in a few minutes, in which hypothesis is this that we just covered. This is my hypothesis.

A

Number three: no.

E

E

So it's I think they answer your question tying at the the way. The hypothesis word is a little bit sort of like underspecified, because we're not talking about like the impact that the potential impact- if this is true on our productivity numbers right, so we're saying we're saying we think this is true. We think people get promoted in management, they are less productive, but are we saying this is the reason why our productivity dipped in the other graphs or not I, don't.

D

Think that we can conclude that, based on the small numbers of people who made their criteria, seems more like an overall trend. Okay,.

E

So then the I would add: I proposed ahead of the beginning of iPods s like string clip over clear. Our overall productivity has declined because longer tenured, IC employees, population, cellos, productive risk manager, population have gotten less productive and what we're saying it sounds like his managers have gotten less productive, but that is not why our overall productivity is to client because percentage it.

D

Seems like everyone has gotten less productive and managers have been sucked up in that as well. Okay,.

E

So now let's say we: actually you got the right answer, but I think we actually falsified this hypothesis as I just recently reworded it. So it flips from green to red, but that's still a good result and then that that helps justify why we can't determine what an intervention was, because if it's green there is else tells the other definitely something that we should do, but whereas for so there's not in this case, does that make sense to everybody. I want to make sure I'm not like.

A

It makes sense to me: I, don't I, don't know if I would ever I don't know if I would ever the veracity of the hypothesis doesn't always necessarily translate to me I'm, like I, don't know if I would necessarily okay and you're saying why you want to make it change the wording so that it does this hypothesis fashion. You know it's a hypothesis if we get a result, that's validated, but it's not something we can action on. That's that's! That's! Actually an acceptable outcome in my book. I think.

D

You've been the way it was originally worded, it would still be read because the ten longer tenured IC employee population is not still as productive. That's the overall thing that we're investigating right.

E

That's true, so there's a third result bullet here longer tenured.

D

Yeah, the in that graph that the chart the red line and the Green Line earth started the red line and the black line, the overall line and the IC line were very matched over the whole whole course of history.

A

Yeah definitely say right now: we were only promoting a subset of folks.

A

If we ever got to a much more aggressive state of promotion, it would start affecting this line would be my like, like we validated the hypothesis that if you convert somebody from a more experienced we just in this case, it wasn't statistically relevant, because the number of population was twelve well to the group, which is less than 10%, but if I think it'd be a lot more dominating factor. If say it was twenty or forty percent, not that we would necessarily hit see that in this situation, but that is a possibility of organization.

A

E

You could make an argument that that change actually has happened right like in 2018. We did a lot of bottoms up building and some top down building, and then we learned in retrospect that it was more effective to build teams top-down, so we've been hiring outside managers earlier and putting people in interim positions earlier, so I think that actually has occurred, but had done things anywhere near twenty or forty percent right.

E

We're still like that change, I just thought it was probably between ten and twelve percent, which look like that, because we still have pretty broad spins closely yeah like there's, there's technically something there, but these are to take our first statistical hat and put on our business at and just say, like it's not worth spending time pulling on this thread.

A

Cool that means we have number seven.

A

Issue maintainer said struck over time, which has Lee impacted our average. Mr / author.

A

We had that associated with number five and the fact that it was taking longer to merge.

A

And what- and this is uh this is actually me- jumping on the sword again week, two or even had I haven't had a chance to investigate this.

F

So if I may jump in Virginia, this is this is a great hypothesis hypotheses that cycle time would help us dramatically. So, if you remember from our conversation, I was talking about determining wearing cycle time, we're spending so much effort having the breakdown in our cycle time algorithm to say this is how long the review cycle was would basically help us make a determination here. So this in any case, while you know we're looking at it and Christopher, will look at the data and we can discuss in the future I.

F

You know this is a great case for cycle time.

A

Cool and then I'm hypotheses a comment to Jeff an update.

B

So this one I cut to evaluate a little bit more kind of what we're trying to achieve. It seemed like we're trying to figure out why longer tenure engineers are more productive than less tenured engineers, so I kind of broke it down to a few things that we can test. We can send a survey but I'm not 100% sure, but what the purpose behind hypothesis is. So it's hard for me to figure out whether there's something we really need to dive into deeper.

B

A

So if we found my my supposition is that if we found that a fact that matters is that they're, the onboarding is cumbersome or maybe non-existent right like if you come with your own methodology and you're continuing to use that methodology, and it turns out that that's not an effective way to operate, then that may actually in turn, you know your remote employee, you don't have any feedback loop, consequently, you're, not necessarily seeing that result right so like that, would just push us towards better training, associate things to say: hey, here's a way to be more effective, or at least you should consider this, as part of you know, pretty great efforts associated with that I.

A

F

If we could tailored this one around, can we can we graph the complexity of the product? So that's one of the the things that developer mention is that it is more complex today to contribute to the product, and it was you know, with a 25-person engineering team, which is you know the longer 10 year folks who've been around.

B

We could also potentially gauge like the time it takes people to do that. First, mr I know we've started to do that for onboarding as a metric, but trace it back to older engineers and see how long it for them to get started. I would be a better potentially a better representation of onboarding and I'll be ramped up faster yeah.

A

I'm a little resistant to the first, mr because first, mr isn't gonna get us to I, say 10 MMR's per month right like like it's not a it's, not unnecessarily a it's causal, so I'd like it to be from that perspective, but I I was trying to figure out if, if, if, if we could validate or invalidate the hypothesis before, we were down the path of how how we best solution a you asked the question of what would we do with the information think we'll we do with the informations, then figure out how best to you know, get people up to speed or quickly, I'm being productive, which is more than skiing, up people, this beaver, it's getting up, people to speed and productive right.

A

It's right way to put it.

E

B

E

The same thing I mean if you're up to speed, I think it as soon as productively, because that's the verdict, 2d cuz, that's the job, can I ask some questions like one when we say less tenure developers with less tenure come in is that refer to them having less tenure at gitlab, which is of course through new developer, or does it mean we're hiring people that have less tenure in their career like.

A

Dating it to reflect that, which is the.

E

B

A

It was is that kid lab so like a senior.

E

A

Comes in but they're still new to the company, because one of the things that uh Clemente and Dennis found was is that I sees with over two years of experience, I think was actually we're more productive, which I was really surprised by I was like gosh takes two years to get there. That's that's. You know surprising. At least I thought it was surprising two years at get lab two.

E

Years at gala, and then the the different methodology thing like what's the, what are we looking for here was some example of the different methodology.

E

I think the two things I could speculate on are of like yeah. You come on. You just don't know the code miss it's like three million to two million lines of code just takes a while to learn that, but then also the only thing that, but that's not really methodology. The thing that was striking methodology is that we're measuring Amar's.

E

We have a culture, particularly long-standing culture, really doing small granular bars and being very iterative. That is probably the main thing the new developers have to learn if I were to speculate and yeah we've actually been doing that, and we've been seeing that a positive increase on this average number, but are there other things that yeah.

A

As an example, if the ringette lab on a small box, prototype man being somewhat theoretical here but like if it turns out that our senior senior developers are running on a small box, prototype versus using a cloud since it gives them quick, quicker iteration that could be, that could be a determining factor right or they open up the RMR right away and submit upstream to- or you know, basically get the armored mr open as quickly as possible. Even if it's in draft form, because I want to verify that they haven't broken anything else.

A

Rather than bring the test locally as an example and ring that parallel things like that, or it could be the opposite way. Right from that perspective. Mm-Hmm.

E

How about you, Clemente, or maybe you Dali like, do you have a sense of what people need to learn.

F

E

Like a two-year tenure disease as I, guess, they specified yeah.

F

I, like elaborate on kind of the complexity of the product. What I heard is that you know by nature being a monolith, there's so much more to learn where people who came from a more services oriented. They feel that there, the expectation is that they can learn their service really well and get up to speed much faster. So that was one comment from you know new engineers and- and this was a senior person, so they are capable. But just by factor of you know how big our application is. It's it's a lot to consume quickly.

F

So one of the things that we're doing on ops is striking about component ownership, and how can we very well define the areas that the team needs to focus on I'm curious, if that's possibly a factor for other engineers around you know in the other groups, as well as.

E

Simulating the maker services yeah.

F

And I mean III I feel like it's a conversation we we should talk about as we grow. We we have more engineers and more lines of code and you know we want to operate efficiently so.

E

I think that's actually a separate thing right because again, that's not a methodology, but I do agree. That's important, so I think there's almost a separate hypothesis here of like people have too much surface area to learn quickly and that's why they're slow to learn and what we should do is sort of create synthetic microarchitectures just in the form of areas of the code that teams and individuals are responsible for and therefore they don't have think about anything outside that and then separate from that.

E

There's also methodology like work breakdown, small m ours types of things, Christopher mentioned other stuff that we could actually put as part of homward. This is how you know how to be a developer and so I've hired. You decide to be a good web developer.

E

Yes and then I would take, tell you your point move to this one.

A

If that makes us abandon eight as a investigation hypothesis, that's no amended necessarily a outcome, but it's this. It is well. It is an outcome in regards to it as an option here. No, we want to focus more on number nine. It's supposed to eat.

E

Well, I mean the you know the data would be do we may already have the data, because if you were to try to confirm this, you would say like well how many? What's the average mr by tenure, we've already collected, that data we've shown that longer tenured employees do more Amar's on average than short-term employees, so the interview may be ready for an intervention.

E

This one and just say you know, do something in the onboarding process to really be coaching people about how to break down work and how just how iterative we need to be and I can definitely say from a managerial perspective. Since said our iteration value is gonna challenge me I'm, like yeah, whatever.

E

No we're sure enough that that turned out to be true quite unusual to ship such small units of work, frequent reorg small process changes much smaller than I've done in the past, so I mean I, mean I, couldn't I see exhibit eyes with engineers going through the same thing of like that's an M R.

E

That's what would be doing anything you know, naming I, think you, you know like people like Clement, you probably have more of a ground truth on that, but I would say if people feel like, if that's data they could falsify twenty have that data, then I would say we could jump to that intervention and try to do more onboarding around iteration, let's say or clear things. I.

B

Think there's always gonna be improvements. We can do to help people do that shift and thinking about smaller version requests. I know, for me, a big part of it was a big part of my shift when I joined was when it was like: hey the salary calculator, it works, it's not perfect, it's not UX approved, but it makes improvement. So let's merge it and then he merged it. So that was that was really big for me and understanding. What iteration really is.

E

So like with the data about tenure, where was that what I bought this was that.

E

Wow 21 references to tenure in ER. That's.

B

Number two I think.

E

E

Then I could just copy paste this down and say.

E

So this would apply to me that we've confirmed at least for this one small, smaller MRA's example. We've already demonstrated it and we could actually identify an action to add to our onboarding. But we want to keep this. When is that a good idea, and then you'd want to keep this open and look for other methodology.

A

So your your philosophy is is to really push on the smaller air Mars aspect of it.

E

Well, that's the that seems to be the methodology difference that we demonstrated that we've got data to support. We accidentally collected it in another office, Nicorette.

A

We collected that they were more efficient at producing a Mars. We didn't actually validate that that was smaller in Mars and that data did. We.

B

We didn't compare the size of the merge request, yeah.

A

So it feels like maybe next step is to actually go through and see whether or not you know take you. Don't necessarily do it if you do it for all that's great, but if you want to do it for like take to the foot from the population of two plus years and two from the population of say three or six months of tenure and do a comparison of their, you know lines of code as an in curve.

A

You know merge count associated with that and if we see a dramatic difference between those, then then that wouldn't be validation. In my book.

A

So I should be writing that down.

A

Which I guess you have is number one compare size of that Mars by tenure.

E

Yeah, so then, then, we're not ready for an intervention, but that's a good action can be.

A

Sampled, like I'm, okay with.

E

Thank you I think that's. Why do a medial account we'll do a double blind to manual counsel.

A

All right does that cover 89, well, no furs or other aspects. Clemente, do you feel like you have enough that you can run with it? You know.

B

I'll tackle I'll, keep tackling eight I'm, not sure about nine, though.

E

E

Yeah you're talking about graphing the complexity of the product. The other thing we do is we just ask people their impressions about learning it. But what would you be idea to graph the complexity of the product Talia.

A

Dropped off so it's going to hurt for an answer that question she had an interview, I believe I, like the idea of asking people, it's cheap, I, wonder if what we do is a survey, question and survey question of ask how often you look up something.

A

When writing code or designing like it's, it's kind of a it's gonna, be a little bit of a hand wavy, but I'm, just wondering if I like, like we like, generally speaking, when I'm learning something and I'm trying to get more productive or than it being at the front of my memory, because I'm still trying to figure it out, you know I have to spend a lot of time. Like you know, oh I've gotta go with this reference.

A

I gotta go check this out as opposed to just having it built into my head or there'd, be the checklist to sign up for an M, R or those aspects. But if other folks have better suggestions on what questions survey question I asked her. You know whether we should maybe just go interview a couple folks, that's the other option. I like.

E

That because I think the alternative suggestion is we a graphing. The complexity of the product actually would mean like taking the history of every developer and seeing which areas of the code that they touch running through k-means and get like a nonparametric answer to areas of the product are there and then we name them, and then we turn that into architecture. So I think that's like a seven-year dissertation project for the PhD candidate, when we can just kind of get people sense and chart something out. It's just the same answer: okay,.

A

All right, I'll take an action to talk with Dahlia about that and see how we can best get some traction there. From that perspective,.

A

Cool, so that's the hypothesis half an hour in we're. Finally getting the point where we can welcome folks carrucan. It was the beginning of the discussion. Sorry about that, so welcome, Lyle and Virginia. That's a group Mike has that listed below and then also mark, Remy and Jason are also joining, though I didn't get them added early enough, probably so I'm guessing. That's the reason why they're not here but they're, welcome as well in regards to adding them when we expand it or charter.

A

We had a bunch of more folks who will start showing up here. Mike did you have anything to add.

C

Both mark and Remy on vacation and now expected me to be joining next week. Okay,.

A

Well, we'll talk about that. A second here cool and then numbers for April are in I, haven't had a chance to do the okay, our calculations, just because they came in about 10:30 this morning, Mrs / author is up again eight point: nine three yeah and also M ours, total M. Ours has gotten to 1750 1775, so really really good results from that perspective, and then it was just curious for feedback from the team of you know at the risk of any more work.

A

Is it better to have these issues and hypotheses tracked and a board, or do we like it? Its current format from a document perspective.

A

And I'll assume no feedback. Is we.

D

Should dog food huh any issues we.

A

Are using in shoes to track things we're just not using this really aboard from that particular, so the question is: is you know, is it better to track these as issues within the Dockers? If there's a board associated with it, I vote.

D

E

And then is that for each hypothesis is one or does that mean each action item? Is one hypothesis.

A

Exit exit criteria and any other actions that we need to track in this form because don't forget our trailer expand like I, was hoping to sneak out the door with exit criteria, and it turns out that we added a whole nother set of metrics who want us or Basin things around.

E

Yeah great Tanya, wind out dog food I.

E

On that note, heard Brinkman reach out to me, because I mentioned that we determined through the course of dog fooding on our management board, that we would love to have got lines and turns out that got prioritize. I think that's coming in the next release, so it works. Do.

A

You like that, out of curiosity, do you like the to label that you have to set up words I'm, not a big fan of it. Sorry, the to label yeah. So if you want, if you just wanna, have a board, that's like like for like multiple projects, and you want to bring in issues, you have to tag them first, with a label to get so gets them on the board, and then you had to have a separate one depending on which column you want it in associate with it.

A

So you have an engineering one and then I think you have like you know. Vp of engineering development director or in a development department used to the pur ones listed yeah. So.

E

I actually I like how composable they make boys personally I do think it creates a lot of visual clutter, so I created an issue to actually you know like art, many projects I'm checking out this is our engineering management board and itself and all the directors different departments work.

E

Every issue on this board needs type of engineering manager. That's how it appears on the board, so I have an issue to just say hide this issue from all of these things put a title of the board up here. This is an engineering management port and just have something in parentheses, about the scope. This is like you know, has this label, and then these gray ones just kind of like disappear. Then you can do the same for each column.

E

Every issue in this in Christopher's column here has this orange development department label hide it from this column, but just put it up here like leave it right here, as it is just imply that like yeah, it's all here, then, if you want to see every issue, you can see it in the detail view or this preview. You can see like the completeness on initiative that me says to me, but product management isn't really it's not really flying for them.

E

They don't want to create a button that either hides all labels or shows all that was and I said well that you know rather than a sensible default. You give me a button with two states, neither which is terribly useful to me generally like that, but I wanted to be sensitive, not pulling rank and advocating too strongly for any one thing. So what I'll do those they've got that issue right now?

E

You all can move on next topic and, if anybody's passionate about my idea or has one of their own doing that issue and sort of pile on about it and that's how we'll win. That's all when people over yeah.

A

If you want pile on to that trying to think if I've got it up here- and this is a we're getting way off topic, so I apologize about that everybody else. But if you look at the petrol cord which I think I can pull up really quickly here,.

A

This is this: is the retro board with that model?

A

Okay? So if you, if your choices, turn them all off, it seems like that's a really bad idea, but like removing one or two around that when the case when you have like you know, 20.

E

Here's what I did I was bored one evening and my board I mean I had important stuff to do. That's procrastinating, okay, except sharing and I'll jump in any share. Yeah I found the issue yeah there we go so I went in the chrome inspector and I just deleted all the labels and took a screenshot. So this is what the board looked like at the time with all the redundant gray and colored labels and then without it you can see.

E

You know you only see the labels all the sudden start to jump out at you. You realize that, like oh, this is important. This one's important cetera had.

B

The effect that I was looking for I.

E

Was anyway, I will put that issue in here and if you want to go to go for it and you will change their lives, okay,.

A

Cool all right and then I had an action item from last week, just to update that which is training as a set of criteria. I talked to a couple of our directors and they definitely would like to managers to be held accountable to the number. So we need to set up, but basically push for training associated with that.

A

So that doesn't change the actual document, because we already had training as an item in there and then the.

B

Next one I had on there.

A

Was is this author versus versus org assignment debate that I think we had early on where we talked about?

A

You know using either a group or some some component to basically measure the number of people, rather than doing it dynamically based on I, looking through the authors and then and then adding them up, I think I, think I've convinced my wife I've certainly convinced myself I'm trying to convince the rest of the team that we really need to use the orga Simon methodology here, two two keyframes 1, if you don't have the order to assignment the metrics you're looking at like if you look at average m RS per month average for 10 hours per engineer per month, the reality is is that you won't see that number grow, because it's just highly dependent on when authors get involved, because, like you had a high population at the end of the month, you've got involved in did a single mr for the month.

A

You would actually see a decline in your average, mr s per month, associated with it.

A

The other thing is: is it doesn't account for people outside the develop organization who maybe aren't spending as much time, and if that puppet, as that population grows, it's gonna actually force us to force us to be explaining that aspect of it, which is you know, hey the rest of the population, is crying but they're, not necessarily either held to the same metric or or looked at in the same way from a Productivity perspective, they're helping that we're helping the product grow. That's a good thing.

A

We don't want to discourage that, but by optimizing to average in Mars per engineer, without really being definitive about it, it's it makes the problem more difficult to look at mech, you're gonna. Tell me the technical reason why we can't do this. I.

C

Don't think you have tracking into product, for this can.

A

We well, you got members right, like you, have developers and maintain errs right.

C

So if we keep a separate list to track the members and then plot it out in the chart, it could be possible. I made my trip: let's go at it, I.

E

Was gonna say so Christopher I understand you're saying is that like, ideally, we would have groups who represent these different product areas and we would look at the average of our per this group and then we would take on the burden. Could people have the right group attached to them? I think my instincts are. Is that if it's not necessary to be in the right group.

E

Necessary to have that to do your job, it's almost a hopeless battle to keep that up to date, because there's many similar things like our Google Groups and other things that, like we rely on our onboarding issues and people ops to do for us and managers and like the compliance on that stuff, is like 65% at best, which means this data would never be good.

E

What what we could do that would be a fan of is that if we tackled values went up above about creating the synthetic notion of microservices in the codebase, we just looked at those that theoretically would never change. It would be a target and the code would always be coming there and I think then we would have 100% of the product covered and, and that would be stable and I'd, be much easier to keep up-to-date.

E

If we could solve that first problem of directories or files or something to say this is the create code base in the monolith.

A

Yeah, I, guess I guess I'm less concerned about it on a per team basis. So that's that's a good point. Cuz, like the the one thing I was distinguishing between was overall organizational aspect versus team, and once you start getting, the team does get unwieldy from that perspective. So that's that's. Probably a good aspect to kind of call out.

A

I think the thing that I'm trying to figure out is is you know the way we actually decide by team is is or by by stage is by labeling I think as opposed to by any other method right. So.

E

What's the necklace, your instinct on like how good a job we're doing as a company and historically on sending things the right labels we're.

C

Getting a lot better, we leave for one in force, singularity on on stage labels now and there's an effort by Remy just to switch from group labels to new stage labels instead, so I think we should double down and we can double down on that as a better accuracy than just team labels, and one thing that I would say is: we are somehow double counting in some areas because we don't enforce one team label 4mr. That's not something I'm happy with.

C

If we switch to you stage, they both and it'll be improved, but it'll be pegged on the stage notion, not up like a individual.

E

So, even if so, if I understand, you're saying graphically we're- probably not good enough at this today to trust data that come out of it and even if we fix it, we're not historical data and.

C

Not accurate, we can try, but it won't be accurate because we infer from a team level to reapply those stage labels and if we run into two of you had to pick one or the other, and it might count to some other stage stage group which is not accurate. So.

E

I'd be I'd, be skeptical. We have the the data I gene here to extract a lot of meaning from it. We do it's.

A

Also, the it's also the major like you know, it's the major countable number that you know we're being expect to be measured to write.

E

You're you're sort of advocating for your directors and managers see their slice of it, so they can be accountable to their number right, which I understand. But at least you can see your whole purview. I would bet on labels being easier to achieve than groups, because we've already got triage bots they're, like MEK, it seems like you might be able to enforce that one, and only one rule a lot. I guess this. A little closer I mean.

C

Let me write this down.

E

My second point here is sort of like yeah. If we solve cold fusion, we can definitely do this second thing: Natsume.

E

A

The Rose I'm, hesitant on the microservices I, think I think. Well, you know you.

E

Can argue that the micro-services.

A

Maybe your repos, like that, that's actually how, like you know, one thing: I haven't gone and looked at it's like the distribution team works edifice of a repo, so but their very nature are they more productive because there are much smaller repo as an example, this iPod.

E

A

But what I'm work we're trying to figure out from a top-level prioritization is just the fact of you know. Hey like, like you know, you're supposed to be in trying to encourage people to contribute, but again not everybody's job within the engineering organizations is to be. You know, a hundred percent development all the time all day long and you know as those populations grow. You know what our expectations around the phone are super.

A

If you don't have some key market demarcation and it feels like it's it's uh it feels like it feels like we're, not building a good measuring. Stick from that. We.

B

Haven't do an idea.

E

A

It and we can kind of dance around it, but, like you know it's it's it's it's definitely no slice and dice the data every month, the don't look at it to really figure out. What's going on.

A

But to your point, it's it's hard to solve the problem of tracking it accurately. So maybe I'll like. Why ask for now we'll see how things go, but I just had a feeling that you know if yeah I have a feeling there's at least one cause the customer most likely government-related, where they want to be very direct about. Who is who is working on? What and what pieces of it? And you know those kinds of things.

E

Make is the tree sort of this? Is the triage spot part of the product or is it separate and outside making customers use it users? Yes,.

C

They can use it. Actually, some companies are already using it. It's a it's a public project, that's um anybody can just walk out and set up in their good life instance, and set up rules were heard about to do all these things. Okay,.

E

So this, but it's a separate project: it's not you make it laughs. If I, just download, install good enough I don't get, the ability is just going to start configuring. This right, I.

C

I do have an issue, though cuz I know we talked about it earlier before, and the easiest way to get this is it's open, an integration at the project settings and this it's gonna install this for you and then you can define rules and it's gonna run on a CI run it for you, but yeah. There's there are plans to integrate it to make it easier, but we just don't have the time to work on it right now. I could find the issue first.

E

Well, the way you said is this not a high enough priority to work on it, yep.

C

And before we believe this is a bullet item, I want to said, I have more confidence, there's not a lot of confidence at the data at the orc level. I think that's accurate. It's not double counting the day to day dashboards. The date the dashboard foolish group are helping engineering managers to execute their day to day attaches it's great, but we are indeed double county in some areas. If multiple labels are added to anymore or issues has to be enforced in.

A

C

And that's gonna! That's where a more.

A

Interesting the metric, to be honest, then at the individual teams, or even at the stage, is necessarily because I think you know if you read anything about story points as an example like you expect them to very heavily between we're. You know groups of organizations. So then you have to kind of factor that in as well Emma's.

E

A

Little different from a complex, a perspective, but it's still when it surprised me at all. Actually, the complexity levels of our engineers are varied.

C

um Is this how about this in like very boring solution we could in for singularity on on team labels right now on em? Are you can do that without help and retro actively apply that retroactively? Li remove double-counting labels on in Mars? That's like an NBC.

A

Else with the labeling issue, don't know if it gets to the average shimmer store author issue. That's a population right separate from the labeling issue right.

C

Yeah, it doesn't trickle to that level, of course, yeah and I. Don't think we want to do that at the average number, because that's gonna hone down to a team of group I thought we want to keep the average for developer at a very high level without a pointing any fingers.

A

Well, I'm, looking more at like numbers right so like as an example, you know, I had 10 people to my organization and I have a hundred person organization. You expect it to crease by 10%, but that's only if the way we do em every gem ours per author.

A

The way it actually works out is that if I had 10 people, my organization and 5 other people get added other organizations where there, let's say those folks are going to MMR's per month, and the other groups are to a Mars program, not gonna, grow 10%, I'm, an X, that's much smaller right, I, see.

C

I think, okay, okay, that makes sense right so.

A

It's it's. It's art, art art. Our distribution is getting more diverse,.

A

C

In the end, the end goal for each group dashboard: do we one average in Mars per that group? Eventually, that's what I'm getting here right that will help you with like slicing, which area of the organisation needs.

C

A

I actually I think I think as a top-level metric I I wouldn't need it then look at throughput numbers for individual teams and then kind of go back and figure it out from there I. Wouldn't it so look at average on Mars, like the only time you look at average on Mars is when you look at the overall, whether you think that the group as a whole is is at the same low for productivity or gain more or less productive, productive right, I.

C

Still think in forcing singularity in the team now group labels will help because it will help with the truth right now, if you add NMR, you have two labels. He was showing both teams words so like we at least we address that problem and like now we have a discussion in mr which team actually needs to apply this label if they're just only reviewing, and it should be adding that label in the group.

A

Label I think I'm in interest of time, I'll be beaten in the submission right now right to reserve the right to bring it back up and another month next year. The next thing yes.

C

This would be a short so after looking at the data with with Tonya, we realized that we are potentially missing some data, but looking at the trend, I think, even if we count that missing data there trying to be the same, it's increasing, but we need to address label hygiene and proposing some really boring rules here to help with surfacing bugs faster, and it will involve some helpful support as well. For example, like there are there issues that don't have a customer label, we can actually parse the issue.

C

Description, do a regex magic that has a sales force or send desk lingo and only add a customer label if it's filed by a team member, if it has security at a bug there, there issues with performance, but no bug label there, issues in security that no buck label and there are issues with with bugs customer and no severity. So I think these are boring solutions that we could do to help improve the hygiene and um yeah. That's I'm. Just gonna put it here for transparency and we're gonna jump on this. If.

A

C

Need my support on.

A

That, let me know, and anyway, thank you from perspective if we find a role, non label or speakin of conversations and.

C

I found one with both buck and feature and I'm like. Is it a new feature or is it a bug? So that also needs to be enforced? Legionella pick one it's an either at the bug or a feature and that things like we're having as well my next one so I'm designing a new trash, funnel process and potentially defining levels of trash completion. We have rolled out or agreed on at least the first iteration that the first level tries before it reaches engineering.

C

Man and PMS needs to have a team label needs to have severity needs to have a customer if applicable. So that's the that's the process for that. I was looking to reinvigorate the engineering reaction, a rotation to, and let's help right now, we've enlisted everybody in quality engineering to do this and they're gonna work with their counterpart teams, but any help areas appreciate it, and maybe we can take it offline, as you would decide how to remodel this process.

C

And the second blue item is the the trash levels in a funnel that we are designing right now so yeah. This is the first of the improving time to resolve boxes, assist at the first part, and we talked about how they scheduled another insurer fixes or make being done fast later. Okay,.

A

Cool and then the last one I had was, is reviewing the extra criteria which I will share the screen.

A

So the big thing to know here is that we've expanded our exit criteria, so 20% increase in development department throughput, which I haven't done April's numbers, but we'll figure out how roughly we are doing there. We've got a keep you guys pretty close over a visca level, three need to establish level four aspects of those maturities and then training. We decided that we definitely need to do that from the prospective, and then we have a whole bunch that got added in the last week that we'll have to start tracking. So it's pretty reading the board.

A

One of the things we'll do is take an action to create issues around these, so that we can kind of have issues to track these exit criteria. Does that sound reasonable to everybody?.

A

Cool anything else to discuss.

A

Alright, thanks all.