GitLab Manage Stage, 6 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Measuring DORA4 Metrics in GitLab

Description

Haim & Orit discuss the implementation plan for DORA4 metrics in GitLab

Epic: https://gitlab.com/groups/gitlab-org/-/epics/4358

A

um Okay, so today we we decided to talk about dora metrics in gitlab, uh so my name is rita and I'm a group product manager at gitlab- and I have heim here, who is the senior product manager for the optimize group, who's responsible for both dora and value stream and other analytics.

A

um So in order to do that, I'm going to share my screen, and I want to talk about this epic, which is called measure dora for metrics and give up now for anyone who is not familiar with dorametrics.

A

Dora is a devops research association and they started by creating a report back in. um I want to say 2019 where there was a lot of question. How do we measure roi for devops? How do you measure roi for a process? And how do we know that teams are improving in their day-to-day?

A

This is basically based on the book accelerate, which I recommend everyone to read, and they proposed four different metrics to value to assess how our software development process is behaving.

A

So the four are deployment frequency lead time for changes, time to restore service change, failure rate, it's very important to state that the doral metrics all relate to the production environment. They do not look at dev testing or other environments. This is based on production environments, because what they want to see is how measuring this impact on the end user. The end user only has access to the production environments.

A

So the way that the report.

A

Is built- and this is based on the old 2019 metrics, there's, there's a new report for 2021 that came out, and we probably should update the metrics here, but in any case, uh based on that report, what do we see? We have four different classifications for how organizations perform elite performers high performers, medium performers and low performers, and for each one of the metrics there's a different definition, so deployment frequency is how often I deploy to production where our elite performers deploy multiple deployments per day and our low performers are very waterfally.

A

They have one big release, maybe once a or twice a year, and everyone else is in between lead time for changes is how often does my code from the minute I started coding it until it reaches production and we'll talk about how we measure that in gitlab, because it's a little bit different, but just in general, I just want to give an introduction. What is dora in the textbook definition and elite performers?

A

You know if they deploy multiple times a day, then, theoretically, my code commit can, in the same day, can reach production um time to restore service is if I have an outage in production. How quickly do I repair it and the service gets back up, and this is also a little bit tricky? What is a service impairment? Is it? You know the entire service is not working or is there's a bug.

A

There's a lot of philosophical questions around that and change failure rate is how many, how much percentage of my deployments actually resulted in a failure in production.

A

um Now you can see the top two metrics they're, how they're related to my velocity? How quickly am I releasing things, and this second two are related to stability? How quickly do I fix things in production? Any questions.

B

um And um so with these categories, so uh usually, when we're looking on those kpi, we look them up like two different categories or, like uh everybody, look on the four of them.

A

We usually look at the four of them. However, we have only two supporters currently in gitlab today, so right now we're looking only at two, but we want to complete them to before.

A

um Really it's a combination, because if we look only at velocity, we might be releasing lots of buggy code and then you know my stability goes down, so it doesn't help me to just you know: release stuff out to production without testing it properly or fixing it in time it. So you really have to look at the two together. It's not um really wise to look at only half of them, but they do mean different things.

A

B

Another question is about so from existing customer today, uh uh so I I know we have a lot of uh interaction questions uh uh so uh uh do you? Do we have numbers or typical customers or specific segments? Let's say enterprise sas company, usually uh adopting doral metrics. You know in their production environment.

A

So I've seen everyone being interested in this enterprises and small medium businesses. uh Usually the enterprise customers are willing to pay for it, and really this at the moment is only available for ultimate customers. So the majority of the customers that I talk to are enterprises. However, when you look at the at the dora reports, usually the elite performers are the ones with the new newer technologies.

A

So it's the people who are using kubernetes cloud services. um They might not be those large enterprises.

A

um The monolithic architectures um and the non-scalable technologies have a harder time moving between the performance indicators, so I would say it's a mix. All the enterprises want to move there. It takes them a little bit longer.

B

Yeah and uh usually they are kind of uh measuring their project per group.

A

So it depends who your persona is and that's really tricky for dora, because on the one hand we are catering to the executive persona to the person who wants to know what is my roi. I just put a ton of money on my devops infrastructure in my organization and I want to know what's in it. For me, it's really hard to slap dollars on it.

A

So there's the exact persona that is interested in seeing improvement over time and at the end of the day, you can say that if I deploy frequently to customers, that means more people are getting new functionality, maybe they're paying for it, and that means hard earned cash, which is great, and you can also think about it at the other side. If I'm, if I don't have an outage or if I fix outages really quickly, then I don't have to pay all these penalties that I have for slas and other things.

A

So again, this translates into money. It's. You can't really see the money in in front of your face, but it's there behind the scenes um and on the other hand, we have the development teams themselves that are interested the more autonomous teams we have and the more independent they want they are. They themselves want to understand how they're improving, how they're improving both in terms of quality and in terms of the velocity um and the more automation that they put in that translates into this velocity and scalability.

A

So it depends on who is your persona and we have different personas for dora metrics. It can be the team level and it can be the exact level, and that's also why we're also thinking about different ways to tier this and to chart to charge. In my mind, everyone should enjoy dora metrics, and that's uh we'll also talk about this. It says that here in this epic, how we plan to do that.

A

For now, we started with everything in ultimate.

A

A

You have here what this looks like in github, for example, you can see production change, failure rate and so on, and um this is talking about the mvc. We still haven't completed the nbc because, as I mentioned, we only have two of the metrics and not four of the metrics, but we're getting there. You know yourself because we just put in the morning this in the planning, the other two uh api requests.

B

Yeah time to restore.

A

And change failure rate. So what what are we talking about? In the nbc?

A

uh We actually already completed the two that were uh defined as the mvc and showing um the different trends in different levels, and what we said would not be in the mvc was the ability to sort by milestone additional metrics, which are not the ford dora ability to add annotations. This is really important.

A

Select, custom dates, time to restore service uh by the way custom days already is in the planning time to restore service and change failure rate we're already test uh planning, so we're a little bit beyond the initial mvc, but I would still call it an nbc fedora because without the four it's not really the nbc.

A

So, as I mentioned, the nbc starts with everything in ultimate what we want in the future. So, as I mentioned in the future, we want everyone to enjoy dora metrics, even our open source community on our few free users. This is our stewardship promise to to make sure that, even if you have a free um license, you can benefit from a lot of different things. So what we plan to do is in the gitlab core in the free model to have these tile views.

A

So you, you can't really see the graphs or drill down into different data into it, but we will. You will be able to see a numerical value for each one of the door. Metrics.

A

Okay yep, um these are this is old uh ux because uh we already implemented a bunch of stuff. So we can definitely update the epic. But it's not that important in premium. What we wanted to do is get uh give the customers on the project level access to to the graphs and the tiles like the free. The reason is because um the premium is targeted towards the team view and collaboration and then in ultimate.

A

We wanted to to put it in the group level, and this should probably say workspace and uh instance, and in the workspace level the idea is that the executives and the director level persona are interested, not an individual team, but rather as an organization as a whole or a group of groups, and that's why this makes sense.

A

Okay, so this is in general, what we want to do that then, underneath this epic, I have a ton of different, uh smaller epics, which we'll walk through um there's a lot of really interesting information here, uh where the about the documentation and links to different customers that want it in the github roadmap, and I don't even remember what this video is, but it's probably worthwhile to look at and of course, all the google dora documentation, because google purchased torah yeah okay.

A

So let's talk about the big ones and then we'll go into the individual ones. So the first thing that I want to walk through is zora api support and aura. Api support is important because again we have different personas.

A

The executive level is going to want charts and visuals, but the devops engineers are usually more interested in apis, and so we started with adding api support. This also helps because our graphs are based on the apis themselves, so implementing.

B

This approach, it.

A

Makes sense um so we want to add each one of the four different metrics and we wanted to add it in every single level, on the project level on the group level and an instant slash, workspace level.

A

Okay, and you can see, we already added support for the two metrics that we discussed deployment frequency and lead time for changes both on the project level and on the group level. It's actually a really neat api.

A

Let me open up the documentation.

A

So it's a cool api because um it's the same api for all the metrics, and so you can get all the the metrics or you can select an individual one which is really fun and the api for projects and groups is very similar. So you can see the projects, one is the get project id and the group level is groups id so they're very, very similar, and they give you a lot of data. The api is super flexible. You can see all the options here. You can change the start and end date.

A

You can also set the environment here. So I mentioned dora. Metrics are only the textbook definition. Is that it's for a production environment, but.

B

Using environment, you can also do testing or.

A

You can do whatever you want as long as the the environment here is defined in the gitlab cimo file. You can you can query for any tier, and this is really important for different reasons. A lot of people, enterprise customers that I spoke to that are in highly regulated industries, for example, banks. They can't deploy to production easily. They have a lot of hoops to go through and even though the their velocity is really quick and they deploy things and finish the coding really quickly.

A

They have to get a lot of different manual approvals to get into production, so they're actually really interested in measuring dora for staging and not necessarily for production. So this is really helpful.

B

Okay, so so this is customized, every customer can do it for himself and uh usually with the using the api they eventually visualize the data through to meteors or for through kibana, whatever.

A

B

So the devops team use it for the phone itself. Anyone do you know if anyone using it also for getting kind of an alert or even a predictive kind of notifications before something been kind.

A

Not yet not yet, but it's planned, it's a good question uh and and save your question for later. um For this, this is really very basic api support.

A

Again this is only available for ultimate customers right now, um so something that's really interesting. Besides the basic support I've heard from a customer, they wanted the ability to exclude environments from dora, so, for example, they wanted. um I don't remember exactly what this customer wanted. Maybe I wrote it um okay, so the group level dorimetric is they aggregate the data it takes all the the data from all the production environments and adds them together based on environment tiers.

A

But maybe I remember that this specific customer that I spoke to had different um geo locations for their production, so they they have a production environment in europe and they have a production in the americas and they wanted to be able to compare them. So in this case you can exclude it and then get only a specific production environment.

B

A

um So that's basically the the.

B

A

Support level epics, I think, um and where, as we mentioned, we're adding uh these two in the upcoming milestones.

A

A

And then there were specific ones that didn't want the tier. They wanted specific environment names. So not only group everything right here, but if I have something called production1, I want only production. So that's api support.

A

um The next epic that I want to talk about is the ui.

A

I think I see that it's a little bit disorganized, but.

A

But let's talk about that so, for example, I have an epic for deployment frequency.

A

I should mention that um as a way of starting implementing dora, we use the gitlab iteration value and since this started from work from the release team on ci cd, this was added under analytics cicd. But I think dora is important enough to have its own separate location, which needs some ux work. But we haven't done that.

B

Okay, so the name of the so in the navigation on the left side. So each one of them, you call them a report, a page, an analytic page.

A

Can you repeat the question.

B

When, when we navigate under the analytics section in the left side, the left menu, so each one of these you will, uh the terminology is to call them a report to call them a page.

A

B

A

A

So what we had planned to do- and this is just a representative of one of the dora metrics- was to be able to show a graph, and I opened it here for deployment frequency, for example, okay, and what you can see here is um it tells you what you're seeing okay and you can see that we can toggle between last week last month and last 90 days, and you can see that in gitlab we're pretty good in deployment frequency. It's pretty predictive uh drops are usually on the weekends.

A

So today is sunday and.

B

We'll deploy on sundays, okay, nice yeah.

A

So saturday and sunday you can see it's really the same job every time, saturday and sunday, saturday and sunday. um So this is great. It means we're very consistent.

A

Sometimes we have some more um deployments and sometimes not- and if you remember when we started talking about the epic elite, performers deploy multiple times per day, and this is on a daily basis. You can see 160 deployments per day, we're doing pretty well.

B

A

That's more than multiple times per day, so um even on the weekends we have a few deployments. Oh, why isn't giving me a tooltip? I don't know. Let's join us see even on saturday, we had twelve and.

B

uh Yeah, even in the holiday season, in the end of december, there was uh yeah. The velocity was nice yeah.

A

So we're still elite, even though uh it's the weekend and most of the people aren't working, but you can think about us two who are working on sunday and we also applied to production. So um so uh this is a little bit of an explanation.

A

What we can see here um is the time the time this is it's on every day, and what we can see is how many deployments we had per day and the average for the last seven days, because this is what we selected here- you'll see the average for the month. It changes based on what I selected.

B

A

Some things that we added recently was the average bar here, which was something customers asked for, because they wanted to kind of see how they're doing how they're fluctuating based on the average. Something that we want to add is a target so that customers can add. I want to be at 120, for example, and so I'll see my average and I'll see a bar. That represents the target and I can see if I'm above or below target, and we can change target based on how well we're doing.

B

And is there a place in the gitlab platform to add those targets or.

A

We would need to think where, to put it probably understand.

A

I'm not sure if we opened an issue, I'm pretty sure that there's an issue we'll check while we go through the epic um and then we also added this visual, which already existed on value stream analytics, uh but it wasn't shown here so it makes it really easy to understand in the context work that I'm in how many deploys that I have in the last month and what was my frequency right and you can see it's the same deployment frequency.

A

So that's pretty good and then there isn't a lot of things that I can do on the graph, which is things that we want to improve on. First of all, is this timestamp? It's not very flexible.

A

If you look at value stream analytics, for example, uh so this is something that we plan to move over also to dora. So that is the same, consistent improvement experience and in the beginning we started with with the last 90 days because of performance issues. So the only thing is that we need to be mindful for performance.

A

We don't want people to get stuck but allow people to customize their dates in a more flexible fashion and also everything that we can do in the api level. I would like to see that here on the graph level so being able to select a different tier right now. It shows me only production, but maybe I want to see staging. We support it on the back end, there's really no reason not to do it, except for capacity constraints.

B

So if you have customize the yaml they can, you know we can give the user eventually to customize everything in a page and you can um yeah.

A

I would imagine there would be a drop down here and you could do multiple selection or a single selection at the beginning, with the different deployment tiers and then after we add support for the environment, name have environment, name and again. The problem here is that it shows me only one month, for example or one week. Maybe I want to see a comparative view.

A

Maybe I want to see you know one week next another week we could possibly sell that by opening two tabs and selecting different time stamps, um but it would be more elegant to have that option here.

B

To compare yeah different tools and.

A

Have trends you know? Last month I had a thousand deployments, so I'm going down something nice and visual. um So going back to uh the epic that we had for the specific one, the only one that we haven't completed was creating a download historical data uh option.

A

Other reports in gitlab, for example, testing, allow you to do a download to csv for your data for the last 90 days because of uh because of this limitation.

A

So it would be helpful if you could have that to compare over time. So.

B

That's the last report. I saw several requests. I think two requests to have this downloadable uh csv and I guess this is as well something very easy to customize.

A

I believe so because we have the api that returns all this data, so it shouldn't be too difficult to create a csv. It's just a matter of prioritizing it, but since we're walking through the epic.

B

A

A leadership over this you can decide what is more important in my mind, completing the the four is the most important thing at the moment: um okay, so that was this epic. Let's see if we have anything else in epics, besides everything, that's not organized in the epic and feel free to organize this. However, you want okay, we have comparative view.

A

B

A

Is really interesting, and that is how do we want to show the data.

A

A

One thing that we can do in terms of comparative view: one of that isn't even listed here, but what we discussed is time by by time, let's just create a placeholder, so we don't forget.

A

So we just discussed that I'll. Let you fill that in later.

B

A

And one of them is by a different environment. So let's say I want to compare my dora metrics my deployment frequency on staging versus production. Maybe I want to understand why something small slower? Maybe it's because of manual approvals? Maybe it's because of something else.

A

Maybe I don't have enough owners for deployments in the production environment, so there's really different actions that, based on the comparison, I can understand and make myself more efficient, a dashboard based on projects and subgroups see what this is. I don't remember, but we'll open that up and see so customers who are running gitlab as a service for the tenants, one dashboard, the comparison door and metrics between tenants projects and subgroups.

A

So imagine that you're a consultancy, you created uh different uh projects for your different customers and you want to see how to compare them between themselves, which is really interesting um and.

A

This one for self-managed users- this is interesting, so.

A

Let me open up a really interesting chart. I'm gonna stop stop sharing my my screen for a minute and then I'll share it again.

A

We created a really interesting chart uh with a data team. By the way this data is available. We haven't put it in the product. It is something that we should think about adding to the product that shows uh deployment frequency for all the users on gitlab.com.

B

A

B

A minute I need to sign into the.

A

System so I need a sec.

B

I will be back in a minute.

A

A

Okay, so I told you we created a really nice dashboard. It takes data aggregated data from different users that we have in the system that shows the average monthly deployments and medium and the low performers and the high performers.

A

I feel free sharing this because there's no customer data here, it's all aggregated data and anonymized, but you can see here that our high performers had 1 800 monthly deployments and the lower ones had four okay.

A

So imagine that you open up your door metric and you can see how your organization is doing based on other organizations on gitlab.com, it's pretty powerful.

B

Kind of an index for my performance.

A

Yeah actually, um based on this there's a few options that we can do that one of them is to use the dora report and just every year update what is a high performer and based on the objective metric.

A

Are you below or above give you that the badge based on the zora metrics, which is an option or use our own data to to compare to other users on get gitlab? What I was thinking in the future uh way way in the future? If we could categorize organizations that are similar to me and what.

B

A

And and have that data? That would be really amazing, like you know, maybe they're using the same tech as I. uh Maybe they have the same number of users, uh maybe they're in the same industry. I don't know different ways to categorize and give give this data, but it's really really interesting.

A

Okay, so that's comparative views for dora. I'm sure there's other options here, um but that's what we have currently. So, let's go through the sporadic issues that are unbucketized and on anything else.

A

This one is really interesting as well, and this is coming up more and more in conversations that I'm having with customers, there's a trend in the market where teams are becoming more autonomous and and based on their performance, they kind of get their wings. If we have an.

A

For example, for flight uh flight attendants, so uh pilots, the the better you do, the more you can do it by yourself. So um this goes into different permissions and rules that you can do in as part of the software development life cycle. So maybe, if your team is really really high performing um you, can you don't need manual approval for deployment to production? Or maybe you can? um I don't.

B

Know skip a code review whatever different functionality.

A

That you can that you can do, and so this has been coming up more and more customer conversations, which is really interesting on giving team permissions or user permissions based on how well their team is doing their.

B

A

Yeah, so very interesting topic um target for graphs is actually what we discussed before, uh which is allowing people to set a target right. You asked me where so adding a setting under ci cd named dora4.

A

um I don't have permissions in this project. Let's go to a project, but I do have permissions and I'll show you what where that is.

A

um So if I go under settings under ci cd, we could add here another category called dora4 and then you can add a target here. Maybe add, set permissions based on performance as an example and others.

A

um Metrics, on top of dora 4., so I think we talked about this in a previous conversation that every time I talk to a customer, they get really excited about analytics in general and not necessarily dora so having that ability to add additional um data, for example the number of robots that were done from production.

A

This is okay, it's interesting because it's related to change failure rate. There is some kind of loose connection to that, but it's not one of the top four uh non-standard release frequency. I don't even remember what this was: let's open it up and see um on how often he's deployed previously unplanned work to production. That's not part of the planning frequency.

A

It's okay doing.

B

The freeze period, or just uh so for post-production to keep kind of trick about.

A

Keeping track of unplanned work, so it wasn't a feature. It wasn't something that was in the planning. All of a sudden, you had some bug in production and you imagined yeah release frequency, rent metrics. I don't remember what this is either.

A

So it's it's all the I would say, probably like a major release, not any security patches or something like that.

A

um Download historical data at the group level group level links to project level ci cd. So today, if you look at group level, ci cd analytics, doral metrics, let's go to the group level.

A

If we go to the group level analytics, we get an aggregated data of all the projects, but there's no way for me to understand which projects contributed to it, and so it would be nice to see at least a list or a table that says this project here.

B

A

Yeah, because on the group level it looks on the project level, you you see the project itself, so this is aggregated data. I would like to have the option to see whether either aggregated.

B

A

Data, maybe I want to deselect some of the projects have this. You know, flexibility.

B

So so from a data structure, if, if I'm on the again in the group level- and I want to see this background of project- so if I will have the option to to customize the query to the yaml- is this something that is uh you know, you think it could be a low angle fruit? So this is something that requires.

A

A lot of second, the yaml exists only on the project level inside the source code. The group level doesn't have it it just aggregates the data, it's think about it as a container that holds the different projects. So um the way that I'm thinking about it is okay. I can see, for example, um sorry. This is just the the download button, but in if you look at test coverage, which was my inspiration for this issue here, you can select which projects you want to download from.

A

So we can actually reuse the modal that they use.

A

Of course, as an nvc, you don't have to have selection, you can we start with just download everything. Then you can deselect the date range or not. The date range give a default date range select all projects, multiple select projects, and so on.

A

Does that make sense.

B

A

As I told you all.

B

The kind of uh uh is still kind of um mystery from his uh um again. Why it's uh you know uh uh the difference of uh the perspective when I'm, uh you know looking on analytics from a group perspective or from project perspective,.

A

Think about it as a um a pane of glass project is the smallest component and that's what's really interesting for our team. Once I go above that unless I am a team leader and I lead a few teams, I want to see how each team is doing and then compare them or not. Compare them or just understand.

A

Then I look at the group level. Group level is more for the leadership level rather than the team.

B

But still, I guess, when you're working on the project you have several, you know.

B

You want to see the breakdown over there as well.

A

So let me try to help you out think about it that project in gitlab um equals equals a repository, a code repository.

B

A

So when you're looking at a project you're looking at that specific code, so don't think about a project as a project that you worked on in your previous group company, where there were different teams that were working together to create one big project.

A

That's a group in gitlab.

B

A

Does that make sense.

B

Yeah yeah yeah and then I opened there yeah. I found my note also from one of the onboarding training yeah, okay yeah, so uh uh I got it. Okay,.

A

So maybe an mvc even starts with having this ability at the project level and just having a link from the group level to specific projects to download. Then later you can aggregate it. I mean you need to kind of think about the different iterations that we can do, but we want to give value to our customers as quickly as possible. So what is the smallest component that I can do that helps them out.

A

um Okay group level links, so I think I combined the two together when we were talking. I didn't confuse you, but the idea is that um I can press and get to each one of the projects individually.

A

Hopefully I can also see the data, but uh but we can start.

B

Small starting with adrenals yeah.

A

Okay, dora best practices. This is more of a documentation issue, um a lot of the customers that I talk to on a weekly basis. They are looking for guidance on how to get started. What should we measure? What should we measure? What should I do so? I think we have a service to do here to our users to kind of guide them through it, so creating a blog post about dora.

A

I might have already done that um and and talking about what what it is, how do we use it, um creating a survey so that people can assess themselves how they're doing based on the survey results? We can point to documentation an area to improve not only based on the survey but think about think about combining the different analytics that we have in gitlab.

A

Imagine that we have even without combining imagine, we have a change failure rate and it's really low, I'm doing really poorly. Maybe I can tie to users and tell them hey. Did you know that we have a feature that does automatic roll back and in case you have an incident? It does that and that will automatically improve your change failure rate, because the service will be up because it indicated uh automatic flight go back. We can help them because git lab is a huge product.

A

People don't know about every feature that exists, so maybe we can help our users.

B

Suggestions and insight, yeah yeah.

A

um Show users where things are located uh where they are located, on.com versus others that we talk. I showed you the periscope dashboard, but also we can think about just putting the numerical values of door reports once a year.

A

Points to different documentation and okay- we already talked about this, so really. This is all about making this more visible to users and helping our users out when they get stuck, but also think about how we can connect this to value stream metrics and how value stream metrics dora metrics are more of. How am I doing versus the industry?

A

Where is value stream metric is? How is my specific process.

B

In my organization.

A

Doing so, they should have an impact on one another right.

A

A

Okay exposed dora csv, I'm not sure what the difference is between this and the previous one.

A

I think maybe maybe the duplicate um create door for widgets tiles in the project ci cd. We started by the way.

A

We started, I showed you on the ci cd analytics that we already brought in the deployment and deployment frequency. um Next, we should do the same for lead time, because value stream already has this data. If I'm not mistaken, uh we can just copy it over.

B

A

Okay, um though, I think there's a different definition, so it might be tricky. I think this one is calculating um median time. No, it's the same. We can. We can move it over and.

A

I don't know I feel, like we already discussed this before uh addadora badge to the project level, so this is. um This is actually something really neat that we thought of doing. um This is related to the decision that we have to make whether or not to copy the numbers from the dora report, but assuming that we do decide to do that, we can already give our users badges based on how well they're doing, which is really great, because it's gamification it's incentivizing.

A

This can be the base for giving permissions based on uh how well you're doing and so on, which is really neat.

B

So do we know where let's say what is the door 4 kind of road map? You know now that it's been owned by google um yeah.

A

I was in recently in a webinar that they uh presented the their findings this year, the most interesting finding that they had.

A

There were actually two interesting interesting findings, one they measured the impact of documentation on dora metrics and they found a correlation between how well things are documented in the code and in uh in user guides into how how that affected. Dora 4 metrics- and there was a high correlation.

A

I don't remember the number, but there was a higher correlation, and the second thing they found was that.

A

Everyone became much more competitive, there were a lot more elite and high performers this year than in previous years. However, the low performers got worse, so it looks like the people who are struggling a few years ago.

A

It even became worse because either they were adopting, let's do high velocity and the quality was going down, and so they had a lot of buggy area or that they're just having a really hard time moving from legacy software to higher performer technologies, from monoliths to microservices and so on.

B

And is there any place that people kind of collaborate and sharing their results.

A

But I'm not, I never looked into it, so I don't know.

B

So it makes sense, of course, for for the gitlab.com. You know, customers that to have this benchmark and to have this kind of uh comparison index, but people that working on-prem, the self-managed and, of course, not gitlab customers that measure their doors so how they can kind of evaluate their results.

A

Well, the fact that someone is self-managed doesn't mean they can't evaluate the results they. The numbers are uh published in dora. So if we look at the 2019 numbers, you know if, if you're here, if you're here, if you're here, if you're here right it doesn't matter if it's a cloud platform or if it's uh self-managed, on-prem installation, you would know that anyway,.

B

A

What the you wouldn't know is if we do decide to use the metrics that we have on other users, and I did open an issue specifically for self-managed users that I skipped, but we can go into it.

A

So if we look for self-managed users, my idea here was to present a tool as part of your hiring process. You saw that you have access to the comp calculator, so I was thinking it allows you to insert your own data based on your self-managed data, you that you yourself know, and it will show you where you're located.

B

Okay, very nice.

A

So it's an idea.

A

I also think that um our self-managed users, if they're they already, can use the dora metrics. They don't have to use it on gitlab.com, and so, even if we don't do a tool, uh we already know their numbers and we all they have to do really is if we decide to insert the dora textbook measurements.

A

But once a year we can have that in the graph that is pulled on a self-managed doesn't have to be sas, and then it can be available for self-managed users as well, which would probably be more simpler, simple than what I suggested here.

A

um Okay, where were we badges? Okay, mean time to results? Okay, so let's go into the two last metrics, which was time to restore service and change failure rate.

A

So the way that that you measure um the time to start service is really when, when a service is down- and as I mentioned it's a philosophical question- because what does it mean that a service is it a bug? Is it everything's down? So the way that I decided, I thought of idea to mitigate this was to use our own gitlab incidents and measure mean time to resolve incidents as the the time to restore service.

A

If you have an incident, that means probably the service was down and measure that um it's relatively simple to measure, and so I think it's a good measurement. The downside of it is the number of users that can benefit from this, because, even though we're uh we wish that everyone uses gitlab for everything uh in reality, a lot of people, for instance, don't use gitlab.

A

uh They use uh other tools, so they.

B

Would I guess, most of them using servicenow gr well,.

A

Jira for not not so much for incidents, that's more for planning, but servicenow is a good example. It comes up on a lot of customer uh calls.

A

People use different tools, so this is this limits, the amount of people who can use our door for metrics, and so we would probably think that we need an integration with other tools in order for more users to benefit from this.

B

But we should start with the guitar.

A

I think yes, and maybe people will start using gitlab incidents from because of it.

B

And are there any kind of existing apis for github incidents today, about you know mean time to repair and all those metrics yeah.

A

You basically, if you think, about incidents you can think about as like an issue, it has a time that it was open. It has a severity and how the time is closed.

B

So the label is incident, that's it.

A

Okay, next, we have uh comparative view, which one is this.

A

I feel like we've discussed this over and over again, I'm I'm I'm debating whether this is duplicates.

B

No, it's it's about it. It's kind of again, if you can, uh you know emphasize again about uh uh you know using door for the dollar matrix for the value stream, analytic, yeah,.

A

So it combines everything together, which is really nice, and it shows you um it's a comparative view for your organization. So what does that mean? It doesn't look at dora as an industry event mark and it doesn't look at dora based on other customers on gitlab.com.

A

What it does is it looks at your organization, so it will compare different projects or different groups and show you how you can see other projects in this instance so think of instances as a group or a workspace, and this compares different organizations.

A

So this is actually pretty cool.

A

B

Do we have left.

A

Oh, this is what we deci discussed on the tile view. I think.

A

We already present deployment frequency, so we wanted to copy the lead time for changes into intensity.

B

A

I think the graphics need to be updated, but that's the idea. Okay, uh this is actually outside the scope of uh dora, but I'll give you a little bit of a history. So you understand, um if you go to the group level, if you go to the group level analytics under ci cd you'll see, there's release metrics okay, um this was it has nothing to do with doral metrics. This is kind of bonus points that we got when we introduced dora metrics at the group level.

A

There was no such thing as group level, ci cd analytics.

A

So what we did was we created release analytics, which tells you how many of the projects within the release are associated to releases. um So it really has nothing to do with dora, but we had to add a.

B

Landing page kind of.

A

We had to add a group page, it has good data, it's just not related to door, and then we added all these other metrics, even shared runner usage, uh which is really not dora, but just so you have the context. Maybe we should move it out to a different epic.

A

um This api, we discussed and widgets for ci cd insights at the group level.

A

Okay, so it's adding the charts once we add ci once we add the change failure rate and time to restore we'll, add the charts similar to the way that they look today. So today we have a tab for deployment frequency and lead time for chases. Imagine that you also have one here for change failure rate and one for um time to restore service.

B

A

I'm pretty sure that completes everything in this that's included in the dora for epic. um There might be sporadic issues that we forgot to to combine, but I think this was an extensive overview of the epic.

B

Okay, so I'm happy we have this recording, so I can come back uh to review them and uh yeah. I think that in the short term I try to at least you know, capture some low-hanging fruits and to see what can be done and for the long term we will. uh You know, try to look in the kind of and more end-to-end kind of uh perspective.

A

Okay, do you have any other questions about this.

B

No, I have a lot of uh to do to read and to do it more, I will follow up for more information. Great.