GitLab Pipeline Insights Group, 22 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Verify:Testing Group Think Big #16

Description

Think Small today about open metrics and the custom metrics reports available in GitLab.

Think Big portion: https://www.youtube.com/watch?v=bWqoHtS1Mco

A

So this is the think small for the verified testing team. uh It's april 22nd 2021 last week in the video that I'll link below uh we started with think big, around open metrics. Now that that standard is actually published, it's used by our custom metrics reports feature so we're interested in exploring new use cases. The other ways that customers could use that users could use that.

A

um Just to sum up last week we talked about an ideal outcome, for users would be to create custom metrics as part of their builds, our pipelines, job timing package, size total pipeline time. Those are some examples of things that they could track and then see how those trend over time, at the project level, probably roll that up into the group level.

A

One of the examples that kevin from the uh release monitor uh team came to us with was a prospect he had talked to somebody who wanted to gamify uh pipeline time amongst their teams uh have a leaderboard of bits which is super interesting, so grouping those things together, either by group or some other logical unit, which is a whole other ball of wax canon worms that we've talked about for various other bits that we want to group together, not in our normal grouping logic.

A

So that is our kind of high level ideal situation uh for customers creating, oh, creating and capturing custom, metrics utilizing open metrics uh did I miss anything in our recap or from the notes that I reread this morning.

A

All right: well, let's jump in um talked about kind of the big vision. You have a awesome graph. That shows you how everything you've defined for custom metrics is tracking over time. um A bunch of other cool, smart stuff uh drew mentioned a lot of extra analytics that maybe we can make available um like here's, the 95th, percentile 99th percentile, here's things that are outliers to that, where you should pay attention, um that's our long-term vision towards that. What could we do in our next milestone? If we put together an issue today for fortino?

A

What would it look like that takes a baby step towards that minimum, viable change style.

A

B

Put a number somewhere.

A

Put a number somewhere.

B

What number and where I don't know.

B

We have custom metrics in the.

B

C

uh No there's there's, I I like this idea put a number somewhere. This is this is great. Well because what something about this, what what numbers can we? What number would be easy for us to automatically put there right to like essentially like seed the data structure right? So so people aren't starting from nothing right. Maybe they don't know what the open metrics format is off the top of their head. They don't have to google it like we just throw in like any old, dumb metric.

C

That says, like the number of seconds your test suite took to run and.

D

It's already there, so somebody could.

C

Just look at it and like copy paste, the line or something instead of needing to go, look up the standard like any. You know like any.

C

Number number of people who were have commits in this merger quest, one like whatever it.

C

C

That, if, if we do, if we put any metric in there, no customer will have to come up with their open metrics reporting from a whole flag from whole cloth. It'll, always just it's always easier for me to go into something existing and add one more thing just like it than it is to go, invent the whole thing.

C

So if we do the inventing with some dummy number, it should lower the barrier for everyone.

D

We have to make sure there was a way for them to get rid of our report. I guess they could just delete the artifact or something like that. Well, it.

C

It's though, if I remember correctly, it's this just one artist.

D

C

With right with a format in it, yeah.

D

C

You know have your brackets in here and your braces here and quote your strings like this and your integers like this. You know that like yeah, so they.

D

Can just overwrite.

C

It or something like that yeah, if we we copied like a readme thing out of the open, metrics documentation and put it in there, so they could like type to work. They could replace our thing with a different number like.

A

But if you're doing it as well.

D

Auto open metrics, yeah.

A

Yeah, if you're doing it as a template, you can include a link to documentation and other examples or a library of other examples. Right.

C

Yeah, I would we could our template could write something I don't know.

A

So if we just pulled something out of ci, for instance, because you you have to be running a pipeline to utilize this, because it's going to be a job that tracks it right, um total job time might be available or total pipeline time. That's always going to be something that you have. If you run a pipeline, you might not have tests, you might not have deploys um right. Yes, you should have commits- maybe probably maybe not trying to try and think about what is the most stripped down.

C

A

You can run.

C

Well, you might commits is tricky because you might be in a merge request and you might not so what yeah commits it like there will be a commit, but a number of commits on a branch pipeline. That's just going to be number of commits, like in your main branch of all time,.

D

I know personally, I have a pipeline that just builds my code and does nothing else. It's not tested, doesn't matter just builds and then hey it passed we're good.

D

um So I mean that's, probably the most basic strip down pipeline that I can think of.

D

They're open metrics that we can do off of that. I'm not sure.

A

Can we get timing like of the overall pipeline like if that's all, we tracked and said, yeah we're going to compare every single time? That's the very first baby step thing that we would do is give you your total pipeline time.

C

There's uh there's a problem: there's a time travel problem in there that we can't write the amount of time that the job has taken while the job is running, yeah.

A

Forget about that.

B

Can we write the amount of time that we're at while it's writing like time from zero to then.

D

Doesn't really give us much does it, though, is there a way we can on pipeline completion um edit an artifact or something I don't know if that's very, if I don't know if.

C

D

Enough for us to add that ability.

C

Still the time travel problem where, if it's running, then the pipeline's not complete you'd, have to use like a downstream pipeline and then reflect back on the first one.

A

This nbc gets complicated when you introduce parent-child pipelines to it. Yeah, uh okay,.

A

There's probably something else: a number of jobs scheduled to run, or something like that that we could include. um That is a little more there's less time.

C

Scheduled it's great.

A

Yeah job skills.

C

You know that right at the.

C

C

And uh it's also just thinking that um making the number like especially useless, might drive engagement.

C

Because people might see it and be like well that metric is silly. What's that doing there yeah and then they'll they'll go make it useful yeah.

C

I don't know, I don't know if that's like a weird anti-pattern of like annoy your users into using your software.

A

D

You give them something.

A

Even if it's not useful, if it runs really quick and it's example of how you could track this thing, that it encourages them to look into. How might I use this to solve my own problems if we then link them to a library or examples of other things that we track, but then gitlab like hey, we're tracking all of our gem sizes, we're tracking, whatever else we actually track with custom metrics, because we use it for a lot of stuff that it will prompt them to figure out.

A

Oh, this is what I could replace that with apologies miranda. I totally stomped over what you were saying. I it was.

B

Mostly as a joke, but we could have the starting metric of metrics that you've set up so far zero just to annoy them.

D

Are we at all concerned um for customers that are.

D

Concerned with storage space, artifact storage, space.

A

As far as like this would be a big artifact.

D

For them I mean it, wouldn't really be. It wouldn't be a big artifact, because I mean the amount of it's a text file, and so I don't know we could probably count the number of bits that it would take for us to like take up, um but if they're running, like tons of pipelines but they're still concerned about their storage for artifacts or something I don't know, I'm just concerned that there's gonna be one a few customers that are like hey. Why are you automatically creating an artifact? We don't want this.

D

How do we opt out of it without having to edit our ci config of erasing this file.

A

I wouldn't put it like when you create a gitlab ci yaml, that this job automatically inserts, I would think about it as here's a template that gets. You started with custom metrics and it just got.

D

A

So you they would have to opt to do it still, because this is about.

D

I was imagining yeah. I was imagining that it was going to be an automatic like hey. You ran a pipeline. Here's open metrics for your pipeline yeah.

A

D

Okay, that template makes more sense.

A

And we could even do it as a informal template like not actually spin up a project, but here's an example of some syntax that you could use to insert um go from there, maybe beyond what we have already in documentation today.

D

A

And there there.

D

Was benefit of running it automatically too, but I don't know what did you just.

C

What were you gonna say? I'm just thinking it's. It seems like more of a not like include this template for the complete solution, but include this template as like you're getting here, you're getting started with open metrics like you include this, but you'll be expected to expand on it, because it's a completely open-ended boarding system.

A

Cool, I think that's that would be a good mpc that would prove out people actually want to use this if they, if we see it, start to be included, if we start to see more engagement with the custom metrics, both upload of data and the engagement with the mr widget, which we track both of those things on our dashboard, so we could track.

A

How is this trending over time so moving on to, I guess: what's our what's the riskiest assumption we're making here uh if we're assuming things that, if they're not true, it would blow up, and we would never get to our ideal outcome for a customer.

C

I feel like we're assuming that people like.

A

Openmetrics yeah.

C

Like people might see like oh, I can include that template for open, metrics and just like barf, and never look at it again like, but we could get zero engagement to be fair. You might know that I feel like I'm assuming that so.

A

Yeah, I think we talked a little bit last week about open. Metrics is still a little undefined uh by and large prometheus follows it. But people follow this um kind of the standardization that prom has put together, and that would be the format um and that it enables people to potentially output those artifacts or grab those artifacts and put them into other places as well and for visualization.

C

Purposes is the prometheus standard, a superset of open metrics.

C

Okay, because it would be convenient if, like open, metrics compliance means it's prometheus, compliant.

C

Just if prometheus is like a de facto standard and there's a conflict, I mean doing the open standard is kind of right in principle, but I could see it being harder to gain traction if people already have a prometheus standard ecosystem and we're gonna be a conflict.

A

Yeah we create a bunch of metrics that don't work with their existing grafana dashboards or whatever. It might be right.

C

Right and it could be some tiny little thing, but if it means they have to think about it, that could kind of kill the momentum.

A

A

Yeah, I think I think you're right on with that. That's a risky assumption that people want to use that. um I think we've proved or we have evidence that implies. Customers want to track additional things that we're not giving them as part of their builds.

A

I think a lot of the really interesting problems to solve are around build metrics themselves pipelines which jobs take the longest. um I think that we did it. I hadn't really thought about the you can't get metrics on a pipeline until the pipeline is done running, so you can't do it within the pipeline problem before so.

A

That's interesting to expose and think about, but I think there is our appetite there to track other stuff as part of these pipelines, which is interesting um any other assumptions around the use cases that we talked about either the things small or the think big.

A

So I'm thinking about both our think small and our think big, um our ideal outcomes here and the things that we could potentially build in our next milestone. What's the hardest problem here to solve.

D

I feel like displaying the metrics in a consistent way, because it is so open-ended. It's not really like a solid chart that we can do other than hey. It has changed.

D

So, making a display yeah people, people can easily parse and and and quickly see the results.

C

The I think, you're right about the open-endedness being the the chart is an ins is probably the most obvious place where this happens, but we don't know what people's solution is going to look like like we, we don't have an end goal to be like. Oh, we can automate this much of the process. We have no idea where people want to go with it.

D

C

Have to just be communicating with customers a.

D

C

Yeah um and I internal on dot com, we should probably do some monitoring some informal uh quiet surveying of metrics artifacts to see what people are actually using it for that people want a time series chart is already an assumption of what kind of data is going to be in there.

D

Is that going against confidentiality if we open up their metrics reports.

D

Though, probably I mean the only other thing.

C

We could do, we could.

D

Just we, we could just survey customers, you know what are you using it for um yeah? It's not going to be a complete solution, but always give us like a trend of like what are what people are primarily using it for.

A

Yeah, you could probably dig into open source projects or public projects without a problem and start to poke around there and get some ideas. Yeah.

C

Yeah any yeah any place with public ci. I guess.

B

Invite folks from gitlab to make their make our metrics better make different types of metrics that match up with the data types and chart types that we want to support.

B

Your unfrozen sky, maybe maybe not.

A

Cool, I think, along with displaying that data, is storage of that data. That's probably it's something that we're doing more and more often lately as a team in some of our feature sets, but that's always something to keep keep in mind how that grows, how we deprecate, how we not deprecate, how we sunset data at the end like what our window is to keep those tables small, potentially.

A

A

So it sounds like our very first or next thing that I will create an issue for and put into I'll put it into 14 0. To start it may end up into 14 1, which are next to refinement milestones out to just create an example.

A

Blurb of here is a single custom metric that you could use to get started and from there we can build on that with some additional documentation, or example, projects of here's things that we use in gitlab and what we track them for and how we act on the data as it changes.

A

I think that would be interesting to go down that path and see how people are using this today or would react to those kinds of examples of how we use it today.

A

Awesome, the two minutes left uh any other things we want to dig into here on openmetrics. Anybody would just open up the format the documentation and read through it and pour over it and record that put it on unfiltered.

C

James record, a two-minute speed run of reading the openmetrics documentation post that to our youtube channel. There.

A

A

All right: well, I'm going to take those two minutes now, one minute that I get back and try to go, create that issue so that everybody can take a peek and comment on it and we can refine it a little bit I'll drop it into 14-0 cool all right thanks. Everybody appreciate it good session cheers.