GitLab Pipeline Insights Group, 13 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Verify:Testing Group Think Big #15

Description

Today's ThinkBig was about how GitLab can better make use of OpenMetrics in the Custom Metrics Report to solve problems customers have. We had great insights from Kevin Chu the Group Product Manager for Monitor, Configure and Release about what he's heard from customers wanting to track data as part of their CI/CD Pipelines.

You can see an overview of why and how Testing does ThinkBig sessions here: https://www.youtube.com/watch?v=UAaOnnbIud0

A

This is the think, big uh for the testing team for april 13 2021 today. Our think big topic is open, metrics uh and I'm gonna give a quick overview because I started this issue, so I started this because the metrics reports- functionality that we have is driven by openmetrics or it says that it uses the openmetrics format.

A

That format was in draft as an open source project for a long time and I'm guessing four months ago, because that's when I created this issue, it got finalized and is an official thing now um yeah. So it's it's out of its rc or beta kind of period and there's an official open metric standard. So I wanted to think big about how can we expand on this feature? We see some decent usage of the feature. It's a feature that's sitting in premium.

A

I can describe some of the problems that we see people using or trying to solve with it today, based on some of the conversations we've had and just what we see in usage and the questions that we get about the functionality but wanted to open it up and talk about. How can we really make this a great experience for somebody who wants to track something as part of their build or test process that we don't let them do today?

A

So that's a quick overview, um I'll jump into what are some of the problems people saw. I know. Gitlab uses this to track gem size uh or some of the gems that we produce in the build process.

A

I've heard some folks who are doing there was an embedded processor project that was using metrics for something and now I cannot remember what it was specifically, but people are using it to track timing of things or performance of things, custom code, quality stuff that they want to track over time.

A

There's any number of things that somebody could track: api errors, for instance, basically things that we don't provide tracking for today that you want to see how it's progressing over time or changing against your default branch as part of each of your merge requests.

A

So that's the job they're trying to do with our custom, metrics or openmetrics solution today,.

B

Is the is the format standardized enough that we could automatically apply a bunch of statistical measures? Can we provide because you know we have like a lot of it- is like we just we compare the diff right of like you know our bundle size was this now our bundle size is this. um Couldn't could we talk about? Could we look at some metrics and say well not only between these two, but in the last 90 days of these here's, your variation.

B

You know, here's your standard deviation! If, when people, if people are tracking things like you know, error error rates across you know we have 700 000 tests. um Our error rate is, you know, the air rate here is 0.2 percent, and but what is we could put that in some in bigger context? You know we say that the the variation is 0.5, so 0.2 is is normal, um but we could do some, I'm thinking.

B

If people are tracking complicated, metrics using open metrics, we could sort of get ahead of them and say: oh well, you're, probably tracking something a little bit complicated. We can attach some of these more thorough analysis to it.

A

I would imagine so um it's built on top of the prometheus format or prometrics, so we should be able to just push a lot of that data into some of the existing monitoring, tooling, that we have today and that's why I'm really excited that kevin has joined us um and I'm hoping that that's why kevin did join us to see where there might be some crossover with the monitor group here exactly openmetrics.

C

Has in some ways been a vein of my existence in my previous life and the relic because, like the promise is so great by like and literally every single observability vendor created their own variation and more recently people started kind of honing in on, like prometheus is kind of the default it doesn't claim to be, but in practice it was, and then it's like. Why is this open, metrics taking so day long?

C

So that's my engagement and but I frankly I don't really have any ideas um uh beyond like hey. I want to hear what people are talking about when it comes to open, metrics.

A

See I think that that's a that is something we could solve for and there might be some really interesting cross stage opportunities to introduce within the monitoring stage. Here's monitoring for your open metrics from your build system yeah. It could be really cool.

C

Yeah, like actually, I just got off a customer call where they're talking about exactly this. It's. How do I understand? What's going on with my build system, which is the galaxy icd pipeline.

C

Feels like if, in the context of open metrics, if we are capturing and storing data in the open, metrics format, this makes that data very transportable, so people can also take it and dump it into their observer blue platform of choice. Since we don't have one currently.

C

They can do things like if they have a grafana dashboard. They want to graph it then so standard. It makes it really quick and easy.

C

Not sure what else.

A

Kevin I'm going to put you on the spot a little bit. You were just on a customer call. um How would you describe what persona was that customer? I guess what's the yeah, which of our personas would they fit into? uh He is the boss of.

C

I think priyanka, so he he, his team is responsible for making continuous deployment continuous delivery, a a real thing. He works for a bank and the bank just hired a new cto. He came in with great ideas, saying hey: why are we deploying it once a month or whatever this? Can we deploy every day, but then the system's right in place? So he hired this guy to try to figure out how to enable the development team to be uh efficient and he's his primary problems like using gitlab.

C

We have the environment dashboard, but it's kind of crappy. He doesn't actually understand what's going on with with how the builds are going and he can't empower the developers to self-service yeah.

A

And priyanka is the platform engineer. So if we were talking about priyanka's boss, they might be the manager. Even director of you know, I've seen that as manager of devops or manager build systems there's sometimes some crossover and system administration, as well, depending on where your organization might be in their devops journey, uh cool yeah, so priyanka and priyanka's boss, don't know in this case for use of openmetrics hey, what's going on in my build systems, how long are things taking?

A

Is that something that they could export to this open metrics and then trend that over time.

C

I think the first thing they want is just to see it in gitlab, because gitlab is their tool of choice and they would like to visualize it since it's a they're already in there. They want to see the performance and hiccups of their pipelines within gitlab and a big thing is like it's: it's multiple teams, so it's like how you slice and dice the dashboard views to fit.

C

The particular user is an important aspect of it.

A

Okay, hey yeah. I like that use case. That's a really interesting one. I think.

A

So for our original time, because we got started a little bit late today, we only have about five minutes left. I have time after I can go over, um but before I jump into the next question- and we get six minutes into five minutes left- uh is everybody else cool with going over or should we try to truncate this and we can pick it up a little bit later.

C

I can go top nine.

D

So your note taker will be uh we'll have to jump. I do have a one afterwards, but um I could be here for the next five.

A

Let's, let's crank through as much as we can five minutes uh so for priyanka and priyanka's boss, if they're tracking stuff within their build pipelines um or their cicd pipelines. What would an ideal outcome for them? Look like as part of doing this job. If I need to track things that are happening within the pipeline over time,.

C

Yeah, where they are at the moment, is they like there's different maturity within the different teams and he his idea. He wants to gamify it so that by making that information visible to teams and visible to the organization, they can kind of stack rank. Oh these teams are doing good. These teams are not doing good and use that as motivation for improvement.

C

So that people see oh we're in the bottom 10, we got to fix this because we're looking bad uh the then go ahead and fix it. Whatever it's like, causing the slow down, whether it's they're using some tool, that's just not good enough or they're using unnecessary steps, but like competing with speed is he wants everything to flow smoothly? So lack of a failed pipeline is really important because what's happening is people have trust issues when they're not bought in they're like oh, this always fails me.

C

It slows me down, so I'm just going to remove this testing portion of it, and so that's a repeated pattern over and over again so by understanding who are consistently broken, hopefully like someone like priyanka can go and address them so to to kind of build up the trust among a bunch of teams that don't trust the system that they're moving back.

A

So it sounds like I just want to make sure I understood if I'm tracking something over time I want to be able to so, if I'm within the team at this project, I want to see this thing is trending this way or this way or if it's a binary pass fail. I want to see how often it's failing like if it's a job, we can start to find some flaky jobs or inconsistent jobs, and then their manager would want to say great here's all of the projects that we cover.

A

The teams that they're associated with here's kind of our stack rank for how everybody looks today- and this is where we want to be in a month- see where you can move within the ranking.

C

Yeah, okay, cool.

A

And they would want to extract that data in each of their pipelines or maybe a scheduled pipeline against their default branch. So they have kind of that baseline and then be able to present it somewhere either within gitlab or outside of gitlab.

C

Yeah uh potential, the outside of gitlab- it's I'm not 100 sure so for observability stuff. I can see that as a legitimate use case, because about actually even for things like pipeline, probably because, at the end of the day, the people that look at these dashboards- probably all wantings one place similar to how we just want to consolidate all data using sizes um like yeah.

C

If it's metrics related, it's helpful to see it all in one place. Therefore, if we have, we, we've subscribed to a uh standard format that makes transporting data from one one tool to another. It's much easier.

A

Okay and then in our last couple of minutes, if we were to implement something like that or if we were to make it uh possible to do this with our open metrics or our custom metrics, how would we differentiate from anybody else who's out there?

A

There is the secret sauce here or the the killer feature that the customer can do this within gitlab. They can see it within gitlab. They can operationalize it within or gitlab there something else that can help us differentiate from. I can extract metrics out of my build system and present it somewhere.

A

I haven't dug in, but I have to imagine you can do that with jenkins, somehow, with a plug-in, probably.

C

So the format is definitely not the special sauce. Generally, the the data holder holds most of the power in these relationships.

C

I don't know, drew parker.

A

See drew you had some interesting thoughts on. Can we preemptively do some things for you.

B

Yeah, I the usually when I thinking about how to make somebody like something more either. I immediately go for give it to somebody before they ask for it or like automatic or flexibility. So if we can make the metrics more of it- and I know less about the competition for this- um but if we can make this, if we can make things like this very easy to consume in you know, sci-sense, whatever chances are there's, they have some kind of dashboarding software going already.

B

They look at charts somewhere we're using a popular open standard. It should be super easy for teams to say okay, the data's here, but this isn't the the place I'm going to look at it. I just I I want it here and I want to get my pipeline data over to tool x with as easily as possible and stop thinking about this um that I think, generally, when I'm thinking about you know data features. uh I think people tend to like that engineers. At least we like apis yeah, do other stuff on it. Yeah.

A

Cool we're at time or overtime for folks who wanted to hop before their next meeting. I think that wraps us up in a pretty good spot for this abbreviated. Think big though um I'll drop, the agenda link into our local chat or into slack and folks, can contribute asynchronously, and we will do our think small version of this next week same time, cool thanks. Everybody.

C

A

So much everybody, it's a great discussion appreciate it cheers.