GitLab Distribution Team Demos, 14 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Distribution Team Demo 2022-04-14 - Pipelines dashboard for monitoring

Description

https://gitlab.com/gitlab-org/distribution/monitoring/-/issues/4

A

Hey everyone: uh this is the distribution team demo for april 14 2022 and today, I'm talking about a pipelines page that we've added to our monitoring application that we use um in distribution and before I get into the application um I kind of want to highlight the problem, um usually when you're creating a dashboard or a dashboard, page you're, trying to get answers to questions that are hard to answer with the current system, and so in this scope, we're trying to figure out the relative priority um but get a sense of you know.

A

Distribution has several projects and we want to see how well they're, how healthy the build pipelines for each project are in relation to each other. The thought is that this will help us better prioritize. um You know the maintenance that we do on pipelines for our different projects.

A

So on my screen, you can see I'm sharing the issue for this um in the browser and we've got an mr open. uh This is in the final stages of review, so it is still on a branch but should be merged here shortly.

A

um So back to that problem, if you're looking at our different projects and trying to get a sense of like the duration, the relative durations or the relative success failure of our projects up until now, what we do is you go into the analytics, slash, ci, cd part of each project in gitlab, and it shows you these nice graphs.

A

The problem here is that, unless this is kind of like a prepared statement, that's being returned from graphql, and so you see these pipeline durations for the last 30 commits. um I I looked in the graphql quite a bit as part of implementing this page, and you can't change that. 30 commits that's just part of there's an api route called like pipeline statistics for project and it passes back exactly what we're seeing on the screen.

A

So if you want to say something like you know, what were the durations for the last three months, then this isn't gonna, do it for you, you're gonna have to either use the graphql or the http api yourself and dig that out.

A

So this is from the monitoring project. But, of course you know this tab is for the omnibus project. I talked about durations a little bit, how we're stuck to the last 30 in the upper left. Here you see, we have these overall statistics which is like for the entire direct or since the project was created, the pipelines, um I'm not sure how useful this is like. uh uh Overall, like I I I guess I don't know what questions you would have.

A

You would answer by knowing that you know, since its creation omnibus passed its pipeline 75 of the time, it's usually more like temporal right. I want to know for the past two months, three months, one year versus the year before, like sort of uh analytics like that um and then on the bottom. We have the pipeline charts and similar thing here. We give last last week last month and last year, but this is also part of that prepared statement.

A

So, if you want to say the last three months, then you're not going to get it from this page or the query that generates this page. So what do we do about this? um We have already got a monitoring application that we used on distribution for the monitoring, the merch requests that are ready for review, we're all familiar with this page, um and so it's nice that we already have an app adding a page to an app it's a lot easier than standing up an entire app and then adding the first page.

A

So um since we have two pages now part of this um story, I I added some navigation. You can see now we have this back to slash which takes you back to home. Otherwise you wouldn't be able to navigate around, but this is the new uh pipelines page that I'm talking about, and so I'm going to cover the the web element a little bit and then I'll dive more into the the code and the graphql.

A

So we can talk about how this happens, but you see immediately that we have all of our projects listed alphabetically and it's showing the success rate of the default branches, which for us right now, is either main or master, whichever we're using for each project. It's it pulls from that, because we don't care what the success rate is on. You know feature branches and stuff, they break as a series of working on them and that's a normal thing. So it's not too compelling to track for branches.

A

um So you see here like this might be. We see the reference architecture tester is at 48 on its main break or its default branch. So you know this might be. I don't know for sure, but this might be an indication that we need to put a little more priority towards uh the master, the main pipeline on for that project. You know something else might have happened. Of course you have to apply some context here, but at least we can see them on one place down.

A

So that's the that's the middle column here the success rate and then we've got the duration in the right column, which we're doing min median and max of the duration of the pipelines.

A

I use we're using median instead of average, because outliers will skew an average and they won't skew a medium. So you know we uh there there's a there's a little filtering that goes on. Here too, I noticed working with the api. Sometimes you'll get back a duration of zero for a pipeline, which I suppose there's any number of cases that that could happen. Maybe you pushed it and the pipeline um configuration the ammo was incorrect.

A

I think that results in a zero duration pipeline um there's a few different cases for that, but for this application we filter those out because they're not so useful. um You can see in the projects, on the left hand, side that this is not all of the projects. The distribution is uh responsible for we're filtering out pipe or projects where there are no pipelines for the for the duration um that we're talking about.

A

So there are more projects than this, but unless they have pipelines they're not going to show up here and that just unclutters the interface a bit, you don't have rows where things are zeros. That's just kind of noise.

A

Okay, um it's kind of it for the page. Did anybody have questions before I move into the implementation.

A

All right so, like I said we are using graphql, I'm looking at the branch for this right now we're using graphql for a couple reasons. um One is that, and you can see here in the lib now we have a graphql query: it's an erb template. Let me make it a little bigger that uh we were under it via erb, so we can eject these projects and stuff.

A

But when we first started looking at this, uh you know the obvious thing to do is to use the rest api because we already have a gitlab rubygem api rubygem, but starting to look into it into the api routes. So we want to. We want to think two bits of information here right. We want uh for pipelines, we want to know the duration and we want to know the status of the pipeline and the rest api. If you call the route, if you say for project 125, I want all of the pipelines.

A

Well, the objects that are returned to that pipeline's list. Query do not include the duration, so you have to make a subsequent uh rest rest api call for every pipeline to grab the duration out of the object, and you know we're this app is one month right now, but where is is a one month window right now?

A

But this is just the start, like we've already been talking about how we want to do like comparisons between you know, maybe this this past three months and the three months before that, if you want to do quarterly or month by month, so we're talking about you know potentially tens of thousands of pipelines and uh that's a lot of rest api calls which is slow, which slows down the build of the the static site we're using here so looking into graphql a little bit.

A

There are some interesting things about it too, but I think it ends up being a better framework for making these sorts of uh more complicated queries that might return a lot of objects, and so you can see here I'm sharing on the left. This is the query that we're passing into graphql, um and so you see here in light 14 we've got the query.

A

uh You know what a name um now we're iterating through the projects that are passed to either render to the query and then we're pulling out a few fields so like id name, web url, and then this main and master. So you can.

A

This is basically just saying only give me the pipelines that are on these branches um there's not a way. Unfortunately, ref doesn't take an array that would be nice if you could just pass an array to ref and not have to like create two different nodes with different edges.

A

uh You know graphql is an edge or, as it is a graph based database. So when you get results back, you've got nodes and edges that you might have to like clear out and trim up before you pass it into further data structures.

A

One nice thing, though it does have, is this idea of fragments. So this is pretty much like a reference in yaml. You can define it once and then use it throughout the project and that's what we're using here. So this fragment is saying: these are the things I want to know about the pipelines that I'm retrieving and then down when you're specifying the branches you can just use that reference instead of you know, writing that code again so yeah. This is the graphql query.

A

Let me go back.

A

We've got a new pipelines model, um so in the in this lib folder lib data sources folder, we already have merge requests which is using the gitlab client that I talked about previously to find those. Mrs, um the pipelines is it's very similar like it uses all the same. um You know this. This is manic right.

A

You have to have an up method, that updates your data source and an item method that returns um slices that base dip into paths, so you can retrieve them later and so we're not using the galab api here, but it's pretty easy to just hit the graphql endpoint here and uh with graphql you post the query as data, but it's not like you know from http.

A

You would think a post would affect the state, it would modify the state of the system, but that's not how it works in graphql, um a post is just a query, so yeah we render that template, send it back and then kind of build up our own data model with the relevant bits.

A

One interesting sort of uh challenge here is that uh so, for you know we're making queries that are not just going to return 50 items they might return hundreds or thousands of items. So we're talking about pagination now right like same as a rest api, but the way that graphql implements. This is what these cursors um and a cursor is.

A

Basically, it's like a base64, encoded string that refers to a position in a list of graphql resources, but that it's it's not like filtery and red in a rest api where you're giving it, because what we?

A

What we're doing here is we're saying that um let me switch back to the query, we're saying that we only want pipelines updated after x and or after uh the cursor, and so that's how we're getting things from the past two months and not getting everything in the world um and so the way that this happens I had to uh in the in the graphql here you can see we're looping through the project ids that get passed to the query, but, uh and that would be nice if we were able to just say, hey give me all the pipelines for all the projects.

A

That's one query for graphql versus you know how many hundreds or thousands for rest, but it gets a little tricky when you're working with queries, because because of those cursors, and so if you you're pulling back multiple projects with multiple sets of pipelines, then you have to keep track of multiple cursors in relation to each other in order to make a following graphql query so that all just kind of got a little too complicated for me, and so basically, what we're doing is that for every project we are making a graphql query.

A

So for this page you know we're making like 10 queries to the database, which I feel like is acceptable, um and that's just so. We could have an easier time like uh with this pageant issue with the has next page of the cursor. So we can grab all of the data.

A

um Yeah, that's most of the source code. For that uh this, this pipelines is also using this projects.yaml, which is the list of the project ids we're responsible for. If so uh we had talked about this when we were doing merge requests, but it's nice to just have a list of static data that any of these pages can use to grab the project. Ids.

A

um Yeah, so I think you know this is the current state of things. I think this is a good mvp, but, uh and it helps us answer, questions like how are how are projects doing uh you know, success, fail and relative to each other. What's a heads-up overview of the duration and the range that we're we're seeing in our pipeline times, but there's some obvious jumping off points here.

A

You know it would be nice to be able to uh go back further in time or maybe go back in ranges of time, so you say instead of just for the past month. I want to do the past three months or you know, maybe some of that happens at gitlab or something or something that event in our team. Like I don't know, we we all take two weeks off. We hire so many new. You know whatever uh it might be.

A

Nice to you know view this data for a range of time, so you could say like from last november to this february.

A

uh What were the statistics, and then, along with that, since we're talking about range of time being able to see- and this is this- is what this cic cd analytics page shows like for it shows the durations for the pipelines for the last 30 minutes and then showing some historical trend. Data here for the success fail, but I think that uh some version of that being able to say well, our pipelines, are getting slower over time faster over time. You know this kind of data.

A

All heads up in one place will help us, I think, uh put some priority on. You know what projects we focus on, because we can't just give 100 to every project all the time.

A

B

Justin yeah, just to give you an idea, there's something that I have done in my case when I was working with uh some of the time sensitive data like, for example, I'm generating each week, I'm generating what I've worked on by going to the apis, I'm just creating the static page. Just like you do here too, but what I also did is the pipeline that runs it.

B

It has the default values that just generates for the last report for the last week, but on the flip side you can actually run it with the custom values and produce.

B

Whichever section you want you want you can produce for a year for two months whatever so perhaps they'll be just you know, maybe it's not ideal and it's not immediately available, but at least one can run the pipeline and take a look at the results for themselves as an alternative to this, because we don't have any database behind it to actually store the time series to create a nice graph. Not yet anyway,.

A

Yeah right yeah, that's a good point. uh I I kind of thought about that. You know because yeah every time we refresh this page, it's pulling it fresh from the api and there is some caching with danuck, but it's kind of like a file level. Caching, like it's just sticky stuff in a build directory, so that's not really distributed or whatever um so yeah I mean.

A

On the one hand, I think it would be nice to have a data source, so we could query more, but that's kind of what graphql is right: we're making queries into the giant data source, that is the gitlab api.

B

One other thing I would kind of see as being able to do right out of the box again, because it's a static content that we could what we could do.

B

We could pull like. We could publish the data as json or yaml whatever, that is as an artifact for each job and just query. The last successful job pull the data and then you can display the delta here and we can key that by you know the execution. If we combine the two suggestions that I've made, so the customer execution will not publish the stuff into the artifact, but the scheduled stuff will and then you can see the delta it went up. It went down.

B

You know that kind of a thing I don't know just yeah.

A

Right yeah, I like that, I mean it makes a lot of sense, because this is data that is not being updated. Right, like these are past pipelines. This data's never going to be updated so querying for it. The same data every time is kind of wasteful in a lot of ways, but yeah I mean I'm looking forward to you know getting this merged, and this is the first pass and I'm sure just like with the merge request application. You know we'll make this better as a team and more usable for us.

C

I'm curious with the graphql pa. How long does it take to generate this report in comparison to the mr report.

A

uh This takes uh well, it's it's kind of hard to tell because it we it compiles them together, but I did notice. It was taking locally like five seconds to compile the mr and then once I added pipeline in it's now 15 seconds.

A

Yeah I mean I did notice that when I bumped it up to two months, uh there's basically a linear relationship between the size of the graphql query. So it took twice as long.

C

And just in general, how was it working with graphql? Was it pretty easy to iterate on and was there good documentation just because I see there's, probably opportunities elsewhere, that we could use those endpoints instead of basic um rest, api endpoints? You know.

A

Yeah they're, the the document gitlab documentation for graphql is pretty much all I referenced. um I'm trying to get uh so the documentation is good. The api is fully documented, I'm sure I'm getting to the graphql explorer page uh somewhere in here yeah. So this explorer was really helpful.

A

It's uh yeah I get lab graphql explorer because this is just a console for graphql, and so when I was building um this query here, this query and even the fragments and everything you can put all of that into the explorer, um and so that made it nice because you could just change things and then one thing about the explorer too, is that it auto-completes, and so if you're saying pipelines, semi or you know, parentheses and you start typing something, it will only it'll auto complete all the fields for you, so you don't have to like guess about it and then, when you click into, um if you, if you click on a field name, it pops up the docs, which are the same as from the gitlab graphql docs, so I'd say over.

A

I I didn't use a whole lot of like outside graphql blogs or documentation or stuff. Other than to understand the cursor stuff a little bit, um but they get they get lab. uh Graphql docs explains that briefly, but yeah I'd say it was a positive experience. I didn't have to read like a ton of documentation.

A

It was obvious where I needed to go in the docs too. You know it's just alphabetical the api docs for graphql, so you can jump down the pipelines and find it pretty easy.

B

Yeah yeah one thing that I did find with the graphql when I was kind of trying to trying it out for other purposes, and that was not all of the endpoints are exposed in graphql. So certain things you cannot query on period and that was like, where I stopped, because the stuff that I wanted was not there. So I ended up doing that.

B

A

Anyway, right uh yeah, I mean, I guess. One thing I would say is that you know we're working on this. One of the things we needed to do was to have this. uh Was it updated af? What did one of these? uh I think it was updated after was not or maybe it was.

A

I forget, one of these fields was not there when I started this, mr and so had already created an issue for graphql to investigate it, and then charlie blitz, hopped on and helped us and actually added this field. So I'd say if the graphql api doesn't have something we want like it's. It seems like it's pretty easy to get some help to you know, add fields or routes or whatever.

C

Yeah that was awesome. uh Two quick, more questions then so in that uh explorer it will actually run the query and return data for you in the ui.

A

Yeah, let me actually I'll just paste in a pipeline id uh get rid of this.

A

It's also got all the shortcuts of like vs code, like I just did- uh control d there to select multiple things, so it's kind of like an ide but yeah just id here, quick, so got the id for get lab operator.

A

Put it in here.

A

Yep here we go.

C

And then, if you need to hit, uh if you need to get any like private projects, you just pass an authorization header and then.

A

I think I'm already authorized uh I'm already authorized by being logged in to this, um because I was testing this with security bearers too, and it worked yeah using your.

B

A

C

A

C

Oh sorry, in that, um in the ui you were showing, where it kind of had presets for like prepared statements for data, is um the gate lab code base basically doing the same thing like posting, an http post request to get that data, or is there some kind of library that makes it easier to interface with graphql and not have to do that data translation step? uh I think.

A

They've pretty much they they're.

A

It's in the prepared statement, all that logic like. uh Let me just fight it here, quick, it's pretty easy to find project statistics.

A

Yeah so there's this project statistics item um in the.

A

Or it's pipeline statistics- I don't know it's somewhere in here, but basically there's not a whole lot of stuff going on on the front end here like at the javascript level or anything basically, the the they. They created a special route for this page that just passes back the data already prepared.

C

Got it so there's nothing, there's no like handy library to handle that data, you know pagination and wrapping into objects, etc. Just to.

A

Go off yeah yeah. We did it from scratch for uh this so far, it would be nice to have uh you know, we're doing the pagination here with a while loop uh and that we're building up this project item but yeah. It would be nice to have some of the syntactic sugar. That's in the rest, api gem, where it does this, because one of the other things that I ran into, I guess with graphql objects, is that you have like edges and nodes objects, that wrap other objects, and there is a javascript library.

A

I think it's called like graphql printify or something, but it strips out all of those extraneous objects and like furnaces vertices and all that um that doesn't exist for ruby yet, but that would also be a another nice improvement because that's one reason we're creating a data structure from scratch. Instead of.

A

Instead of just parsing the the data that comes back, uh you know and returning that I think that's what you do more for http or like a json api is you you know you don't have to translate the data too much before you pass it to the view, but um I don't yeah. I don't know if the if there are plans to bring graphql into the gitlab api gem or whether that's another gem, I don't know like what the community standards are. If you do two or just do one.

C

Yeah, it makes sense because, in your opinion, it probably wouldn't make sense to use graphql for the mr dashboard right, because there's so much less data and it's a simpler query, is that fair.

A

uh I mean it might be a little more work on the ruby side to do it, but I guess I it seems kind of like case-by-case basis right, because uh if it might be the case that to pull all the data you have for the mr dashboard, you might be able to do that with one graphql query and then page it a through it like we're doing here uh and also it might be.

A

The case, though, that, like dimitri, says, there's a route, that's missing or something so that turns it into a little more work but uh yeah. I think it's gonna have to be case for case by case right now.

C

Got it yeah, it just seems like in the future. It seems like it'd, be nice to try with graphql first to see if you can get it on one neat query and if not, then you can fall back on rest.

A

Yeah I mean I, I think it is kind of nice to having the query uh you know, even if it is a template in the project like checked in versus, like you know, uh in the ruby code, I don't know just for me. It's it's it's nice to have this as an example to look back at like when I forget how we did fragments in two months. I can just look at the query instead of like having to read through the code so.

C

Yeah yeah, the syntax, is really unique, but it's pretty readable.

B

Yeah, that's one problem I found with the graphql is that if you're not in it all the time it kind of flies out of your head? It's not sticking that really well.

A

Yeah yeah, I can see that, but I mean I've got. You know a lot of uh rest. Api experience a little graphql experience, so I don't know it's the new kid on the block. For me.

C

Oh, that was awesome.

B

Thanks dustin.

A

Yeah, thank you all right, uh yeah. I think that's it for this time. So we'll get this recorded and published and we'll make this app better as we go thanks, everybody.