GitLab Pipeline Insights Group, 10 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Pipeline Insights:Internal Stakeholders Sync - Nov 2022

Description

Reboot of our internal stakeholder monthly sync.

A

All right so hi everyone. This is our uh first newly uh rebooted pipeline insights, uh internal stakeholder, sync um on November 7th, um so, um like I, said I'm new to this, um so I'm I, I populated a couple of items for discussion here um and then uh you know. Obviously uh everyone please feel free to to add items as we move along here, um but just as like I guess to to kind of get us kicked off. um You know. Thank you all uh I know.

A

We've had a couple of product changes here in pipeline insights as James, and then um Jackie was interim PM for a little while um and I started in the spring. So um it's it's been a wild six months six plus months, um but I uh feeling like I, have a significantly better idea about the team's Direction um and where we want to go so I do have a recording that was done about two weeks ago.

A

um That's the general pipe site pipeline insights Direction, um which really highlights like the remainder of this year, which is going to be focused on build artifacts um due to the storage enforcement limits. uh We do have a couple of higher priority features that we do need to get out, um uh but then some of the other items that we're really focused on um uh around the code testing and coverage like flaky test. That's something that Gina is going to start doing more research on um to help move this forward in that direction.

A

So um the video does have a lot more. uh But um if there's anything like General high-level, Direction I'm happy to answer that here um or you know, please feel free to to watch the video uh and ask any related questions.

A

Cool um and then I just had two items that I wanted to share out and then um obviously, uh if you guys have any other questions or any like feedback for the team is part of like our planning process, you know I want to make sure that we get your feedback um and then we can make sure that we're we're planning ahead um to incorporate those features.

A

um So I guess I'll just keep talking um so fail. Pass, testing, Mackie and I talked about this a bit ago. um So I've been doing more uh deep, diving into this, um and it looks like in terms of where we are from a maturity perspective like this really only supports I, guess Ruby at this point, um so we potentially might be I think I'm concerned that we might be a little light on support here um in terms of getting this set up for for customers.

A

So I don't know kind of what your thoughts are around there like what I'm concerned about is we launched or we put something out there and then we suddenly get a lot of issues and questions from the customer and I. Don't know if we have adequate support right now to be able to support that turn. I.

B

See is this on a roadmap, because, from from our key review, uh Sid seems to think that we have failed fast as a feature that we could immediately tell the world about it. That's part of our part of the action, and we took us to sing it sync up with the product management team, making sure that our customers know that there's something such as a fail fast.

B

We are open to link to things that we are currently working on the customization on top of field file, so the users know that they can do this, um maybe copy our CI templates or something along those lines.

A

Okay, um I, don't I, don't officially have this on a road map.

A

um We we had a meeting this week with sea lights um and they do a lot of the the end-to-end testing and I would say a very similar feature. Actually I, don't know if you're familiar with them at all, um but they look at like code lines of code that have been changed and what are the tests associated with those specific changes to highlight like hey? uh These are the tests we should run um if we run these first right. Here's all the pipeline savings time that we would get uh through that.

A

So they do offer that as a service. They also offer a service that essentially says like hey here's, a lot of code that changed and there is no test uh to test that, and so that would also highlight to the um to the user as well.

B

I'm looking it up so sea lights is not a gitlab product.

A

That would be an integration partner.

B

A

So that's that's One, Direction uh that we're looking at in terms of bringing this forward faster um and then I think another option is um working directly with the ml Ops Team to see what it would take for us to build this capability internally.

B

um I I thought we already have uh the fail fast, CI templates. That's what we're using.

A

I think they're, very my understanding is they're. Very uh infant staged.

B

Right and we have some customization on top of it, and that's what we're currently using right now for for the monorepo yeah.

C

I'm I'm not sure what we're doing back in in engineering productivity with it, but the fail fast feature was based on the the tff gem, so it was Ruby, rail, specific and and the template we had was I believe pretty specific as well.

C

um I'm gonna post a link to the docs that we have on it right now.

C

So if you want to take a look at that, that'll give everyone I think an idea of just how much coverage we have so so I think Jocelyn was was correct in that it's it's pretty uh minimal as to coverage right now, but maybe Jennifer could comment on on what they've expanded on in engineering productivity.

D

um I'm not actually very familiar with the fail fast scripts or the template so definitely haven't. We I personally haven't been involved in any work around this, so maybe the team hasn't really done anything either.

C

Yeah I think it was a was it.

C

I'm trying to remember who it was who worked on it, but but I think it that was something specific for git lab that that engineering productivity was working on. um But when it comes to uh customer facing it's, we're limited to rails. Right now, really.

B

Yeah and for the spirit of iteration, if, if this works for rails, uh we should put it out there right, because it's if the customer is using rails, they can use this start to use this immediately and give us feedback iterate on on top of it. So joyson is this on your roadmap at all. Are you prioritizing any work here so.

A

I, haven't uh specifically prioritized work around this previously. um This is something that I'm I'm, trying to figure out.

A

um Sorry, uh yes, I'm trying to figure out you know, is this uh another category that we need to be focused on um my uh talking to James. This was something we worked on under code testing and coverage.

A

um But there hasn't like in his his recollection, like he knows that, um like Albert and I, think Drew did some work on it, but there hasn't been a lot of work done on it. um I would say easily in a year.

B

Yeah, it's been, it's been roughly a year, Albert was on our team and then he moved over to to the product teams to extend this.

C

Yeah, that was, it was Albert. That's who I was thinking.

A

And um kind of like one one wrinkle to that is I, would say pipeline insights for the last 18 months, roughly has been really focused on build artifacts.

B

um That's good to know: I wasn't aware of that. Yeah.

A

So I it's I mean I I love. This idea and I certainly want to do more around it and do that research to see you know. Do we integrate or do we um do this like in-house, with ML Ops um and build out like a data model to make it intelligent? Because I know there is, uh like analysts have said, like kind of the future of CI. Cd uh really is, is dependent on like the intelligence piece and there's a lot more of the Intelligence being folded in um we.

A

Just as a team for I mean we're, definitely focused on build artifacts through the end of this year, even um because we need to our our customers have a big management um problem with our build artifacts, and so we've just really been got.

B

It okay, so I I'm trying to get us to like a minimal iteration, at least to pin something that we already have and get feedback from users out. There could could we at least link to the field fast template, and you know, even though it put a caveat saying that it's it's still in its infancy and we um uh use it or contribute to it. I mean that's, that's how we work right like like we. If we can prioritize working on it at least the community, if they're using it, they can contribute back to us.

A

um Yeah, so we can certainly do uh like maybe update our documentation around this.

A

A

We can um Gina is this something that you think would be like a candidate for like Banner feedback kind of stuff like if someone were to like to is there a way we can kind of use that Banner feedback idea.

E

uh We could definitely figure out how to solicit feedback in the UI I. Don't know if Banner would be the right thing, but we could do this yeah and also we could maybe create a blog post if there's not already one out there and, and that seems to help get the new features like get customers to understand what we have. That's new.

B

So, if I'm reading reading the reactions we we want to shy from putting it in front of users, because it's not mature yet is that correct? So what I'm hearing that we're exploring other venues to put this on, that is not front and center or on the customers page.

A

So we can, we can definitely like put. uh We can try to put in an MR to put this on the customer page. um We don't control that content, though.

B

Right: okay, like.

A

Like you know, I, don't excuse me, I, don't know specifically like from like marketing or product marketing, whoever uh if they would want to uh I guess use the real estate um for this purpose.

B

Yeah yeah, it's it's premium real estate and the marketing team governs that space that how I'm reading it correctly yeah.

A

B

Okay, got it yeah that helps a lot. Okay, then that I can take some of that con context back to uh back to the key review and said so.

B

He understands where we are uh I, think landing on like a uh an iteration, that's palatable for everyone, updating the dock or or uh or being a blog post, maybe too heavy, because I'm not sure who has the bandwidth to work like on that right now, let's, unless um Albert wants to chime in it's been a year, so he may have to, um we might have to resurrect some old artifacts for this.

C

If, if it helps at all just uh understand what usage um I I, don't know what we're doing with the templates, but the if you look at the test file, finder gem, um we haven't had an update since June of 2021, but that version had six and a half million downloads. So it's oh wow. It is yeah. It is out there.

B

C

Using it seems like it.

B

E

So, let's just go to try to get feedback from those people that are using it. Is that what we're kind of doing, or are we trying to make this thing more discoverable.

B

I would say both uh I think I think it's discoverable already, if we're not putting any any voice behind it, and we have that much download, so I think I think that's a signal to us that there's people that want to use it and I I suggest at least putting something out there to get feedback, because if people are using, it they're, probably running into limitations that we may know or may not know, and when Jocelyn, when you're ready to pick this up again.

B

uh That could be good input to the product maturity and the direction too.

A

D

I I just realized fail fast uh in this context, is, has been uh I've been referring to as test intelligence. Apparently these two are the same thing. I didn't know that we had the plan to actually make it a pipeline configuration, so that was that was the part I didn't realize. We.

E

D

A team to use it a lot I just we don't think we made any improvements over the past year, like what's that past pointed out.

B

Jenny, maybe it's worth it. I know that Remy probably know this apple tablet's head, but there's those those customizations. Sorry.

D

B

Customization is probably on top of the fail fast uh and test intelligence is, is what was discussed in the key review. I think we need to set that record straight. It's it's fail fast or unless joyson you want to rename this to test intelligence or or um uh create a new category for this group for more Investments, that's also offending for you, too,.

D

It doesn't matter as long as we get our language consistent, so I, don't I, don't keep tripping on I. Think it's referred to um like I. Refer it as test intelligence than somebody else. Call it something else like text selection process, I feel, like the words are being used interchangeably here, and it's confuses to me. But I just realized fail fast. It's we may we basically just uh based on the the test file finder and as well as the crystal ball gym.

D

That's how the gate lab project, it's adopting the fail, fast approach, but it's the idea that we're trying to make it into a yaml configuration is that what the idea is.

A

I, don't know that we we have a final solution um at this point, uh so I don't.

E

A

If the answer is like a final yaml configuration whatever we do, we do want to make sure that it's something um our customers can can configure I think reasonably easily. Like one of the pieces of feedback we get from review. Apps is people saying like it's not out of the box, it doesn't just configure.

A

um We do want to avoid doing that twice, um especially knowing that we have that feedback.

D

I see, okay, so have people adopt the approach in a in a consistent way, so they don't have to create their own scripts because it looks like just by looking at how gitlab uses the fail fast or it uses the test file. Finder gem, it's a script that we built and then we implemented in the R-Spec helper script.

D

So for contributors to or our customers to use this they would probably have to build their own Scripts.

A

Yeah I I think if we want, because part of success isn't just finding it, it's actually able to use it and so I'm concerned that if we put something that's like really, we don't have maybe all the right documentation to guide someone to configure um like we're.

C

We're not going.

A

To get real usage, we're we're probably going to get discoverability we're going to hit that challenge. um Work.

A

um And then we still won't know what what features that to actually build out for it. At that point,.

E

B

We get them to use it one time, but then it doesn't result in sustained business value. Yeah, okay, uh I'm curious about your plans with sea light. Is that a confirmed path that we should align on or like? What's the importance between our in-house features and Integrations as I see it with like tab, nine and gitpod as well? uh What's the level of or what's the prioritization uh levers on that.

A

Yeah, so we're currently just working with them to understand their offering. um We are also investigating um what the integration could look like. um They have a dashboard that focuses.

A

Largely on like usage for for QA teams and not so much the developer um and so.

A

So then the question becomes like what what data do they have that we might pull into one of our Pages, um where we could display the information that that we're talking about right, like code, changed, here's the test files that we should be running um or if there is no test file, here's what we should be running or someone should go at least manually test um designs. So we it's still unclear, though, like in terms of what data that would actually be willing to give us like.

A

We would have to Define an API um for that data exchange. So I would say it's still very early like I would not say. This is something to double down on yet right, because we've had discussions with with companies like datadog or datadogs like no like. We just want get lab to give us more information, but we're not going to give you information um to to assist in your dashboard. So um it's really going to be dependent on what they're willing to share.

B

I see okay yeah, they look pretty interesting. I wish we had this building house, it's kind of like the VSM for testing almost.

A

Yeah, it's a really cool dashboard um and, uh and they did a demo, um we could certainly ask I, I suspect they're, going to be doing more demos with us, and so I can certainly ask that the next time they do a demo that you're also invited. So you can see what they offer.

B

Yeah this is this: is the final I want to walk, because there's a there's a push for us to use our own stuff versus using something outside. It's always going to be that that questioning from our stakeholder yeah or maybe not using our stuff. Why are we not using things but I yeah go ahead.

A

Part of working with him too right is, we need to see like who their actual customer base is. I mean they are pretty small um and so for us to actually go and build. The integration like we have to have enough I'll, say overlapping customers to.

B

Really make it worth a while um yep that makes sense. Okay, for the sake of moving towards the common iteration I think we we agree that we want to make the document more discoverable, um either a blog post or just an MR to discuss with the marketing team. If, if it's Worth, to put it on that real estate, uh is there any anything else? You actually that's regards to testing like test intelligence or anything else. That's not fail fast that we can Clump into one package. So it's more meaty.

A

Yeah I mean one thing we don't have yet yet that we would like is um around flaky tests as well, um because I think that's also that's a gap in like test intelligence like in terms of like how, when we say. Oh, these are the tests. One should run. uh We also need to be able to, like I, think, pull out those flaky tests as well.

B

Yeah, that has been the manual customization by inch fraud and I, think we have like an influx DB that we store. Is it right Jennifer like.

E

Something in some.

B

Sidecar database, that's not actually part of the application.

D

That's right: there is an endpoint that we can query. We will list out the the flinky test. It's like a report that shows you the line numbers the file name uh and it gets updated every time when Master a pipeline runs on a schedule, so it gets updated, pretty pretty fast.

B

So to answer that choice, we have it, but it's it's like a sidecar foreign.

A

Yeah so I think maybe.

A

Is there, is there like a piece of documentation or something that I could.

D

Yeah, let me just pull up the endpoint. I was just looking at it.

B

It's probably an hour for documentation.

D

D

B

D

And this documentation for.

B

This almost the time to.

D

I have to fill in the notes async for the documentation.

A

All right thanks, Jennifer um and then um so yeah, so we'll look into those um and then also I. Just like wanted to share out. Like the big thing, the team's been working on um Gina's done like a ton of work um in this area around the artifacts page MVC.

A

um We uh what's a good way of saying this um yeah we're giving more visibility to our artifacts.

A

um And so uh we missed the cutoff for 15.5, uh but we will have it in 15-6 uh as well as some additional features on top of the MVC.

A

um We'll have we'll have some sorting and filtering um as well as it looks like uh the Mr for deleting single artifacts um was also is also in review and being merged. So we'll start we'll start giving folks more more control, more visibility and more control over their artifacts.

B

Okay, I click on the link. It's not running, okay, it's finally rendering. Oh wow, it's been open for a while yeah we need. We need bigger investments in CI, CD I. Think people want many. Many things that I can empathize is not not having enough Investments I think we're finally trying to get that this year. I believe.

A

Yeah Gina I, don't know if there's anything like you wanted to to add around like our artifacts page stuff, I.

E

Think you covered most of it we're this is for a project.

E

um It's the first level, so you'll be able to see all the artifacts across your project, and it was largely meant for making it easier to manage artifacts because they roll up to storage.

E

um But we also know that it's really difficult to find artifacts after a pipeline is run or a job is run or even tracing it back like a couple weeks later and trying to find them again. So those were our main use cases for the page.

A

um But yeah so because the storage thing, that's, um that's really hit our team really hard um with the storage enforcement. So that's um that's been a that's been a sore spot for for the team and that's really been I. Think what's gonna occupy us through the end of the year.

A

There's a light at the end of the tunnel end.

B

Of the year well, it's this year right not next year, calendar year. Yes, so like two months left.

A

Yeah, okay and then books on these other things.

A

So cool so um so uh Cadence is set up for monthly right now, um so the first Monday of every month um and then I'll I'll get like the agenda template out and then um please like, please feel free to add things that you guys want to talk about um in advance.

B

Cool thanks a lot uh yeah I'm glad we are finally rebooting. This uh conversation again, I think we we haven't been having that and it's on it's on us to maintain the relationship. I think this is. This is a great.

C

Start and Jocelyn's, if it's okay, I'm gonna, add uh Tiffany Ray and Richard Chong from the QA to.

A

This meeting people I wasn't sure who to add, like I, looked at the old meeting, yeah.

C

I was the original, because I was the only SCT in verify at the time, uh but now we have. We have two scts that work within pipeline all three and pipeline execution and I'm. Sure they're gonna have some valuable feedback, I'm still interested, um and it still affects me but um I think we might want to add those in Mac. Unless you don't want to spread us too thin enough with distractions.

B

Yeah, um we should definitely involve involve the team in uh in there.

D

One quick question about the artifacts page: I was just looking through the screenshots Linked In. The issue and I didn't really see the size of the files in the artifact page, because I remember last time we talked about that was like one of the pain points. Is people didn't know like how large their files are, how many they should delete.

E

Yeah I think it's in a separate issue. That's why we broke that down.

A

E

Many I'll I'll link it in the agenda, so you can see it, but we are adding that.

D

ah Sounds good! Thank you. Yeah.

A

And- and we are working uh through some calculation issues with the storage size as well still so that has been ongoing. uh Just as a heads up, um there are we've seen a couple of reports of negative uh storage values so.

D

um So you probably don't want to show those yet yeah.

A

We're not quite ready uh for prime.

D

Time, okay makes sense.

B

Cool uh yeah, looking forward I, mean I, think we have a path forward and yeah looking forward to the next the next catch up. When is when it's gonna be the next catch up? Is it gonna? Oh.

A

It's gonna be the first Monday of December.

B

Okay sounds great.

A

You've probably been monthly um or I think at one point they got moved to every six weeks, um so you know for sure. Like I know, I scheduled this for 40 minutes. But uh let me know if you guys think we.

E

A

More time or if it's uh as we're kind of catching back up, if we want to bi-weekly for a little bit, uh you know we can we'll continue to adjust the Cadence.

B

Okay, uh why don't we do this? Let's do a thing for now to hash out the the current next steps and I'm sure when we finish the I'm sure you would love time back to deliver the artifacts page and everything that we probably better it's better to pick up uh once that is completed, sounds.

A

B

Cool well, thanks all appreciate it and um yeah uh looking forward.

A

Yeah. Thank you all thanks. Thanks bye.