GitLab Engineering Productivity Showcases, 12 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Engineering Productivity Showcase 2021-10-12

Description

Albert discusses Selective Test Execution initiative

A

um I'm just going to go through the selective test execution, so I think, first of all why what it is so um we, the idea, is.

B

That we want to run more targeted.

A

Tests for back-end and front-end code, so we um given the size of our code base. um We it's the hypothesis is that we don't. We don't necessarily have to run um all the tests for every merge request, because uh many of the merge requests will probably just change we're changing like one.

A

A couple of files will be a handful maximum and given those we should be able to identify tests that are directly related to those changes and only runs those specifically so with that um we hope to be able to get a faster pipeline so that we get. We get faster feedback from that from the less number of tests being run as well as saving the cost, because then we have much less uh runner time being used.

A

So how is it done? I'm just going to share my screen and then I'll go through so also you can see.

A

So I'm using my ipad as whiteboard, so we will see how that goes.

A

So um I think if you want to look at our code base, we have um front end code on one side we have backhand coach, so this is like javascript, and this is mostly ruby.

A

And from test perspective, we have just for the front end and we have prospect.

A

So so far, what we have I mean previously. What we've done was that every back-end code will run almost all the rspec tests and then every front-end changes will run almost every uh all right. Oh no, it will run all the jazz tests that we have. um So we want to be more selective in running, only related jazz tests or front end code as well as um related aspect code for backend. So then the question is: how do we? uh How do we link the two together um for the backend? We have?

A

We are using crystal ball, so crystal ball, trace the exact um code coverage of the test and the source code, and then with that it will do a reverse map and identify what um aspect code needs to run for specific ruby files.

A

So what we are seeing with crystal ball is about 85 um accuracy, meaning 85 percent of time.

A

It will be able to cover 100 percent of the changes of 100 people like be able to catch 100 of um the the test that needs to be run so that when so that we don't get into a like a master, broken situation or uh we get a regression where the test fails without us, knowing so that they're missing, 15 percent can come from like things like uh various ways of meta programming, which then doesn't get tracked in the code coverage.

A

um So we have done this so far um and then for front end code. We are actually using js itself. So just has this thing called find related tests, which has been pretty good. It's close to like more than 95. I would say accuracy.

A

I think that even the I think it's even close to 100, because um what I've seen was that the fairness that we found was actually uh master, broken failures that also exist in um the main branch. So it's not really because jazz was not able to identify um the.

B

A

Between the code and just and yeah, so so with so now we have these two, um I think js we uh pretty much um yeah.

B

We are running.

A

Just just related tests for all front-end changes, um so so the challenge is now between the back-end code and just so, we have this sort of dependency right now we run. We still run all the gest tests that we have for every backhand code back end changes because there's the fixtures in the middle, so fixtures, um the.

B

A

Are generated from backend and then it's loaded by just for to run the test. So then um we need right. Now we don't. We don't really know how.

A

We can um identify what js, which friend test needs to run given a change in back-end because of the fixtures, so um there are like two parts to this so first, so one part is linking between the ruby code to fixtures.

A

So we don't have this um right now and the other part is linking the fixtures to just so. This is um right now in progress like about maybe 80 done, I would say um we have been migrating.

A

The use of fixtures in just to use the import so using import um then just will be able to identify specifically which fixture file is used in which test files, and because of that, given a change in a fixture, we can then uh trace to specific test files that we need to run.

A

um So the missing pieces missing piece will be. Then right now is this, which is how do we track um ruby to fixtures? I think we have ways uh we probably can find ways to do it because of how the the test is written. It's an aspect test which just generates json or html and save it into a file and given a specific file name. So we can probably trace that and then sort of use similar um like crystal ball or something just to keep track of.

A

The coverage like to use the coverage and link that to the fix the back end. Fixture spec form yeah. So.

B

A

Now the missing part is really this um yeah.

B

I think yeah so.

A

That so that's how how I see the entire code and the test being dependent on each other and where is where's the missing gaps that we need any questions so far.

A

Oops, okay, I didn't see the chat one minute's up. That's okay, yeah! uh No questions. Okay,.

C

uh I I I would just ask one question just to clarify here: um yeah.

A

C

Selective tests or like.

A

C

Related tests and crystal ball those run prior to approval right. Yes,.

A

Okay, yeah um yeah, I'm not gonna. I decided not to go through that here, because I think most of us are aware about that, and I think that was the part that is not very clear is how all the pieces are fitting together, and I think I thought it's best to give that overview sounds good yeah, so yeah I'll probably draw this up and then put it up in one of the issue.

C

B

A

I want to share is just the chart, so I'm not sure if everyone knows about this.

A

Yep questions, uh given that our.

B

Fixtures are also written in our specs.

A

So can we do this then use crystal.

B

A

I guess that's the thing uh yeah, I think that's one way um we might still need to run all of them and I'm not. I haven't really thought through in detail but um yeah. I think we can.

B

A

B

Okay, cool yeah.

B

So the the plan, if we can do that like if we can from front-end front-end fixture spec, uh find the rb file that that would cause this to change yeah. um Would the plan be to uh only uh regenerate the fixtures that that that uh subject to change.

A

Yeah, I think we still need to regenerate all the fixtures because there might be other tests that are dependent on the front end code and not just not the back end, but not just the front end, but it still requires uh another fixture file that didn't change from the bracket doesn't make sense like we might have two um fixtures.

A

A and b, for example, um fixture a is not changed, but fixture b is changed by the back end, and then we have a test that requires both of these fixtures and in that case we still need to generate both of them not just b.

B

Okay, and um do we still plan to um to actually uh check in the fixtures in the repo.

A

um I think it's not necessary to do that for this purpose. I think um that can be a separate um initiative. I would say like yeah, it's it can't read a couple because I think um checking in checking in the fixtures is a more com. It's a more complex process so far. I think it requires a lot more.

C

A

More changes to uh developer workflow for the front end engineers, especially so I think we should separate the two.

C

And just to add to that, I chatted with ip last week on that as like and treating it as a separate initiative.

C

He kind of point- and I was pointing out- we don't really have the front-end workflow expertise to know how to do this without creating impacts to all front-end engineers, and um he also said like he had a really good suggestion of bringing a member of the ep team one or two back end engineers and then one or two front end engineers from foundations to come up with a solution for how we can manage fixtures in a more reproducible manner versus generating them every single time so being more efficient there it it's.

C

um It seems that, because of developer workflow on both front end and back end, there's just more considerations than just let's commit them and let them let them go there. There's a there's, a few more impacts that we need to identify and figure out how to mitigate.

C

And I agree with albert it's going to be a separate initiative, maybe something that comes up in q4. We can talk about that related to okrs down below too.