GitLab Backend Pairing, 16 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: BE Pair Programming - 2023-08-16

Description

Today we paired on fixing a master broken issue

https://gitlab.com/gitlab-org/quality/engineering-productivity/master-broken-incidents/-/issues/3317

A

Okay, so this is a back-end pair session and we are going to look at a broken master that is ongoing right now,.

B

Yeah, so um if you, if you look in master Griffin, there's there's a few different failures, but.

C

B

Mainly it's mainly this issue with rack attack and, unfortunately, the tests are excruciatingly slow to run um I'm, not totally sure why I think it's, because it's sending a bunch of HTTP requests.

B

So if I try and run all the Tiff, um it takes a very long time to finish.

C

B

But also, if I run the test individually, it passes.

A

Oh, um that that happens to me a lot um because I feel like it's like some sort of context or like order-based failure um and the elastic tests regularly. If they're set up in a certain way, they could take like one or two minutes to run locally. So I'm I feel you on things being slow. Oh, okay, cool! So that's an example of one where you ran one that failed before and it completely just.

C

A

um uh If you run it locally, it fails every time. I guess yeah.

C

B

Like at least at least it's reproducible locally, um the trickiest wins are the ones where you run it locally and it passes and you're like yeah.

A

B

Usually it is I.

A

Can I've been somewhat lucky with using that seed that the pipelines print out, I wouldn't say it's like 100 of the time, um but.

B

Yeah I think I see.

D

B

Of the test was the flu um there's a shared example here and I think it looks like we're sending like 10 HTTP requests for each example.

C

B

That's where it's blue.

B

um But it's testing startling so.

B

Kind of kind of have to think a bunch of HTTP requests. You know.

A

uh What is the error I'm? Looking at the error? That's happening right now: money failures.

C

No 16 minutes in CI. That's not that's a small one: okay, um okay,.

A

Oh interesting, okay, so one of the pipeline failures that I saw it has a really weird error in it um that it's mentioning about feature flag. Invalid operation cannot enable percentage of time for a fully enabled featured flag.

B

Yeah I'm not sure I'm, not sure if that's related yet.

A

B

A

B

Well, there's two different failures:.

C

B

I'm not sure if they're like two different independent barriers or if they're related to each other uh but yeah I've, tried I've, tried running both files at the same time, and you know the the Fair start popping up when you do that.

C

B

I I haven't actually gotten to the feature sticks yet because the wreck attack stick takes a long run, um but some.

D

B

Here is interesting, so we're expecting a hash, including uh message wreck attack, yeah.

B

Dennis429 um I.

D

Don't see the status code so.

B

Maybe that is why it's failing, but it's it's kind of hard to compare these gifts.

A

Yeah it's definitely not. My favorite thing to do is try to figure out where the differences, but that status, okay, I wonder if there's been any recent changes to.

B

Yeah I do not see the status code anywhere in here, so that is probably it.

B

B

D

B

The status 149 gets added here. That's interesting.

B

B

Yeah, so we have, we have the picture flag issue too.

D

B

The output is so big that it it overworked my terminal history, so I can't go back to city pool after oh.

C

B

D

Direct attack but.

B

Packed if you run them individually, yes, but the the feature expects.

D

B

C

A

Feature I'm not going to try to run them locally, I, don't I, don't know I, don't think, there's any benefit in me. Running them locally. I will go I'm going to look at them at least.

B

C

Once my IDE stops almost.

D

B

Picture specs do the same thing.

D

B

What if we have both.

A

It looks like there is.

B

That's bizarre so.

A

There is an open issue for the.

A

Feature invalid already, so I don't think this is like a new um I, see an open issue for the uh spec requests. Api feature specs, um that, while all of the errors are not the same, it does have the invalid operation failure.

A

And I see a few others that are related.

C

B

Wondering I'm wondering where this.

C

Came from like.

C

A

Person created it, it looks like I wonder you can like create an issue from a failed spec, and so that might be how that got created.

B

A

Done that before, if you have like a filled pipeline, you can just like create an issue from it, and it does it for you. um I wanted to see if there was any major.

B

I was kind of absolutely.

A

Not in that spec file, but this issue is a month old, so I feel like for some reason. It's become way more flaky.

B

B

I'm thinking about running both the specs again and sticking a debugger in one of the failing examples: okay.

A

Yeah, let's do that.

B

I'm trying to find I'm trying to find a line number for the tip. Oh.

A

Okay, let's see I know those are hard to identify for me, sometimes where I'm like. Where was this in the spec um I'll see if I can get it from the pipeline, I was looking at.

C

I had a pipeline open.

A

Did I override it.

C

I did have one though okay, here's.

A

One of the jobs that I was looking at I'll, put a link in the chat again.

A

That might be helpful to get the.

C

B

A

uh 456 is one of them, I think or it called from.

C

Let me see that trees, yeah.

B

You know what I'm just gonna, I'm, just gonna run it again and then, as soon as soon as one of the tests fails, I'll cancel it and it should tell me which line it was on.

B

D

B

A

Oh I, used to okay, like I, figured out how to read that like one time and then I never figure out again, but it is possible to trace that back to where it is in the code. But when it's a really long spec um and then you add, like the shared examples into it, uh it gets a little murky for me as to how to get back from that.

B

A

You might be able to do like fail fast on our spec, and that at least would fail in the first failure that you get um to make it run faster. But.

C

I don't know if that'll be helpful.

B

Yeah but I think.

B

I'm pretty sure this one is one of the.

D

B

B

D

B

um Yet the the shared example kind of makes it trickier to debug to yeah.

B

The issue is like The Lager, so I think I think it's this one right here.

C

B

B

B

React attack Global, throttles and authenticated webcraft.

B

I can't even find it using them. Oh.

A

Okay, so throttles Okay so um hold on, go back up, it's like rate limited and authenticated requests.

A

So that's the top.

D

B

A

Unauthenticated web requests, so I think will be like the context and then hopefully there will be an update, behaves like that. Has the next part in there.

A

And then, in that it behaves like you'll, have to find the part of the text that matches when the throttle is enabled.

B

A

B

It and it logs architect, info into structure logs, so I think right here.

C

A

C

Then right yeah.

A

Because that's the matcher.

B

The thing is, I, don't want to.

B

Okay, so I'm trying to uh do like a conditional breakpoint so that I don't stop on every single request.

B

So uh I'm going back here and.

B

So here I do signing up crying if you're around it does not require authentication is oh.

A

C

Are signed in that's like it's a good idea, yeah.

B

Okay, so hopefully this is gonna. Stop us on this test.

B

um It's the 27th one.

A

You could run the same seed and then it will run the same because I think it randomizes things. Do you have the seed that was used by chance? No I.

B

Know, I, don't think it's seed related. It feels consistently every time I run these two specs together.

A

B

So I don't think I, don't think it has to do with the seed, but I do feel like it has to do with these two. These two uh specs running at the same time, okay,.

B

Okay, we got into the debugger.

B

Oh, this makes it more difficult because I actually I actually picked a big point at the Run Spot. This isn't where the logger is.

A

B

I actually need to.

B

Actually need to have the breakpoint here.

B

But what I can do is.

B

Yeah I think if I continue this, it's not going to.

C

Oh, it was very unhappy.

C

B

Okay, we got here.

B

Stand as of now, that's interesting, how.

C

Did this happen well,.

B

So so I managed to get into like the failed.

A

B

I meant to get into this method and.

B

D

Status is nails.

B

So I wonder if we're actually calling Track event when we're supposed to call throttle.

B

Yeah so I guess now um I think there's a way to get the back trace on how we got here right.

A

Oh gosh, um yes, I, don't remember how someone just posted it uh about where.

B

A

B

Tree maybe this is what I went yep.

C

Yep, that's pretty nice.

A

All right, I'm gonna, have to move some screens out of the way.

B

I'm kind of having trouble getting it to show me the whole thing.

B

A

Or so I feel like in the innards of okay there. How did you pluck that out? Okay, 484.

B

Yeah I feel like I feel like we don't want to be any of in any of the gyms unless and let the gym it actually was talking about.

C

Oh did something get updated with that.

B

With that, maybe for rack.

C

B

B

This could possibly be.

A

Calculated yeah.

B

Yeah but I wanna I wanna keep going through this first.

B

um So so one thing that we can try is I want to do this back trace a little bit more and then, after after I give up, we can try reverting that commit and feed the tip.

A

B

Yeah we have Rack Attack, shared examples. uh 9484 I think this might be our entry point.

B

D

B

B

I think we want to work upwards from here.

C

I think this code hasn't been changed in so long at least I haven't seen it.

B

I wonder if we're actually like in the test code, but.

B

Yeah I feel like I, feel like we're in the client code rather than.

A

Being in where you want.

B

B

B

B

D

This is more helpful, probably.

B

Yeah yeah there we go okay, so we can take that track.

B

Yeah that that confirms that we're coming in through here, okay.

A

B

Since the test has status, 429 I think we're expecting to be coming in through here instead to throttle.

A

Which would explain why status is nil in the test. Yeah.

B

C

B

Can keep finding this back Trace Active, Sports subscriber.

B

Yeah as soon as I go here, stop making things.

A

B

It's um so looking at the back chase. You have no good error and now going to uplogger and so we're in track, and then you get here and you're getting invoked by Finn.

C

B

Active support subscriber I'm I'm like how is this: how is this calling.

A

B

It's like as soon as soon as I got into this fire I'm like. How do we end up here.

C

D

A

Oh I guess from.

A

All right, so are you able to show the spec again so if you're.

B

I have so many files left to know.

A

I know I'm with I I'm I also forgot that, like my GDK was all shut down.

C

And it was not in a good state, so I'm trying to fix that locally.

B

D

To speak, spec is here.

B

B

Lover to receive error argument.

C

Oh those Speckles gigantic, okay,.

B

So I have a feeling it might be. This Rack Attack instrument thing.

B

So I think I think this is like doing some logging internally and maybe when we upgraded the library like it changed how it does the login or something but.

A

Yeah I don't know, um um let's see.

B

I'm Gonna Save this back Choice and then I'm gonna exit out of this.

B

Oh wait pass what.

C

C

A

I did see okay, so this there is a git lab rack attack and I. Don't know how that interacts. With this.

B

I'm not sure either, but.

A

I'm gonna send it to you, because that file has a five hour old change and so now right now it's very suspect. uh Let me send the file and look and see if this file involved in any of this.

D

A

Shared examples are referencing the git lab Rack Attack class, so maybe they are involved at some level.

C

A

I wonder if you like reverted that commit locally and run it? Is it.

B

Yeah I just did that um and I'm running the fifth now.

A

You did it was so quick, I look away for a second I'll, be practice because I know we all have like on-call, and sometimes it's really difficult to jump in and see how people figure out what caused a problem.

C

um I'm, looking at the changes and.

A

Nothing is jumping out at me, but.

A

I'm, sorry, oh dang, okay, well,.

B

It's just failed, so the the upgrade.

A

Oh, that's the object, okay, so the upgrade's, not it. What about the changes that were recently to the get.

B

A

B

A

I sent you that.

B

A

If you revert that, does it start working.

D

B

If this could possibly be it, so, let's see.

B

Five hours ago seems kind of thing: yeah.

A

It's really close, and it's touching um the shared examples.

B

Yeah, let me see.

D

B

A

I'll look at the Mr and maybe that will have some more written explanation about what was happening so.

B

It does seem promising, but I wonder if it just added the tips that are failing, I.

A

Mean it might have I'm, not sure.

A

It's really hard to tell uh so it looks like um it.

A

Modified the it logs Rack Attack info into structured logs, it didn't add it and you're I think you're. Looking at the same yeah you're looking at the same section.

B

Yeah well, I, don't really understand is why it's only fails if you're using the feature stack of.

A

B

I, wonder I wonder if the feature spec is like enabling a feature flag and it's causing a date leak or something strange.

A

Like that, oh it's maybe.

A

C

Okay, let me see it.

B

Does it does look like we're burning in this fixes? The failures, though,.

C

A

The feature spec, um oh.

C

Yeah I don't know why.

C

A

Unlike looking through this code, changes to see if there's any feature, respect or anything funky.

B

It looks like this fixes it, though,.

A

Well, that's great, but is it I guess it's good. So what's the like.

B

Like we have no idea why everybody fixes the problem, yeah.

C

A

I guess um how do you recommend we want to handle this? You.

B

Know no, it adds a column.

A

I don't see a database migration, though.

B

A

B

That, maybe maybe the description is lying to me. No.

A

I think it I think there was a separate Mr, maybe for the database migration. It looks like there's a separate Mr.

A

B

A

The database migration, so maybe it's okay,.

C

Add a new column which will be used later for the rate limiting.

C

That got merged.

C

C

A

That yeah, that Mr, is where the um application settings were added.

B

A

I mean I feel like if it's a broken Master. My inclination is just like revert the Mr and the person needs to look at it and figure out the problem. If we have like a reproducible step of tests, hey run this. This reproduces it like you're, like I, don't know what else to do.

A

B

Not afraid of reverts.

A

A

B

It doesn't look like anybody else is looking at this.

A

Yeah, that's um I, think that's fine I mean we can't leave Master broken.

A

A

Yeah I will say uh like open a revert, Mr and then.

A

Maybe leave a note on their original Mr, with the link to the revert we like this need, like this cause. Master broken here's the issue. Here's the reproduction steps um like we didn't see a clear fix.

B

B

Okay, I'm gonna do that.

A

Foreign I still don't quite understand why the feature spec is related to it, but I think that your idea about it, maybe being as state league, is a good one.

B

Yeah possibly I feel like it might be something strange with uh with that flip it on registered groups because, like we don't we reset this and then we don't register the stuff again afterwards. So.

B

So if there's like, if there's like the existing state, is being lost here, I think.

C

Register models; registry.

A

I, don't know much about how flipper Works behind the scenes so.

B

I have um I have made like some contributions to uh the future product stuff, but yeah I haven't looked into the flipper internals either.

A

Yeah I think just like having the bias reaction to reverse some of his causing Master broken like let the person have time to look at it rather than let it sit out there, especially with it being so close to the end of the milestone.

C

A

Oh I see this was uh a jihu contribution, so.

C

C

Oh I did review one of these Mrs, not.

A

This one but I did review one of these Mrs because I remember looking at it.

C

I, looked at it and passed it along to somebody else very possible.

B

Okay, so you want to review the reverse request: I.

C

C

As somehow I didn't have 12 emerged.

B

Reviews, oh my gosh baby here.

A

Oh yeah, you want to, um you, can add master broken.

C

And we'll trigger a new pipeline, it will be faster.

B

C

What are the other? There are other like I wish it had more. Okay,.

C

Link to the right thing: okay, remember that word: uh it will apply pipeline expedite and.

A

Master broken: oh, he already did it. Okay, you're quick.

C

C

C

A

Think I'm, okay! I! Can we don't need a second review on this I feel like.

B

Yeah I mean reverts are usually pretty thick yeah.

A

So so I'm just gonna just double check and click all the things and then I'll click.

C

um Merger, okay and hopefully it'll be faster.

B

Yeah I started a new one.

C

I just said it to version, it succeeds so.

A

But um well that was a good debugging session. Thank you. So much for that I can stop the recording. Now, since I feel like we have at least gotten the MRI to revert and then um I didn't know. Would you mind leaving a note on the person's Mr that merged it that we had to revert it so that they notified.

C

A

They probably won't see it for a while, so I've stopped the recording them.