Rust Programming Language Incremental Compilation Working Group, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: incr-comp Meeting #5 (2020-10-20)

Description

agenda/notes here: https://hackmd.io/Ban950hJTRyKzmq9_7fHsg

A

um So this is the uh what fifth meeting of our of the working group. It's like that, doesn't really matter um all right and um we have a hack md that wesley just created it's linked from the zoolop stream.

A

And maybe we can uh go over, see so participating attendance.

A

We have uh david um felix.

A

Santiago wesley.

A

Two faces that I admit I don't recognize um because I don't remember: um I've had probably one of yous I think been here before, but I'm not sure about that. So I know so um could uh could aaron hill raise their hand, hey.

B

Hi erin, I don't know if I've actually seen you seeing you before. um This is my first time here. I'm aaron 1011 on github.

A

Yeah yeah! No, I I I remember you like your contributions. I just uh didn't didn't face, and then we have a julia julian wallace burger um down yeah all right. That's me.

A

I am the first time here: okay, great um okay, so we're just adding in um any items or announcements that people want to put on they can you can type it yourself on the agenda and we'll go quickly through it after everyone adds whatever they want to add. um So maybe just take a moment now. If anyone wants to write writing down, you can throw it on the items um or, if you want to say it out loud, I can try to add it um and uh let's uh get it for that.

A

I'm going to be starting a new job myself um next monday, so this either means that I'm going to have more time to do stuff for this, or it also mean I have much much less time to do stuff. That's working, congrats, yeah, yeah, it's good! It's.

A

A

All right, um it looks like we probably have all the items that people are going to add at this point. So let's let's go through this.

A

um So the first agenda item is uh david david's reporting on the continued work on split dwarf, which we spent a little while talking about um last last time, um but we can go ahead and quickly review it now, since not everyone was here last week, um so david, you want to speak on this, both just quickly say what the purpose of split dwarf is like what it's accomplishing and then um review what you've done since last week, david, oh you're, not muted, but we don't hear you um okay, so I don't.

C

Hear you, okay, ximena yeah, um so yeah splat dwarf, lets us separate um a debug info that doesn't require relocation link time into either a separate file or to adjust the dwarf in the binary such that the link I can skip it. um I'm working a pr to add support for that to rc um it's currently working um last week.

C

The problems we were facing were that we made a dwarf object file per every o file and we had to link those together somehow because otherwise we'd end up with lots and lots of them one for every compile unit our code unit.

C

um So I found out that lvm has a lvm dwp tool which I could use for that and it resolved the concerns we had with the dwp tool that ben utils has, which was only when I was aware of the time, so I've adjusted things in the pr so that that tool is invoked and where we normally do linking and for other objects, and I've also made some changes so that lvm's dwp tool is shipped in our stature like we do with led, and so that gets a lot closer.

C

I think it's just reviews and then working out whatever final little things need change to make it usable.

A

Great great and just to be 100 clear, so we are still going to generate individual files for each.o file. But then it's just that we gather them up up at the end to produce.

C

Yeah, so those still exist, if you pass say, like save temps, for example,.

A

Great great, okay, very cool, um very, very cool, okay and then there's a pr from santiago um that just has a pr number here. um What is the um pr like? What is the title or purpose of the pr that you have santiago? Let you describe it yourself.

D

So uh this is there's nothing new here, it's like the same stuff. I have done like last week. It's just like I mean I I don't want to to like uh what it is. Not not. um It doesn't need like a lot of a lot of extra. I mean I I didn't do anything basically to sum up like this is the same stuff. I have just replaced the thing and we have.

E

I think the only changes I'm going to approve it today. um I just want to write up an analysis of the perf results, since there are some regressions, but it looks like the regressions are in uh really small tests, so we're looking at like a regression of five milliseconds or something in a synthetic benchmark. um It might.

D

Be a constant thing: okay, but it shows up as.

E

Like a five percent regression in one specific case or something so, um I'm planning to approve today, but I'm gonna write up a comment for the perf triage people, so they hopefully don't freak out when they land on it.

A

Okay, great yeah, it's probably good to just yeah, say we've already investigated it's a good idea. Okay, um all right, and then we got some more time here. So, let's, let's go ahead and look over the um the open. Let's see there's so uh the issue triage. We have a lot of open um a comp issues, 106 of them and then six open, um wg incremental comp issues. This is a lot of stuff to go through. um It's we're not obviously not going to be able to do it in this meeting.

A

A good task might be to figure out some way to um get through these, like you know, and categorize them in some other way, but also you know could be useful for right now. I want to see I want to look quickly and see if either of these have any other labels like p high on them, which I doubt they do. um But let me see. Oh some of them.

A

Some actually are tag p high, um so it's possible that a good thing for us, at least in the meeting itself, would to be like look at the key high and the critical ones um which, unfortunately, I don't think could does github have a way to access or to search. um I I can't recall um so the I.

D

A

D

We have p criticals, but.

A

Okay, I thought yeah, I don't see it. Let.

D

Me make simpler for you, but.

A

Yeah it will. That would make things easier for me. Well, there's no p high wg, ignorant comp issues and there's no and then for p critical right. You said I, I don't think there are either of the p hot, any pre critical, um but there are some p high, a common comp issues, there's three of them.

A

So, let's I think a decent thing to do in the meeting itself would be to like look look through the three that are that are there and they're they're two of them are assigned, two of them are assigned to me um and then one is totally unassigned, so it probably is a decent idea to and they're all very old. That's the other, very important thing to note here, like that.

A

There's some worthwhile, like you, know, potential for us to just close them or determine if they are uh no longer an issue at all, um so let's just go through them um and see what their status is um or assign somebody to look at them. um So the first one I'm gonna mention is uh six two, six, four nine, um which is the ice in an ice in chemical compilation.

A

After you substitute trade for struck like this, and somebody like that struck a trait item and replaced it with a struct item um and this somehow caused breakage- and um I I do remember this because we spent a while investigating it, um and the only question is like whether there's actually more work to be done, because my memory from looking at it was that somebody landed a work around um to prevent the ice and then, uh oh, is that not true hold on?

A

I thought that was the case that there was a workaround that prevent the ice. I.

B

Think there is, I had opened a pr that I think I I'd revived somebody else's pr, but I wasn't able to fix the perf issue so eventually got closed. It's at the bottom of the thread. I can oh.

A

B

So if I remember correctly um so, there's like a hack in the in the the depth graph system that um let me see if I can find where it is, but I think the issue is still there. It's just it just there's, hopefully a better way of doing it than the current hack.

B

So let me finally get a blink and I'll put in zulu.

A

um Okay, okay, um oh I see, are you aaron? Are you talking about the that was aaron right or is it wesley sorry who's speaking just now? Oh, that was me. Aaron okay, right see the pr you're talking about. Is it seven five, six, four one, um because that actually linked the issue? That's.

B

uh That's right, I yeah, but it had some major, perfect questions and I think yeah.

A

Okay, yeah, okay um and the point is in part this does still does it still exist at this point, then, um I'm trying to understand because, like because I see that in the thread that there was some discussion, that that there was a workman that landed, it was just the issue was just landing, the more fundamental issues that caused the problem, but then I saw some other mentioning of people mentioning that, like certain kinds of compiler flies could cause the thing to happen.

A

So do you know offhand whether that you can observe the bug in any case, even with the workaround, or is it something where the workaround, as far as we know, doesn't means the bug is? Is not reproducible not observable.

B

uh I'm not certain about that. So I know, though, what's happening in the issue is that the dependency graph is supposed to everything that accesses the hir is supposed to go through the dependency graph, so that um when it gets changed, we invalid we mark it as red. We don't try to read something that doesn't exist, but um I think what's currently happening is that we have this other table that we populate that doesn't go through uh incremental compilation.

B

It doesn't go through the depth graph, and so we may recompute that we may try to read from it and we'll read from something that that no longer exists without going through it and I think, there's a hack that marks it as read when we, when we get to that error, I'm not familiar enough with the mental compilation to know if that could cause any other issues, but I think it's, I think, whoever wrote it did not intend it for it to be there long term.

A

Yeah I see right and that hack does I mean that sounds like that would be conservative, like you really shouldn't see the ice anymore, I think oh well, it depends on what kind of ice it is, I guess in terms of weather, but anyway, all right all right. We spent enough time on this. I guess it sounds like there's potentially still stuff to investigate here um at the very least trying to get rid of the work around all right.

A

um I also assume aaron that you have other things that you've been working on and don't really want to try to resume working on this given or in the short term, because there's other stuff you're doing that is more important or would you are you interested in trying to figure out some way to resolve.

B

This uh I could try to um open my pr again. The main thing was that it was, I think, content-wise it was working. It was the these I revived somebody else's pr that I think had fixed it, but um right it was the. Let me see if I think the perform is still here and I think it was pretty bad and I'm not super experienced with performance related.

A

Okay, let me see so there I see the perf run results. They are still there.

B

It was not great, it was regressions across like a one like a one percent.

A

Refreshing a ton.

B

Of benchmarks, that's not it's not really good.

A

B

um Looking at the average, it's more like a two to three percent regression on incremental benchmarks and you know effect on full right.

A

But also not great, um okay,.

E

Interesting, a lot of the worst regressions are in small test cases. um If you look at the larger crates, it averages more toward one percent, so.

E

E

Just to say it's not as bad as it looks initially.

A

All right, um let's uh all right, see yeah. It doesn't sound like something that has to be resolved in the short term, um but it would be good to try to look into it, um and the only question is whether to try to revive the previous approach or figure out something else.

B

To do here, um I did.

A

B

That the hack and I linked it in the tulip where we're currently so yeah what's happening is I I think um so we're.

B

I think the way incremental works is that um if the so we we compute to be hash, the def id and get the def hash, and if that we're supposed to in the whole um incremental system check if that still exists, and if it doesn't, we bail out, we mark it as red, and that means we have to rerun the query, but um I think what's happening here is we we think it exists, but it's it's not properly being tracked, so we're trying to read uh the wrong def, node gotcha, so yeah, I'm not oh yeah.

B

I I can't I couldn't say if this could cause other issues. I'm not. I have a little experience with the incremental system, but not too much.

A

Okay, let's leave it open, obviously, and I guess p high. um The only the other only question is whether to lower the priority on this, but I think I'd want to keep the priority at p high for now um and um I'm gonna. What I am going to do is assign a taran, because I'm certain that they are not looking at this, um and the only question is whether I should also sign myself.

A

Realistically, I probably should have signed myself because I'm not really looking at this- I'm not telling you look at it in the near term. um Okay! Next up. um Let's look at another another issue, um so the next one on the list.

A

Is um five? Eight six three three uh x dot py check, failed, incremental, builds enabled this is again an old bug. So you know it's it's questionable whether it's uh even still a problem.

A

A

Oh right, okay, so this bug is something where there's like certain unused attributes, um or rather there's there's links for unused attributes and um something about the incremental system is causing like the lint system is relying on certain things being you know, processed or tables being off populate or whatever, and then the incremental system doesn't actually necessarily populate those tables.

A

If I remember correctly- and so you end up getting these things flagged as unused because it doesn't encounter the code, that's uh actually using those attributes and we basically worked around it in um the compiler bootstrapping by just saying, allow and use attributes stop you know, warning or denying them um and honestly it's hard to say whether this is like to me it's something where I would think that the same problem, I'm trying to remember if it's an artifact of our bootstrapping process in terms of being specific to like certain stages, and it wouldn't actually happen in real world crates, um but I'm not 100 sure it definitely sounds like something where you would think that if this problem occurred outside of our strange particular, you know bootstrapping system, then we'd see people complaining about it um elsewhere.

A

So it's a hard call to make in terms of whether to just close this or to leave it open, because because you know really it's a question of do we treat this like something, that's a more fundamental bug that represents it just as a design flaw somewhere or do we say you know what we're going to try to fix this.

A

um I I don't know.

B

uh One thing I'm noticing reading through is that all of the affected issues uh appear to be internal like that's. The only attribute so allow internal unstable uh any of the rusty prefix ones. So I don't know if that's just that's what people happen to come across, or maybe that points to like. You said this being something specific to bootstrapping specific to working on the compiler that can't actually affect users, because the some of the rusty lints are implemented.

A

Yeah right, that's a good good point in terms of it being a lot of rusty. Attributes are often implemented in a way, that's sort of not the standard lint system, which would definitely explain why normal lints don't encounter this, because, probably we you know got we do things the right way with the normal lint system and then um the rest of the attributes sometimes are just sort of hacked in um and don't go through the standard path.

B

Yeah because uh lintz, if I remember correctly, they're like normally done as like hir visitors- and I don't think hr is done- we always we always run lintz on the full thing. I think I don't think that's these ast isn't incremental, so we have to lower that each time you know we could do. We.

A

Could actually we, instead of um try to fix this bug as it is? We could get we could stop using on unused. We could make a new unused attributes thing like unused, rusty attributes or rusty and use. You know, rusty unused rusty attributes, something like that. That would just only white list the uh rusty attributes as if they're unused right and stop and stop basically something that interacts with the unused attributes links, but in a way that's specific to rust, c, and that way we get.

A

We would be precise about saying this is just for these attributes that are already you know, unstable et cetera, and it would if this hypothesis, that it's just this only is affecting rusty attributes is correct. Then we would get the benefit that the compiler developers would still get um the behavior of like linting on other kinds of attributes when they're unused.

A

So it's worth considering it's I s, though I mainly mention because it sounds to me like a very easy hack versus trying to figure out what's wrong with the implementation of each one of these rusty attributes that they're, not you, know, being properly tagged by incremental compilation.

A

um What do people does anyone have any opinions about whether it's a terrible pack or a decent one.

E

It doesn't seem like a terrible hack to me. I I'm reading michael left, a comment uh on december 19 2019 that uh having heard what aaron said kind of, I read it a little bit differently and it sounds to me like he's saying that unused attributes just doesn't work. The unused time just doesn't work with the incremental system, that's more troubling, um because he says we have a solution for side effects.

E

In the incremental system for diagnostics, it could probably be expanded to unused attributes if there's no cleaner approach, which sounds like unused attributes just doesn't play nice with incremental at all, but maybe I'm reading too much into that.

A

But then I think that's funny: do you think people just blow away their internal state when they encounter.

B

This and don't don't report as a bug so anecdotally. I was hitting this a lot in the past and haven't hit it in quite a while.

B

So I don't know, if that's just me, if that's just I'm not running to it, for the region or if that indicates something has changed, uh and if I remember correctly, the way marking attributes is used work or these some of them is that there's just there's this global field and like one of the ast globals or something and that doesn't interact with incremental compilation at all, and I think the reason it just doesn't come up is that we we always re, we don't try to do anything incremental with lints.

B

We just run across the whole hir. I could be wrong about that, but I think that's how it currently works.

C

We've had a pr land in april that allows unused tributes and stage zero incremental builds for the standard library, so we wouldn't had it anymore, even if it was still a problem.

B

Oh, that explains it yeah.

A

I see so when aaron said that they were hitting it themselves. You still meant only on rust seed bootstrapping, not in other other projects, right aaron,.

B

That's right: I've never seen this in any other project. So, okay, it probably very much probably is just the ignoring them for bootstrap, then or allowing them. I mean.

A

Well, a way we could definitely so certainly if we did the approach of what I said that the hack I was grabbing of having rusty unused rusty attributes.

A

uh You know as a separate thing, then that would allow us to confirm um or deny the hypothesis that this is a more general problem than just the rest of the attributes right. We would very quickly very quickly presume if we then turn that on still hit other unused attributes, um presumably elsewhere.

A

Okay, let me um yeah okay, I'm willing to leave this open and assigned to me if it's already, assuming it's still assigned to me, and hopefully maybe look at one of those approaches. um It definitely should stay open. It sounds like because the potential for it to be a real bug- that's observable elsewhere, still worries me. Okay, let's move on to the next um issue. The next issue is 58368.

A

Which is uh it's five? Eight three, six eight. This is a big regression in tokyo. Web push simple, simple, opt um so a certain benchmark regressed by 88.

A

And they identified the uh the pr or the commit that caused it to be a certain querification, um that's interesting, so that caused the incremental violation to have more code to lvm.

A

uh But then zany walk says they can't find any obvious.

A

Culprits, okay um and no one's had a time to investigate this more deeply.

A

B

Well and the perf run looks to be gone from the perf server, which seems super unfortunate.

A

Yeah yeah, you know it's it's, it is too bad. I wonder about that. I think you know I it'd probably be good. I've encountered a couple cases like this. Where the performance get you know automatically deleted, we probably should consider having some way to art like archive individual performance either. You know by like either letting people say. I want this archive like tag just keep this because I'm linking to it somewhere, that's important or even just automatically, do it for linked ones that are actually linked from github.

A

um Let me bring I'll bring that up with some similacrum um because I think they're, the ones who drive the perf service right, um because it seems like I've encountered this multiple times, but nonetheless, because somebody did try to commit, then it's the potential to be able to still track down what caused this.

A

um I'm gonna leave this open um and signs myself, because if it is a tokyo um in terms of like, if it's something that affects tokyo, which it's not clear to me, whether it really is or not um but there's reasons that I think I'm gonna care about it. um So I'm gonna go ahead and sign to myself and hopefully look at it, maybe this week or next week, okay, um and that's all the issues that are marked p high. So the last thing I want to sort of then address is this question.

A

We have a large collection of issues that are not pi. um I in fact that are not tagged with any label at all right um in terms of prioritization, and it might be a good idea for us to at some point. I don't think this meeting.

A

I don't think today's meeting is the right place for it, um but we might want to plan to try to process this backlog um in some incremental fashion, so to speak, and so um I you know, I propose that maybe we just like um each each before each meeting every two weeks this at the current cadence we allocate like you, know, 10 of the issues or something like that and just put them like and cue them for being prioritized um and actually what we could do is we could just take 10 issues right now and enqueue them and then, if we don't prioritize them in some way before the next meeting, then we do it at the meeting itself.

A

Does that seem plausible to you all? I mean I mean that's the first question: does it sound reasonable to say we should be trying to process this this backlog of issues in some fashion? That's, I think, we're the sort of best people to do it, given that they're tagged, there's no compilation. So the only question is whether people think that's actually a useful thing to do.

B

That sounds good to me. I could definitely help out with some of them.

A

Okay, um so then yeah, I think, rather than rather than saying, we will process a certain number of issues at the next meeting. I think it's more effective to say to a cue the work right now and that way, if it happens, asynchronously that's awesome and if it doesn't happen, asynchronously then we'll force ourselves to do it. The next meeting um or we'll try to force ourselves we'll see how much is on our plate at that time. Okay.

A

So what I'm going to do right now is on the um hackmd is um just write down the encude set of issues. If I can get my browser to actually respond to me. um Well, it's really nothing's working! um Oh, it's just super laggy, okay,.

A

And by to be triaged here, I just I basically just mean mark with a p label, which I know not all of you have done um because that's usually that's the work of uh the prioritization group for the most part, but I think it's a useful thing to.

A

I don't know, learn about so um let's um so we can talk about that asynchronously in terms of the process there like what you know. What um are the criteria that one uses? um I look lord. I am just so yeah. The first ten issues is what my plan is to put on here. um So I'm putting seven seven one, one, eight, eight, seven, seven one, eight one, eight, seven, six, seven, two: zero, um seven, six, two five one wait are these prs or no? They are all issues right, yeah.

A

They are issues; okay, sorry, um seven, six, oh three: seven, seven, five, nine, nine, seven, seven, five, nine hundred and seven, four, nine, four, six one: two: three: four: five: six: seven: um seven: four: eight: nine zero, seven, four: three: seven: nine and seven: four: two: five, eight! Okay! So that's that's ten issues um and, like I said we can, I think it'd be great.

A

If we can try to just prioritize prioritizing asynchronously um santiago, do you think that it'd be good to have the conversation, so what the prioritization group does is they allocate individual issues?

A

Sorry, the allocated zulu topics for each individual issue.

D

Right, if we, if we add the label the hypothesization label, it will automatically create a topic in the very stream.

A

Right right, so that might be the simplest thing to do here, because the only question to me is like you know, would we want to keep the conversations about the prioritization here in the um incoming compilation, working group stream, or do we want it to go in the prioritization working group stream and personally, I think it'd be fine to have a go in the prioritization working group stream. Even and just have the people who are in this working group also, you know, go join that particular topic area.

A

um I don't know if they need to join the whole stream in order to participate in the top individual topics or if they can um somehow join a topic on its own. Without being you know, subjected to the deluge of of notifications from the prioritization working group, um but it's just one one detail here, but there's anyone else so yeah.

A

My proposal is, we, we've leveraged the existing infrastructure and tagged these things with I prioritize, and then we just go and try to process them in that other stream, um which I know might be counterintuitive in terms of saying you know we're part of the question working group, but I think it's good to give some visibility to this work um in that in that area. Okay,.

D

Yeah, I can. I can also comment in the working group saying that uh we're going to do this and there are, there, are going to be 10 issues that will show up there and it's. The idea is that uh the this, this working group is going to help predispose.

D

So exactly or we want to be aware of this.

E

Okay, are you thinking you would tag them as prioritized now or tag them, as we actually decide to prioritize them.

A

I think that they're gonna get prioritized one way or another. So therefore, there's no reason not to just tag them right now, um so yeah, I think adding that. So do you simply just add the label santiago, or is there a comment that you put on? You just add the label here. So just do we just add the label. I prioritize to each one of these issues. Then it'll um the bot will do the rest right.

A

C

E

My only my only thought there is um the prioritization working group is pretty good about jumping on stuff, as it comes up.

A

E

They might be prioritizing.

A

I see I see okay, all right, so, let's not, let's not all right. So, on the one hand.

E

A

E

Be totally fine, but if the goal is for us to do some of the work and not just spam them with work, yes,.

A

That's exactly right, that's right! So let's wait! That's an excellent point. Then, let's not tag the issue until once, once you're once one of us is ready to actually dq one of these issues then tag it with I prioritize and go to the stream.

D

Like and also if we want to go another route like we could totally have the have the list in the hackmd and not tag without prioritizing just go ahead and add the prioritization level. We we think it it sure.

A

I just want to make sure the comment I want to make sure there's a cover, a place for the conversation about the prioritization happens right and so that either would happen, and so either we're going to try to hide this from the prioritization working group right entirely or we're going to you know, make it something that we alert them to, or at least you know make observable in their space and to me I I'm fine either way.

A

I don't see much reason to hide it besides, making them feel like they're being presented with more work, um which is an excellent point that wesley brought up, but I think we can handle that if you know when someone has a chance to chew it off and then writes in the open street open topic area, that this is something that the working the incremental motion is working group is looking at right now in terms of prioritization, then I think that'll be okay, don't I don't think we need to entirely avoid using the prioritizations working groups infrastructure here, although I would love to know if they you know at the next, if you could bring it up at the next meetings on the get feedback about or maybe I'll show up to the next meeting I don't know uh unlikely, but you know the next time that you have a governance meeting or whatever for the prioritization working group.

A

um uh You know getting feedback about whether they think this is a terrible idea. That would be good.

E

Yeah this scene, that seems fine to me. I just want to make sure that we don't create a bunch of work for them and then they could do all of our work for us. So yeah exactly.

A

Okay, great great, um that's everything that I sort of think we you know it's good for us to cover. Is there any other uh things that anyone wants to bring up before we end the meeting.

A

Okay thanks everyone for attending this has been great and uh yeah I'll, hopefully see you all in two weeks, all right, bye, bye, bye, bye,.

B