Rust Programming Language RFC 2229 Working Group, 30 Oct 2018

Previous Meeting

⏯

youtube image

►

From YouTube: closures and upvar capture (blitzerr, 2018-10-30)

Description

Discussing how closure desugaring and upvar inference works in rustc today with blitzerr on 2018-10-30

A

A

So I think what our plan was is we were going to talk about like.

A

A

How the closure stuff works today with the goal of hopefully getting to see. What's on my screen.

A

The goal of eventually in figuring out how we're gonna do this closure, the sugaring, RFC right, okay, so figure out, which first problem with my workspaces. You should choose.

A

Let me make a little branch with you guys.

A

All right, so, how much do you know about the the way US models, closures? Actually I.

B

Just read the rusty guide about.

B

A

So like the first thing is just to understand that when you, let's show you a few examples here,.

A

A

Okay, so the first thing is.

A

When you make a closure, invest this this, this basically corresponds to some structure that implements the FN phase right. That part.

A

So the interesting thing is: if this closure should use a local variable, then we compile this.

A

We basically analyze and figure out. Well, we have two different modes, but in the default mode we analyzed how this local variable is to use and figure out what the discovered version should look like right. So essentially we're going to make some kind of structure. It's like the closure struct and now we have some choices, but for this field, X that we captured, we could either store a reference. It's assumed X's of you 32.

A

We could store it a reference to X or a mutable reference to X, or we could just capture it by a value and the way we decide between those three is by looking at how X actually gets used so so like here.

A

It's even though X is mutable we're not actually doing anything to mutate X here we're just reading from it, then we're gonna capture it as a shared borrow, but if we were to.

A

Mutate X now we would need a mutable borrow and we're actually going to get an error here now. The problem is this: the problem is that we specified we need an FN closure, but if you kind of discover this you're going to have something that looks sort of like this.

A

Something like that, that's what the do sugar and the problem here is that if you have shared reference leading to unique reference, immutable reckons that that mutable reference isn't really unique anymore right, because you could copy the shared reference and then you have two paths that lead to it. So that's the type checker that gets unhappy, so that's kind of the root cause for this error right, but the way you see it as a user is that we're telling you hey, you can't use a shared FN closure.

A

You need an MMU closure, because in order to mutate the state from your caller, you need to have unique access.

B

That part, that's not required right like that, will be the dish you got part, but if somebody happens to write just the main then and they remove the line, it's the sequel to the last line. Basically, we are incrementing X by one and they should be able to compile it. Just fine. No, but.

A

B

B

A

Mean like, if I just assign the closure to a very poor yeah this. This should work just fine and in fact it's also okay. Well, if nll at least.

A

So this this will work. Fine, you get if you're, if you're in MLL mode, you wouldn't have to it'll, compile just fine, because.

A

Because the closure is dead here and it's Baro has ended, yeah yeah. The problem is more that we're falling through okay. So the last thing to show is that there's another option, which is this might be.

A

This you can either so far, we've seen cases where we have our sheriff Baro you've seen case with a mutable Baro, but it could also be a move all right and it's there's a common misunderstanding about move. Closers I haven't even talked about, yet uh you don't need a move closer to have a move. You can use that this code is going to insist. You drop it's going to remove X from its parent.

A

So when the closure is constructed and needs ownership of the vector, so it will go ahead and take ownership, which means, if I, try to call X here. I'm gonna get an error because, like it says, the value was moved into the closure, and so that means I can't use it afterwards. So this is kind of the analysis. I was I, wanted to kind of walk through how this works. I think at a high level.

A

A

One way is, you can dump the mirror. This actually generates quite a lot of stuff why'd, you dump well, let's make one that actually compiles.

A

If I do dump your equals all, for example,.

A

Now I'm gonna get you're, gonna get a directory called mere dump and it has a whole bunch of files. Each one corresponds to some version of the mirror for some function, as it goes through the transformations.

A

So in this case we want, let's say the version of let's look at the main function and we might start with like.

A

No I usually look at the NLL ones, actually because they're kind of in a good level of oops so.

A

So if we look at like rusty, got named, I want a log and all that in here this one here this.

A

A

What does this contain? Let's just clear out this directory.

A

Okay, I, don't think I ran with in all, so you don't get that, but this this first file I think is the initial mirror that we build, and then we do transformations on. So this is for the main function. It has a lot of junk the thing that somewhere here this most of that stuff. Okay, so all this code- I just skipped over this- is just building the Hector.

A

It's right around here, it's a little hard to parse, but what's happening is this is the equivalent of saying this like we're building a struct, but the name of this truck is this closure?

A

That's it's, and you can see here that we actually moved the field and if we change this to doesn't change our input to, let's say just era now we're mutating it, but we're not moving it and we regenerate these files.

A

Now we can see that we brought in underscore 6, which is Namsan, so you can really see what's happening.

A

Ok, so let's see go back to my notes. Ok, so, first of all you can view the disha good form with side.

A

A

B

A

They're, not that different I just didn't happen to be using it all. In space there was no nitrogen. If we did run with an oil, then it would look like basically the same. The main thing is: if you look here, there's all these different versions as we transform their color they're, actually, mostly not that different from one another. So you could probably pick any one it'll be kind of okay, but in some cases we do like optimizations that might make it like yeah, okay.

A

So, for example, here in this version we no longer have that underscore: 5 equals loser, X, we've sort of rewritten it to just say: closure dot x, equals because it's mortgage. um So that's why I say like the NLL or this this very first one he's early enough that we haven't done much so looks similar.

A

Yeah, so so what happens in the compiler now so oh one last thing to mention it is: is that all of this happens if you write a default closure, if you write the move keyword, we don't do any of this analysis. We just always move. That's all, um but but but like I said it's not the only way to move it's just a way to for certain um and the way this works is there's like a couple of phases. The first thing is somewhere I.

A

Don't exactly remember where we have this notion of what the free variables of the closure are. There's a square here.

A

Yeah, so what this does is you've sort of familiar with the Russ query system, the rest of the query system right so.

B

This file is all the query. Is it yes,.

A

This file is a why you can't see the very topics a view, but basically it's like rusty type library, there's a section in the rusty bucket. It's probably the best thing, but yeah lists all the queries and then somewhere there's a function that defines this query. Some reason I'm not finding, probably because some of the queries are set up in a kind of like.

B

A

Somewhere around here, somehow there's a notion of what the free variables are, and this is just the local variables that the query happens to talk about. So in this sorry, not query, but the closure happens. So in this case the free bars would be X and we say three because they're not balanced, like there's no definition of them inside the closure. So there's C.

B

A

In general, a free variable is a variable that is referenced, but not to find wise by something.

A

So you can say, the free variables of the closure would be all the local variables that and it talks about, but it doesn't define any more.

A

So in the we go into the Lib Russy.

A

If we look at the lip, Russy type check sure.

A

A

Yeah I just like where to start from I'll start from here. There's this file called up for RS and you can sort of see some of the documentation. It has the job of figuring out for every free variable. How does it get used and that that determines the mode for that free variable? What we call the borrow kind, should it be a shared borrow of mutable borrow or no borrow at all, actually there's another.

B

A

Talk about which is a unique and beautiful borrow, not particularly important, but it's kind of like basically there's just these levels. That's the main thing based on how you use it and.

A

And that in turn defines sort of the minimum which traits the closure can for implements, because if, if all you do is, if you have no free variable, so you just read, then you can implement the FN trait the SMU trade and the FN one straight, because you can be called many times in parallel or sequentially, which is like FM or exactly once. It's fine. But if you do more advanced operations like, if you move something, then you only implement FN once because you can't be called with other ways.

A

You have to be called exactly once you require ownership and similarly, so we kind of infer all of this and the way it works is uses some kind of old and grotty color. But anyway, that I was someday like to remove called the expression use visitor and what this does is it walks the walks, the here like the I/o, so at this point, we're representing rustiness you're familiar with it here, it's like the syntax tree yeah.

A

So, what's the source of the closure right in here and figures out, it basically calls you back for everything that gets moved, mutated or borrowed those kind of things. So it was used to implement the old pirate record. It still is, but so basically it it's kind of exactly what we want here. So in this case it might say you go back to our example.

A

If we use X, let's say we use X twice, but y equals x is zero. This is gonna, be a bar of shared borrow of X, because we're reading- and here we have X of 0 plus equals 1, we would get to Paul batts right one year saying there's a shared borrow and one here saying: there's a mutable borrow and I'll also tell us what was borrowed, and so those callbacks get invoked on this. This delegate.

A

By the way, just to the part of the reason I'm videotaping, this do I know it's a lot of information, cuz I plan to post it on youtube. So you know it's some of these notes and things we could probably find haters, so the infer Barrow kind. So the delegate implements this struct, this struct as a bunch of random stuff like context information, but the main thing it has.

A

Is this field? That's tracking? So, oh here it says up far, that's the same as free variable. That's a tournament! I! Don't know why! But rusty uses it. Rivers I've never seen it outside of rusty for meaning a variable from the surrounding context.

C

A

Somewhere up, yes, so the up foreclosures. This basically is saying for each of our. What moment, what mode did we require and as we see those callbacks will adjust it right? So we'll start out saying we everything is just read because we haven't seen any uses yet and then, if we see a move, a mutate we'll bump it up to mutate. If we see a move to bump it up to move, they basically need the strictest one that we actually see and I think so they up for a capture map.

A

This is just a hash map going from the ID of the variable. Just some I get some in it index to is up for capture which tells you it's either moved or it's borrowed and what kind of borrowed.

A

So the actual callbacks there here we get like consume you get this consume callback. If, if we see a move of something, we get the borrow callback, if we see a borrow shared borrow or a mutable borrow or something and the rest of them mutate is when we see a assignment to something the rest, I guess don't really matter so and in the callback we get. This thing called a CMT, which is an old data structure, stands for like categorization suitability and tight. Yes, but basically it represents yeah.

A

It basically represents what was your borrow removed, so in mirror. We call this a place so like. If you had back to our example here, I have X of 0 plus equals 1, it's actually a bit of a let's just, simplify this. For instance, I just had this.

A

Now the the CMT would just be for a local variable X, you see it later so when we call one of these methods, basically based on the kind of thing that happened, we call a method saying adjust the bar Oh kind for this sort of access, so this would be like for a mutable borrow. This would be for a unique borrow. I didn't talk about, but that's the point in between. So if a shared borrow, let's we make aliases and requires immutability and a mutable borrow is unique and allows you to make a change mutation.

A

A unique borrow is unique, but still a new book if the Pilar uses them internally, because there are certain rules of rust who don't make sense to be open, but they're not really something you talk about. That's normally.

A

That's not so important, but we come into this adjust thing, and here we look. We look at what was the.

A

What was the thing that got borrowed? Basically, that's what this EMT got at that stands for category. It's like look at the category of this thing. Is it a DRS? So that would be like you know, sans star, X.

A

Or an interior means like access to a field. The details aren't well, you can read them on your own time, I guess so, but basically this basically propagates down. So the idea would be. The simplest case is when.

A

The simplest case would be when.

A

Yeah I, don't know the simplest case would be like where it's just a local variable. Basically, and then the more complex cases are like. If you had. Let's say we had a tuple here.

A

Let me add something like this, then we would get a CMT telling us that this is X dot. One.

B

A

And actually, there's a service. That's an important point in some sense that I'm skipping over I should say because I think that the CMT we get back actually looks a little bit different for those two cases, so you will get callbacks in that cult like, for example, we would get a callback here too, and this is a true local variable of the closure. But in this case, in the case where you access an up far, it's gonna look.

A

C, like this, that's the actual path that you'll get a callback for here. So we sort of see the D sugared path. Only there's a look I forget exactly how this works, because we don't actually know yet I think yeah. We see this path, always we always see a star and then so. There's some code. You see here that if it sees a dear F of a borrowed pointer, that would be this case star.

A

It's gonna go and look and see.

A

So we have this extra notes basically says this note thing is like saying: where did this dear F come from? Was this implicitly inserted because it's an access to a closed referral and if so, then we'll? This is where the actual thing this is where the actual code is that changes.

A

Okay, then, in that case we need to upgrade to a write for X dot, one you know, because now we saw a mutation or when he's upgraded to a mutable borrow of X, whether it's this code here so so you will see this distinction that way um so yeah. So that's that's! Basically it in some sense, when it's all done once it's boxing the whole enclosure. You have you've observed all the things that the cloture does and you figured out.

A

You have adjusted this map to be immutables or borrows or shares as it moves borrows yours whatever, as appropriate.

A

A

Then we just kind of store it and store it in our. So the output of type check is this whole bunch of tables hashmaps telling different things about the RUS program. We just hijacked it. In this case we store and say here's associated with disclosure, yours the set of borrows that we did or the here's the news like the modes for each capture and then we'll also figure out it's the closure implements FM what and store that information.

A

But so that's that's. This part of the code. I wanted to walk through a little bit. The question would be now.

B

A

Good question yes, and no so one option is: there are two ways to go about this: you can build them. You can disable optimizations and build a build with debug info, and then you can use gdb or or better yet RR. If you know, there's sort of similar I, don't know which platform you're on actually are you on Mac or Windows yeah.

B

A

So, in that case, I think LOV B is the debugger of choice. um This I can point you it's in material tonight and as far as I know, it should work. Okay, but I. Don't have a lot of experience with that, because we usually because disabling optimizations takes a long time for so the problem is that the compiler you build the compiler and then the compiler has to build the standard library.

A

So if you turn off optimizations in the compiler, then it runs really slowly and takes a long time for it to build the standard library we'll see. So so, usually we don't turn off optimizations because it's it's kind of annoying, but you know if you're willing to wait. It might be worth it, especially if, if you're not if you're, just putting to like walk through and not make changes, then maybe it's worth it. The other thing that I usually do you see already all these debug statements.

A

My usual strategy is to turn on debugging just dump everything save it to a file and kind of walk through, but but that's for what you're trying to do. It's obviously less less good, because we have to sort of think of the questions before you happen, so that you can print the answers.

A

So you could try a debug build. I would be curious to hear how those be I.

B

A

There's material about it in the rusty guide, but if not I'll just tell you that you would open your config tunnel.

C

A

Change debug equals false to debug, equals true and then I think you probably want to change optimized false.

A

At least it's been my experience that I love um it's not very good or about preserving debug information after doing optimizations, and so the end result is that, like most local variables, it tells you they're, optimized, gallop and stuff like that. So you can't actually get a lot of information.

A

Main thing, I would say, is: if you do this, just just build when I do a build like.

A

With a command like this, where you just build stage 1 so you're not going all the different phases and you just stop after building the standard library and stage one cuz, that's a useable compiler that also reviews that'll. Do the minimum amount of building with your non optimized compiler that you can get in order to have something. That's really usable. You could, in theory, build a little basket with you or you could just build up or something no mic you're worth it.

A

Thanks just build the poor and write some those standard tests like like foreclosures, you don't need the rest. Babies, but I would probably just do this.

B

A

No, but you do have to do a clean bill.

A

It shouldn't leave all of um but I think otherwise it doesn't.

C

A

I think I think this information regarding the RFC in question I, think all this information is sort of background that you should know it's not well, basically, because we're going to have to generalize this system so that we're not storing just we're, not tracking everything at the level of individual variables, but rather paths like that's.

A

The goal of the RFC right is that if you do like X dot one and that's the only thing you use, then you should just capture X dot, one but I, don't actually know I, guess I have to reread the RFC and think about it, a little bit like I'm, not sure, for example, it might be that right. It might be that this this there isn't actually a super deep changes.

A

I think that's the hope, anyway, that we don't need deep changes to the code here, except that we sort of figure out a set of unique paths to get accessed to an extent, and then we track those.

A

Otherwise the system remains the same, but worry about it later.

A

Yeah, you don't have to worry about that latter case I, don't think because they all sort of come up in the borrow checker that if you've borrowed a field, so I think from from your point of view. At this stage in the code, we can think of all the local variables as disjoint and then the borrow Tucker's job is to ensure that this is true like as much as it needs to be true.

A

So it's probably okay if they overlap as long as it's only shared borrowers fix anything but I think the challenge here is gonna. Be that there's a bit of a gap between like.

A

The mere operates on D sugared stuff and here in the type-check phase, we're still sort of in a semi sugared world, and some of these pads can get kind of complex like the theoretic, although to do other things and I'm not sure we have to sort of we'll have to work through that I imagine. We can start with very simple cases where we don't have where those things don't apply so probably where we will start I.

A

Would imagine that a good first PR might be to try to write an analysis that just basically to generalize the free variables, so we might yeah that that makes sense. The first refactoring company is to changed so that we're not representing free variables always is just local variables, but rather as something that can be more. That can accommodate more complex paths, not sure exactly what that is. Yeah but like the very first refactoring is probably just introduced a new struct, so that we can make changes to destruct later he's doing so.

A

Like these maps, like the there was that math, you were looking at.

A

You know well this with free bars, for example this this tells you all the free variables and in this in the closure, and then we we have this up for ID. Everything is key violate so like.

A

A

This map stores the mode shared mutable remove and it's keyed by upper ID. That's the battle. Probly have to change to like a far path or something.

A

If you see what I mean so that we can even talk about the idea of capturing something, it's not a local variable and then right now we just can't even represent it and what so I that's sort of. We can start I. Think we I don't have I've done my head, though I think this series of PRS will be something like. First, we have to introduce a new type here and we'll just push that through, but that new type will just be a struct, the wraps up our ID. So like none.

B

Of the code really.

A

Changes but there's still a lot of like grunt work to push it through and now once that's in place, then we can like start adding a new some new cases, more complex cases, and we won't use them at first, but we can you can sort of beside. What's the right order to do it that, like maybe, will generate well. Okay.

A

What I imagine is that we'll have well add some more complex cases, it'll be feature gated and we'll start with the easy stuff like if you have a tuple and you use dot, zero and dot one. That's like no couple realtively a few complications get that to work and then we'll gradually expand. So the next piece of code is will probably be something. How could we do that?

A

Grunt work that looks at looks at the paths that the closure uses and decides what what the up fart has that it captures ought to be, and when we first write that code it'll always yield just the set of our ID so always simplifies and then we'll go anyway. I think well we'll worry about that math later, but that's the basic plan. I have.

B

The place where the concept of like that is basically come model.

B

A

Kind of path, I mean, is more like a dot B dot C like a series of fuel excesses that there isn't a very clean representation of a path, but that's one of the challenges we have to figure out how to overcome, but I think the closest thing is this CMT's, which they have a path? Basically, so if you look at them, they have a category. The category is like: do you rep, so that would be like star AK, star P, where P is also a path, and then they recursively have a structure.

A

It's a a dot, B dot C would be a series of interiors until you get to the local variable or the up farmer to the a or b so.

A

That's kind of what it looks like, and would this data structure isn't really suitable for what I'm talking about I? Don't think I, don't think we have a good data structure for what I'm talking about, so we we probably have to make like, in other words, this up far path. Data structure that I was referring to doesn't really exist.

A

We have. We have a way to talk about. You have here expressions.

A

We have a couple of things: I, don't the gain. Time are quite right so off to think about that.

B

A

Cmt might be okay, maybe we'll use that, but it but I sort of like that too. I always sort of like that to go away. It's very old code and it's very grody you could you can sort of tell that it's old because, like it doesn't follow our naming conventions, for example, and right. It starts with a lowercase letter, whereas modern rust struck start with an uppercase letter. This predates that convention, among other things, but so I sort of rather a clean new thing. I will get rid of this whole thing.

B

A

You think, do you feel like you, can try to write down some of what I said.

B

A

I'm gonna, as soon as we get off or start encoding it. Thank you, sir, takes a while and then I'll post it up and I'll, send a link it'll be on my like I'll. Send it to you, but I I have also a playlist of like videos like this. So I'll put it in there.

B

A

I need to UM advertise, attend anomaly: I, don't.

B

A

Has anything else that you might want, but you can all send anyone.

B

A

B

A

All right, I gotta, run hungry.