Rust Programming Language Rust Linz, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rust Linz, October 2021 - Rust Closures by Rainer Stropek

Description

Rust 2021 came with a significant enhancement regarding closures: Disjoint Capturing. In this session, Rainer Stropek starts by describing how closures work in Rust. This is for relative beginners in Rust who know closures from other programming languages and want to know what's different in Rust. Based on that, Rainer illustrates the new disjoint capturing feature in Rust 2021 based on a "fairy tale" code sample.

https://twitter.com/rstropek
https://twitter.com/rustlinz
https://rust-linz.at
https://sessionize.com/rust-linz

A

Good so again welcome from my site now the community news is over and we are going to focus on the next session and it will be all about closures, closures, closures, okay, so my talk is based on a column that I wrote for high cde. uh It came out a few days ago, I think, on wednesday last week, so I will take the material from this uh from this blog post as the basis for uh for my talk.

A

If you are interested in reading the entire talk it's in german, then I can share the link in the comment section or put it on discord. I will do both. I think, that's okay, but today I will do it in a talk format, so I have taken all the samples- and I will walk you through and describe what closures are all about and what's excitingly new in rust 2021.

A

So for that I have taken up. I have taken a sample project and I've created some snippets because in 45 minutes I cannot write so much code and those who have seen my talks before I always use- or I frequently use snippets so that I can bring in more content without you watching me, miss typing, so I've prepared snippets and I will walk you through the entire content. Good, let's get going.

A

Let's start with a very simple example, and my first one is this one: okay- and let's run this guy- let's run a cargo watch down here, so you always see the result what's going on. I think I can. uh I can move that one here up and I think we are good with that. So let me quickly zoom in a little bit and show you oops. First we will do the following sorry for that, and now I will yes now I can make it larger and smaller, just as we like. I think that is good.

A

So let me quickly introduce you to closures in rust. That's the 101 of closures! If you are not that experienced with rust, this is what a rust closure looks all about. You might be familiar with terms like um like lambda expressions or lambda functions uh or arrow functions in in javascript, and this is the this is the syntax of such lambda functions or arrow functions in ros. They are not arrow functions as you can see, because you use the pipe symbol to specify the parameters, and then you specify the body of the closure.

A

Just as you can see it here, I was told that this may be similar to how it is done in ruby. I've never written any line of code in ruby. If you are a ruby developer. Let me know in the comments section I would be interested, but for me, coming from a c-sharp, typescript and and javascript background that syntax was a little bit new.

A

So let's take a look at the code here we take a variable x and we have a get answer variable here and this variable is essentially a closure, because when you take a close look, you will see that this function, this one captures the variable x from the environment, so the environment has the variable x. It is captured inside of this closure and at the end we get the result 42, because we call the method, as you can see it here, with the parameter 21. So that's nothing special.

A

That is how a closure works, and in the next half hour we will dive a little deeper into how this stuff really looks like. So, let's take a look at the next one here, so, okay, this one is empty. That's by design- and let's put it um here- let's do a bunch of demos about syntax, let's first focus on the syntax.

A

Please take a look at line five and six. These are the first two lines where I would ask you to uh to take your to take a look at in line five. We define a regular function, which is the add, function nothing special here and in line six, we assign the add function to a variable f and what you already see from the rust analyzer, that f, is, is getting the type fn add i32 i32 i32. So we see this small letter fn here which, which is essentially a function pointer.

A

This is technically not a closure. It's not capturing any environment variable, so this is just a function pointer. This is not already a closure. You can use this one whenever you need a function pointer and, of course, you can use this function pointer to call the method. You see it here very simple and you get back a result and it works perfectly fine.

A

Now the idea of the of the the closure syntax, um we can try that one here uh goes in that direction, that we let me zoom in a little bit that we essentially remove the fn and the name. We don't need to come up with our own name. Instead, we take the variables, as you can see it here, and we move them between these two pipe symbols. The rest is essentially the same. The return type is inferred automatically from the return expression. Everything else works as expected, but we can simplify this in rust even more.

A

We can do something like this. If you take a look, if this is just a single line expression, we can remove the curly braces to make the syntax look. Even nicer.

A

Rust is smart, especially when it comes to type interference, so we can make this one even smaller. As you can see it here, I removed the i32 and i32 here. This is inferred this type by the rust analyzer. This is not part of the source code and still it works.

A

Why does it work because it, the rust compiler, takes a look at the call to f and it defer in it in first the types for x and y from the call to the function f.

A

This is still a function pointer, a lowercase letter, f n, but this time I use type inference to let the rust compiler come up with the correct type.

A

The last example that I would like to show you here is not the best coding style in rust. It's not considered to be good coding style. We will get a clippy error, but still it works. So I decided to let it in and to show you how this is done. What you can do is you can define an inline closure. As you can see, it here same closure as before, and you can immediately call this closure directly here after the declaration of the function.

A

So this is possible, it works fine, but clippy complains, so clippy says: hey assign disclosure to a variable, and then you can call it similar to up here where I assign the closure the function pointer to a variable- and I call it down here so this is considered not the best coding style, but still it works. Now you have seen it okay. Maybe it was a little bit interesting good. So this is the basic syntax of closures in rust. That was 101.

A

We have to dive a little bit deeper. Let's do that.

A

Let's take a look at this example. This example defines, as you can see it here, a function add I will zoom in a little bit, so you will see it again. This defines a function add as we've seen it before, but the big difference to the very simple samples that we saw before is that we are now doing a little bit of functional programming, so I define a second function: let's use a different color here and I would like to draw your attention to the third parameter, a strategy for calculating the result.

A

This time in this function is not hard coded. This function gets two parameters x and y, and the strategy for calculating the result is passed into calcan's print, using a function pointer. So please note the lowercase fn here. This indicates a function pointer which will receive an i32 and a second i32 as a parameter, and it will result in an i32.

A

So therefore we can and let's go back to red again, we can pass in the add function which we have defined up here and it works perfectly fine, because this is compatible with the definition of the calculator parameter.

A

We can also use a closure, although calculator is not a closure type which we will learn about in a second in this case, it's a function pointer, but this is perfectly fine as long as the closure does not capture any environment variables, any vira, any any variables declared outside of the closure. Everything is okay, so this one is now the calculator without a function. Definition it's an inline function and and specified as expected, as we have seen before this i32 and i32.

A

The types are inferred here, so I do not need to specify them manually works as expected. If I run this, I will get the result three and three two times because add and this closure they do essentially the same.

A

So this is a function pointer and now you have learned that you can use closures wherever you want to have a function pointer, but when you really have a closure, a closure that captures a variable that is declared outside of the function of the closure function, then this fn does not work anymore. So if I go here and try to do something like this, I get an error and this error is by design. Let me show you what's going on here.

A

I have defined another variable, which is called z, and I have used this variable c inside of my closure, so I'm now capturing a variable from my environment, and this is no longer a function pointer.

A

In fact, if we think about how this can work, it is super important that this does not work, because this expression, this closure, cannot be represented by a simple point to a function. We have a function which needs access to external state in this. In this case it needs access to the c variable and therefore we need a different data structure at just a function point that is not enough anymore. So this does not work.

A

We have to think about how we can solve this problem and we have to think about how we can make this work with a different type than just the small letter fn here. So let's remove that one and go into the next demo. This demo looks very very similar to what you have seen before. Let me zoom in to make sure that everybody can read it properly this time. If you take a close look at the code, I'm not using the lowercase fn anymore, but I'm using the fn trait with the uppercase f here.

A

Uppercase f is, as I said, the fn is here a trait. Therefore I need a dynamic dispatching. I need a trait object which I have to put on the heap in order to make it work. So why is that important well think about here? This works as expected, we only have to put the function pointer in a box, and then we can pass in a regular function. Pointer check works as expected.

A

If we have a regular function, as you can see it here, a regular closure which does not capture any environment variable everything is okay. We only have to be explicit in putting it onto the heap in a box, because that is how rust works. Rust is very explicit. There is no con magic happening by convention. You always have to type what you really want to do and therefore we have to put the closure into a box in order to be able to pass it to our calculate and print function.

A

The interesting thing comes down here, because this time we can take this closure, which captures the c variable, put it in a box, and it works perfectly fine, because now we have a trait object, what's technically happening behind the scenes, at least conceptually not technically at the very low level, but conceptually what's going to happen is the following. Let me show you that one what's going to happen is we are going to get a kind of. As I said conceptually, we are going to get a structure where we store the captured variable.

A

As you can see it here, and the structure also holds, as you can see it here, the logic of the of the closure we will get at the end of the day. We will get a structure, and here you see it and we will pass in a trait object which represents, as you can see it here, the structure consisting of the captured state plus the method which operates on the state and, of course, the parameters so conceptually behind the scenes.

A

You always have to think when you talk about closures with the uppercase f, that behind the scenes you get kind of a structure with data state and functionality.

A

This is also the reason why sometimes let me go back one uh one element here that sometimes you have to specify: oh, my god, a lifetime for a closure, because in this case, rust needs us to specify a lifetime, because it is important that the lifetime of the variable fits to the lifetime of the function. That's essentially the structure behind the scenes. In this case, we don't have to write the lifetime manually.

A

We can use lifetime elision here, because rust will come up with the correct lifetime, but it would not work if you do not specify this one here. If you try to do that, we will get an error and you can dive into the error later. If you are interested in why we have to specify a lifetime here. I only today have 45 minutes, so I will not go into the details, but at least kind at least a little bit.

A

It should become clear when you have when you remember that having a closure is essentially another form of writing, conceptually a structure, okay and there we already know that sometimes or often we need lifetimes.

A

Okay, that is the uppercase fn. Unfortunately, the story doesn't stop here, because this uppercase fn is just one trait for closure. For closures. Out of three, we in total we have three types representing closures in rust. Fn is the first one, the fn, the the uppercase fn trait, means that we are not mutating the captured variables.

A

We can mutate the captured variables if we use the trait fn mute instead of fn, and this is my next demo.

A

This example is split it in two parts and I will zoom in and I will walk you through. The code first draw your attention. Please let me draw your attention on these three lines. As you can see, this calc result. Closure calculates x, plus y, but the result is not returned, but the result is written into a captured variable. So in this case I'm mutating result.

A

So, therefore, this is not just a closure, it is a mutating closure. If it's, it is an fn mute. We don't see that if we just use these three lines lines- five, six and seven, because here you see I'm just calling it result. Everything is fine. Everything is good, but we it. It becomes very clear if we take a look. Oh that's, maybe a little bit too wide.

A

Let me do it like that and zoom in again, if we try again to store this, this disclosure in a variable and write the correct type, we will see that we can no longer use the fn trait here we have to use the fn mute trait, because only with that we have a mutable borrow of the captured variables.

A

I can show you that if I remove, if I remove the f the mute here, I will get an error here. This does not work as you can see it here, because the captured variable is an fn closure and therefore we cannot assign a value to the variable result.

A

What we have to do is we have to change that to an fn mute as you can see it here to get a mutable borrow inside of the closure, and then everything works as expected. Let's make it here. We put it in a box as we did it before, and then we can call the result. Calculator and everything is okay.

A

We can drop the result, calculator, that's fine here and then we can print the result. That's perfectly okay, that was an fn mute. So, let's quickly recap fn for capturing variables, which, just with just a borrow, read only immutable, borrow fn mute for capturing variables for mutation, so the third one I told you that there are third there are three is fn once fn once means that we are moving ownership into the closure. Let's take a look at these two examples. I think they should fit on the screen quite nicely.

A

Yes, let's take a look at the first one. I would like to draw your attention to the first four lines of code. As you can see, I define a vector with some numbers and then I get an iterator into numbers iter.

A

Then I would like to do the sum over this iterator and in order to do that, I have to move the ownership into the closure. I can use the keyword move to essentially move all the captured variables into the closure. Now the closure is the owner of this iterator, and with that I can call the fn once here and I will get a result in this case. The sum of the numbers one, two, three four one, two, three: five: six.

A

Okay, I've got the four, I don't know why it becomes even clearer if we take a look at the second part.

A

Let's use a different color here this one, because in this case I'm storing the closure in a typed variable and the important type here is this one, as you can see it here, this uses the fn once trait and the fn1 straight is okay, because we are again using the move keyword here we are moving the numbers iterator, the the ownership into the closure, and therefore the result will be an fn once we can only call it once yeah. We can only call it once okay, so we have fn for immutable.

A

Borrow fn mute for mutable borrow fn once for taking ownership.

A

You might ask yourself: why do we need that? Why is this move so important? And why does rhino tell us about it? Well, one of the most prominent use cases of fn once with move with a move keyword is working with multiple threads. Let me show you a quick example.

A

In this example, please note that we have a counter and a counter string outside in the main method.

A

Then we spawn a thread as you can see it here and we are using the move keyword to create an fn once we are capturing the variables by taking ownership.

A

As you can see here, I'm just incrementing the counter.

A

If the string is not empty, I'm adding a comma here and then I push to the string, the counter value and after five elements I'm breaking and we are waiting a little bit so at the end, I'm joining so I'm waiting for the thread to uh to to to continue, and if we try run this one, the counter string here it doesn't work. The counter string was defined here in the main method. So you might think that we still have ownership here in the main method, but we don't because of the move keyword here.

A

We moved ownership of the variables into the thread and that's important, because we explicitly want to give ownership to the thread and we do not want to have a chance to make to to, for instance, access these values by design in the main method that might result in some kinds of race conditions or or similar bad things, and therefore the move keyword is super important. Here we are moving the captured variables, the ownership two of the captured variables into the uh the closure here.

A

Okay, so this is an fn once an application that you see pretty frequently when you do multi-threaded programming, okay, okay, good.

A

Now, let's think about what has changed in rust, 2021 now you have the basics. You have learned about function, pointers. You have learned about the fn trait the fn mute trait, the fn once trait and now we can build on that knowledge and take a look. What changed this is the sample that you frequently see when we, when you take a look at what's new in ros 2021, let me quickly describe it and I will describe it by switching to the edition 2018.

A

So, let's wait a second and, as you can see, we get an error and this error is by design. Let me walk you through this code. Okay, now here you see, you have a structure, something it consists of anything and anything else.

A

The constructor assigns foo and bar to anything and anything else. The important thing comes here: we create something and when we explicitly drop a member of something anything we drop only one member, the other member is still okay.

A

And if you now take a look at the closure down here, you will see that we are capturing something dot anything else.

A

We do not capture something, not anything as you can see it here, but we capture the other member of the structure and up to rust 2018. This was not possible because in this case, rust would capture the entire object, the entire instance of the structure.

A

So, therefore, we cannot access st dot anything else and we get an error use of partial, partially moved value st. This is exactly what rust 2018 gives you now. If we take a look here and switch that one to 2021, let's wait a second until it is continued. We see already in the output down here that now we get a result.

A

So if we take a look here at the main method, oh sorry, I still have this clippy error, but for demo purposes it's perfectly fine to call it here. The the point of this demo is that I no longer get an error. We still capture st on anything else. It is perfectly fine to access anything else, because we have only dropped anything and not anything else.

A

So the capturing is no longer done on the on the structure level, but it's done on the member level, and this is called disjoint capturing inside of closes this disjoint, because we only capture one element and not the entire structure.

A

You can force rust to to move everything in it if, by by just assigning the entire instance of the structure to a local variable inside of the closure, and then you can enforce the behavior that everything is captured, but by design. This is a breaking change by design. It only captures the member of the structure.

A

Now you might ask yourself: why do I need that? Where is the advantage? Where is a use case, and I tried to come up with a simple little bit funny and entertaining example that should bring uh to everything to the point.

A

um I don't know about you, but when I was a child, I frequently heard the the cinderella the tale of cinderella um ashenputtle, you maybe know it from from german. If you take, if you have taken a look at the cinderella movie from disney, you might not know how it really worked in the original book there cinderella when the the bad sisters threw out the lentils on the floor, then cinderella had to go through all these lentils and um and and and put away the small ones, the bad ones, and keep only the good ones.

A

The big ones and a lot of pigeons came, and these pigeons uh had the I have to say it in german, because I don't think that they translated it in the film in english. It said the good instruction inscription. That means they had to eat the bad ones and keep the good ones, and I want to implement this cinderella this cinderella tail using rust in a way that I can describe how this joint capturing enclosures work.

A

Let's see if I can do that, okay, so we start up with a very simple structure, which is a lentil and it just contains a size now. The second structure is the interesting one, because in this case we have a cinderella task. This cinderella task comes in with all the lentils that are spread across the floor.

A

This is the vector of lentils, and then we have a strategy. The strategy that cinderella gave the birds in order to help her okay, she said to the birds: hey eat the bad ones and put the good ones into this box.

A

Okay- and this is exactly what we have here- we have all the lentils and then we have a strategy where we pass in a trade object based on the fn trait. Can you remember that's an im: that's an immutable borrow of a single land till and then we pass. We return an indicator whether the bird should eat the lentil or it should capture the lentil in in the box, so that the lentil keeps keeps the same okay.

A

So this is the the kind of cinderella task that we have here now we implement the cinderella task, as you can see it here, and uh here we see here the magic comes in because in the implementation of the cinderella task, I implemented a method which is called sort. Lentils sort lentils operates on the cinderella tasks and it uses the retain function, which you might already know it's going through the vector, and it only leaves that one that really that really are fine, and in this case I am capturing as you can see it here.

A

I am capturing the eat method here, uh yeah the eat strategy up here, the the um the closure. This is what I wanted to show you. We pass in the lentil here with the eat method, but we still access down out there, the lentils which are also part of the cinderella task. So if this method here would capture the entire self, then I could not access the self.lentilseal.

A

Let me prove the point here: let's go to cargo tomml and switch to 2018, compile everything, and it does no longer work, as you can see it here, because we cannot borrow the eat function here, which is uh in this case um a closure on itself, and here we again borrow the lentils and this does not work only in 2021.

A

This is possible. Now we have disjoint closures and these disjoint closures enable us to here capture the heat and here access the lentils and everything is fine. So let's check whether this really works. As you can see it here, I define some lentils and these lentils are of different sizes, five, six one and two, and then I set up a cinderella task where I define that a lentil has to be eaten. If the size is smaller than five, then it's a bad lentil. The birds are allowed to eat it.

A

That's my strategy and this strategy is formed in uh here, so you can see it here as a closure. So I put that into the eat strategy. Together with the lentils they become the cinderella task, then I call the sort, lentils method and what remains are two lentils, the lentils with size, five and size thick six, and that's exactly what I want to have.

A

So this is hopefully an example where you see that these disjoint closures are really interesting. They are useful. You can use them in real world examples.

A

Nice, this is what I wanted to show you. So let me switch here and let me quickly summarize what we did. I wanted to show you how closures work in rust. We started by taking a look at the basics, the 101 of closures we took at the we took a look at the the syntax of closures. We compared it to other languages like javascript typescript. We saw that we use the pipe symbols instead of arrow functions. Here we saw how rust is smart in inferring types of closure is worked, it works pretty.

A

Well, then we analyzed what is the what's the difference between the lowercase fn and the uppercase fn one is a function pointer which is just a pointer without any state and the other. The uppercase fn is called the fn trait. The fn trait is really what makes a closure, so the uppercase fn represents, behind the scenes at least conceptually a kind of structure where we have data inside of the structure, together with the methods that operate that capture these variables.

A

After looking at the uppercase fn trait, I showed you that there is also a mutable fn trait and an fn1 straight for taking ownership. So we have fn, we have fn mute and we have fn once and now. If you get an error from the rust compiler that something is not compatible with fn once or so something like this now. You know that you have to take ownership and use the move keyword. So I hope my description helped. You understand the messages of the rust compiler and you had helped you understand.

A

What's going on behind the scenes. Last but not least, in the last few minutes we took a look at what's new in rus 2021 disjoint capturing enclosures.

A

I showed you in a hello world sample that rust is no longer capturing the entire structure, but only those parts that you really use and at the end we did a little bit a fairy tale programming. We used the cinderella tail and I showed you how we can use the disjoint capturing in a data structure where we have the data and the closure operating on this data in a single structure, and we used capturing and closures to execute the the closure onto this data. This is what I wanted to show you.

A