Rust Programming Language Macros, 5 Nov 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustFest Zürich 2017 - Macromancy: an introduction to Rust macros by Geoffroy Couprie

Description

Many times you must reuse code
And your generics aren’t commode
Can you deal in the macro dark arts
Recompose code of similar parts
Or would the compiler explode?

Rejoice! Follow now our ways
Unlock the compiler’s high praise
Listen here to this initiation
Each eldritch arcane incantation
Summons help ‘gainst the terrors we face.

About Geoffroy Couprie:
Geoffroy handles security and quality assurance at Clever Cloud, develops in Rust and researches on parser security at VideoLAN. He thinks a lot about cryptography, protocol design and data management.

A

Okay, so hi: let's talk about macros yeah right now, good yeah, so I will talk about macros first yeah, he introduced me. I do Nam and other stuff I also work at the hosting company, and we can toast you rust among other stuff.

A

So we're writing a lot of things. I have lots of project. We can talk about it afterwards, right now, I'm doing a video toolkit, it's quite fun, so we were talking about macros, which macros procedural, macros, syntax extension.

A

That kind of thing no, but this is quite cool, except that it's not available in style and sale or maybe you could talk about custom derive stuff. So this is quite interesting. Like okay, we cannot have compiler plugins that are stable, so let's just do an annotation that will and you get a function that we received the code and we write code. That's basically you get, the string will give a string and you have a rest parser. That's called sin that use none fork of nan desu call sinem to pass press code everywhere.

A

So whenever you use that there are lot of macros as well, but this is still not the point: I'm, basically talking about decorative macros, like the macro rules and whatever. So why would you use those macros I know why? Because I can?

A

This is quite nice effect, so roast macros are ugly yeah, but they allow some flexibility. We don't have basically in the language like generating a lot of codes from some kind of templates, so the kind of thing you can do with macros you can have one that will generate structures and limitation from just an identifier and some types you know like the arrays with a static size. You can do implementation fondues, so there's a future communing with The Associated clones that are kind of thing.

A

No, it's in stable right now, but before we had to do like not close everywhere else stuff, like that, all of all over the place in the standard library, where you have a macro that will go through integers and make the implementation for each size of array or a Perl or whatever, it's quite funny to watch.

A

You can also use macros like PrinterOn and over ones, they're quite useful. So it's a very common component universe, but most people don't reuse them. They have a bad reputation, but thankfully there are people like me that really really like those. So let's take a look. This is a basic macro block. You saw the previous one like the generate struct, so this is how it could work. Basically, you define the name of the macro.

A

You define the pattern like what will go into the parentheses in my macro and then you write the code like what the thing that became that begins with the dollar sign will be like a kind of variable. It will be replaced inside the code.

A

So here we have an ID and it will be named the natural expects kind of name and we define a struct called with that name, make the main implementation to generate a new one, and you can then write that for any name, you have a hello stroke, you have a world street or whatever it would work. We're not limited to just names like that. We have lots of things available in patterns.

A

So that was the ident you can have a pattern like you can do, can have macro that will take some stuff. That will be using pattern. Matching. You can use types you can use what's interesting. The expression I use that a lot like the McCoy expects something that will be an expression, so it can be a block, it can be. A function call can be whatever anything that will return. Basically value, and this is difference with like see.

A

Macros is they're kind of smart, but what they expect in the input, so they will check things a bit and make sure the code is quite all right. So here we have an oculist quite simple.

A

Another interesting part is that you can specify different patterns so like this one, we have the default implementation for something and we have not the main one, the text that takes an expression as default default number, and we have one where we can call the macro. We had the second argument and it will put 0 in its place, so you can make macros with lots and lots of different alternative like that and basically call them differently. Like you know, very logic, functions like the Castella thing, that's the way, println and other things work.

A

An interesting part is the order in which you will declare its alternative is important. So sometimes you will have annoying issues in the way the macros are interpreted and that's why they are a pain to the back, but we're not there yet.

A

Another interesting part is the macarons are hygienic, so no wait there.

A

So, basically, here we have an example like we defined variable, that's called state, and we pass that to the macro log. It says from the the main documentation and inside the macro we have another variable called state and we will use the expression we get in in reg'ment. So you can have this. It's really really useful.

A

Like basically macro it will be nicely isolated. There's still. One annoying thing is, if you want to do like strucked method method, calls on strikes and stuff, you have to pass self to the macro, because self will not be available. That kind of thing. This is the kind of issue, but still this is quite useful, like if you've debugged, some SEMA cruise you've seen that, like hygiene is just painting, yes to use them, import them and stuff.

A

Sorry, my notes are sled. I have by one should not have an interest. Oh yes, a repetition! So maybe you see like this dollar parents staff parents plus. Maybe we can expect multiple arguments, and so we can apply the pattern multiple times. So this is the the thing that was used in the our impt before so here we define the clone the current rate for the arrays of different sizes, and this is quite small, starting to be a bit harder to read, but it's still doable.

A

I use that kind of thing a lot all over noon. So with that like, if you have already a lot of tools to start and write your own macros, so now you have to expose them so yeah yeah, basically to import macros from a crate. You use the macro use annotation. You can use macros with parameters to say, I want this specific macro or that specific one. It's not useful muscle time, but still, and you can export macros for moderate etcetera.

A

What do we have next yeah? So a part that's been very confusing for people, it's like. How do we declare macro? Do we use the the parens the square brackets or whatever and like when I wrote, the total was just panicking, hey I've always used the same one over and over and I don't know if they do different thing, so, basically all the same stacks they do. The same thing is just whatever you want and they will just expand to the same stuff.

A

But there's something interesting you can do is put another layer because then it will expand to a block and then inside the block you will be able to like import stuff, isolate your card stuff. So this is a good technique to use in your own macros. So this was basically how you write macros and if you stop there, you like at non 1.0, which was painless to debug, which was made of huge macros, still make a fusion backwards, but they're much more manageable now and it was really not nice.

A

So how can we prove that first? Where am I again? Okay, yeah printer in micro. So it's the example. I was talking about about a pigeon. So now we can do some funny stuff norm is full of macros. That call over macros, again and again and over so to match a macro. The idea is, you take an ident, so the thing called sub Mac you put a bank, then the parents, then the arguments of the macro. Then another parents and then I match something that behaves like a macro and I couldn't call him.

A

I can't call it afterwards, so this is quite cool to do which you can combine stuff very easy that way and you can go a bit further. This is an example from the documentation to write HTML from like square brackets.

A

Basically, you have macros column across over and over over there's an issue you can run into with this is that it's a pain to sometimes the compiler will just show up and say: okay, there's just too much recursion. It's a pain like for the recent path. I made the compiler side, okay, there's a 64 M it on recursion; no, it should be 128 because you do too much and then I write the next parser and say: okay, 128 was not enough.

A

Yet I had to refactor a bit of stuff, but it gets really into the bag. At this point, so maybe we will need some tricks for that. I should just yeah.

A

So let's debug the macros, because that's the biggest pain we have yeah, you just put a parameter somewhere and something else explodes, and you will not know from where it comes. So it's pretty pretty annoying. So basically, I have a list of tools to use. First, one is the pretty file.

A

Basically, you can pass arguments to Rossi or kangaroo and it will expand all the code like if we show you the code, just after passing the macros and importing stuff. It's quite useful can be a bit verbose at some time, but it's quite nice, so yeah here we have a macro that will just take a value and replace that with the value plus one and the code in the end looks like okay, we declare a macro, but then the main function is just value plus one which was zero.

A

You can do the trace macro stuff. Trace macro is unstable, but I still use it extensively. Basically, you call trace macro through before you use your macro and then you can trace micro files to stop using because it will just run for every macro and it will just print how it's expanded and it got just really nice, with recent versions of the compiler before I was just treating everything as fast as possible.

A

Now it's trying to to make it more readable, and here we have just a very small layer, but in non when you have macros calling macros macros. If you say ok, so this thing expended to that and due to that and to that and that- and you will see where it breaks. So it's quite quite useful.

A

There's the lock syntax macro she's see which will show like how it how something has been called, there's last one that still useful is string if I like. Basically, you call you wrap whatever we string, we file and you'd be able to to print it. Basically, if you make a string, so this is how I make the beginning stuffing in non. It's been, is basically I.

A

I have a macro that I wrapped with a debug, and it will print the macro and call it as if it was the macro itself, so lots and lots of tools. It can still take some time to to the back then understand so. I have a few few tricks to make sure that you might probably walk correctly. So basically, just.

A

Yep, so, first being in a macro, the place where the macro will be called from may not be inside your crate. So if you have a reference to something, that's inside you crate, you have to use the dollar crate name to import like to create name inside otherwise people. We have to import the new modules everywhere, and it will be quite annoying.

A

You can do that also for the result of shown that stuff, and so since I wrap correctly with the parents and stuff I can do my import locally and then have something that looks kind of nice in in Roscoe I had before everything was fully qualified, because I could not do the importance it was a bit annoying. This is quite nice to it right like. If you see the example of the Tri macro that was mentioned earlier, yeah, we fully qualify everything, but to try macro is okay.

A

If we have okay, something with written, we give the value. If we have never all. We just do an area return, but it's not really nice to read that way, because there's dollar create everywhere and with the import stuff it gets more manageable, so I've been reading. Writing all of them like this. It's quite nice.

A

Another thing that's interesting: to fix there with the macros that expect different stuff, depending on the order. You have one macro at the at the end like this, that they expect, like the user facing code, it's how you will call it, and then it called the implementation that will be prefixed with imp so that whatever people are calling it will not get into those those alternatives.

A

So we can make a private part for your macro, but it will not be called backed by people too tightly before I made like a separate macro for that. But then there were people complaining because they were trying to selectively import macros from none and so like they got the alt macro, and then they the Saudia, to also import the art, passer macro and or alt in from whatever, and it was quite annoying. So you can just fold everything inside one use. The macro like that.

A

A very nice trick, I really liked now macros will just match patterns and when you have people that are misusing your macros, usually they will all you misuse it the same way. So maybe you can match that pattern, and so on error message, and so there was a feature that was merged very recently, thus available in stable hope. I will just show an error message at compile time.

A

Sorry, so the idea is, if you match that pattern, you shown our message with the correct syntax. This will make the code much easier to debug for for people, since not macros are a bit we had to use. This is quite good.

A

I'll never good one I talked earlier about using stringy fire stuff. This is a macro, that's used in Dom, where, if you get a collect value, it will return the value. But if you get an error, it will print the the macro that was called and the input we have a hex dump. I can do that kind of stuff, it's quite nice to use, and it's just like reusing components we had before like we have stringify. We have we matched a macro and called called it inside of 25, but also inside the match, match pattern.

A

That way, it's quite interesting and the interesting thing is the dump. I can just put it anywhere to wrap any macro. It works.

A

Concat I- don't it's quite interesting as well, might not work for all the use case. You might have in mind, but basically you have different names. You can concatenate them. Sometimes it's useful, sometimes not.

A

This one is quite fun, basically, basically stop seeing. Basically, it's not basic there. You have the name, the macro in them that used to generate a function and you can match on attributes and patterns and stuff and put them on the function, and it will work so you can have documentation, that's passed to the macro and will be passed to the to the function afterwards.

A

The trick is quite tricky.

A

Takes time to sink in yeah, this is the total passer. This is not the worst one. I wrote you have to call different passers in the sequence and return the result as as a total, and so you see. The first argument is the input and in each pattern. But the second argument is like the expected name of the function, the expected result, and so we passed through all of the macros one after another and accumulate the the result inside this stopper and in the last I can just know in the last one.

A

We just return the the second argument when we have no more no more macros. So this is a pattern that you can use a lot to accumulate stuff because sometimes it's like, when you request, you accumulate data over and over, but instead of.

A

Allocating anything like it will just write exactly the code you want with one variable. So it's quite interesting so.

A

Those were a lot of very weird tricks to write a macro. Basically, you don't have to remember much stuff just that you can write them so that the user friendly when you present them to people they away at their. How to read so make sure that you can do in parts correctly that you can separate in smaller macros that call each other it's much more manageable, that way and they're quite fun to write really because you just start writing code, and then it generates a lot of stuff and it's quite cool in the end.

A

A

Okay, we have a few minutes for questions. I have a suggestion. We can do questions. Oh, we can see the weird, the really weird stuff, Oh, weird stuff, rest, your hands. Okay, you asked for it, don't regret it so I showed you the non macros. So the thing is with the first agreement is the input, but most of the time I will just where am I yep.

A

So most of the time, yeah, the first argument is the input, but when you call a an on an imposter, you don't pass this argument. That's because the named the name function just pass it there. So you see a lot of microscale in each other, and you say that where is the input and it's been passed automatically and you don't see it, and this is confusing a lot of people because they try to call it on just. But what's that I argument? That's used everywhere.

A

If you don't you call that input yeah, maybe but one character is enough.

A

A

The permutation Combinator is a really great one.

A

Where am I yeah it's there, so the idea is, you will try to pass stuff to pass stuff in the sequence like you have a list of pass or to apply, but you don't know it which order they will go and just know like each of them will be called once so. First, you need to like do permission in it. This is where I create a total of the right size, which is like basically a just a long list of known that will be replaced by some something.

A

Then you have the permutation iterate, that's where we call one passer after the next. The ID is okay, don't forget that yet the permutation later you call if, if the current value for the current wizard, for that passer is known, we call the passer, and if we got a value we press by some. Otherwise we try the next one.

A

So the interesting thing is when we get the result it's like when we know that all of them are some something. Instead of known, we want to unwrap all of them, but to get access to one of the parameters. One of the other elements of a tuple. You do like my topple dot, 0, my topple dot, 1 or whatever. So this is why I have this I? Have the success?

A

Succession Combinator the I call macro with 0 and macro and will pass the next integer and when I get number and the topper topper dot, zero dot one and whatever and I will do all the unwrap track that this? This is quite interesting to write. Really it took me like a few weeks to just think of how I would do it and then just mad evening.

A

The whitespace Combinator is quite interesting at some point for people that want to write languages. They complain that they have to intersperse space it in parcels everywhere and I say: oh yeah, that's knowing. Maybe we can write a Combinator for that, so you are do WS. Combinator can just wrap a list of macros and it will automatically insert the space sitting passers everywhere, like interval, you double will be visit in the end space like free space, study, space and the way it's done is well macros calling macros like the pair separator like what we had.

A

It called the separator and calls each macro recursively. We have specific versions like every Combinator for that and it's a lot of God. Amazingly it works, but this is still not the worst finger rods.

A

So may I introduce the lapa project, which is M AMQP client in roast, like RabbitMQ JMS, that kind of stuff. So this protocol is generating from specification, that's written in XML, basically that the Z, so they give you this huge file and they say. Ok, these are all the methods, all the things you can see on the protocol, and so you should probably generate you cut like that.

A

Let's say: ok, let's make.

A

Handlebars template, but what we call in the build script and in that handlebar template I, will make non macros and I will also use cookie factory, which is a nice little project. I made it's kind of like Mak like Nam, but for serialization, so that's macros, all the way there as well, and so here's the template, so it's handlebars and syntax, so the mustache stuff everywhere and then at some point we start defining non parsers and generators with cookie factory. So here it's like for every every method.

A

Within the in that class we will generate stuff and we generate when we say, as we said eyes. First, the idea of the class, then the name of the method and whatever it gets worse and worse and worse from there.

A

But it was not that hot right, like you, can generate code and you can do very great stuff, but it's so useful. It's not theoretically clean, but it works. Thank you.