Rust Programming Language Rust Linz, 4 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: macros_rule the_world! - Alvin Ramskogler - Rust Linz, May 2022

Description

Ever wondered what the bang ("!") after "println" means? Not anymore! I will show you exactly how macros work, how to use them, and how to write your own macros.
This is the perfect talk for you if you are using macros, but you always wanted to know how they work and how to implement them yourself.

A

Okay, so this is me: I'm alvin raskograph, I'm 14 years old right now. I actually have birthday in about two and a half weeks, and I love programming as a hobby, and I also love creating compilers fiddling around with custom programming languages. That's really fun and I love birds.

A

Okay, let's get into the talk first, I want to talk about c macros to understand where macros come from, what are they they're, pre-procession pre-processor directives? Maybe you've seen things like include in c programs.

A

They start with a hash and then some name- and these are c macros they're snippets replaced by the c preprocessor and they're nothing more than spring replacement. So you can't do complex, syntax matching, that's only possible with rus. They look somehow like this. This is an example we have pi is 3.1415.

A

Blah and macros can also take parameters like in this case. Circle area takes a radius and squares it and multiplies with pi.

A

This is a more complex example. It takes the same macros and it calls the macro macro calling looks like this. We just put the name. It looks like a function call, but it's actually replaced by the preprocessor.

A

What do you think this is going to print now? You don't have to pull out your calculators.

A

This is the output and that's that's correct, but there's a problem.

A

If we add plus 2 in this position, it doesn't output what we expect it to output, and this is the first problem. I calculated the correct, correct result and was 153 point blah blah, but it actually outputs 27.708.

A

What's the problem in this case? Well, they're, just simple string replacement.

A

So if we look at what it what it replaces it with it's pi times, radius plus two times radius plus two- and we can fix this by just adding parenthesis around- and this will output the correct thing- and I have another example here, which is a very simple macro called sum- we actually don't need it, but this is just for an example.

A

The parenthesis are placed correctly here. Everything works fine, except for this, because this is going to give us the same problem as before. It's one plus two times two, so it will print five.

A

That's also not correct because it should print six.

A

That's the second problem and, as you can see here, this is what it what it expands to it's one plus two times two and that's, of course wrong, and we can apply the same fix as before, just saying parentheses around here, and this works fine. This is what it expands to and rust solves all these problems.

A

You don't have to think about string replacement. It correctly groups together expressions, so you never have these problems, but I ex actually experimented with c macros. I found out that they can do some very crazy stuff.

A

Let me show you something here. This is a hello world in c, but if I say include java, this is pilot. I've written, I can actually say public plus main, like this and copilot, actually knows what I'm trying to do.

A

This is valid code. I can run it it outputs, hello world. If you don't believe me, I can change this here and it still works.

A

I look into this file. This is the entire macbook thing. I actually got that from reddit. I think it was in c plus, but I copied it for c. It was a bit hard, especially with these things here, to correctly reduce the syntax tree, but it works, and I even prepared something more, let's say: type script.

A

Function main console dot and I'll look at this. I even get autocomplete. I can now say log hello, world.

A

And it compiles and outputs the correct thing, but if I say console error, this.

A

Outputs a second thing: this is now red, and if we look into this file, it's actually just one single macro called function, which does all the things needed for this toolbook.

A

Okay, we said rust solves all these problems.

A

Now, let's take a look at rust macros. What are they? I'm 100 sure you've already seen them, because they're in the hello world example and they're called using the bang symbol.

A

This is the rastel world and, as you can see here, print line is a macro, and this bang symbol means that it gets called as a macro. If you don't have this symbol, it won't work because the compiler will say a print line is a macro, but you try to call it as a function.

A

Now. The thing is in rust: there are two types of macros: there are procedural macros and there are declarative macros and the interesting thing is they're called both in the same way, both with name then bang and then a argument list.

A

Procedural macros are basically a sub-program, so you write rust code that gets compiled first and then executed while compiling your program.

A

The problem with this is that you have to compile code two times which is slower than just one time, and that is once declarative macro solve they're like function, definitions just a bit different, but they don't need a second compile cycle because they're just like functions or types they're in your program, they don't need a separate crate, they're just inserted by the compiler, and I don't want to talk about procedural macros today.

A

That's a very complex thing, but it's what about declarative, macros and for simplicity, I'm just going to call them macros, okay, a very basic example: how do they look like this? You have macro rules, bang say hi to this is the name in that case.

A

This is the argument list, and this is the content of the macro and macro variables always start with a dollar symbol. Then the name of this variable colon type.

A

There are multiple types like expression in this case or statement, visibility, attributes all these things and they're called like the print line, macro using, say, hi to bang and then the arguments the problem is, they have to be defined before being used.

A

You can have recursive macros that works, because when calling they're already defined, you can even define a macro above this or below this. You can define a macro below here, call it in here, but it is defined when, when this gets inserted, because we call it below the definition of this other macro- and I have to apologize for my title- it was just it just fitted so well, it should have been syntactically macro rules, the world.

A

This was my title, macros ruler world, because that's true in rust- and this would be syntactically correct. Do you see the difference between this? Let's diffuse, simple charts, but the other one would be correct.

A

Okay, how old are they build? This is macro rules, say hi to in this case. This is the name, and this is the macro content. If we, this is call signature, how it is called- and this is the body of the micro- and you can actually have multiple of these blocks, but I will get to that.

A

Okay, a call signature. What is the call signature? It's basically a list of tokens. You need to use matching bracket pairs which are brackets and braces.

A

They can have patterns like we saw before using dollar name colon type, and this is an example. It has to be called using. This is an example, colon expression, semicolon expression and every literal token like in this case. This is an example, colon and semicolon they have to be directly inside. You and patterns can be like an expression or visibility or statement or whatever, and they can have multiple tokens and they're correctly parsed by the compiler.

A

The problems are solved that c macros have if we change. If we change the signature to that, that's a bit simpler than before, and this is the implementation you see, print line, blah blah expression times, expression two and we call it using two plus expressions: two additions: rust correctly groups them together. We don't have the problem like in c, where it's just inserted the simple text replacement, but it groups it it groups together correctly and the same thing also applies for return values.

A

So if we say like, if we replace this print line with expression plus expression two and we call it and we say times 2, it will also work correctly.

A

Okay, there is more complex syntax. In macro headers we have like a variable amount of tokens, so called repetition or optionals, and a simple example is the vector macro. It looks like this. We can say dollar parenthesis, then again, some list of tokens or patterns closing parenthesis separator star and inside. We can also do it same way: dollar, open parenthesis, content, close parenthesis star.

A

If we just try to use dollar item on its own. This will not work because it will say variable still repeating at this step. That will be a compiler error and you can actually use different kinds of brackets. When calling the macro, you can use parenthesis, you can use brackets like in this case, or you can also use braces these ones.

A

This is also a very simple example which says which just prints all the expressions passed inside you limited by calmer in this case like you, can see here and any amount we can now change this to plus, to require at least one. This will still work perfectly, but if we don't specify anything, the compiler will say unexpected token in macro expansion or something like that, I don't know exactly the error message, but this will not work.

A

Now the problem is if we want to have a trailing camera. This will also not work because it will say, expected end of macro invocation, but it got the comma. So what do we do?

A

This is where zero or one are also called optionals coming. You can, instead of the star, you can use the question mark to say either nothing or the things inside the parenthesis.

A

Now I talked about multiple headers before you can have multiple call headers in the same macro.

A

You need them for alternatives. Rust macros, do not support alternatives on its own, like you can say, a repetition, and then either this or this that doesn't work.

A

You need to use multiple call headers, and this is an example print named with a colon, some expression and we print it using a equals blah and the expression or b and c. This is the other output and another problems.

A

Another problem that c micros have is the so-called hygiene, and this is basically that when we pass something to the macro that has the same name, a variable that has the same name as something defined inside the macro, we will overwrite it inside the macro and don't have access to it other later than, but in rust we don't have this problem. Variables are scoped, and in this case we cannot access num, it will say, cannot find value number in this code, because the macro hygiene is defined outside of my function.

A

We can solve it by passing a identifier token, that's also a different token type and another way to solve. It is to define the macro in the scope of the variable like. If you need a macro, only in one function, you can define it after the variable has been defined and then it will correctly access the variable- and this is the fixed example from before.

A

Okay debugging in visual studio code is pretty weird. It's very inconsistent. I tried some examples and sometimes it works. Sometimes it doesn't really. It works very well at the call side. So if you like pass the statement inside the macro, then everything works well, but it doesn't work always when you try to debug inside the macro.

A

I have an example here: if you try to set the breakpoint, for example, at line 25, this doesn't work in this case, but in line 33, this works fine.

A

They can be used across files, but it's not as easy as with functions where you can just add a pub keyword and then in a different file. Are they using at the top because they have to be defined before being used.

A

You actually have to use a attribute called macro export, because pub is not supported on micros, and this is an example. I have a file macros.rs, the macro, my macro, which takes two to expressions and adds them and prints them, and this is the attribute called macro export, which is added to my macro and in the main dot rs.

A

We can say mod, macros and just right after it, my macro blah blah. We do not. We do not have to import it because it's automatically imported, if it's in the same crate. If we import something from a different crate, you actually have to add a use. Blah blah like in lazy static, I'm pretty sure, you've already seen laser static. If you work with regex a lot, then you say: use lazy, static, colon colon laser static and that imports the macro, so you can use it.

A

There is a feature that is called decl macro. This is a nightly feature oops. I forgot the brackets here, update that and it's an unstable feature that basically works like functions where you can say macro blah blah. So it's just a like function, definition with functions.

A

You have f and name, and with this feature you have macro name, and in that case you can actually use pub and import it, so they don't have to be defined before being used, and I looked at some internals and it seems like it works so that the parser, the rust program is parsed, then it's checked and then outputted and the parser, which turns your text into the syntax tree, does some magic and it it basically inserts a placeholder in the position where you call the macro and parse later when he has the information from the from whether from where the macro comes and it can be exported and imported with pub and use, and this is a simple example with pop macro, my macro just like with functions.

A

The argument list, which is the same as the macro call header we saw before and then the body which gets inserted- and this is the main file with feature decal macro mod macros use macros my macro, and then we can call it. If we don't add this line, it doesn't work because it's a declarative macro, a declared macro.

A

And I'm going to give you some examples now some very basic examples. In this case we have the hello world example. This is what every rust programmer white writes as it as his first script.

A

If we look at this, this is the macro, and if I press command, you can see it highlights it, and I can go to the definition and, as you can see, it's actually just a macro in the standard library called print line. It matches it. If you don't pass any arguments, it just prints a new line.

A

Otherwise it takes any amount of so-called tokens.

A

Tt is a token tree. It's either a single token like comma or colon, or matching parenthesis or brackets with anything inside like multiple tokens. Whatever.

A

And it calls the print function with the format arcs micro.

A

And I've prepared some examples here now in the slides. I showed you this macro here.

A

This is not really usable for production, so I thought we could use something that we learned before so-called repetition. I'm not going to try to rebuild this macro so that all these things here work. If we now look at this, no rules expected token b. I deleted this and also no rules expect is d.

A

Let's add some simple repetition here, like this limited by camera, so everything that's inside here any amount of times and between or with a comma now it still isn't work, because we only expect an a at this position, so we can say dollar name.

A

I then expect any identifier here and this almost works. Now we have a problem with this this here you have to always put it inside here and now, which is yeah. I can show you this arrow variable x is still repeating at this step, so this doesn't work. We have to add the repeating thing here too.

A

Now we have another problem: unexpected end of macro invocation, missing tokens in macro arguments, which is here so it says here we don't expect any more tokens, but we pass tokens here. How do we fix this?

A

We can add an option from here a comma. Now this works.

A

If we run this run, as you can see here, it correctly outputs all these things, but it doesn't output the identifiers, it always says a. We can replace this with a format arc. But what do we put here if we say name we'll give an arrow? Can it find the value a in this code because it tries to access the variable a now? There is a macro, that's built into the rust compiler, and it's called ring by this, and if you, if you watched carefully, then you will see there is something missing.

A

You will say: expected function found macro string. If I forgot the bang symbol here, but now everything works. Fine, the stringified takes any amount of tokens and it returns the string representation.

A

If we run this now a b c d e and the correct values, so everything works fine. Now this is a very simple example of a macro, but it can sometimes also be useful to have these kinds of simple macros, but I have a more complex example here.

A

This is a pattern you will see really often up enum color some variants and then display for color and just some string formatting. You can see this really often and many crates work like this.

A

And if we say that color.

A

Red in this case- and we say print color- this works, but it's not very beautiful. As you can see here, we have a lot of code that we actually don't need, but we have it every time we write this kind of eno.

A

So, for example, I have written a macro called fmt enum, not macros. This is a file that I will get to this enumerator this micro and we can now say enum.

A

And I can move in this color enum delete this here.

A

This now says no rules expect the token comma. Why? Because, I have to say equals.

A

In this case, it's like this and then I can say, equals green and blue, yellow goprolog. Copilot even knows this.

A

And like this here, we can see.

A

A

And isn't that much more beautiful? I don't need this anymore. This is way more beautiful than the old version we had before, but it does still work you can see in line this here.

A

You can add some more prints.

A

Red and it still works like before, so how does this macro work? Don't be scared of this? This is normal things you see when writing macros, but I commented it, and this is the first thing you see these two match, which is to code, highlights them, and this says everything inside here any amount of times without a separator. If we don't put anything before the before the star symbol, it has no separator, then we can have any amount of attributes.

A

This is called matter. A meta thing is basically the thing you put between these brackets when defining and defining attributes. Then we have visibility. This can be either nothing or pub or pub crate. All these things, then we expect the literal token enum.

A

Then the name is an identifier, and then these braces here, which are these things, and then we have any amount of enum variants, separate by comma, with an optional trading comma. This is what we've seen before it's an optional comma after the list of comma separated values in here we have attributes again for the variant and then variant name.

A

This is a bit complex.

A

It's basically this here we say we have an optional tuple variant, which these these parentheses here with any amount of name colon type, also separates by comma, but we need a minimum amount of them like we need at least one- and we also can add attributes to this and the same thing for struct variants.

A

Which just uses these braces and then after this the equal symbol, which is this symbol here or this a literal which is a format string and then, after this optionally this here- and this is an interesting thing- we have an optional and inside this optional, with a percent sign an optional percent.

A

What would happen if we would move this.

A

Out of here like this, this is still valid everything here, but we can now say.

A

A

Now save here percent: this is valid.

A

We don't want this. What we want is actually inserting it inside here.

A

It gives an arrow here, because it would expect if it sees a percent sign. It expects a parenthesis, because it knows it is inside here, so this has to exist this optional. So it expects a parenthesis after and this parenthesis is just a list of expressions separated by a comma, with an optional trailing comma.

A

Remove this again and that's the entire macro header and the implementation is not that complex. It's just what we've seen before the enum with all the variants- and this is the trick, I'm using some kind of pseudo alternatives here, because rust will give an error. If we have both this and this, it would be valid for this head to put both.

A

But it's not valid here anymore, to put both, then we implement display for this name and this basically just right. That's the format string here, and these are the format arguments and that's the entire format. Enum macro.

A

And I have a very interesting tool here, which is called cargo expand which expands this macro. That's a tool you can install using cargo, install expand cargo, expand. I don't know what the name exactly. I will send it later in the youtube chat, and if we run this, we will see exactly what it outputs this is. What comes out is the edom. It's exactly what we had before. I can remove the hello variant.

A

Exactly what we've seen before and pretty much the same in here- and this is something very interesting, because the right macro is a macro itself here, and this gets expanded to f, write, fmt core fmt arguments, new version, one all these things and that's actually what happens in print line two.

A

It says print an empty string, then a new display for the color and then a new line- and this is the same for these two cases. This time with rgb and here with red.

A

And I talked about lazy static before I have another example for cargo expand here. This is a very simple laser static example. With static graph example. String is hello world to string. This does not work when we don't use laser static, because tostring is not a constant method. You can see here. If we add const, this would work, but it's not a constant method.

A

We need lazy static in this case and we can print it using the draft operator, for example,.

A

This actually works, as you can see, and if we expand it. This is what comes out it's a lot of code. You can copy it into the file.

A

This is what comes out a feature prelude import. This is some rust thing. It's like the all the often used things like vector or option. You don't need to add usings for them. You can just say.

A

Like this, this works because rust imports this automatically.

A

Then it uses the standard library macros and now this is my code that I've written here use laser static, lazy static, and this everything from here to here is what the lazy static macro gives me. This is a lot of implementation detail. I don't really understand what it does.

A

It's some very complex thing with init things and uninitialized memory. I don't understand that, and this is the print line micro again, and we can actually look at the definition of this.

A

It says any amount of meta attributes and then we have static ref. These are literal tokens, name type expression and then any amount of tokens. Again, that's basically multiple. I can say.

A

A

Like that- and this also works because inside this, which is this here- it passes all these tokens back to the lazy, static macro and that's macro recursion. So first we go in here. We pass everything into this here.

A

We look at the first thing and we pass the rest back to this macro, which, in return passes back into here.

A

And these were my examples.

A

I have some takeaways for this talk. Macros are a very powerful tool.

A

It can help you a lot they're, safe, they're way way better than c macros, like hygiene or the grouping things they do have problems like having to be defined before being used, but that is going to be solved soon with the decal macro feature I talked about before, and sadly there are no alternatives in micro headers.

A

I imagine having something like macro rules.

A

So I can say I want any amount of either a or b. That's how I think about it. That would be cool, so I can call it using a b e, a a b, a b and so on, but that doesn't work currently but they're very useful for your everyday's work.

A