Rust Programming Language RustConf 2018, 6 Sep 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustConf 2018 - My Little Procedural Macro by Chris Wong

Description

RustConf 2018 - My Little Procedural Macro by Chris Wong

What if Handlebars was a macro?

Three years, 376 commits, and countless breaking changes later, I may have an answer to that.

Maud is a macro for writing HTML web pages. It's like format!(), but for markup instead of text. This design lets it catch typos at compile-time, as well as run blazingly fast.

This talk will cover lessons I learned while working on Maud, including: tips on designing a macro's syntax, some common pitfalls with the procedural macro system, and dealing with Nightly Rust. There will also be a pony.

A

So hi everyone,, I'm chris, I'm super excited it about procedural matters is, you may have noticed by now.

A

And I'm going to talk about a particular procedural matter.. I've been working on.

A

What lessons I've learned from it and hopefully give you some advice that you can apply when you write your own procedural, medeco.

A

So, first off what is a procedural macro? Exactly.? Oh.

A

And the single sentence, I could say that a procedural macro.

A

Extends the syntax of rust, let's unpack that sentence robin.

A

So when you tell the rusts, compiler or cargo to compile your code.

A

You start with the source code and.

A

The first place it goes is into the parser, which takes your source code and.

A

Converts it or parses it into a form, that's easier for the rest of the compiler to understand.

A

Magic happens after that, which we won't go into for this talk and.

A

After all that we get the finished binary, either an executable or a library, crate.

A

Where a procedural macro fits in then, is.

A

Inside the pazza, you can see the bagel there.. It's a bagel by the way,, not a doughnut, a bagel.

A

And what the procedural macro does is whenever the parser sees a call to this macro.

A

It takes whatever is within that macro call and.

A

Hands it over verbatim to your procedural macro.. Your macro will take this input.

A

Processor process it in its own way and then hand back.

A

Generated rust code to the pawza.

A

Which will understand it as normal rust.

A

In that sense, a procedural macro maps, a custom syntax into rust code.

A

Thereby extending the syntax of the language.

A

This might seem a bit abstract,, but in fact procedural macros are everywhere.

A

You've probably heard about 34 serialization.

A

Diesel for databases or the rocket web framework.

A

But in this talk, I'm not going to discuss any of these libraries instead, I'm going to talk about something called mod.

A

What is mod good question.

A

um Put simply mod is a macro, a.

A

Procedural macro that enables you to write html in line with in your rust programs. and yes,.

A

It is named after the my little pony character., it was. It was a while ago, at least no judge.

A

Moving on to help understand how lord works, let's take a look at this example. This.

A

Program generates a bunch of html and prints it to standard output.

A

And, as you can see here, it calls a macro called html and inside of that macro call is completely custom. Syntax.

A

The rust compiler itself doesn't know what.

A

This particular syntax means the thing in purple.

A

But what mod will do is it will take?.

A

This piece of syntax, that's given by the compiler and.

A

Convert it into normal rust code that the rest of the compiler will understand.

A

And this is what you see when you run that program by the way, the string, hello, pinkie pie within paragraph tags,.

A

Now, if you were to run something like cargo expand to view the code that more generates.

A

You would see something similar to this.

A

The real code is a bit more complicated for various reasons,, but this is pretty much the gist of it.

A

As you can see there, all it does is create a new string object.

A

Push a bunch of literals to it, and this render to method does html escaping.

A

But other than that, there's not much to that code.

A

If you were to try to write an html template by hand.

A

You would be hard-pressed to write something that's simpler or more efficient than that.

A

And that's one of the biggest advantages of using mod or any other procedural macro solution.

A

The really low runtime overhead.

A

We can see this in the numbers as well a while ago.

A

I wrote a bunch of benchmarks, comparing different templating engines, just to get a feel for it and.

A

As you can see, here, ward is by far the fastest one.

A

I was expecting some applause here.

A

A

But also,, if you look at the label on the bottom,, it says render time in.

A

Nanoseconds like if you've done much web development before you'd know that.

A

An average web request takes around a hundred milliseconds,, which is about.

A

Thousands of times, bigger than even the biggest bar on this graph.

A

So in the real world, most of the time.

A

The speed of your template engine doesn't matter that much,, but I think it's still cool that.

A

We can create something that is that efficient.

A

Now another advantage of maud.

A

Would be static, type checking and in fact,? I think this is the more important advantage.

A

If we were to take that first example,, we had and replace that y with an e, so it's his best "pone". Instead.

A

The compiler wouldn't let us run then, and in fact this is the exact same error that you would have gotten if you had made that typo anywhere else, because at the end of the day,.

A

We're just generating normal rust, right?, and that is in comparison to other templating engines which might parse and run the templates.

A

At runtime such that you wouldn't know about any typos until you try to run your app and everything falls down.

A

Now, I've hopefully convinced you all that maud is amazing and you should all use it.

A

But how do they come up with this idea for this library?? How did it all start?.

A

Well,, it started like many things by reading the official documentation.

A

This is an excerpt from from the old rust book it has since been rewritten by thus the old one and.

A

One of the sections in that book is about macros and in that section is an example of another macro.

A

That generates html and when I saw that I thought why don't I...

A

...Take this concept and develop it into a real-world, practical library.

A

Now this isn't the first place that I saw this idea.

A

I had already known about this general concept from other languages like hamlet and haskell,.

A

Slim and ruby and razor and c-sharp.

A

In that sense, maud isn't really original, except for the fact that it's written in rust.

A

And I don't think that's a bad thing at all.

A

In fact,, it's a good thing, because you already know that the idea has been proven elsewhere and.

A

The nice thing about rust is even though there are existing libraries, which are really high-quality.

A

Just because rust is so young as a language.

A

There are still a lot of places where the library support isn't that good. There.

A

There's still a lot of low-hanging fruit around, and so, if you have a concept.

A

That you're really passionate about, but can only be found in another language.

A

I think it's a really good idea to take that idea and try to port it to rust.

A

You could make a really good contribution to the community that way.

A

So I got started developing the library, but early on I.

A

Ran into issues developing on rust nightly.

A

Because the thing is even now most of the procedural macro api is still unstable.

A

There has been a big effort to stabilize much of the api lately,, but the stuff I use is still nightly only and.

A

The funny thing is, if you look at the history of.

A

Maud, you'd notice that its first release was in the beginning of 2015.

A

But rust 1.0 was released in the middle of 2015 and.

A

The keen-eyed among you would notice that I started working on.. This thing before rust was officially released.

A

And in a sense that led to a sort of trial by fire,, I guess I.

A

Didn't really have a choice to.

A

Figure out how to work on nightly. uhm, I had to.

A

Learn well, I was forced to learn to deal with these breaking changes coming in all the time, and that gave me some really valuable skills for working with.

A

Procedural macros.

A

Here are some of the strategies are used to manage all of these breaking changes.

A

Most importantly. I scheduled a daily build using travis ci.

A

The exact kind of continuous integration doesn't matter that much.. The important thing was, it was a daily build.

A

Because the thing about rust nightly is that it happens every night, right?. I.

A

Know I'm putting out the obvious here,, but.

A

That means that if you have a daily build and you notice your build breaks.

A

Then you know with relative certainty.

A

What exact change broke your build and that..

A

Narrows down your search quite a bit.

A

Research rust commit history, blame and pr's.

A

The nice thing about the rust development process is that it's really open.

A

I think that was a major theme in the keynote this morning and.

A

So if you find something that breaks.

A

It it's just a matter of a few minutes of poking around and you'll, probably.

A

Find out what changed and why it changed and what you can do to fix it.

A

And finally, subscribe to relevant tracking issues.

A

This this builds on that all rust being really open thing.

A

All unstable features, including the procedural macro api.

A

Have a github issue that tracks all of the work that is done on.

A

A

So it's a generally a good idea to subscribe to that particular tracking issue.

A

Now one other problem I had with developing maud was keeping the library simple and well scoped, because it was largely a one-man job.

A

One way I managed this complexity was in keeping this syntax really simple.

A

In particular, I put distinguishing markers at the beginning of every expression within their syntax.

A

One example of this is in the "let" statement in rust.

A

If you've done any work with rust before,, you know that the "let" statement starts with the keyword lit and.

A

Yeah,, I know I'm saying a lot of obvious things: here.

A

And the nice thing about that is from a parsing point of view. Is that.

A

Once you see the "let" keyword, you know that the rest of that statement is going to be a "let" statement, no matter what.

A

There's no way that you can go half way along that statement and suddenly realize.

A

Oh, no,: this is actually a while loop, instead, right and.

A

That means a parser can avoid backtracking. That's what it's called.

A

I'm having to go back and parse everything again.

A

Use the limiters parentheses square brackets and braces as much as possible.

A

One of the few things that the macro api gives you is that it groups these delimiters.

A

Automatically and so if you build your syntax around, these particular delimiters.

A

Then not only will it look more consistent with the rest of rust...

A

It will also make your job much easier because you don't have to match these up yourself.

A

One example of where these two principles apply is in the syntax for.

A

Substituting-In dynamic content, so if you've used the templating language, like handlebars or moustache before you'd know that to.

A

Include the value of some arbitrary expression at one time., you would do something like.

A

The example in the middle, you would wrap an expression in braces.

A

And I had to come up with a similar syntax for maud as well.

A

The first sentence I came up with was the one on the top.

A

Starting everything with a leading carret and while that followed the first principle of having a leading marker on it,.

A

It didn't follow the second and.

A

The problem with that was well imagine there's more stuff around this particular expression.

A

For example, suppose there is a pair of parentheses after.

A

The caret thing.

A

When the parser see that,, how do we know that...

A

The particular expression ends there and.

A

Not somewhere after that pair of parentheses like this could be a method call right or if there were square brackets there. This.

A

Best could have been an array and we could have been trying to index it, and so I ended up coming up with.

A

Really arbitrary rules around where a particular expression is supposed to end.

A

These days, we have libraries like sign which let you.

A

Factor out this sort of parsing, but it still makes life a bit difficult.

A

What I eventually settled on was the syntax on the bottom.

A

Wrapping everything in parentheses, and now this follows both principles, and it's really easy to parse.

A

That's what I still go with today.

A

Another issue I had was.

A

Well,, I guess a piece of advice would be.

A

Don't fear the heap by the heap, I mean structures like box,, string and vec,, which that you choose.

A

And an arbitrary size for a particular data structure at runtime and.

A

The nice thing about rust is lets you avoid heap allocations altogether. If you want to.

A

But that doesn't mean you have to do that.

A

And in fact, zero allocation code can be much harder to read than the equivalent code that does allocate and.

A

Without benchmarking,, it's hard to tell whether avoiding allocations actually makes your code faster.

A

I think sometimes it makes it slower., I'm not quite sure.

A

If you can imagine, early versions of maud did not follow this advice and.

A

I try to write a version of the library which didn't do any heap allocations.

A

Problem is when you're generating an html string.

A

You have to put all that data somewhere, right?.

A

If you're not allocating a new string to hold all of that.

A

You have to end up writing all of that data to an existing buffer and.

A

To support that I had the macro: instead generate a closure which took in an existing buffer and brought to that. Instead.

A

If you know about returning closures in rust, you, you would have heard of impot traits and.

A

Not being able to name the type,, you know there are a lot of issues around that and in particular, if.

A

Your closure ends up referring to anything in your list of arguments.

A

That means that you have to annotate the lifetime of those particular parameters.

A

Explicitly and that's the apostrophe our thing there and.

A

When I saw this, I thought this is way too much effort just to generate a bunch of html, and I came to my senses and ditched the idea.

A

Now so far, I've covered more user-facing aspects of more like the kind of things that you as a user would see.

A

For the rest of this talk., I'm going to go more into.

A

Implementation details things that you might not see as a user but still help you.

A

Help you structure your procedural macro internally.

A

First off using an abstract, syntax tree or an ast.

A

By that I mean, rather than taking the input to your macro and directly mapping it to the generated frost code.

A

You should instead take that input and.

A

Rearrange it into a sort of tree a more.

A

Explicit or simpler or more abstract form that is easier for the rest of your library to work with.

A

Early versions of maud didn't do this., you might be setting a theme here and.

A

I actually,, I thought that was really cool at first, because.

A

Well, everything is done in what a single pass.

A

But the problem was the moment I try to do any sort of analysis or optimization.

A

Everything just fell down, and I ended up rewriting it to use an ast anyway.

A

So, even if you don't think you need it now,.

A

Please use some sort of syntax tree because you'll thank yourself later.

A

Another thing is: keep the spans.

A

A span is a particular procedural macro thing that tells the compiler, where a particular.

A

Piece of code came from by that I mean the source file.

A

The line number the column number and.

A

Various stuff about hygiene and the nice thing about well,. The reason why you should keep the spans all the way through is that it improves compiler errors.

A

Imagine you generate some code and that generated code has a type error in it.

A

If you follow this advice and keep the spans all the way through to the output, then when the compiler encounters this type error.

A

It'll understands that this generated code originally came from this exact location in the user's code, and that would express this error in terms of what the user originally wrote.

A

Whereas if you didn't keep the spence, the compiler would just drop its hands and say.

A

Well,, the type error is somewhere within this file., good luck, figuring it out and.

A

It also avoids weird scoping issues., like if you've heard about hygiene,. This is probably it.

A

I can't claim to understand it., but if you keep the spans, that should all just work.

A

One example of we are using an ast and keeping the spans.

A

Was really nice is in a particular error message that maud gives.

A

As you may know, an html elements can have attributes attached to them and.

A

One of the rules around attributes is that you can't define the same attribute twice on the same element as.

A

Maud there's an html template language.. We have to enforce this rule as well.

A

And if you have a closer look at this error message.

A

You'll see that it explicitly marks out the parts of the original code.

A

That are relevant to this issue like exactly and.

A

This wouldn't have been possible if we didn't keep an explicit structure that represents.

A

The user's input and it wouldn't be possible if we didn't keep track of the locations of every single.

A

Syntax node in there.

A

To cap off this talk, I'm gonna touch on where maud is going to next.

A

General generally, right now, maud is pretty much ready.

A

It's usable people are using it in production for some strange reason,, but.

A

It's pretty much.. Okay, now.

A

So most of the work at this stage is polish.

A

In particular,, I want more diagnostics, more error, messages and warnings.

A

In particular, maud doesn't warn about invalid html elements in attributes. Right now,.

A

And I would like to warn in the future and also rel="noopener" hands up if you.

A

Didn't know about well, no opener before I mentioned it.

A

Wow,, that's a lot of people.. Well, everyone who raised their hand just then will benefit from this particular thing.

A

Yes and working on stable rust., this isn't a really high priority right now,, but.

A

As more of the procedural macro api stabilizes somewhere along the line, it'll tip the scales and it'll be worth pursuing.

A

So that's all I have for today, thanks for listening and we'll enjoy the closing keynote.

A