Rust Programming Language FOSDEM Rust Devroom 2018, 6 Feb 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Oliver Schneider - Reaching const evaluation singularity

Description

An introduction into miri and Rust's const evaluation

The Rust interpreter miri has been merged into rustc to be its new const evaluator. This merge not only fixed various bugs in the old const evaluator, it opened up the avenue for many new features. Ever wanted to do a for loop in a constant? Want to parse a toml file into a static Config struct and report parsing errors as compile-time errors? Well, now you can do all that (pending RFCs for the details). In this talk I will present miri's design, its usage in the compile-time evaluator as well as future features that are enabled by it

PhD Student in Software Engineering for Safety and Security Critical Software at the Karlsruhe Institute for Technology since 2014. Rust tools peer (clippy, miri) since 2017. Likes books, embedded development, programming languages and dogs.

Rust devroom @ FOSDEM 2018
https://fosdem.org/2018/schedule/event/rust_miri_const_evaluation/

These talks have been recorded at FOSDEM (https://fosdem.org/)
This work is licensed under the Creative Commons Attribution 2.0 Belgium Licence.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/2.0/be/deed.en

A

Hello, perfect, okay, so how do works now? So at the end, you get a thirteen, so it doesn't go country valuing works. Well, this actually called constant folding, but the difference is moot. So now we get to more complex stuff. Imagine you have a function, it has local variables and it has some loops and stuff. So how do you consult that? Like turn that into a tree and start collapsing it like that's, not really possible?

A

This is code that you walk down from the bot to the bottom and you execute its instruction after each other, like you would do at runtime, but this is not possible. If you try to combine things like, you cannot combine. Why divides equals two, because what are you combining there? There's there's no two values to combine to something else, so we were trying to look up for a look at something new where we could do local variables, which are not possible in constants right now, where you can do mutation and where we can do loops.

A

These are things that we want to be able to do it like right now. You cannot do them at compile time you you just can do them at runtime, but even if the, if there's an error in this program, you won't detect it at compile time it'll panic at runtime, so the compiler developers decided to do it right. They created a mirror, which is a structure that allows you to do the sequential constant evaluation instead of the folding.

A

And then somebody wrote something called Miri, which is the mirror interpreter which allows you to evaluate Mir code, and if the mir code contains solely constants well, then you can just evaluate it to the end and look at the end result and then store the end result and when you started doing it, they were looking at like integers and and adding them and similar things like that.

A

But at some point they kind of got crazy, and then we got pointers and pointer, arithmetic and calling C functions and calling syscalls and yeah right now we're not quite at reading files yet, but we can do print line and reading from s Citian.

A

So if you just pipe your file until the SDN, you can read files, that's a little bit much for currency valuation, but well we try to get there so last year, I added Mira to Rusty as an experiment where we were running the old console a little next to the new one and comparing them so just to make sure like we're, not introducing any weird stuff that we don't want to have because right now we just want to get the new we're married.

A

So we can get new features in the future, but we don't want to break anybody's code, so that was running well in December. That was a big pull request. A trust fest which had like a 1,600 commits. It has 7k lines of code added to the rest compiler, which is a lot, but you have to remember this is something that's essentially a virtual machine running your rest code at compile time and producing a value that you can then compile down into your final binary.

A

The next step was well removing the old console valuator, which touched another 6,000 lines of code, but it doesn't add anything new. Instead, it removed all the old crap that we had in there in the old compiler and well now it's gone. The pull request isn't merged yet, but it's running right now is the rest. Compiled developers are merging it at the moment. It's basically already approved just checking every single crate and crate sale, whether we don't break it with the gnu compiler, because well we don't want to break anybody's code.

A

So let's go back to the bottom. What is Mir Mir is called the medium intermediate representation where what's that? Well, it's basically an assembler assembler language, but it's very close to what how rust builds here, your code. So you have types, you don't have any registers, you have local variables and it's already used for a lot of things like Baro checking is done completely on the mirror. So that's where we're getting nonlinear lifetimes in the future, from from Mir and Baro checking on the mirror. Also in the future we might be able to do guaranteed.

A

Optimizations like tail call, optimization and so on. On the mir like everything, that's rust required will be done on a mirror. So mirror looks a little bit like that. It's, like a big text, blob that you can't really read it's essentially, if you, if you read all the small steps there, it looks like rest code, but there's go to us in there like a lot of go tooth and you don't want to read code, let's go to us, so instead you can convert it into a graphical representation.

A

So this is exactly the same code this, and this is the same thing. The only difference is all the local variables declared on top are turned into virtual stack and all the blocks in there with go-to are turned into a graph was arrows between them. So you can actually read your code, and this is actually a tool that you can use to step through your your constant code to check like what is the doing. What is the the virtual memory of Mary doing? Well, what's going on and yeah.

A

So Mary is a mere interpreter, so you can execute all the code. It has no undefined behavior. You can do pointer arithmetic, you can do whatever you want and if you try to do anything, behavior compiler will simply tell you. Well you try to do anything any behavior here. Please don't do that so it'll give you an error message that, like it looks like a rest, compiler error message.

A

It just tells you at this line of code you're doing any fine, behavior just stop and try again- and this works for a lot of things like like. Please try to break Mary. It's really hard, like there's a few things that we know about like you can actually mutate non mutable variables, but that's basically by the definition. A mutable variable is just a lint telling you you are allowed to mutate it, but whether you like that at a background memory is mutable. That's completely well you're completely allowed to modify it actually, there's! No, nothing!

A

Restricting you from modifying immutable variable yeah!

A

So I already told you me read the tool it can do a lot of stuff like read from command line or do some syscalls some easier ones, but well we don't want all of that in compiler. So what we're doing right now is we're separating Merida tool and Mary the constable Raider and the constant value that does everything you expect from constable.

A

It computes your plusses, your multiplications. It calls functions and so on, but it doesn't do print line. It doesn't use those calls, because this calls Jenkins revaluation. That's just weird, but we might do it in the future, but we'll need RFC's for that, because we don't just randomly want to enable random features that maybe are bad. So, but what you can do instead was married. A tool is well, you have a test suite and you kind of getting undefined behavior somewhere.

A

You don't know where, so you just run kygo miri instead of Karger test and it'll run all your tests with miry through the console valuator. So if you do any undefined, behavior it'll just stop and tell you this test at this line of code is doing anything.

A

Behavior and to top off on that it'll print, a stack trace like if you're inside lip STD in the vector function push inside something whatever deep it'll tell you a stack trace down to where you're doing undefined behavior it's not like gdb, which which crashes at some point when you're at an actual error it. It stops you right when you're doing a defined behavior like right before it, and then you can even look at the memory and say: okay. Well, how does it look? What?

A

Where am I all right now and then you can find out how your code is doing anything behavior. So this is the tool side effects of using all of this and cons evaluation. Is you can start doing heap allocation in constants? Well, you can't we disabled it, but we will be able to build, and if in the futures they can do strings, you can do serda decoding of JSON files at compile time, convert that into a structure, store that structure and static memory.

A

And then you have your configuration already parsed already check for errors, everything at compile time in memory. They could just say: okay, we're like what's the setting and you get the setting you, you don't need to check anything, you don't need it. I'm gonna have the file present and you have like a big configuration file like a JSON or XML. This might take megabytes or something convert them down to binary is the down to kilobytes or even less so you save all the computation time at compile time.

A

You save all the memory and you even get error, checking at compile time so yeah. These are a side-effect.

A

There's still some things that we need to do in the future.

A

For example, there's some very complex things when people start using associated types and associated constants like technically, do you think so all right, but Miria will just say sorry. This is too complex for me, the old Kong sevillio said the same thing, so we're not losing any and any features here, but the thing is was Miri. We can actually start doing these things like we can think about. How are you going to do? Is we?

A

We have a structured way to approach this, for example, think about you have an array, and the length of the array depends on a trait implementation for this array that you don't know the length of. So you need to look at the trait implementation, but to get at a trait implementation, you need to have to type for which you need the length for which you need to trade implementation, yeah, you're, gonna end in a cycle, so, but maybe that value you need for the length is completely independent of the length that you need.

A

So there's no cycle, it's it's just an information attached to the array in general and to do these things. That's called partial evaluation because you don't know everything, but you know enough to compute the value, and this is the big thing that we're gonna be working on this year, but it's gonna take time and yeah we're not a big thing that we're working on is allowing, for example, gdb to run rest code with miry on actual physical memory.

A

So instead of trying to actually execute real code, you're gonna run these things on your actual memory, but with the miry debugger checking the code for things like undefined, behavior and so on in your debugger. So you can run gdb and then you execute a small command. It got compiled, it'll get run by Miri and I'll. Tell you some results that you usually could not get, because this kind of complex evaluation just isn't possible in gdb.

A

Also, one feature that we want to have if you compile your code and you have a constant evaluation. Well, compiler could stop at that point and offer your debugging window telling you exactly what went wrong like you usually get a stack trace that tells you okay. It went wrong here and there and there and there, but you don't know any values. You know you don't know how the memory looks.

A

Maybe you did some weird pointers, things and your constants, you don't know exactly what's going on so the next step is to allow you to call your compiler and then just stop when something goes wrong and inspect what exactly exactly is going on. So these are things that we are working on, but what's even better for you is, you can help too and you can write RFC's because we have a lot of features that are basically just error needs RFC formats, some texts about what kind of RC we need.

A

This is literally just to check if something is not allowed report this error and below us the code that would execute if it were allowed. So the code is there like. We just need our seats, you need to write a text, and that explains how to how this affects our things, but you need to write it from a user perspective. You don't need to care about implementation. The implementation is there. You just need to write how you wanted this feature to work and we can just turn it on so yeah. Please do that.

A

We need RFC's and we don't have enough people annoying the arrest, compiler or developers to actually enable the features if the features required by enough people we're probably going to get it pretty quickly.

A

Yeah one other thing you can do: you can go to the rest, compiler and look through the documentation and think this function doesn't do anything weird. It doesn't do anything that depends on runtime value is no randomization and so on mmm it could be a constant function. This function could be evaluated at compile time. So all you need to do is open an put request to the rest, compiler writing Const in front of the FN and you're already helping us, because you found a function that we couldn't make Const. There's a pull request.

A

You're talking with the rest, compiler developers about it and we're gonna turn in a constant, and then it works. It's now a constant function.

A

These things are really easy to do, but somebody needs to do them and we already trying to do hard saying so would be really cool. If you guys help us there and yeah make the rest compiler even more awesome than it already is Thanks. So.

A

Does anybody have any questions, yes,.

A

Okay, the question is: if we can automate the process from finding functions that could be constant and making them making these suggestions automatic. We probably can that's I have no idea why I've never thought about that. That's like I'm, developing Clippy, like every few weeks, but yeah. That's that's really good question. We should totally do that. It won't work for everything like it's really hard to detect.

A

If something is considerable, if it were easy, we just do it automatic and you wouldn't need to annotate anything, but it is basically the halting problem like sometimes to check whether function is considerable.

A

You probably need infinite computation time to figure it out, so that might not quite work out, but for the easy things we can definitely do that. Yes,.

A

The question is where they're adding cons to a constant to a function so making it a constant, would be a breaking change. No, it will never be a breaking change to move from FN to conserve n, that's impossible to be breaking change in by itself.

A

What would be a breaking change is going the other way, quantum constant to FN, to say this means you need to think about whether this function would ever in the future of rust do something that could be not constant, for example, access random number generators or do something other like actually in the current time or so so on. If it would ever do that in the future, then you cannot turn it from FN to constant right now. This is something that wouldn't be automatically checkable, because it's just semantics of the function.

A

A

Okay, I am prepared for this question because Pascal already asked me this question so sufficiently advanced incremental compilation, it's indistinguishable from a ripple, so the rest can pilot if they're doing incremental compilation things we might have heard about those. This is to speed up compilation so, instead of having to recompile for adding a small variable or anything, the compiler just recompile supports that you changed.

A

If you get this incremental compilation good enough at some point, you won't care if there's no repple, because you'll just be weak, like the repple, will be just a wrapper around the rest compiler that just adds new stuff to a file and then reruns the rest, compiler and memory. The country, valuation memorizes, with Instragram incremental compilation, the values from the previous compilation. So this will be a batch script with like three lines.

A

A

The question is since mutable immutability is to default for locals and so on. Would it be better like if Rath started in you to make everything default, consonant and then half of a flag for making it not constant I personally would think that would be the correct solution. We had a narf see about that where there was a big discussion about it and at some point we just realized well, switching around would be a breaking change, so.

A

It it's questionable, like you, can't really test it like with immutability and mutability. It's easy to just look at big database, a big big code basis and look how it's done there, but Wisconsin ofin. The problem is: there's not much previous experience about that, like we have other languages that do best effort console valuation, where you, for example, enlist where you just throw random statements in there and they're gonna get evaluated if they are valuable, and here we try to do it upfront by by deciding whether it's constant variable or not.

A

It's a really hard decision and it would probably be better to do the other way around, but that might just lead to people randomly throwing not constant on it. So right now we're just opting for the Lindt option. That tells you when to make it constant and then you're done with it.