Rust Programming Language RustConf 2019, 16 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustConf 2019 - Taking Constant Evaluation to the Limit by Oliver Schneider

Description

RustConf 2019 - Taking Constant Evaluation to the Limit by Oliver Schneider

Have you ever wanted to write a static with a complex initial value? Are you programming microcontrollers? Do you want to run as little code as possible at runtime or are you trying to reduce your memory footprint? If you answered yes to any of these questions, then you probably want to evaluate all-the-things™ at compile-time with a constant evaluator.

This talk will focus on explaining the inner workings of the new theoretically unrestricted constant evaluator in rustc (nicknamed Miri for "MIR interpreter") and give a short outlook on the future features Miri will enable in the Rust language.

A

Hi everyone I'm oli, so you might have seen this logo. That's me and I work for kasnian, so we take security, research like multi-party computation and homomorphic crypto and like put it into products that actually protect your privacy, so yeah trying to do cool stuff here anyway, you're here for constables. Oh stop talking about me! Why constable well I guess everybody has used constables somewhere, maybe for an array links or something so usually use constable to save resources.

A

You want to make sure that something happens at compile time instead of at runtime, so he just like, especially if I'm embedded, you don't have many resources at all, so you don't really want to use them up.

A

Other things that he can do is like you can reduce some types of some trickery like sometimes you end up like working around weird, weird things, so you're working with the system to get it to do like the type system get something to do that you want to do, but it's very hard to do so, you're doing like weird tricks and it just gets annoying. So constable saves you from that, because it just allows you to write stuff in real rest code, so it can do whatever you. You were previously doing comp in complex ways.

A

Also, you can prove constraints at compile time, so you can make your program secure by ensuring that some things that simply check the compile time error out at compile time and, most importantly, because we can that's my reason so there's a bunch of levels of constant evaluation that you can get at. So there's like the omnipotent level. You know it is from scripting languages.

A

You have an invalid function, you can just involve anything and you get a result from it and you can just use that at compile time and then use the values and the during compilation. So that's like omnipotent, like in crazy levels, because you can do anything in there. You can open that for connections and like connect to whatever database somewhere and then do weird stuff on the internet.

A

That's not what you should do it in console, but you could then use restricted omnipotent, which is like well you're not allowed to do Network to the internet, because that's kind of yeah weird. So let's not do that and then there's inherently limited, which is basically a glorified calculator and allows you just to like compute some values and- and so we are right now at be very restricted. Omnipotent, so big shout out to Scott Olsen, who wrote the so called Mira Mira interpreter, which is the BAE thing that we are using in Constable in rusty.

A

It is basically all-powerful. It is a virtual machine interpreting rest code. It is a virtual machine in all of the definition of a virtual machine. You can run anything in there theoretically and we want to go to restricted omnipotent. So we want to keep some restrictions around. We don't want to run anything to be possible at compile time and there's a bunch of reasons for that. So, first of all soundness.

A

If you allow anything at compile time, you get some weird issues where, like you, compute a value over there and you compute it over there and it's suddenly not the same value, even though, like the logic is entirely the same. This happens, for example, if you have random number generators as like at compile time, that's a bad idea.

A

Then again, though, that all boils down basically to determinism. You want your constants to always be the same thing. So if you have a constant and you look at it and you look at it later again, it should still be the same, constant and finally like for us implementing constant evaluators. We don't want this to get more complex entities, it's like already a thousand lines of code inside a rest compiler, that's pretty big for just a constant evaluator, but we'll get to that later. So we want some restrictions.

A

For example, we don't want mutation of global state. Mutation of global state essentially boils down to random number generators because you can have some global variable and you're mutating like you're, accessing definite, differend them or something getting random numbers. Whatnot mutating global state is a bad idea because you get like the initializer in another sized initialization order problem that you know from other languages, so yeah, no global state.

A

We don't want any non determinism, so you're not allowed to get check the bytes from some pointers because, like pointers to some things in memory like they'd change, the address between compilation and the order is not defined that the actual addresses are not defined. So we don't really want that kind of things, and we also don't want Cindy again back because it's very complex to implement, like thousands and thousands of similar instructions.

A

So there are some restrictions that we just want to keep around and we have to work around those, but we have loads of unnecessary restrictions. So one thing I often get is like. Why can't I have like new I mean BEC new doesn't even allocate it doesn't do anything it's like just like create some vector that has no element. So why can't I have this? Well, there are some problems with with that, because internally it uses things that are a little bit more complex than doing nothing and those little bit more complex things.

A

They aren't really possible right now so, but we're working on that there's no Canarsie. For that, then another thing is box new vector, so anything that kind of like needs to heap. So we want heap during conservation I'll get to that later and explain in detail how that is happening, because it's actually a very hot topic, because C++ just got heap during constable 4 to some 20 so but yeah so loops and if conditions so any kind of branching.

A

We can't do right now and we want to get rid of them and cool update yesterday morning and you pull request was open that actually will help us get there and another one is in the work, so I'm pretty positive. We can have this in the next month, well, reverie unstably, and not for anybody using it unstable but like it'll work at least then calling trade methods. You cannot call a trade method in constable right now.

A

We have concept and since the 2018 edition, so you can define your own consequence, but you cannot call trade methods. You did there's no way to specify those bounds, so there's an open RFC for that and yeah once we have that and we can implement it and then we'll have that. So this is work in progress, but we're getting there format a space, Atika lamination of everything that you saw above so we need trade methods we need.

A

If we need loop and we need heap allocations and then you will be able to call format and when you can call format, you can basically like end up using like all thirty, because that just uses the same stuff. So my goal is personally to get Sri at compile time.

A

So you take like a configuration file for like a Tamil file on your source and then you run it through surly at compile time and you get struck at compile time with all your configuration and you get an error at compile time telling you that the file was bad instead of like getting it during run time and having your product crashed somewhere.

A

So the last restriction that is basically unnecessary is that we're not really allowing you to do I'm safe right now, you can do unsafe, but only if the unsafe contains nothing that is actually unsafe.

A

I'll get to unsafe later ant, if is pretty cool during comfortable and yeah. I have some examples. What we're gonna do there, but everybody here probably came here not to hear what you can't do. You want to hear. What's gonna happen, how it's gonna happen, and so let's continue things that like aren't constable but well. They are now so Siouxland just got constable like yesterday, it's not nor yet, but like it's working and so it's empty and all these kinds of functions so Stephan standard Larry is getting constable all the time.

A

Basically, if you want something to be canceled, I'll, just open it, an issue about it and like we'll we'll fix it like most of the things can be constable nowadays, so conce offense. So we can just do that. So panic panic is constant and it's pretty useless because you don't know if so, it's just gonna panic at content there all the time, which is an error but like hey it works, you can have let let moot so. You can actually like write some pretty linear code and you can have to be structuring in there.

A

So you can create things and destructured them again and so on, so that works and there's all kinds of functions on you to your triplet integers that are constant now, so you you have like overflowing and wrapping operations you're like. What's it count, zeros count ones whatever like there's loads of operations on integers that are now consonant and you can use them and there's probably more that we could make constant. So if there's something you know, you're missing just open an issue about it.

A

If, if you're telling us about it, then we can probably fix it but most of shortly, but probably a lot of people really want is pointer. Offset from you may ask: why do you want that? If you were to offset off its like to calculate the offset of a field instruct to the base of the truck? So you may wonder why you need that?

A

Well, it is used in so many kinds of abstractions, where you're doing some some neat hackery at compile time- and this feature is constant now and we can use it again on nightly, obviously, but like it works. So these are kind of things like library, wise that we get, but there's more than that, you can basically end up making anything Const of n, like literally you pointed it, and we can make it constant, there's very few things that we can I give you restrictions earlier. Everything outside is restrictions, make it constant, tell us about it.

A

If you want, it will make it constant. So what kind of stuff is being happening as console fans now, I already told you about looping, if there's accepted RC for that. So this is just like implementation work and it's happening right now. So there's some open RFC's which give us like calling trade methods and vacuu and so on- and this is gonna happen, sometimes sooner I guess we don't have an RC it for heap allocations.

A

Once we have CR RC through for for trade methods, then we'll go to the heap allocations and with HEPA locations, people can start doing all kinds of crazy things. You can start allocating graphs at compile time and just putting them in a static, and you have them available later and the computation is like all happening at compile time or perfect hashmaps. Well, you computer perfect hash map at compile time right now. This is all like procedural, macros and all kinds of other black magic.

A

But then you can just run your normal code that you would run at runtime, just run it at compile. Time put in a static and you're done and of course, I already talked about unsafe, C, unsafe shenanigans and there's lots of fun section again. So there's super super fun, but before we go into all of these details, let's give an example about heap allocations because he fell occasions, especially in the presence of C++.

A

Getting isn't 2020 a very hot topic, so keep allocations have lots of curious problems, most of them boiled down to this you have a constant and that constant is of type string. However, you get that constant I, don't care like it's you somehow computer constant, and then you use that constant. You say let x equals foo and then at some point, fruit X is being dropped. The runners go up, you drop it explicitly. Whatever the question is: what happens at this assignment like X equals foo?

A

Usually, when we didn't have heap allocations, there was always a very simple solution. We just bit copied a constant into X, and then we have the value right there, but if we bit copy the constant string object, we are referring to a static heap allocation. This is like in static memory. This is this is in your in your data section somewhere, where you can't get it back out, isn't you can't deallocate it? It was never allocated. This just exists like a static would be like calling free on a static as like undefined behavior.

A

So this is illegal code. We can't have this, so the question is: what kind of things are K and what kind of things are not? Okay, and this is what the we could. We were able to take a lot of information out of the C++ discussions about heap allocations in C++. This is illegal you're not even allowed to compile the Const definition, so it becomes foo string is jut, not legal, and we are not going to make it legal either because it has this obvious problem here. So what kind of things are?

A

Okay, so, as I showed you, the first thing is bad, but if you put the thing behind a reference, suddenly it's okay, because you can't move out of a reference and rest so anything behind a reference. Even if there's heap allocations in there, it's basically fine. You can just access this as and you can't move out of it. You need to clone it and once you clone it, you create a new heap allocation and everything is fine.

A

There's there's no problems there, and even if you convert this to it's no problem because then it's just a it doesn't even show you that there's a heap allocation there and you can't move out of it anyway and any kind of other non heap types. So if you have integers or custom structs or slit that don't have keep in there, you can just put them directly in a constant and everything stays fine. So this kind of setting, where is something behind the reference, is okay and something not behind the references but may be problematic.

A

Maybe it reminds you of anything like it took us like two weeks to figure this out when we were like, like sitting there and just discussing the whole time about all of the steep stuff, and at some point somebody noticed well, it's almost like sentencing, it's like not quite sentencing but like we could create two new traits. One is Const, which means you're allowed to put it in a constant and the other thing is concept safe, which means, if it's behind a reference, then you could put it in a constant.

A

So the ideas will create the system and we'll just do an unsafe, employ constructs a four-string, because we'll say we checked, we check the implementation, our string and we are sure we are very, very sure you allow to put it behind the reference, because once it's behind a reference, the only thing you can do is clone it or turn it and just roll and just like work with the slice. But, like you can't do anything more than that. So we have a system forward here and now comes the difference to C++.

A

We are allowed to put heap allocations into constants as long as they are protected, somehow behind references or something else, and in C++ they decided it's too dangerous. We can't do it as because we can't possibly ever check that this will work out.

A

So the only time you can use heap allocations in C++ is during your computation compilation, but the final product does not have any heap pointers, even if you converted to an ampersand STR your if you do the same thing in C++, if it's a child pointer that came somewhere from a heap allocation, you're not allowed to do that, it's just forbidden, but in rest we can do this because we have our awesome type system that has stuff like sentencing that we can now reuse for this kind of setting, so we'll be able to do a lot more and we can do it safely.

A

We can do it completely safely, as the user will never have to worry about it if it compiles, it's fine to do and here's the example which will make all of this possible if it's concept safe. This is the T bound consular safe here, then you can make it con safe if you put it behind a reference, so everything is being done for you, as a user will never actually have to touch. This is all happening in the standard library will expose it like send and swings.

A

So if you're doing some your own shenanigans, you can just implement the appropriate unsafe trades and you'll be fine and so yeah. We have a system now to get heap allocations there and this system it doesn't look very complex, it isn't very complex and we are pretty positive. We can get this through, maybe in the next year that at least you have an on lightly and so we're not playing catch-up to C++. Here anymore, we are like on the racetrack just taking over not having taking over yet but like we're there. Okay, so keep allocations.

A

Now we're coming to unsafe, unsafe is awesome because unsafe gives you a compile time undefined behavior the best thing I love it.

A

So indifferent behavior isn't really that problematic at compile-time, because you just get an error, but sometimes you don't get an error because we are not like we did. We haven't taught self the halting problem yet so you can't really figure out if it is really undefined behavior. So what we're doing is like we reserve the right. If you do anything, behavior we'll just break your code in the future, but by breaking code I mean like will emit an error later, so don't do undefined behavior. It's not a good idea.

A

For example, what is undefined behavior well give a boolean. Let's take the Y first, you have a boolean and you transmute, a 3-2 boolean, a boolean memory. Representation is only allowed to be either 0 or 1. If it's a 3, that's undefined behavior, because a bool with number 3 enter, and there is just a legal value.

A

It makes no sense, or if you cast a an integer to a row pointer and then the reference that and 1, then that may be okay, if you're like on embedded and you're, giving like the address of some hardware thingy, but at compile time like what is on the address, 42 like what I'm, what am I supposed to do is a compiler developer like there's nothing there. So these kind of things are undefined. Behavior. There's lots more about like well all kinds of under sign to find behavior that you already know but like.

A

If you do any of these things, they were like League. If you compile time error telling you exactly why you're screwed up, so you can actually use this a little bit to play around with undefined behavior to see like the boundaries of what what console I can do. There's lots of fun things to do there.

A

So if other things are not 90 fine behavior but like you still need an say for, for example, here we create a reference to the number 42 we convert that to a row pointer and put that in a constant and then from another constant. We dereference that raw pointer and this operation is basically the same operation we had in the previous slide, where we would be referencing a pointer that was actually just an integer address, but here it's an actual address.

A

It's an address to the number 42 somewhere in the stack and in a static memory, and we can be referenced at totally fine. So the dereference operation is something we really want at compile time, because people will be able to do these nice, three mannequins. So to do you be or not to be. That is the question and the problem is. There are much stricter rules at compile time, because we just really can't figure out what you're doing here like. If you put this pineapple variable and puff pizza stuff will happen. That's not good.

A

So, for example,.

A

We are doing the same thing we did earlier. We take an address to a statically allocated, 42, convert it to a row, pointer, convert it to you size, and then we divide that you size by 99, but like what does that do like we don't have addresses at compile time. There's no memory like what is dividing like magical address of something bit by 99. Like we don't know like okay, we could add, like symbol, symbolic, executors and all kind of set solvers and whatnot and go crazy, but like do we really want that like?

A

This is probably not a good idea anyway. So and I'm not talking about a pineapple of pizza I like that. So sometimes we have less stricter rules at compile time too. So we don't have any concrete rules about the alignment at compile time. You can just do weird alignment thing. So if you have like an array of you, eight bytes and you convert that to you, 64, like an array of just you, eights can be anywhere in your memory. It doesn't have to be aligned. It just definitely has not to be aligned like you.

A

64 needs to be aligned and if you do this code at runtime, that's only fine behavior. If you do this at compile time, it'll compile, but we reserve the right to break your code in the future because you're doing undefined behavior so they're very careful rules. What you can actually do- and you can do more things like you- can do an undefined behavior at compile time. We might not catch it, but it gives you the right to do all these kinds of cool things.

A

If you're, just as careful as you should be at at runtime, so summarizing we don't really have in limits, but we are restricting ourselves on purpose just to stay sane, because at some point like we, we don't want to put all this complexity on users, because that's what lots Rus is all about: taking away to complexity, making the same for everyone to work with code, and we want to do the same thing for constable and there's lots of work happening right now to make everything easier for you.

A

So don't have to write any complex trickeries to get stuff to work at compile time. So just write normal Rosco and just execute the same thing at runtime at compile time and you'll be all happy. Many features that somebody might want need a load. Look like small base features, so there's lots of work, that's happening where it's like we're. Implementing three base features just to get one major feature, so example, as I said for 30, we'll need like heap allocations. We need.

A

If we need loop, we need traits the others, even four, so yeah, and what do we actually need as more people telling us what they want to do at compile time as like? So we can prioritize there's so many things we can do, but like there's, very few issues opened actually about people wanting something to become Stefan. So that's my call to you Oh more issues telling us to do stuff at constant at all time yeah. So this is basically everything.

A

I had prepared, but I have a small appendix because people were telling me of the last few days, all kinds of stuff that they are already interested in. So I prepared a few small things.

A

This one is a compiler flag. You can add this flag to a compiler and it will simply take off the guardrails like it'll, just be like. Okay, sorry, I'll, stop trying to be sound. Just just I try to run everything that you give me and you can do loops and there you can do if you can do end amout of stuff and it's a very bad idea, we're using it just for internal testing, never use this in production, never ever some of the tests that we have actually error out.

A

No panic actually because you're doing something unsound that the compiler just doesn't understand. So yeah, don't try this know them, but if you want to play around with the consequent and see how far it can go use, it I want to shout out to add aesthetic assertions grade. If you want to check something at compile time, you use the static, a surgeon's grade it. Basically that's this weird trick and some other way.

A

So if you want to check if a boolean is true and error out, otherwise at compile time, what you do is you create an array of one element? You convert the boolean to you size and you need index another one element array. If the boolean is zero, you get something if your bully is not zero. You panic and panning at the compile time compile time error.

A

Wonderful, so we have static, surgeons, use it it'll, get better, better error messages and so on once we are developing more things at a complaint in the compiler, but I've been told that it's gonna go as 1.0 soonish the static assertions great so you can use it and it won't change its API. We're just improving the Diagnostics that you are getting, so you can just use it everywhere and now a final small anecdote.

A

So you can compute the Fibonacci sequence at compile time unstable, but you'll get a compile time error or two and a compiler error will tell you the Fibonacci number, that's being computed.

A

So this program uses wire loops, it loses if conditions and so I yeah you don't really want to use that, but what we it's doing, its computing the length and a Fibonacci number and using it as the lengths of the type of the underscore variable here and then assigning an array of the links for t2. So you'll get a type mismatch ever before that you get an unlimited expression type.

A

That's the error that Eric actually gonna error out, but we also get some it's net type error and there you can see it correctly computed at 1:44 for you and then they are allowed died.

A

Yeah, so that's it from me for today, if you have any questions, just grab me at a rest of the day and also be here for the in place tomorrow. Two days ago, somebody had a problem. They wanted a fixed and constant valuator. The pr is open now so just come to me. We can usually do this very quickly because many things I just don't know about don't work and they often can be fixed pretty quickly. So yeah, thanks for being here and.