Rust Programming Language RustFest Rome 2018, 9 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustFest Rome 2018 - Tshepang Lekhonkhobe: Behind The Scenes Of Producing An Executable

Description

If it's a mystery what actually goes on when you do a 'cargo run', then this talk is for you. It explores the sequence of steps taken to get to a working executable, from high-level code, to machine code, and what happens in-between. We'll do a simple example and see what each representation looks like, while touching on the steps in-between. In the end, we'll look at the generated RISC-V assembly.

https://rome.rustfest.eu/sessions/behind-the-scenes-of-producing-an-executable
https://media.ccc.de/v/rustfest-rome-8-behind-the-scenes-of-producing-an-executable

A

Hi, my name is Japan like they said so, I'm going to talk about the steps that are taken when you build a was executable and I must be. It was okay.

A

I proposed this talk, knowing that it was a rather ambitious because I've never worked on a compiler before so, and at this point, if, if I get any of the details right, it would be good if some of the money illegible people can approach me afterwards, but otherwise just a bit of a disclaimer, so um anyways, interestingly I just wanted to imagine it as I met. I met. Two people yesterday actually were like. Let's say we are hit.

A

I met, somebody who works on a v8, oh well, who used to work on a v8 engine with the Google team. Another one was a micro as expeditious, quite interesting because they were talking about some of the steps like wow. Actually, those are the people who should actually give that talk, but anyways anyways um about me. I did some I studied electronics formally and I was a pattern friend for a while, but now for some reason, I chose honest I.

A

Actually, don't even remember why, but actually quite dangerous, I think it's very, very well designed yeah I work for a small team I work for a small team in Johnny's back. We actually, we built yeah. We bought distributed that framework yeah so and would actually be interesting to know what other companies on the continuous do something like this, where they actually have full time as developers. Waking waking on in production, so yeah anyways.

B

A

Let's, let's get to the actual leader steps, so this is sort of an overview. This is this is what happens when you actually get to to produce a working executable like if you look at that. The first step is a chalupa sing, the source code, and then, when you pass, this is called you produce.

A

What is called an ast is called an app's, abstract, abstract, syntax, a3 and the second step there is ast processing, actually don't know what the correct term for what is I'm gonna show you what's going on literally with a EST process, but I don't know what is there the normal, the normal name for the steps in work, but somebody can tell me a bit later and anyways so this there's a few steps: they're lowering hir type-checking lower into mir bar checking translation to LLVM. I, our code generation linking will get to two others.

A

So first step is passing.

A

So, with passing, the source code is turned into something like a it's like a meaningful structure, so it just takes basically ticks or raw text and then turns into something meaningful, which is called the est and example it creates. It creates a bunch of paths called tokens, and this is just a few examples they may see. You start feel the weight loss blog or whatever. That's there's actually like a bunch of them like dozens and what you see there. What's everything there is actually types type names in the compiler which is like?

A

Is they actually and internal details? They can change anytime. Nobody should rely on this kind of code, but he has that kind of does that I came across as I was looking at this and yeah okay, so we're done with passing, and then we get to ast processing, so there's a bunch of ones once you actually have that est this there are things that you do to.

A

It example is code, elimination and then we'll get to we'll get to those, and then macro expansion name resolution note that this that they that they are not necessarily sequential, is in one happens and then one like, like you, don't do code, elimination and then make all expansion and then name resolution like they are sort of interchangeable. So you actually have to look at the code to know exactly what the steps are yeah. So talking about code elimination. This is if we ever see C++ background.

A

This is where they, if devs they're, like it's like an equivalent of if there's away yeah, this is quote. What is that they call it code conditional conditional compilation, so that is why I know C plus C C++. You get your if there's and things like that. So it is nice because code, conditional compilation is nice, because you can keep everything in. You know one code base. You don't have to have. Okay, this this part is actually for, for the sake, texture. This part is another catechol.

A

You can just have them in the one which simplifies I, guess debugging. It's also nice event, since that I related I in in one place yeah, and he has an example of conditional compilation where you do use a CFG as an example where you want to compare code for for Windows in in in such a case. Okay, let's just look at an example.

A

This is an example from some code that I actually wrote from some great. So what you'll see? That is that you see that's a one line day that second line. This is CFG windows. It means that unseat em in a valance is a patent. Rep will not compile on Linux. So whenever, whenever you build this Felina, it's not compile only nice about compare for Linux because remember we have cross compilation so what's happening there is that when you compile for Linux is going to be like that, that code does not exist, that and see them.

A

So it's only it's only useful for Windows. So the reason actually did this is to add. Color support for Windows is like by default. Indian. You don't actually get color out of it, so you have to do it feels like a hack. Actually, you have to do something like this, because you know that's quite different from UNIX right, anyways moving on and then remember just a few steps. This now macro expansion, which is kind of a it's kind of a big topic, so I.

A

The three forms that I came across there's a derive, which is which is one example of a derive macro, is when you want to automatically print out the debug information from your from your code. So all you do you just do, for example, a derive debug and then, whenever you print line that and then tell it to give you back information, you get all of these things for free. That is one example, and then you get the second example which is function like this. One is quite common: everybody comes across it.

A

Big example is the print line, that's a that's! Also a macro, and then years of the attribute, the attribute form wishes I Encarta. Yet when I write code with dot, which I'm actually a fan of struct is is, is a crate, that's actually built on top of clip, which does a command line, passing command line interface. Passing anyways, so here's an example of a maker expansion, so you have an asset, yeah I said as idq.

A

That's this actually makes you a good penny, for example, if if your things are not the things that you are comparing are not equal, so this one will definitely succeed, because we all know and believe that the one equals one right so anyways and then this is this is actually what happens so macro expansion takes that bit of code and then creates this, and this is actually just a snippet of it is quite it's quite.

A

It makes it quite hard to read right but yeah this is this was given. You can try to make sense of what's happening like you can see there. For example, it's a match statement, you're, comparing left and right, we lose and it's going to panic with a message with that message that you see a session failed if, if there's problems right, if both sides are not equal, anyways moving on to with with macro expansion, here's an example from struck table.

A

You see you get simple nice code like that that actually gets degenerated into this. This is sort of beautified, because the stuff is so low is all over the place, so I just put it together for readability. So basically the u.s. you are telling it to accept their long option and long option is that is that thing in command line interfaces where you do like a dash dash.

A

So if you're going to give the command, for example, black command dash dash ignorant rect, and then you basically create that the variable called ignorant right and then this is this? The one that's going to tell you that it's actually defined on the command line, and then you can act on it and that's they helped string, and then this is pretty much the good that degenerates, she's quite less comfortable to type. But, ladies, so yeah, this is a macro expansion. Those are just small examples of what's a penny and K.

A

So this is another another step. It's called a name resolution, so that allows you to do to use the same names for like multiple things like things are not that are not the table. Same type like this is pretty much for convenience. I mean, if you take a look at that example, you see we use the same name X for multiple things we make. We we create a type called X, and then we also, and then we create a variable called X as well.

A

That will actually compare so what name the resolution does it that it looks at that code and then notices that they are actually not the same thing, so it's gonna give them different names. That's that's pretty much. What it does actually took. This example from the compiler guide: it's called the rusty guide or something like that. Okay, so it does they're. Also variable shadowing. You know what whatever you doing this is.

A

You can, for example, add on to that and say let Y is equals to whatever type or even the same type or whatever, and then the compiler accept it accepts it. So it handles cases like that. Some not on languages can handle that by the way which I find quite convenient and then it'll also do things, because it knows about names.

A

It's going to do things like typo type of fixes like it can tell if your variable name is maybe too close to another one that this in scope, and then it's going to give you that and then it also gives you suggestions for trace too important. If sin. If sin, however, has a message how good they are, they actually give you those kinds of suggestions like when you try to use that type, but you don't actually have a trait of it that defines that type yeah.

A

You don't have that the trait that it defines a type in scope, so it will give you such a sense for that witches, which is quite cool. Okay, anyways moving on so from the ast there is a step called lowering anyways. What is what it does it? Louis HHI here is a high level intermediate representation of the compiler, so it sort of takes your ass.

A

Your ass coat and then removes all of the conveniences all of the sugar and things like that, and then it takes it into a sort of a raw form which is easier to process for for your compiler. So.

A

Yeah so yeah, like I, said yeah the EEOC's converting to at each other, and then HR is also a type type. Checking is quite important. Type checking happens because yeah it's easier to work on something, a more simple representation.

A

So these are just the commands. If you wanted to see the output of what each hir looks like, that is the that's. What the commands look like so you'll give it.

A

You go into your crit direct variant and run that hir and you later see what the output actually looks like like I, don't know why they use the name and pretty actually and I, don't think it has to do with they look up the code. That's it's! It's quite curious.

A

Anyways moving on I wanna digress just a little bit and talk about Rick's risk v. This wife is a target that I chose for for the output of the code. Cause I wanted to learn a bit more about it. It's actually a new is a is a is, is actually an instruction set. Architecture is a is like an example of it. I say I think the most famous is x86 and you also have um so.

A

His life is like a sort of a newborn competitor in the effort, so I find it nice because I like the well reading, the motivation for it actually, the design is, is good and the people actually learn from them from issues in the design of the a native ASA's so which is cool. It's also. It's also libera. So you don't have to pay royalties and licensing fees and turns like that to use it, which is which is actually cool it allows because because it allows it allows a lot of people.

A

Well, you must be motivated right, but it allows a lot of people to actually participate in process. I design wish wish normal. You actually have to pay a hefty fee or whatever to to be able to do it so which you know so fast. Does it so fastest innovation right I mean if you look at open source, for example, which which was an inspiration for me part of the inspiration of making it open.

A

So I wondered also to note that there is, of course, for Ares 5, but it doesn't have Lipstadt support, so you won't be able, for example, to run it on one Linux. So it's all pay metal at the moment.

A

Ok, so sort of an intro to Ritz 5, there's three basic forms. So there's a 32 bits, it 4-bit, 128 bit and yeah. Those are called instruction sets basil base instruction sets. These are.

A

These are the things that that you have to implement for it to be called the rig's Ares 5, because there's actually a standard out there and then everything else is it's a modular design. So everything else is optional. You only need to implement distance and everything else. For example, you can you can actually simulate it if you, if you want the more abilities you know like, for example, multiplication a turns like that so anyway, so this instruction provides some basic things.

A

Like memory reads, writes these are actually assembly instructions, simple arithmetic, as additional attraction, logical operations like your ends and O's and then branching branching is like jumping to other instructions, because you can just have sequential code right, I mean how do you do mix so.

A

So anyways the difference between the three you'll find that, like cc-64, the 64-bit version will have a bit more instructions which allow it to operate on bigger they. Similarly through 120, that's the basic difference and then and then finally, so, like I said last, the supports there is five target and there's actually two of them, which is that there is at least 80 I'm a crispy 32 imc. And then those are some of the explanations. So I is that base 1mm is a multiply and a is atomic C compressed.

A

So so again, what what this means is that it allows you an option that would be, for example, if you wanted to build a processor that way you don't care about floating points which, by the way, does not support, because that one would be F would be a big F. If you don't care about routine. When you just broke your processor, you don't have to rely on somebody built an entire processors and you only did meet a little only a small part of it, or something like that. So.

A

B

A

Let's just look at a little bit of good. This is my. This is just some poor code that I want to use, so you can actually have a look at some of the some of the output.

A

Yeah I choose something simple, because the output at those intermediate and final expanded layers is actually quite ridiculous. It's hot it's actually quite hard to read, so it makes appreciate using a high level language because yeah it's it's it's it's quite a bit so anyways, because even the small example is quite it's. Quite it's quite big. Okay, so anyways.

A

So that is that this example each our output, for example, you'll see that it added a little bit of code. You know like, for example, the prelude which you normally don't see, which is a actually own Venus, that Roz gives you, but what atomically added they all see? Great Co, for example, we we can access the the core library compiler burdens, which gives you some nice features anyway. So what happens during the each error stage? You get impedances like type type, inference type, checking trade resolution so yeah. The list is not comprehensive.

A

To see three detail, you actually have to look at the at the HAART, which is like super super detailed and then we'll ever look. No! Actually, no, we wonder will give us. It was way too big and anyways skipping. Oh yeah, again, that's that just the the the comment. If you wanted to actually have a look at the diamond, it's quite messy, but this sort of gives you some insight into into how things are broken up so anyways.

A

mmm So the next step is that the HAR is tenth into what is called the mir, which is medium because it says between the hir and LLVM IR. Ir is also intermediate representation, which is how the LLVM represents the causes like sort of the lowest level. So there's like a medium.

A

So that's also. It's also the the concept I also simplified even further here where, for example, your matches are removed and your loops are moves, and then you use you have to use a co2, for example, which is actually closer to what how assembly actually represents it anyway. So what what happens on Mir, something quite important actually, which is Bora checking, which is like a pretty critical part of what makes us special right away. You actually do many memory management, but at compile time.

A

So that gives you save time and there's a few things that it checks for, like wherever's, can be used, an initialized or whatever right, and what else it does some some pre optimization, so that LLVM will have an easier time actually doing doing its own optimization, because it understands because it's a raster itit understands itself.

A

Modern trust, modern LLVM with well so yeah, the output is LLVM ir and then that this example that is example of what Mir output looks like, which is you can tell they're the interesting one is actually that vb0 block where you'll see. If you have a look there underscore and underscore and is actually your argument, then what you do that check multiplication? You multiply that to entry and then there's some magical acid that inside that, for you for buffalo checking enters like that.

A

So there we go, and then this is just simple: l olivia, I our output. It is quite massive, so I only choose a more sort of meaningful example, which is represents. What happens there? If you see that as there's a multiply, they happy and I, don't know what extract they lose the Oz and all of that it's a whole I, think it's a whole talk by itself, but yeah, that's just interesting, looking at what it looks like and then finally we're almost family. Finally, you get to see what the assembly looks like.

A

So, if you look at Maine, they you'll see, I should have put some example. Could yes, you'll see that I was I was actually giving 1983, which is a bit rare in number, and remember, was our if I was actually giving it? It was an argument to a function call and then the this words opinion so basically take 1983 put it in an original called a zero, and then you'll see that, after that, you actually call a function called square and then, if you ignore some of those things, you'll see the instruction called mu L.

A

That's where the multiply actually happens. So remember that a 0, a 0 is just is just 1 right. What what what that does? Is that into register called a 1 put put the result of the multiplication of this to register services a 0 and a 0 at 1983, and you can ignore the rest of the detail. Generated assembly was like 120 lines and yeah note that that didn't go as far as doing optimized code, because that part I don't know I, don't I, don't understand the steps. I thought. Ok, no I will ignore it anyways.

A

So the very final step is actually that LLVM takes the via takes. That quote, generates binaries and then in the end, it actually does the linking way it takes all of the various dependencies and has like the visual. There were dozens, even even with a test for small examples and then finally produces an executable and know that this is not the only one is not unique, rust I'm, not sure.

A

If it does it now, but you'd with examples that I played with you would you would need actually need a link by link uh from the canoe to tool chain which is binary details. So.

A

So yeah thanks I was good. It was. It was sort of easy to lend distance because they say that's a guide. It's a rustic guide wishes which is actually a new project. This I think it was introduced this year so Allen. Sometimes they and I wanted to also give special thanks to respect Paris for inspiring me to give a talk and Russ team is amazing, of course, and yeah, and then rust, intense lights, thanks for to Steve for actually produced that cause yeah, it helped with my doing the presentation, thanks.

B

Okay, sorry, we don't have time for questions now, because we just delayed a little bit. So if you have any more questions, you can find him at the coffee break. Probably, and so please help yourself have some coffee has eat something, and thanks.