Rust Programming Language RustConf 2018, 11 Sep 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustConf 2018 - Embedding Rust in C/C++ by Katharina

Description

RustConf 2018 - Embedding Rust in C/C++ by Katharina

We all know that using C/C++ code in Rust is not too complicated. extern "C", bindgen and build.rs scripts make this pretty easy. But let’s challenge ourselves: what about the other way around? There are huge C and C++ projects that could use some corrosion.

In this talk I will not just show simple examples of how to use Rust modules inside larger C/C++ applications, the common pitfalls you will encounter, and to deal with them.

A

Hi everyone, thanks for coming by I'm gonna, tell you the tale of two unlikely friends that are up there and they are very much in love. I'm gonna talk about myself for a second here, I my name is Katerina Faye I'm, an active open source contributor I like tea a lot and sometimes I, play around with embedded hardware. I do some rust things who would have thought so? I was a contributor to or I guess so, I'm a contributor to the SEL. I working group I write too many crates.

A

Most of them are probably useless, but I've met a few people here who told me that, like one of them isn't so that's pretty cool and I'm a member of Berlin RS, which is a an implementation group, that's starting out to be so. We we work on stuff. Together we organize workshops and little conferences. For example, a few weeks ago we had all mini cons about bind gen, which was pretty fun so before I really started talking about this I want to answer this simple question: why, as in, why would you do that?

A

So there are lots of I. Have people who tell me that if you have to reach for FFI your day, has already basically gone wrong and I think that's very wrong. I think there's very good reasons to use FFI between languages to make languages talk to each other, and especially with rust and native code, to embed rust into a said native code.

A

I could probably like this could be its own talk, so I'm just gonna list like a few up there, and if you find yourself in the situation where you, where you want to grab like where you want to reach for an FFA interaction between C or C++ and rust, just know that it's a valid thing to do, even though there's people on the internet, who will tell you otherwise but I, realize that that's not really the only the only meaning of the word. Why? It's also?

A

Why give this talk so about six months ago, I was working on a project and I wanted to do exactly this and I encountered a lot of roadblocks and a lot of stumbled.

A

A lot and Russ promises to be efficient in its bindings and I mean I know what the people who wrote the website wrote meant with efficient, but when I have to do a lot of research and there's not a lot of documentation about it and basically all the knowledge that I get for this is from blog post I, wouldn't exactly call that workflow efficient.

A

So my hope was to engage people a bit more in documenting things and experimenting with things and building tools that are more publicized than what already exists where everyone basically has to reinvent the wheel.

A

So the second thing I want to get out. The way before we really get started is I want to talk about abis, because a lot of people don't know this. Abi stands for application, binary interface and it's it's basically like an API, but for your linker or your compiler, it's not something that humans would ever interact with. It specifies how a function looks at memory.

A

It specifies the way that data is laid out in memory and, depending on the ABI, you can make certain pieces of code talk to each other, whereas if the ABI is wrong, it won't link and it won't compile in rust. We use the external keyword. So this is a rusty talk. After all, you will sometimes forget in rust, we'll use the extra and keyword followed by a string to specify the ABI that we're using, for example, external c4 structures and enums.

A

We need to use the repressive macro to tell the compiler that we wanted to look like there see variants. Now you might have noticed, that's a that's a string up there. Can there there's other stuff that you can put in there by default. It says external rust. You can put that in your code, it'll, not change. Anything you'll have made your course code worse to read, but it's not gonna make a difference. There is a lot of different abis that you can target.

A

Most of them are platform specific and usually you will just use externally unless doing something super funky, at which point my hats off and good luck. Rest doesn't have a stable ABI in so when rust was made stable in 2015.

A

That was a promise that code that you wrote one day would still work the other and that promise extends pretty much to the language itself, so the the syntactic features and the standard library it does not, however, extend to the ABI there is this issue by Steve open number 600, there's that Russ might want to define an API at some point, because this is also talked about. C++ there isn't one standardized, C++ ABI, it always very much depends on the compiler that you're using and a lot of different things.

A

This could be another 30 minute to talk so I'm just going to gloss over this. There are certain things that are standardized, but most of them aren't. So when we deal when we talk about the C ABI, there is no such thing as a C avi, because she never standardized or specified in avi what we mean instead is whatever platform that we're building on and that is, and hopefully stable, so yeah a little background on that. So, let's get into some FFA and I want to do it the other way around.

A

Then this talk is advertising first, because I think it's. It demonstrates a lot of the same principles and it's a lot better documented. That's why I'm usually called a boring FFA. There is an entire chapter or like a section and the rest book about it, there's a lot of blog post and just generally, if you want to do this route, there's great tooling and generally people know how to do it. So the way you go about using C code with rust is by using external functions that are declared.

A

Has the ABI that you want to use. You need to wrap those function, calls and unsafe, because the compiler can verify that they will actually be there when you run and you may have to make data compatible in a way that C can understand, because rust is smarter than C. In this aspect, the work falls onto you with a bunch of stuff. So this would be an example. We have function called reverse and it takes a string and returns a string or sorry a constant character array.

A

We call this function in an unsafe block and you can see we here. We use the cstr type to transform our string slice into something that C can then understand in the reverse, like I've left, the the reverse, you would have to do the same thing. I won't didn't want to make the code example unnecessarily complicated, so there's two type: there's two modules and the the standard library one is OS raw and FFI, and both of them deal with transmute, like transforming data into a form that C will understand.

A

Os raw, our primitives, whereas FFI are things like string that becomes a C string. It's an owned allocated string, there is STR becomes CSTR, which is just a string slice and, and then there is lots of primitives, so void and c becomes c void and etc. I'm not going to go through all of them when the documentation of this is really great, there's good examples in the standard library, so this is usually not an issue how to like transform data.

A

Okay, so let's turn this around and let's, let's look at what you actually have to do to call some rust code from C. We use the same mechanism with the externship as previously, except with a little difference which you'll see in a second. You have to take data in a C form, so whatever data C provides, you then have to make rust compatible, not the other way around, but there's also a thing.

A

You have to declare your functions as no mangle, because the rust compiler mangles function, names differently than a say, GCC would expect, and as such, you need to declare them as no mangle, so that the the c compiler that then links you and your application can actually find your functions. Usually if you have function errors, it's probably because you forgot my angle, it's because I forgot my mango at least so. This would be an example we, so it's a function called reverse. It takes a string, returns a string, it's no mango.

A

It's ex turns the implementation, isn't really important right now. I wanted to do an example where safely, like reverse unicode and there's a great credit for that, but the example ended up being like half my slide and I just took it out, because it's really not relevant at this point.

A

You will need a few more fields in your cargo towel, so there is the lip section which I don't usually use unless I'm doing this sort of stuff, you can give it a different name if you want it to be different from your normal trade name, and you also have to provide a crate type. Cdy lib will generate a shared object file where, as static clip will create a statically compiled library, as the name might suggest.

A

But something to keep in mind at this point is that cargo does not own your project. So cargo will build your rest code, but it's not in charge of your actual application that falls to whatever build system you're using with C or C++ I. Had a wonderful conversation over lunch about how wonderful the situation with C++ build systems are so whatever, whatever you'll use in that regards will have to understand how to link in a library that is generated by cargo.

A

One thing so in this example here I'm using C make all of this code will be online after like the conference so that you can go through it. One thing of note is the reverse or dot H header, in which we declare a function, which you might recognize so basically, every function that you're rust API exposes needs to then again be declared in AC header, so that your C code can actually use it.

A

Okay, so so far, so good calling this from C is then actually pretty simple. We include the Reverso header, we generate it like we use a Unicode, a very Unicode greeting, and then we call our arrest code. That will reverse this this for us and when we call it, we get that out. Funnily enough, the crater was using couldn't deal properly with all of the composite emojis, so um yeah it reversed them individually, which means computer.

A

A woman is not the same as computer woman, so that's pretty much it then everything works like share and subscribe. Ok, not quite so, I see some problems here.

A

The first one is that I'm, notoriously lazy and I, don't wanna write headers and not just because I'm lazy, but because, if you have a header in your repo and it's out of date from your library and then you get weird linking errors and you don't know what's going on and then you find out that there's like a merge conflict from three weeks ago, because someone like push the button too quickly or github, and that's why your application doesn't build. That's not really cool the the other thing is this was a very optimistic function.

A

It returns a string. What could possibly go wrong, so maybe we might want to fix that, and the thing is rusts makes it very easy for you to forget about memory management. So how does that happen? And the last point is less of a problem? It's more of a philosophy. I guess I want to talk about what even makes a pretty API, because this isn't just about making rusts to talk to see. You can make anything talk to anything. This is about efficiently talking to C, which means making it pretty.

A

So, let's start with the easiest thing first time this could be another 30 minute. Talk, I, think you're, seeing a pattern here, it's a lot of stuff. So there's this project called sea pines gem, which is pretty awesome. It's basically like bind gen, but in Reverse, so you pointed at a crate and then it'll generate header files for you. You can either do this in your build RS in rust code, or it provides a CLI that you can then invoke in your favorite build system which you probably have yeah.

A

So when it comes to actual build system support, you have to do a lot of work, so usually I find that writing a few bash scripts. That hook into some make file, or something is usually the easiest to deal with like invoking cargo building it than taking the library from the target directory, putting it somewhere that your build system will understand and then letting the build system do the rest like it would normally do.

A

But this isn't like this isn't a thing way. You can just go somewhere and look it up. It's figure it out yourself which isn't very nice. So let's talk a little bit about memory management. The thing that rust doesn't really need you to handle so by a default. When you create an object and rust, it creates it on the stack which is really convenient, I guess, if, if you don't need it to be on the heap and in most situations this is fine.

A

There are situations where you might want it to be on the heap, and then you make that explicit, and especially when we talk to some other piece of code where we have no idea what it's doing, what it's doing to the stack. It's not a great idea to use the stack as permanent data storage.

A

So put your troubles in a box, the box is a heap allocated pointer in the standard library and whatever thing you put inside of it will then be on the heap and you this code up there, the extra and see the box thing. What that does is return, a pointer to whatever you put into memory and the thing is we declared our structures repres I, which means that C will be able to follow that pointer and then do something with it.

A

The other thing is that boxes retain type information where they store the data, which means that you can also get the data back. So you can, if you have some piece of code, say PTR and I just realized. That code example is wrong. It should be PTR that you dereference not CTX, so whoops you get a C void which can be literally anything and then in an unsafe block.

A

You essentially cast that to be a mutable reference to my thing, because you can follow the pointer to wherever the thing is a memory, and then you can use that and then you can in your rust code that follows safely access all of your stuff without having to worry about something going wrong.

A

This is what this looks like from the seaside. So basically, yes, the mixing function just returns. A pointer. The my thing pointer is then on the on the stack and we can deal with this. Something to remember, though, is that you can't make your application memory safe, so the rusts guarantee ends at when you return back into c code, which means that you should not make any assumptions about persistence of data.

A

It might be gone, something might write over it or it might just die before you can do anything with it, so always be prepared when you build these api's for them to have a certain amount of hardening against C code, doing something stupid, even if it's like unintentional or a bug, it's something that you have to deal with.

A

Speaking of things exploding in your face, let's talk about errors, so errors and and generally reporting. If and when things go wrong and C are a little weird. So in rust we have these amazing types result in option that let you return something from a function and it contains some type information or some state information about what state it is in.

A

The result realistically communicates the state if it's okay or an error and the option just communicates of its existent, and that is super convenient because you can just return something from a function and it's I don't have to explain this to you. You probably know this well.

A

This is something that you can do in C, but before I go there. I want to talk about something, because this talk is called C and C++, and I've really only talked about C. So far, we have to split this up in errors and C and errors in C++, because errors in C are something that we can deal with. We have a binding to C code and we can deal with interacting with C code.

A

Errors in C++ are either exceptions or some class that uses a template to like, essentially be the same thing as a just in C++, and that's not something that we can really easily interact with, and even if you could, it would be very dependent on your compiler version and it's not something that you ever want to touch.

A

So whenever we we talk about errors and whenever we talk about building pretty API, is you have to keep in mind that you build a layer to see from roster to see and then your C++ code, if you have it, we'll have to use that C code just the same way like externally as your rest code would so you basically have two parts that sort of communicate via a common ground that they both don't really like, but they just put up with it.

A

So this is something that we can do. We we can emulate a result in C. So this is a rust struct. Obviously, and what we can do is we. We have some struct over a generic T. We have a box in there so that C doesn't have to understand how big this is because in the end for us, it's just a void pointer and we can mark that field as ignore me or like hide it or something.

A

As long as the structure knows that there is a pointer in there that points to something it doesn't have to care.

A

So this way you can deal with returning rust objects that you might want to give to other rust functions, but C doesn't have to deal with, while also returning an error code that when it's convenient- and this is what this would be used, then so we call some function.

A

This is C code, we call some function called my function and the return value is this wonderful result type and after the function was called, we can just call code and if, if that's a, if that doesn't trigger- and we don't have to deal with an error- it's a valid thing.

A

There's another pattern: in a lot of C libraries and I've been trying to look at how it evolved over time. It seems to have like a lot of older libraries use this, but a lot of humans as well.

A

So it's not really sure, like I'm, not super sure about a trend that used double pointers and so as a little bit of a refresher, I guess, because rust developers usually don't deal with pointers directly because we don't have to the idea behind this is that you provide the pointer to a pointer to something and then by dereferencing one of them. You gain access to the other.

A

In this case, we have a variable which is a pointer on the stack and we give a function, a pointer through this pointer on the stack at which point in the function we can dereference the first pointer and right onto the stack outside of our function. This is a pattern that you can use and C to do. Error reporting by always returning an integer from your function. You have the ability to communicate if something goes wrong.

A

If it returns a zero, it's fine and happy or it can return some error code or an enum or whatever, while handling data return types with double pointers. Where you make some field on the stack. So we have. This function called get client and we have a client pointer on the stack and then we give a pointer to this pointer to the get client function at which point it'll fill it in and if there is no error, the client variable will then contain the pointer to a valid client.

A

You can do the same thing for initializations. You can also provide parameters, obviously that are have nothing to do with any sort of return, value and just yeah. This is the same pattern, essentially so from rust. This is super organ ami to use, basically what you're getting so, let's take the initialize function where we have a pointer to a pointer and a port. We get in a star star, star, mu, star, mute, C void and a port. We you can check if the port is valid and then returned like an error.

A

If you want to allocate like port -1 or something, and then what we do is we generate like we create a new server in rust, we put it on the heap with a box, and then we use the box into raw function.

A

What that does is it destroys the box and turns it into a raw pointer, so rust does have pointers, they're, just unsafe, and so whenever you want to deal with an actual raw pointers or something when you can seriously shoot yourself in the foot, you have to put in an unsafe block, and you then also have to deal with the lifetime and the cleanup of this piece of data yourself.

A

So you can actually this way you can leak memory quite beautifully, because you can put something on the heap, give yourself the raw pointer and then never do anything with the raw pointer again, and it's just. It remains on the heap.

A

But so it's pretty ugly from rust, but if your priority is to be in line with the rest of a CA, B C API that you're trying to match this is really great because it feels exactly the same. You can even like the server struct can be exported to C so that it can directly access the port if it.

A

If you want that to happen, and if all of your other code uses this pattern or if some of your other code uses this pattern, it's a good way to sort of match that, and this is like literally figuring out the box into raw was why I then wrote this talk, because there's like no documentation about this well, there's now, but so, let's, let's talk about C++ well, I already mentioned that we'll have to wrap some stuff and what we can do is so I'm, not I.

A

Don't write that much C++ code I know that people shout at each other about whether or not to throw exceptions. So I have one example that throws an exception and one, but that doesn't basically what you will have to do is include the the header that is generated for you. The same way that you would include any C header from C++ and then you would probably want to write a module around it that calls into it and then abstracts away any sort of errors into the system that you're using in your code.

A

This is very specific to your project. So if you use exceptions, then exceptions or if you use like a result, structs that your immunity- because there are things like that, you can use templates or something you can return that instead. So at this point it's very much down to you and your project, what you need.

A

The important thing is that, with the patterns that I showed you and see at least you have a good basis for communicating errors in the first place, because the worst thing you want to do is in your C++ code, just throw an oopsie exception. That tells you that something went wrong and you have no idea how to debug it.

A

Okay, so I have five minutes left and I hope that until this point I've shown you some useful information and maybe shown you where maybe the ecosystem needs more work for the last five minutes. I want to answer this question, which has been haunting me for the last six months or so so I think if this isn't your reaction by the way, then I'm deeply worried about you and well. The answer is yes, you can so, let's, let's look at. Let's look at what exceptions actually are.

A

Maybe you've never worked in a language that has exceptions like C or rust or you've never looked under the hood deeply enough. This is based on the C++ LLVM documentation, which is really great. So if you're curious about that, definitely read that it's really well-written, so there's really three key words involved. Sometimes four, but like we only care about three right now: try throw-and-catch what a try does is create a sort of landing pad that you can jump into later and I'll go into the details in a bit.

A

Then, when you call throw at some point in your code, what you do is you start walking up the stack at the same way or similar to how you would unwind your stack. You go up until you find the landing pad and you jump into it.

A

Catch blocks are then basically function, pointers that you jump onto and like call them.

A

The thing there is that there is a bit more logic to handle which again I'll deal with it in a second, so the try, the landing pad actually more information than just how to deal with an exception, because the the try block can be ended in two different ways: either you can have thrown an error or you can have not so it needs to understand the difference between or it understands, the difference between an allocated exception that sets somewhere in a buffer and not at which point it'll either choose to evaluate the catch clauses or it'll.

A

Just continue in your code as normal. So the catch is actually a filter, because in C++ at least you can catch for multiple exceptions at the same time, kind of multiple catch statements or a catch blocks. I guess that that filter for different exceptions- and you can even like use the pipe or logical or operator on them- and maybe you don't have a catch block which catches the exception, and then it gets wreath roan in the hopes that someone upstairs will care.

A

The last part is actually the easiest, and that is nice, because that's what we care about, so the throat keyword gets replaced by the compiler with calls into Lib C++ that allocate an exception, define types with certain mangling rules and then throw the exception.

A

So this is the talk about rust and I'm. Sorry, if I went a little far there and introducing exception RS, so I wrote a create that can throw exceptions, C++ exceptions in rust. This is what it would look like. So you have the exception thing, and then you can like throw some structure with a whatever payload. You want how this is implemented. Well, there's no Lib C++ binding instrument bindings in rust, which is probably a good thing. So instead I wrote a little shim layer and see that just invokes those functions that are exported.

A

So those are the two functions. There's like a third that it's not important for this example. These are the two big functions that you need to allocate an exception and to throw it and those declarations are in the C code exactly that way, because I I can't provide those and I am compiling my code in the hopes that, before it's run, a C++ compiler will look at my code and go oh yeah.

A

I know where to find those so you're very much dependent on your project being compiled with a C++ compiler, or else it won't work, and this is what that looks like now. So you, the car, go bolt release as so that I build the actual rust code that throws the exception and in then you invoke a jeep jeep, Laplace or C++ compiler. That links in the exception, throwing library and then it's well I didn't handle it, because I thought it looked nicer too, to die with a custom rust exception.

A

Alright, so there's one more thing that I've been working on and I kind of got it to work. Answering that question.

A

And the answer is yes, but I'm out of time and I will write a blog post about that, because it's a whole different like can of worms and I died a minute or so. Thank you real and I.

A

So follow me on Twitter I do a lot of posting if you can put up with that or if you have, if you have questions or want to get in touch with me, if you see this online at some point or a year, you're like shy or you don't meet me write me an email, so I I do like getting emails.

A

I want to thank my employer for our systems, the surname you might be familiar with it's the company that I'm working on like in with James- and they sent me here so I want to thank Mozilla for allowing me to be here and like providing accommodation and stuff and I want to thank all of you, the rest community. I I really like the the keynote this morning, because I talked about how the the rest community can be inclusive and welcoming, and it's really one of the reasons why I like contributing to rust.

A

So thank you and yeah have a nice day.