Rust Programming Language Rust Hungary Meetup, 24 Oct 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nikita Baksalyar — All you need to know about the Rust FFI (Rust Hungary #2, 2017-10-24)

Description

The world is imperfect and not every program is written in Rust yet, and while we're rewriting everything, we need to deal with this fact somehow. That's where the Foreign Function Interface (or FFI for short) comes into play: it allows programs written in other languages to talk to Rust. It might seem straightforward at first, but the FFI has many non-obvious parts and open questions: how do you deal with ownership and borrowing and how do you use drop flags to deallocate memory and free resources? How do you pass vectors and slices to external functions and back? What is the proper way to work with strings and why do we need CStr and CString? There's no single proper way to solve these problems, but we'll try to find answers in this talk, learning from the hands-on experience.
Presented by: Nikita Baksalyar (@nbaksalyar)
Web developer & Rust engineer at MaidSafe, Nizhny Novgorod, Russia

Slides:

[Rust Hungary Meetup #2 - Mozilla Developer Roadshow Budapest, 2017-10-24, Mozilla Hungary Community Space, Budapest]
https://www.meetup.com/Rust-Hungary-Meetup/events/244086300/

A

Reduce myself briefly, first, my name is Nikita. I have worked as a West engineer at Maine State. There is a company that works, the kind of very good network technology in our entire. Almost her entire code base is in rust, and actually it is one of the biggest projects out there, except for sir, but perhaps service is certainly bigger, but no, we have like tens of thousands of lines of code in rust and we used SFI extensively in our work and we make a fight.

A

We use it to fight to make our libraries usable from other languages like JavaScript and C sharp and soon we'll be adding Java and let them and other languages to our stack. Today. We want to share our approaches to your dealing with.

A

This is fine and structuring our api's with you, and we learned these things in a very hard way because we have been developing in our libraries for more than a year and there are a lot of intricate details and sa5 that you or we can find in the res documentation, but you can find it and for us to let McCallum and in some like obscure pages in the res documentation and I won't be sharing it too.

A

Many intricate details, butters, huh let's start with the basics like what this it's a fire actually at all that's a fly stands for burning function with your face. It isn't basically it's a way to talk to the external world which not every single program in the world is written and rust. Yet, but yet exactly yeah, we are working on it. So yeah we have to talk to other programming languages and we have to find some proper way.

A

Some way to deal with the other programming languages and with the external world, so edify in Russ, can be considered as a standardized way to sterilize the data and exchange it with other programs, and actually it can be compared to HTTP and JSON in the sense that well, it should be.

A

Is a networking protocol and JSON is a data serialization standard, but they are commonly used to like expose your API from your library so that other pro languages and other programs can use it and the main advantage of HTTP and JSON is that they're supported like by a majority of language. It's. It will be a very hard texture task to find a programming language that doesn't support, HTTP or JSON in some form, and we have been safe actually to use.

A

Dom JSON in HTTP I could put structured like a library that exposed that started a web server that exposed a REST API and that REST API was used by a JavaScript program, basically a desktop application, and it talked to like the rust library via the our REST API Oracle.

A

But this approach also has its disadvantages and the main disadvantage that we have discovered is at the point at the time when we decided to start supporting mobile platforms like Android I know oh yeah iOS, and basically there is no there's, no simple way to expose your your API through the HTTP protocol on mobile platforms, because if you can find your library and your HTTP API to a single app, it will work and maybe on Android on is it might be a bit harder. But anyway it's simple.

A

But if you want to communicate between the applications like, if you have two apps and one to one app to talk to the other app that runs the web server, it will be very hard, like you have to invent some like peculiar ways like running your own custom. Vpn protocol, just to talk to the other, the other program and to send your HTTP request.

A

And it's like it's not easy and we decided that to switch our approaches to FF I, basically and FF is based on platform dependent, C application, binary interface or ABI, and it's called platform dependent because, basically, how it works. You can't, if you compile your library, if I compile or your library on Linux, if yeah, you can't reuse the same binary on other platforms like Windows or Mac OS, you have to recompile your library for these platforms because they use different calling conventions and they're like wearing compatible.

A

It's not even know that the system api's are incompatible. They are incompatible on the lowest system, lever layer.

A

But at the same time, you can be sure that, like all api's on all libraries that follow the same, calling the same c calling convention will be working on this platform, like you can talk to any program, any other program that follows the same convention and it will work and it actually works on mobile platforms as well.

A

So C, ABI and C language itself can because conservative is a kind of lingua franca of system api's, and it's likely that you have used C library in the earth developer life at least once and rust also supports these conventions in the form of their rust 8fo file, libraries as almost any other language. So we can use this rasta Phi API to talk to any other language, basically, but at the same time there is no single proper way to kernel to handle our.

A

If fi calls, because, like it's like with any other API, you don't like you, don't have a single standard to structure your arrest, API. For instance, you have to invent your own way and your own convention that to follow there are only like patterns and best practices that we can follow and I want to share like her.

A

The common approaches that we have found at our company and that approach is this approaches to work for us and like I, want to tell you about them, and, let's start with the simplest, like simplest thing that we can find in our in our FFI api. That is a function. So how do we expose a function to the external world and basically, as you can see, it's practically the same thing as you write your function and rust.

A

It's like this word is similar to the rest of the functions, but you have to add some extra meta information. You expose this function as then, as an FF I function, you have to declare it public because otherwise it won't make sense to make it module private. It won't make sense to you like confine it to single model, because otherwise, how would external world the external world will know about your function right? So you have to make it public.

A

First, next, you have to add these externally declaration, and it basically tells the Russ compiler is that it should make this function, follow the EC : convention and, finally, that last bit no mangle tells the compiler that it should also not mangle the function name, that is, it should not encode the function name in the way that only the Raskin bother will understand it.

A

That's practically it for functions, but you also have to know. Another thing about Phi functions is that you can't use rust types in the function, arguments and in the return types, because rest types are very implementation, specific, like just to use the the resolute structures from the standard library in other languages and to expose them and nfo file.

A

You basically would have to re-implement the entire rust under library in other languages, and you have to re-implement it with every single other new rust version, because they're very dependent on and the RUS compiler version and they're very implementation specific.

A

So what we can do is to use only the only types that are under standard, that other languages understand and we basically have access to only the numeric types, either integer or floating-point, and we also can use pointers which are practically can be thought of as numeric types as well, because if you compile your library on a 32-bit platform, the pointer will be just a 32-bit number and correspondingly on 64-bit that platform, it will be just used 64 and of course, we can't deal with only with numbers in our API.

A

We have to structure, so we have to find some. We have to use some complex structures like structs in rust, for instance, but at the same time our structures themselves are different in rust, because we can't use the the rust tracks as they are. They use different memory layout and others see. Programs and other languages want to understand this memory layout. So what we have to do is to basically restructure these structures as C compatible structures and that's what that repartee attribute does.

A

And we also can use only in numeric types, pointers and other see structures for the Strathfield types and to make it C compatible. We have to make sure that all fills in this structure are also wrapper, C or the primitive types or pointers, and that way we can tell the RUS compiler that by using this attribute that it should make the memory layout understandable by the C programs.

A

But, of course, the available means to structure our API sand pretty basic, and you might be wondering how do we actually build some any useful software? Well, we have access to only two primitive numbers and pointers, but the thing is we can combine all these basic needs to structure our API in multiple, multiple ways. For instance, another way to deal with structures would be to use a pack pointers.

A

Basically, a pack pointer is is just a black box that we provide a user and user doesn't need to convey user is not concerned about the thing that this pointer represents. It has like. The user is not like. They doesn't know about anything that is hostile storage in that structure. The only thing that they should be concerned about is that they should store this value and provide it back to our functions to our API, and we can dereference this pointer and.

A

The main advantage of this approach is that we can use rust structures as well, because the user doesn't know about the internal representation of structure and we can basically represent any type of data. We just need to provide user kind of a black box.

A

So this pointer is there for instantly on the rest site actually.

A

And all that what we have learned so far is very simple and basic until we get to the crucial question of ownership and borrowing or the concept that doesn't translate to other languages in a quite a straightforward way, because how we can pass an ownership to an external program at all, like what we can do is to put that value or, for instance, we.

A

If we have some kind of a structure that is a wrapper C struct, we can put it in a box and then convert it into or all pointer and by converting into it's into our ol pointer you're, telling it. Basically, there you're, basically telling there Russ compiler that you should forget about it and the pointer lifetime will be effectively unlimited until you explicitly free it by yourself.

A

So it becomes our responsibility to incur free this box and we free memory by just by converting pointer back to a box structure. Basically, and when this box goes out of the scope, it will be automatically freed by rust. That's why we have this underscore variable, because it doesn't mean like it. Doesn't we don't care about what this box contains? We just need to dereference it, your friends, the pointer construct the box again and when it goes, when this underscore variable goes out of the scope, the box will be allocated automatically by the rust alligator.

A

And for Bourbon that was about the transfer of ownership for structs and simple values, and for brewing it's a bit more straightforward because for Burbank we just can't convert a rust reference to a pointer, because, essentially, all rust references are pointers by themselves.

A

We, because all the reference represents is just as points to some region of memory, so basically a by converting that the reference to a pointer, we're just taking a dress, taking an address of that region of memory and like basically storing that in a book in a variable or passing it down to some other program- and this is not even an unsafe code, because all rust references are essential pointers and we don't need to do any kind of transformation on at all like we don't need to do anything special about it like we can just convert them into we.

A

We can. We then convert them to pointers. Implicitly, we don't need to write this s against native struct construct. We can just put a reference there and it will be implicitly converted into pointer by the RUS compiler.

A

And now that we have dealt so far with simple structures and simple values in primitive volleys and we need to somehow would deal with sequences and Psicosis are a bit more complicated, but there are at the same time, there's simpler than straws, because vectors in rust are guaranteed to be stored in a continuous memory like in the sense that they have always followed one by another. So if you look at this slide, we have a vector of 1 & 2, which are 32-bit values, but actually they are represented in memory.

A

They can be basically a transmuted to a single 64-bit value, because these values are stored right after another, and this simplifies our task of talking to the external world, because we can just take this vector and pass its pointer to an external program and we need we just need to care about the vectors length. We also need your point to hand it all to hand over to the external world, because the vector itself doesn't contain an information about its length like it's not stored somewhere in the memory is stored separately.

A

That's why we need to pass it over as a separate variable.

A

Basically, this thing about the continuous memory. It applies to any type of a vector I given if you have a some kind of complex structure- storage in the erector, if you have sequence of complex structures that they will be a sorority as a continuous memory as well.

A

And so the fact that vector memories continues actually helps us in developing code and Russ, because it's very fast and efficient. If you pass to an external program, all you need to do is to get an address of this vector and it's like very cheap operation.

A

Ownership is a bit more complicated, however, because vectors have some implementation, specific details that we need to preserve. Well handing when we hunt the vectors over to external programs like in particular, we need to have besides the vectors length, we have to also pass the vectors capacity and the capacity is like we can live without it's in the external world, like the external world, doesn't need to know about the directress capacity, because it's a rust, specific implementation detail in the standard library. But we are now we can't get rid of it.

A

We have to pass it to an external program, because otherwise we won't be able to get that vector back in our program when we need to free our memory. So, basically, what it means is that we need to preserve the virtual capacity to tell the rust allocator to inform the rest allocator about the full vector information so that it can delegate these are vector when we get it back. When we get the ownership of this vector back.

A

And you might have noticed that on the previous slide, we called that man forget function and that might be a bit neutral, but it is a very important part because we need to make sure that crust will not the allocates have this vector prematurely.

A

So what the man forget does basically, is that it tells the Torah's compiler to forget about that variable about that vector, and they forget that this variable exists even existent sashko and mean the fact that rust, the rust compiler forgets about it.

A

It won't delegate this variable when it goes out that scope out with the scope, until we explicitly tell the compiler to do so.

A

And after we have provided the user with the full information about the vector and we handed the effectively, we handed the ownership of the vector to the end-user. We also need to provide the user with only means to deallocate it and the simplest way to do that is to basically provide an MFI function, that the user will call when he wants this vector anymore.

A

This function should take three parameters like the in this similar, in the very same way in how we constructed the vector with to the vector pointer, the its length and its capacity. We need to provide the exact same variables when we delegate it and we basically called this a vector from roll function, and we reconstruct the vector from this information and then when it goes out of the scope, the Raskin bother will freeze automatically.

A

So the disadvantage of finding or transferring ownership to external programs is that now it's the user, who is responsible for delegating this memory to prevent memory, leaks and it's very fragile concept, because users are prone to forgetting about these things. And it's like it's very easy to produce memory leak to cause memory leaks because of that, and we have to be extra careful with transfer of ownership.

A

As for taking values from the external world into rust, it's actually much more simple concept. I just need to have to worry about the pointer and the length of signals, sequence. We don't need capacity because we can do the capacity from the length and we can construct the vector ourselves.

A

So we take the pointer and the length and we convert it into a slice first, and then we can convert that slice into a vector by calling that two ohm's function, which basically copies the memory. If we won't call the two owns, the slice will point to the exact same memory region, and if this memory region is D allocated on the other side, then slice will point you and effectively like the allocated memory region- and this will be undefined to here and actually speaking of slices.

A

We can export them too, and it's not too much different from vectors, but obviously we won't be able to export the ownership of the transfer ownership for slices, because slice of themselves are basically borrowing of valleys and they are basically just references and.

A

We need to use a spatial construct like spdr, because we can't convert these references like if you remember I say that our references are basically a relative pointers in rust as well. But you can't convert your reference to a slice into a folder into a pointer because slice pointers are fat. Pointers like they're, not 64-bit or 32-bit, depending on your the system, they're actually contain 128 bits because they also contain the length of the slice.

A

So what we need to do is to use this either as Peter or as mutable PDR, to take our pointer for a slice.

A

I also, it's very important to make sure that the lifetime of your slice of the underlying sequence, either an array or a vector, is long enough, because otherwise, if you, if your vector will be de-allocated, then the slice will point to a dead memory region and it's it will be undefined behavioral or again, which is not a pleasant thing.

A

So strings they're, pretty much the same as vectors, because in rust strings are basically a vectors of you. Eight bytes and they just contained, contain strings that are encoded in utf-8, but we all have to be. We have to be extra careful with passing rust drinks to external programs because they are not guaranteed to have no zero bytes.

A

So basically, what that means is that they are not compatible with a standard like C string Convention, which says that the length of a string is determined by the zero byte at the end of that string and then rust their structure it in a different way. They have a length of a string along with the rest of the information, so we can't just take pointer over string and pass it to an external program. We have to do some extra steps, like basically the same steps as we have applied to that dress.

A

We need to take the pointer for a string. We need to take its length and capacity and we it's very important to preserve strings capacity as well as vectors and freeing the memory is basically the same as for vectors like we can start. We just reconstruct a string from there are all parts that we get on the previous step and when it goes out of the scope, it will be allocated by rest.

A

But there are also several other different stream types which are suited specifically for fi purposes, and it's actually recommended to use them instead of strings to pass. To talk to external programs, because these string types the yesterday fa5 module are more conventional. So both types, a string and sister are correspond to the standard null-terminated string and they will be. You can be sure that when you pass a pointer to a sister or see straight, they will be compatible with the anyway to work with strings and with other programs.

A

So, to construct a c string which owns its memory region as just as the string struct itself, just call the constructor function, c stringing you and it will do a full memory copy or the strength that you have passed to it and it's well. It will be delegated when you want it to like when you call there when it goes out of the scope or if it, the semantics of allocation and ownership, are practically the same as with recommended with the standard, libraries, rust strings and borrowing.

A

An ownership of the c strings are much more easier than with much more straightforward than the standard cross strings, because we can just basically take a vector of c string and pass it as as it is, and we, if we want to pass on worship of it, we just convert it into a roll pointer by calling that intro and when we want to free it.

A

We just record this string structure and when it gives out, the scope will be allocated, and you might have noticed that this interface is pretty much similar to what we have seen already with boxes and structs.

A

And it works essentially the same, and it follows the same convention from the rest standard library even with the same function, names in the counterpart type sister, which has semantics that are very similar to reference to a string in rust that n % is TR, think so it doesn't own any memory by itself and it can be converted into rust string reference in a very straightforward way by calling the twister function and then, of course, you can further convert that reference to strength into a string by calling to old which will copy memory and hand the memory ownership to this drum to the street.

A

But creating US history isn't safer. You know you have to be extra careful to not pad not perfect. You have to be careful to provide a ball. It C string that is null-terminated, because otherwise, if you provided an invalid pointer or some a pointer to not a ball its string, it will be here undefined behavior again.

A

And so far we have converted the basic. We have covered the basic types of structures, sequences and strings, and so, let's get that to a bit more complicated topic of burying and ownership, and we borrowing volleys from rust is a bit more complicated task, because we can't ensure that the lifetime of alleys which we have borrowed and is guarantee it's not guaranteed to that this value, this lifetime of these values will be indefinite. Because for is this, imagine how you would pass such a barrage pointer as a return volley from a function.

A

So when you, this value, gives our goggle goes out of the scope of this function. It will be de-allocated automatically by the RUS compiler and now, when this wire gets, but when the pointer to this, while it gets you the your external program that uses your arrest library, it will be a like invalid memory. So how do we deal with that? You know.

A

Well is an especially a hard test when you deal with the garbage collected languages like JavaScript mmm now, you might think that this can be elevated by transferring ownership, but it's not sexually the simpler, because so a user will always have to care about allocating memory, and so it will is very easy to produce.

A

Memory leaks and it's very easy to strike of beach from memory regions. You are already allocated in which memory regions you'd allocated and is usually out to use a user, and we can't affect it in any way. So it's a user's desk and it's not- and it's not good, because it's much better to you. If we, if it's us who care about the memory.

A

So let's say if we have encountered multiple cases of memory, errors and undefined behavior related to borrowing and ownership, because it's really hard to write unsafe code and rust, because we don't have the res compiler guarantees and we constantly bumped into segmentation faults and memory, leaks and memory errors, all kinds of them. So we have decided to learn more ways of structure and our API so and we considered several other options. So what we have decided to try is to structure our API around callbacks.

A

It is kind of an unconventional way, but basically we use callbacks to pass all data to external programs and it simplifies the task a lot. If you look at the implementation of this callback thing-, we basically pass the pointer and all relevant data to users, callback as the callback arguments and monthly execution, for it gets back to our function like the rust func, when the execution part gets back to it.

A

It continues the execution and then the vector goes out of the scope and then it's dropped and the memory is they allocated automatically the so user doesn't need to deal with memory allocation. Then, anyway, it they doesn't need to. They don't need to like the allocate memory or care about it at all. The only disadvantage of this approach is that is, it will have to make a copy of the entire data it will have to do.

A

The user will have to copy all variables and all structures that we pass to them and it's not a big deal as we work with small structures and small data types or a small actress, but it one word that read for vectors that contains several megabytes and megabytes of data, for instance, and for heavy vectors. We can combine the call back approach with the ownership transfer and make it a users responsibility to reallocate large values. So it's unlike there is no single answer.

A

There is no single way to structure our api's, but we can combine several approaches.

A

And this way, when we are using callbacks there's an advantage that we can now automate the allocation for our C starts. So, for instance, if your C represented as if you're a structure that is marked as C wrapper. If it contains a vector or a string fills, you can delegate them just by implementing the drop trait, which basically consists of just a single function.

A

That is called when your structure goes out of the scope and the recent addition to the RUS compiler is very nice that we are now are able to implement the drug trade for wrapper, see, structures as well, and luckily the when we deal with this clawbacks approach, it's very simple to deal with it from the languages like C++ or Java Script, because, for instance, in C++, you can just use one of the veteran constructors to copy the values that are provided by rust. And it's very straightforward.

A

You just copy the value and you can deal with it in any imaginable way afterwards and you will be memory safe and for JavaScript, it's and simpler because you have to you have an implicit copy of value when you construct an array. So we just don't need to do much to make it work and I'm.

A

Also a big advantage of this callbacks approach that you get the synchronous codes working for free and without any changes, because so we can use futures and Roskilde and the external interface will be transparent for you and user, because interface will be based on the very same callbacks convention, and we have successfully used this approach in our project, which largely depends on Tokyo and futures for input and output and the network communication, and it works really well for us.

A

So to summarize, burying and ownership has a very complicated topic and if we apply to a fight like it's a complicated topic in the in and of itself and rust, like many people, a lot of people struggle together too, and they learn to us and it's even more complicated. If we apply it to if a fight and again there is no proper, a single, proper way to deal with FFI at all. You have to find your own your own way and you have to balance the approaches for your particular application.

A

But it may say if we have found that generally borrowing is very much- is much more simpler than to do with dealing with ownership and boring, because with ownerships the ownership transfer, we are making our user responsible for all memory box and is a spatial heart in JavaScript and other garbage collected languages, because you can't control the memory allocation in a straightforward way. We have to like deal with black magic to make it work properly, how certain practice really hard? You use a combined approach for heavy binary object.

A

We use a combination of a back pointers and ownership transfer and for the rest of simple functions called box. Callbacks work really well for us, and there are links to the FFI utility library that we use. It contains many useful details for projects that follow this convention and there is also a link to our russ project that exposes the FFI interface in the way that I have described, and you can look at our implementation for an example of our an API that is structured around codex and that's basically, it and ready to answer any questions.

A

If you have any.