Rust Programming Language Integrating Rust, 4 Aug 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GUADEC 2018 - Federico Mena Quintero - Patterns of refactoring C to Rust

Description

Last year I gave a talk on *why* it was desirable to port librsvg from C to Rust. That talk showed cool things about the Rust language, mostly centered around expresiveness and memory safety.

This time, I want to show you *how* the librsvg team (we have a team now!) has been doing the port, gradually, steadily, without breaking client applications. We will present common patterns that show up when refactoring C to make it easy to port to Rust. We'll show how the first pass at Rustification works, but it is ugly - but how a second pass can turn it into beautiful, idiomatic Rust code. We'll show how C code with no error handling can be turned into Rust code that checks and propagates errors thoroughly.

The hope is to show that we can give the low-level GNOME platform another 20 years of life by porting it to a better low-level language.

CC BY-SA 4.0

A

Hello, hi everyone thanks for coming. Can everybody hear me in the back? Alright, let's get started last year, I gave a talk about why I wanted to start porting libros VG from C to rust. This time, I hope to tell you how a bit of how we have been doing that now. That I know roast a lot of a lot better, and now we actually have a team which is no longer a one-man shop. For those who don't know, this is Ferris she's.

A

The official rust mascot now repeat what I said last year: I don't want it to be 2018 and be using a language that doesn't have a cute mascot. So I see a lot of old-timers here so for the few new people in the audience, libera libera, libera sweetie is really old. It started in 2001, it takes an SVG file and it renders it to a Cairo context. That's all it does. It's used in a bunch of places. Genome Wikipedia uses it to render their SVG diagrams enlightenment Emacs image magic.

A

The way image magic can render eyes with ease through a libris wiki and people use it for all sorts of random stuff. We have a team now I'm very, very happy to have people working on the Liberace G, even molitor's key she's, my Summer of Code intern he's converting the filter effects from sitarist. His air give him a big applause.

A

Is Jordan here now he may be stayed in the other room anyway. Jordan has been doing our continuous integration infrastructure and porting. Some of the Roscoe to Butler Borrelli. One of the really really old, timers in genome, has been converting a lot of the liver switch internals to rust, I'm very happy for that. He used to be the 'edit maintainer and a bunch of people doing excellent work here.

A

Most of you use liberals Vigi, probably without noticing the way we render SVG icons, the application icons, the decorations in the in the windows. That's with livers Viji, it happens behind the scenes.

A

The API is very minimal and the way the only one rule we have for the roast port is we don't want to break existing code code should keep working with no changes to the API or to the ABI people who distribute live our SVG, on the other hand, I'm sure they have seen changes in their workflow and there and had their workflow disturbed I'm. Sorry that distros had to start including liver, SVG I mean roast into the tool chains anytime.

A

We in these drawers need to include a new programming language tool chain is a major major pain in the ass. Rust is not an exception, so I feel I feel your pain. We've been doing this at Sousa, and it's just as painful as anywhere else. Firefox is pretty much compelling us to have a rough tool chain. I'm, not complaining, I just wish. Somebody else was doing it, but anyway, libris VG is 20 year old code.

A

It was being written while the SVG spec was being written. So there's a lot of loose ends in there. Lots of ambiguities in the spec had to be resolved in the code, lots of unfinished parts of the spec had to be decided. What to do. In the code the Paris VG comes from comes before the we had Cairo. We used a library called libras Lib art before Cairo, and it was very, very clunky. The presidency G got ported from Liberty to Cairo, so there are some Liberty isms in there.

A

Still we had a Mozilla plug-in before Mozilla could render, as which is on its own Liberty's VG, provided a plug-in to to do it inside the browser it crashed all the time we no longer have that. Fortunately, we even had a GT carry theme engine based on this VG's, and now that GDK is kind of sort of scalable. We don't need that anymore. I.

A

Got this book a few months ago, and it's really good: it has a ton of techniques for refactoring code that doesn't have any tests and doesn't have any documentation for refactoring it gradually into something that you can understand and that that you can write tests for everybody, everybody who has to write tests for code. That is very interconnected knows how hard that can be. So this book has a lot of tips for how to unravel it.

A

Ask me later, if you want to learn where to get a copy, wink, wink so liberate. Vg is not really a typical gunam library. Our API, our public API, is very minimal. It consists of a single, very small, G object, unlike say ttk, which exports you know tons and tons of the objects internally. We don't just use the object. We just use plain structs in C we just leave XML to to parse the XML in the SVG's will use lip Krakow to parse CSS. We use a lot of Cairo.

A

We don't do any fancy I, all just you know, read a file and not even that, because most of the time we just plug to addy iostream and that's that we don't have threads, we don't have cob jects. Your coding. No mate might be very, very different. So take this stock with a grain of salt and expect some differences if you've tried to port other code to to rust.

A

One point I want to make in this talk is that you don't need to know a code base very well to start improving it or to start refactoring. It reporting it I practically didn't know the liveries VG code when I started, maintaining it I had done a few random bug fixes in the past, but they were very, very localized.

A

You know, fix a memory or fix this array index, but I didn't really know the code and I want to I want to reassure you that if you start refactoring some code gradually using standard refactoring techniques, you will get to understanding it eventually. So keep that in mind. So the general strategy, if you want to port a sea library trustees, get the build system going first feel free to copy and paste all the other tools crap.

A

We have in libris which, if to build a roast library and link it with your main sea library, that's boilerplate, there's some work being done in using messin, but it's not finished yet porting computation only code is a lot easier than 14 code, which uses a lot of API calls and roasts lets you wrap on save see. Api is what does unsafe mean here. Unsafe means that they have to refer the reference random pointers. They don't have any guarantees to the they don't have any preconditions or they don't. Let you encode preconditions.

A

If you wrap that with say, frost, ap, ice people who are learning the Rose will come to terms with safety and on safety that works very well and finally, opaque pointers and opaque structures. They are your best friends, they let you have an opaque structure in C that gets accessed from rust or an opaque structure in gross that gets accessed from C, and that works really really well I. This is a slide from last year's talk, so we use out two tools in liver SVG.

A

We built two libraries, the main C library and the Rost internals library, and then we linked the rest internal flavor into the C part. It's out to tools, trickery so just feel free to copy and paste stuff. We can look at this in details later I have musical interlude without music, because I'm sure none of you will have any problems imagining the music so.

A

A

So just imagine the music in your head, I'm sure you know it.

A

A

Where's my focus.

A

All right, let's begin with the code, there is something we have done a couple of times in liebherr SVG and basically we had this massive struts full of data. You know, and one thing that worked very well- was to start moving field by field from the c s-- trot interests trot. So let's look at how to do that.

A

This thing, our SVG state have a pointer, no okay. Our SVG state is a big structure that holds the CSS properties for each element in the SVG file. So, for example, down here, if you have a rectangle element, X Y width height stroke is blood. You paint a you outline the rectangle in black stroke with these two pixels, that value of two gets reflected in the stroke width field over there and there's a boolean. That called has stroke width, because all the properties can be omitted from the elements.

A

But you know the code needs to know if the property was specified or not so we used to store a boolean. There are SVG state also has some independent fields that sometimes come directly from SVG elements, and sometimes they get computed afterwards. In this case, there was a transform property that sometimes gets used just like that and sometimes gets multiplied by the previous transformation. So there were some independent fields up there.

A

The general thing to do is on the sea side of things we added accessors for those from for those fields so that the code didn't use the field, the struct fields directly. Instead, they call some functions. We have all done this with with geo objects before so. The first thing to do was to to make all the code you successors instead of accessing fields directly. This makes the code a lot more verbose initially, but in the end, when it's on the road side, it will become nice and small. Again.

A

This is just a a temporary stage, so let's start draining the strut. What do we do? We add a field that is a pointer, cannot pick structure that will live on the roadside every field. We will drain it through that little hole.

A

We start with an empty struct in rust. We have a state rust over there on the right. We provide two functions, one to allocate it one two frite. This is just like the first one is just like call Ingenio. The second one is like calling g3. The exception is that we do it on the rust side.

A

We take our empty structure and we move the first field to it in levers. Vg, since we had this field has field field has filled thing. We were able to replace it with a roast option. So, for example, fill rule. It specifies how you feel an object in the SVG SVG image, so we replaced has field rule with option fill rule.

A

You may need to define your own types if, if you put them front see whenever they are not available in things like Cairo and stuff here, Cairo already provides a Cairo field, rule ta type that already had a binding in the Cairo rust binding.

A

So we were just able to use that and then we also add accessors on the rest side for the roast field and later those successors can be called from the C code, so we provide two functions: state roscas fill rule which impacts this truck, which sees, if the option is none or has a value and a state rust get fill rule which actually gets you the value. If you are sure it's there, these functions take a pointer to a state rust what they return, C values, because we are gonna, be calling them from the seaside.

A

So we don't return a roast bull in that we return a a.g boolean. Those types have different sizes. Cairo field rule is an enum that is represented the same in CS in Russell. We can just pass it around on the sea side, we remove the fields from the struct because they are in the rest side now and we replace the accessor that we wrote initially with a call to the rust side. Accessor now this might seem like too much indirection. You know I'm calling an accessor from see that calls an accessor from rust that taxes.

A

This is the field that returns a value that returns the value that the code can use television direction, but all of that will go away. Don't worry about that.

A

We do the same for the clip rule and has clip rule. We remove the C version, stroke.

A

Stroke, opacity same thing for every pair of fields we had here, etc for all the remaining fields, and now we only have a few remaining fields in the C strut on the left.

A

My daughter has been learning what her colors and I asked her. Can you draw me various moving baskets from one place to another?

A

There you go, let's fix up ugly whitespace, let's move everything up, so the C struct is much tinier. Now that it has very few fields.

A

Now we go back to the fields that are different, the individual ones, so I had to do a lot of refactoring to be able to move this field into the wrost code because of the way I find transformations get computed. But it's just you know, standard refactoring.

A

We move it to roasts. We remove it from see. Here's the seaside accessor for the affine transformations it just calls state rust. Get fine here is the rust side accessor that returns the real value. So we.

A

Convert the pointer to the struct to our reference so that we can use it and then we just return the value.

A

After all, the fields are moved. We ported the code that actually uses them from C to rust. We here we were porting the fields and adding accessors for them. The next step was to actually port the code that uses those values. The drawing code you know give me the field rule set it on the Cairo context and draw whatever paths we need to draw, but on the road side of things we were putting the drawing colors.

A

How do we access the rust fields, since we only have a C structure with a pointer to the rust structure inside? How do we do that? What we have to do is this is what the rough side of things look like looks like. We want to be passed, a pointer from C truss it to the original C structure. We want to extract the pointer to the rust structure that lives inside the C structure, and you know from rusts viewpoint. This is an opaque structure that lives in the sea world.

A

So we just an empty enum and an empty Onam or SVG state just to be able to take a pointer to it. We export a function on the seaside called ours. Each state gets stay, trust which, given a pointer to an R, is VG States dropped which lives in C.

A

It will give us back a pointer to the state rust strut. This comes from the state rust field down there, we'll create a helper function, state a Trust which gives gives me the the rust side of the struct and any drawing function that needs to access one of the CSS properties when it has a pointer to the sea. State we'll call this function to get to the rust fields.

A

It takes a pointer to an RS VG state which comes from somewhere in the C code, returns a reference to a state rust cause the function that we exported from C to get that pointer and now, whenever we port one of the drawing functions that takes in a pointer to the our SVG State strut, we fourscore. We first call stake a trust to get the rust side of things and we can access the fields normally.

A

So all the accessors that we added before start going away, because once we put the drawing code, we don't need accessors for the individual fields. We just need the the master accessor to access the whole struct.

A

In the end, we want to be taking raw pointers to see structure, since everything will live completely in the rust side and we will be able to access frost fields directly without first calling a helper function, so the code becomes nice and small again, once we have all the code, once we have ported all the code that actually used those drug fields, the C struct becomes empty, we can remove it and we can rename our rust structure to be the one and only one. This is not stay trust anymore.

A

This is called just state and I have not found a good way to make this change.

A

Easily, in a single step, because once you rename this thing, you need to convert the functions that you used to allocate this and free it and change all the Rost code that wants to use the new the new struct and the compiler will complain a lot about ownership and this and that, but that's just taking each error message and fixing it.

A

It takes it's a it's a stressful couple of hours because the record doesn't compile, but once we're done, fixing all the compiler errors like, oh here I was taking a mutable reference for like needing an immutable one or vice versa. Once you fix all of those the code works, the compiler is very good at catching mistakes and anyway, when we're in the middle of this kind of refactoring. Do you remember when we had the part of part of the struct in C and part of the struct in rust?

A

Well, this is what the code looks like when we are in that intermediate State. First, we obtain a state which comes from somewhere else in the code and if it's a pointer to a C struct, we get the rest part of the structure. We can use some fields if they have moved already to rust. We can use them directly. Otherwise, we need to add accessors and it's kind of horrible- that for some fields we can access them directly and for some fields we need accessors. Let me promise you. This is fine.

A

This is okay, all those ugly accessors will go away in the end, as we saw so don't get too stressed about the fact that your code is a very verbose in some parts and very nice in some others, it will all become nice in the end, if you keep doing the refactoring sequence.

A

You may, if you have used rows before you may be asking what about leprosy wrappers see is the pragma that reduces to say hey. This structure should have the same memory layout in C as in rows, so that we can just pass it around by value that works very well. That work that really works very well in Lieber is VG. We only use this for very small shirts trucks that get passed around of all the time.

A

Like small values, for example, we have a our SVG length value and a length value in SVG is just 5 pixels or 3 centimeters or 2 points. There are very small values like a number and a unit. The unit can be pixels centimeters inches, whatever those get passed around all the time, so it's better to have a shared representation between C and roast, but for massive structures with many many types that don't have a counterpart in see like the state trust we had. I have found it better to just use accessors.

A

Now, how do we refactor the first pass of conversion into beautiful Roscoe? The first pass you of conversion gives you very ugly code. It looks similar to see, but we trust syntax and it's kind of ugly. So how do you fix that? I have found that refactoring rust is a really pleasant experience. Roastery factors very nicely, because we have a very, very strict compiler.

A

It won't. Let you have mistakes like like if you rename a strut in C and you pass a pointer to it in a function or, if you add an argument to a function. Sometimes the C compiler will only warn you. You know incompatible pointer type, but it will go on compiling like everything. It's just fine and that's not ideal. You know if you are passing a func, an argument with the wrong pointer type.

A

It's probably a bug, and if it's not, then it means that you really know what you're doing with C and it's kind of scary. So let me show you how we refactored some verbose code into very simple and powerful rust code. This is one of the CSS properties line. Join line. Join is how, if you have two lines, that joining a vertex is that vertex rounded? Does it have chopped off point.

A

This the SVG specification says that the default for the line join property is miter, so you have two lines joined and they get cut off.

A

So we implement the default trait for this type. To say that oh the default value is this is miter. We also need to parse that property, the ignore the API for this parser. Just that means, if the string I got past is miter miter between quotes, then I returned a value of stroke line, join Colin Colin my turn, it's just you know, parsing a symbol. You know, there's many properties like that.

A

Here's another similar property, this time called the line cap line cap is how the endpoints of lines look. Are they square or they rounded?

A

Yes, which is Peck, says that if that the default is bought, which is awesome and.

A

That looked very similar to the previous property, so we refactored it to use a roast macro instead of cutting and pasting the same code to parse symbols. So we have a make identifier property macro, which takes the name of the type we are parsing.

A

We specify the default the default is still but, and we made up a little syntax to say if it's this string then use this symbol value. If it's the string use the symbol value. So that's very legible. Even if you don't, even if you're not familiar with SVG I'm sure you can figure out kind of what that thing is doing. Roast macros are very, very powerful. They let you define a little syntax within the body of the macro. They are hygienic.

A

That means that you don't have problems like you know in C macros, when you have to pass everything in parentheses, to avoid captures to avoid you know, errors with operator. Precedence. Ross doesn't have that problem because it actually knows how to do hygienic macros. So we were able to compress all the cut and pasted code to define individual properties into macro calls, and that's very nice later the macro evolved to support properties that are not just strings and symbols. For example, here we have a call to create the stroke, opacity, property stroke.

A

Opacity has a default value of 1, that is it's fully opaque and for the opacity property we use a type called unit interval. Unit interval is a type in libros VG, which can represent values just from 0 to 1 in that closed range, and it knows that the SVG spec says that if the user specifies values outside the 0 1 range, it should just clip them. You know so unit interval knows how to do that in its parsing stage. That's why we have a new type for that, so it's it's.

A

It makes a code easy to read. The macro evolved a lot. It's even able to define properties that are not simple symbols or simple numbers. For example, there's a text decoration property which can take tons of different sub values and they get aggregated into a struct. You know the macro can now process this, so it's very legible. How to do that. Thank you.

A

Once we finished shaving yaks in the yak shaving boutique.

A

Let me show you an example of turning C code that you saturate rose into rust. Here we are extracting key value pairs from a property bag. We parse the values and we stop with. If we find an ordering parsing. First, we initialize a boolean value for success.

A

We create an enter an iterator on ax, which is attributes which is a property bag. While everything is ok and there is an extra item in the iterator get the key value pair from the iterator try to parse. It update our success value at the end, 3d iterator.

A

If we found an error, the C code didn't really have a good way of propagating that error. Upstream, so I decided to wait until we ported that code to rust to actually do our propagation properly because it would be easier there. The roast version is very small: it's a small for loop for each key attribute value topple in the property box. I traitor, parse value, return an error value. If the parsing is not successful, otherwise return. Okay, how do we expose that I traitor to see well opaque structures?

A

Let's say we have an object, called foo with an ITR method that gives me back an iterator to look inside the object that returns a full I turn the details of how full I try is implemented are not really important here, just whatever you would use to iterate within your your full object, and it holds your iterator state like if you were iterating in an array, the iterator straight would just be the current index. You know roast ask us to implement the iterator trait for full ITER.

A

We specify the type of item that the iterator will return. In this case, it's called my item. It's just an example. We implement the next method for I traitor, Rasta territory to return an option on the type you specified and the option is some my type if we have a next item. Otherwise we return none. If there are no more elements in the right aerator. Let's expose this to see, we create a full I. Try begin function visible from C.

A

This will allocate an iterator on the rough side and pass it on to the sea side and return a pointer. It takes a row pointer to the object we want to iterate on returns, a pointer to the iterator. We check that we didn't get past a null pointer, convert the row pointer to reference call. It either method. We put the iterator object in the heap by boxing it. This is just like calling G, new or jima lock and finally, we return a raw pointer to the value in the heap.

A

Now, let's implement the next function. I'll I turn next, its color from C, so it's X turn C takes a pointer to the iterator, takes a pointer to the location where we will write the value we got from the iterator and returns a boolean. That says whether there was an item or not. We check the arguments.

A

We get a reference to the iterator. If the actual rust side I tried to returns an item, we copy the item to the output location and we return true in the boolean forum. Otherwise we zero out the output value, so C won't get uninitialized data and we return false converted to a glib value. That means that we didn't have an item to return.

A

Finally, to free the iterator from C, we expose a function called ITER and checks its argument and frees the box isolator on the heap by doing box from wrong, and we have now a nitrate or implemented in rust with a nice API that we can call from C.

A

Sometimes I found it useful to do. Printf based debugging gdb is great, but printf is forever as a word of warning. I don't know exactly what was going on, but printf doesn't always seem to flush standard output when you call it but rusts, println macro, always flushes standard out, so I had to make the logging work I had to insert an F flush call after every printf in C. So keep that in mind here I.

A

For the example, I'm gonna give you I wanted to try some complicated code flow, see which functions were getting called and which arguments they were receiving to be able to see the difference between the old C code and the new Ross code. Something wasn't matching in there and I wanted to see exactly where my arguments were getting getting wrong. So I need I'd, really needed some nesting, some indentation you're, seeing it from the other side and really need some nesting in my login.

A

So the simplest hack was to output, my stuff within parentheses, like if it were a lisp, a lisp expression in parentheses, the raw output is horrible. It looks like this, but I was able to load that into my text, editor and indent it with us with one keystroke and that made it very nice to read. So that's a that's a little hack you can use if you want to add logging Within Temptation very quickly without adding a whole logging framework.

A

That knows how to deal with indentation and this, and that this made it possible to find the mismatch quickly. So you know tell useful thing to have around. Finally, how are we doing this? Is the programming languages chart that gitlab is able to display? You can I mean if you have a project in gitlab, you can use this chart. We have more roast code and C code these days, thank to thanks to that man over there.

A

So give him an applause again, the C part of libris, which is shrinking very very quickly, because the only remaining parts are the a few filters, the XML parsing code, which calls into live XML the CSS parsing code, which calls into Lib croco and just the general loading machinery that plugs into G iostream, there's not very much C code left, it's just a lot of API calls and boilerplate. So that will shrink very, very quickly. I want to point you to a number of useful resources.

A

If you want to port C code to rust, the first one is a talk by Carol Nichols she's, one of the authors of the Ross programming language book, the Ross manual as an exercise she ported zopfli, which is a data compression library from c to rust. She has a fantastic talk on how to do that. She had exactly the same problems as with the Liberals V G old, undocumented code, without tests kind of convoluted.

A

She didn't know anything about the compression algorithm beforehand, just like I didn't know anything about SVG beforehand, and it's just the same, just refactoring and porting bit by bit bit by bit and whenever you don't know how to test something. You see what the C code dose to encode, that in a testing rose and there you go, there's a new tool see to rust calm, it's an automatic translator from sitarist, it's very, very sophisticated stuff. It works and it gives you rust code. That is absolutely horrible, but it's reasonably easy to refactor.

A

Why is it horrible? Because in C we, the reference pointers all the time we have shared pointers to everything from everywhere and rust really doesn't like that. So the way these translators work is by doing a lot of fun safe calls in rust to the reference pointers directly. So later it's your job to refactor that into rust references and immutable structures and mutable structures, but at least it gets you started it's it's very interesting stuff. They can even handle complicated, go tools and breaks and stuff it's it's really cool.

A

Finally, Katrina always refactoring talks are absolutely amazing. They are mostly for golang and Ruby, but they they really get you into the refactoring mindset that you can gradually turn code into into a much better looking code, thanks to my daughter for the first drawings she's delighted to know that you are all seeing them Katrina for the technique of narrating code by highlighting it Alberto, my friend Hawke in Mexico incision for organizing the previous good Norman Ross harvests, the Ross beginners channel on IRC Mozilla is really really friendly.

A

That got me started Jordan petri dish for making our CI infrastructure and doing some of the initial roast debugging power liberally. The previous G edit maintainer for porting, a shitload of code from C tourists, he's been he's been doing doing it on the weekends because he has real work to do in the week now. But don't worry, he's starting to use his night time for programming rust as well, thanks to Paolo his competitive code there in my blog there's to tax for all these work, one is the Liberals which attack the other.

A

One is the rust tag you can use them to see. My ramblings.