Rust Programming Language RustFest Barcelona 2019, 26 Nov 2019

Previous Meeting

⏯

youtube image

►

From YouTube: RustFest Barcelona - Zeeshan Ali: zbus: Yet another D-Bus library

Description

In this talk, I will present zbus, a D-Bus crate written from scratch. D-Bus is an inter-process communication mechanism, available and used on almost all modern Linux desktops and many embedded systems. I will start with why I felt the need to take this huge undertaking on my shoulders, followed by the design goals, the challenges faced and how I overcame them during the development.

https://barcelona.rustfest.eu/sessions/zbus-yet-another-d-bus-library

A

Zeeshan: hello? okay, this works. hello, everyone. as the mc said, I'm going to present about zbus.

A

Before I begin, a big round of applause, please, for the transcriber.

A

They are doing an amazing job and it's a really difficult, job.

A

So thank you for that.

A

First of all,, who am I?, my name is zeeshan ali, as said..

A

I work for this small company called redhat..

A

I have been working in open source software for a long time.

A

Nowadays, I work with these cloud weird things in my day, job.

A

I'm into flying- and I love cats.

A

So background story about this talk, right?.

A

It starts with a project called geoclue.

A

Does anyone who, what that is?

A

A

it's a framework for your linux system.?

A

If you have a linux desktop, you probably have it installed on your machine,. If it's a modern.

A

One, and the task of this service is to locate where you are, so applications,, when they.

A

Need access to your location, data,, that's the service that it needs to talk. To.

A

Firefox still hasn't ported to that..

A

It would be really nice if they do that,. If any one of you works on firefox.

A

It's written in c entirely and I have been the maintainer since a re-write.

A

There was a complete re-write because the old score was really bad and stuff,, and that.

A

Was 2013 so it has been quite many years that I have been maintaining it.

A

So I thought:- let's oxidise it.

A

those of you who don't know this term,. It means to port it to rust.

A

But why? well, it would be singing to the choir, but still: crash reports.

A

I get crash reports, all the time., I'm tired of them, so I don't want to deal with those anymore.

A

A

Your location is one of your most sensitive data on your machine.

A

You don't want to give it to any app or you don't want anything bad to happen to the geodata,.

A

This data of yours,, so that is another reason, but, most importantly, I just love rust.

A

I don't do rust in my day, job, at least at the moment,, so in my spare time at least I.

A

Don't want to look at c, and I want to do something nice, like rust.,.

A

So what are the challenges for porting geoclue to rust?.

A

First of all, the thing that came to my mind was this: build system called meson.

A

All the system level services like system d and related services on your system,, they.

A

Make use of this build system, nowadays., also,, all of gnome and hopeful at some point? Maybe kd will use it too if they are not already.

A

I don't know. and so yes, geoclue uses it and it has rust support in it for a very long time,, but doesn't.

A

Have cargo support and it looks unlikely there will ever be any cargo support there.

A

So I thought this would be the biggest challenge,, but we will see later how it will work. Out.

A

The other thing I thought of was d-bus,. How do I talk to d-bus?.

A

What is d-bus?, it's a very efficient interprocess communication protocol and it's very popular on the desktop.

A

And in embedded systems as well.

A

As I mentioned, the other projects like system d and related services and gnome in the same.

A

Two bubbles, d-bus also exists, and it's very popular and I don't see why it wouldn't be.

A

It's very awesome and very efficient.

A

There is a crate, dbus-rs, but it relies on libdbus, which is a c code, so it relies heavily.

A

On c code, and also it is not shown to be a reliable library.

A

There are other issues, but I still decided to use it and not just use it, but I also contributed.

A

In terms of issues and patches and stuff- and I thought I will just use it.

A

Until rust, gnome hackfest, that happened in may this year, and I started doing it, an.

A

Amazing part turned out to be much, much easier than I thought.

A

It's a work-around still but an easy work-around,. I can call meson from d-bus easily.

A

It turns out,, I wanted to use code generation,. It was very, very complicated.

A

Also, sorry,: I have to convey a lot of information here. So if you don't understand, something,.

A

That's fine,, I'm doing a bad job,. Please grab me after the talk and I will explain further.

A

To you. I will be happy to you., that's okay.

A

d-bus crate from scratch?.

A

I thought. Maybe that's an option.

A

So I was like:: how hard can it be?.

A

So let's see what's involved, first.

A

I looked at the pack,, there is a d-bus pack,, a one page which explains everything.

A

It's a really nicely written pack in my opinion, so I started the low level and lowest level.

A

Is just message passing on any kind of socket or any kind of medium.

A

It's medium agnostic, actually, the protocol and there's a wire format that you have to.

A

Implement to be able to talk to d-bus.

A

It's also called gvariant, because d-bus was implemented by non-folks and the main library.

A

Made use of that from the very beginning, and they realised that the wire format of d-bus,.

A

It can be used in a very generic way, so the api is provided to use it in a generic way.

A

Outside of d-bus as well, so it could be like a database and people use it that way.

A

What that is,, it's mostly defining data types and their encodings.

A

By encoding I mean you have to have alignment.

A

All the data is aligned.. If you don't know what alignment means, again grab me after the talk, and I will explain.

A

It to you, what that is all about.

A

D-Bus has natural alignment for the most part.

A

99% of it.. There is one exception.. I will talk later about that.

A

Every data type that d-bus defines has a signature,, which is a string which is a suggested designation.

A

For that particular data type., we will see the use of that in a bit.

A

So we start with the basic data types.. What are they? they map exactly to the rust data types.

A

It's a subset of rust data types actually..

A

The only difference is that they are encoded differently, of course, because they need.

A

To go on a socket,, not in a programming language.

A

Then there's container data types.

A

array maps nicely to a vector or array itself.

A

Structure,, it's a bit weird to have like a dynamic structure with maps actually to struct.

A

In rust,, but of course it's more dynamic, so we can't just use a struct as there is in.

A

Rust. and dict is just short for dictionary and that maps nicely to hash maps and, most importantly,.

A

There's another data type called variant which is basically just generate data.

A

So if you want to put, you can put any of the data types I mentioned before in this.

A

Variant and what the variant contains is the signature of the data and the data itself,.

A

So if you want to transfer data in a very generic way,, this is the way and this data.

A

Type is used a lot in d-bus.

A

On a high level,, you have objects..

A

When you talk to the d-bus, you usually deal with objects that are exposed on these specific.

A

Paths., these are shown here, the object, paths, and the way you talk to them is through interfaces.

A

This is a really horrible example,, sorry for that.

A

There's a one-to-one relationship here between the objects and the interfaces,, but it doesn't.

A

Have to be. any object can implement any number of interfaces and what that means is that, if they implement.

A

An interface, they need to provide specific api of that interface.

A

You can have methods and they have to have the data types that I said, before.

A

Signals is just a method, but in the reverse, direction.

A

If the service needs to notify apps of some change,, they use these signals, and they also.

A

Have parameters,, but only I think, in parameters also out parameters.

A

We are not sure right, now.

A

And, of course, they can have properties that they expose and they can be read only,. They.

A

Can be read, write.

A

So I thought actually it's not that hard after all.

A

Am I going too fast?.

A

A

so after three days,, which includes the hackfest and I had an excuse to not do my work-work,.

A

I could do rust and I started implementing it first as a hack and what I could accomplish.

A

In three days,, the first day I established a connection to the d-bus on the service,.

A

And I called a method. that felt so great. within three days. I could do this.

A

So I was like: yes,. Let's do this., it doesn't sound too hard!.

A

So zbus was born., don't ask about the name., it's a cool name and you have to accept it.

A

So, let's start with the hard part, the low.

A

Level,, which is called the gvariant, as I mentioned, before,, which is dealing with the data types.

A

So I came up with this to represent the data type that can be encoded to and encoded from.

A

D-Bus wire format., it's pretty simple actually,. I think.

A

Encoding is I tried to make it very efficient.

A

Encoding,, I can't make it efficient because you have to copy data, because the d-bus encoding.

A

Is different from the rust encoding, so there's no way out of this. As far as I know.

A

If you know a way out of it,, please do tell me..

A

I would love to know that., but decoding,. I made it very efficient so that you don't have to copy the data.

A

Only for container types you have to allocate,, but even that only the container,, not the.

A

Actual data contained in it.

A

signature,, you might see, there's a constant signature and then there's a signature.

A

For simple data types, it's exactly the same,, the screen that you represent,, it is one corrector,.

A

But for complex data types, the container types,, they have to also specify what is included.

A

In them, so it's more dynamic,. So it's not the same.

A

The constant actually, I have a pointer!.

A

But it doesn't work on this, does it?.

A

No. anyway,, so this signature,, the constant that you see on the top and the signature method,.

A

They don't always have the same output.

A

So basic data types, as I mentioned, before, the same as rust, very easy to implement.

A

I did that. container types were the next.

A

arrays,, as I said, vector,. Only thing is that I had a constraint that it can only have the.

A

Variant type in it., all the other types,, the d-bus support,, not anything else.

A

Structures: yes,, I have to create another data type for that.

A

Captured as simple as I could.

A

You have fields on the structure and you can create them, and then you can use that, and.

A

Then you implement variant type for it, so that you can encode and decode structures.

A

Of course, the most important one is variant which, as I said, contains a signature of what.

A

It contains and the data in encoded format.

A

Now,, this is the important bit that,. It keeps already in encoded format.

A

It doesn't encode when it's needed, but already..

A

I thought that would be more efficient.

A

We will see later how much trouble that gave me.

A

So yes,, this is how you will use variant in isolation without d-bus.

A

There is no d-bus here, so this is the type I came up. With.

A

This is my unit test, which just shows you,. You create a type that is d-bus which implements.

A

For a type and then you create a variant from that and then you can do different tests on.

A

It., you can get the data back from it and you can once it's encoded. You can also decode.

A

It and stuff. yes.. I lied before when I showed the variant type frame.

A

It has lifetimes on it and I did it to be as efficient as possible and I was successful.

A

In making it efficient,, but at the same time it made life much, much harder.

A

Each time I change anything, lifetimes come in the way and I have been bugging lots of.

A

People, especially florian over there,, about how to deal with lifetimes.

A

Oops, sorry,, I pressed the wrong button.

A

So back to d-bus., so this is what the d-bus api looks. Like.

A

You just create a connection, there's two kind of common connections that you talk, to,.

A

One is a session connection, which is for the user, and then there is the system connection.

A

Which is used by, for example, geoclue for system services,, and then you just call a.

A

Method on it., you specify the there's, the service,, the path,, the interface and the method and parameters.

A

Which are zero here,, so I pass none.

A

Then you get a reply and then you get the first out parameter and it's the type of string.

A

So I could get the string and display it.

A

All done with variants, right?, it's all done?! That's what I thought and I thought: like, yes..

A

I didn't talk about an additional part which I implemented really late, but it's kind of.

A

Implemented already, and I thought: I'm done with variants.

A

But I added new test cases for the dictionary.

A

Actually, these test cases were more complex and I was like:. Why is it failing?.

A

I thought maybe I had made the wrong test case or something else wrong with it.

A

And then I had this face.. This is an actual picture of me actually.. That's how I felt.

A

variant. Alignment is all wrong.

A

It's completely wrong.

A

why? d-bus has a few strange. Rules.

A

turns out,. D-Bus is not that easy after all.

A

When you encode, it is based on the whole message,.

A

It's not individual levels.. So if you put an integer in a structure, the alignment is not based on the parent structure.

A

But the whole message when you put all the data together,, which is okay on its own,, but.

A

That's not the only and also if you see at first what I came up with,. There is no.

A

Way with this api,, I don't pass to encoder how many bytes were before,, so this api cannot.

A

Accomplish this, but I modified it soon after and added 10 bytes before so then the encoder.

A

Knew how many bytes were before so it knew how much padding to add for alignment and.

A

Stuff like that, so that was an easy problem to solve on its own, but variants and contained.

A

Value do not need alignment,, which is okay but surprising,, but okay,. We will deal with.

A

This., it's no alignment, better,, but the problem is its grandchildren do.

A

If you have, for example, a structure in a variant, that structure does not need padding.

A

Nor alignment, but all the fields in the structure, they do,. So how do you pass that along, all.

A

The information?, as I showed you before, variant, already keeps things in encoded format. So when you create.

A

A variant in isolation, then you put it in a message: the encoding is not correct and.

A

That's what was wrong in my whole thing,, so it turns out my whole structure,. The whole.

A

Api I built in three months was fundamentally wrong.

A

So I have been on this for a month, now.

A

Mind you,: I only work on this in my spare time, which is very limited,, so one month is.

A

Not that long in reality.

A

So I still have not solved this problem.

A

I came up with multiple solutions..

A

They all had some problem in the end. After all, and I couldn't solve it.

A

Now I have a solution, but before I implement it, I have to kill the lifetimes from the very.

A

End., because that makes when I said, I came up with multiple solutions and many of them failed.

A

Because of this lifetimes, because I just couldn't satisfy the lifetimes, so the lesson I learned.

A

From all this practice is that efficiency is not a religion.

A

I was being so focused on making it very efficient that I wasn't making it more efficient than.

A

I would be in c. in c,, I would be using something called gobjects from glib api and they use.

A

Reference counting all the time., that's what they are based on, and their reference counting is by default atomic. So it's much.

A

More heavier than the rc reference counting type in rust,. So why not use it?.

A

So with our team, I would really like to make use to share data between different points.

A

Of the code, and then it will work out., I have a solution., it will work. if not, I'm in this conference for three more, four more days,. So I will grab someone to.

A

Help me., that's my hope, at least.

A

anyway,. Once I solve that problem,, which I will, looking forward,. I want to separate.

A

Out the variant crate into its own crate., as I said, it's useful in a generic way, and there is one application in rust that the.

A

Maintainers said yes,: if you make a crate,, I will use it, so yes.

A

I have to do that.

A

Then in d-bus, we have an api for receiving messages,. I have to implement signals which.

A

Should be really trivial, actually.

A

"should be", in quotes.

A

Async,, I need to add asynchronous api, and I'm hoping that actually, the api will just.

A

Be asynchronous by default and hoping to use the new async std,, or how do you pronounce.

A

It,, I don't know, crate., it looks pretty awesome, that crate, and also it's good, that I was stuck on the variant.

A

Part until now, so that all the async stories, all in stable,, so I can just use the async.

A

A

high level api,, then I need to add. that shouldn't be extremely hard, but I need to be careful in creating a good api.

A

Fortunately, glib has a really good high level api and I can copy that;, not copy,, but I can.

A

Learn a lot from that, so I don't need to figure out a lot of things.

A

Code generation: most of the d-bus code that I see out there, they use code generation.

A

You give it like, you can define your d-bus api in xml, which people do and you can give.

A

It to code generator and it generates the code, so I need on implement that, and maybe.

A

Also macros for making it super easy.

A

If you don't want to do code generation, some people,, I don't know why,, but they don't like.

A

Code, generation, so sure for them there should be some macros.

A

And a lot more of easy, stuff.

A

Yes,, that's it from me.. If you have any questions., no questions? everything was clear? yes,. There is a question.. Do you get a mic?, it's better. tech team,. Please enable microphone. 2., zeeshan: yes,. When I said receiving messages,, that's the server side, yes., that's coming.! I need it myself for geoclue., that's the main use for me,, so it will definitely be there.. Someone else? it's working,, so I have a magic hand.. So you were saying that performance shouldn't be an issue,, but did you compare it to native implementation in gnome or other implementations??

A

How does it fare? zeeshan:? I haven't done any actual comparisons, but I could see that I'm not allocating anything or copying anything,, so yes,, I made sure of that,. So that's the way. I know it's as efficient as it would be,, but no, I haven't done any real comparisons.. Once I'm ready, once it's done,, then I can look at that. any more questions? no? one more.. Could you explain alignment? zeeshan: [laughing], so some cpus require you to start your messages and to be able to read specific data types from on a particular boundary,.

A

So the memory address has to start with some multiple or it has to be a multiple of 8 bytes,, something like that., that's as far as I know,. So that's why you need alignment.. If you generate protocol, that's cpu or architecture agnostic, you need to make sure of the padding and the alignment.. The way to align is that you do something called padding, and padding means you just add: zero bytes before the data starts, so make it so that that data actually starts at a boundary that is defined for that data. Type.

A

so, for example,, as I said, natural alignment in d-bus,. What that means is that,. If it's a 4 byte integer, it has to be aligned with 4 byte value,. So you have to have 4 bytes of padding if it doesn't start at a multiple of four address. things like that. okay?. So thank you so much for attending and I hope you understood some of it at least. and if you don't,, as I said,, let me know., I'm very happy to explain things to you., I'm wearing the green one,, so you can do that.