Rust Programming Language Rust Zürichsee, 3 Jul 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Rust Zürisee Live Stream 2023-07-03

Description

Topics:
* Rust in the Kernel
* IDE level configuration

Slides: https://github.com/rust-zurichsee/meetups/tree/master/2023-07-03_rust-in-the-linux-kernel

Would you like to chat, ask questions or give a talk? Join us in our Matrix room:
https://matrix.to/#/#rust-zuerisee:matrix.coredump.ch

Support the Zürich community: https://estada.ch/support-my-work/

Follow us on https://rust-zürisee.ch/

Join us on our next event:
https://www.meetup.com/de-DE/rust-zurich/events/293322905/

A

B

Hello, everybody, hello, hello, oh, we need to turn off the music. uh Where can we do that.

B

Thank you very much.

B

All right, let me grab my quick to-do list so welcome everybody very happy to see so many in person.

B

Oh, am I still off no you're wrong.

B

Yeah, that's fine I I.

C

D

B

More or less yeah so we'll just have the mics for recording and for the live stream, which I guess has a couple of viewers now and yeah. So a couple announcements, so first hello, I'm Stefan, I'm, one of the organizers of Ras turisi. Another one is Gary over here.

B

He has the lovely Stitch thrust, logo and Mike, currently behind the camera and our host today's TP and you will give a little introduction later so for the timetable, uh I guess in 15 minutes or so we'll go with the first talk and then Raphael will tell us about the rust kernel module that he made happen.

B

He's very shy, so he'll need some encouragement later yeah yeah and after that. So in about an hour we'll have a quick break, because we need to re-plug some of the things and then also have the last speaker which we'll talk about of uh how to treat your configuration like an IDE with autocomplete iOS promised I. Don't know, we'll see very happy to see. Ronald Roland's pardon is over there. Yes and afterwards, uh thanks to TP will have a rooftop party, but I won't spoil that.

B

Then in August will take a little break because everybody is on vacation and yeah. We don't have a space as well, so the next space, the next baseball looking we are looking for, is in September and if you want to give a talk, then comment find me or the other two guys and if not, there will be a very, very boring talk by me about how to enable continuous integration in gitlab I mean you will learn something, but it will make sure it will be very boring. So.

D

B

Me and we'll find your fancy topic after that in the fall there will be oil rust and we're currently looking into going there as a group. So if you go to others, just chat us up in the room and last but not least, if there's enough interest, maybe you can raise your hand for a quick show who would be interested in buying new plushies.

B

Remember the the ferris ones. Can you hold up my backpack real, quick? Thank you. So it's like two ish, four or five or tennis like the orange ones. Okay, so like 25-ish people.

B

Okay, so Munich will want to have 10 as well. Thank you very much so I guess we'll try to shoot for 200 to 500 pieces.

B

A

B

Gifted away but uh yeah one starts in motion: I will start searching for sponsors or other people that will just float the money up front and then we'll do that. I honestly, don't know how much the cost per piece will be well. I guess 20 is probably the max all right. With that I'll hand it over to you.

E

Thank you Stefan. Thank you very much. Thank you. Everyone for coming regarding the next location. It can be here as well. Naturally we can. We can do that, uh and uh tonight I just want to make an intro about what we do at TP.

E

um We are a small startup and I will just give you a 10 minutes talk about uh what we, what it takes to make a developer tool startup in Zurich and uh provide from the occasion to speak about a rusty part, because uh you should not bite me I come from the or not beat me ever. uh I I come from the C plus committee before and I still like, C, plus plus, but I also like rust. So it's all about synergies.

E

Tonight I mean uh it's the Zurich Meetup, but we have a speaker that is organizer of the Basel rust Meetup, so I think it's good when people work together, uh I think it is good. When you put these things together, we might argue there will be beers, and so so don't hesitate uh to to to say what you think about that.

E

um What I think is the best to uh to put together, however, is beer on pizza, so this is also a good Synergy. So, during the break you can help yourself take any drinks. It's all free, it's written, you should pay, but you should not pay just take it. um We we, we will say. Oh, we don't know where disappeared, no worries, we are startup. You know we know to deal with this thing.

E

uh When you are finished with the with the butters, you can put them in the in the boxes that have written rust Meetup. So they can do the accounting uh uh only make sure that this disappears and uh yeah we'll have the pizza on the rooftop afterwards. If there is not too much wind, so, however, we will reorganize the space and have them there on the there's, also someplace around but I. Think given the forecast, it should look good, so rest Alpha support.

E

So it's Alpha support uh what we, what we have been doing uh to get there is uh creating a startup for developer tool, because I thought: okay, I was a contributor in open source Yannick as well yanica's, uh who is there. My co-founder uh is uh has been also doing a lot of embedded stuff. um A lot of Automotive he's also made some Kickstarter campaign work on so um myself, I was doing.

E

Linux embedded stuff did a contribute to boost libraries, but where once uh the safest, C plus out there and uh then uh did also a lot of cute on the spoke to CPP Khan, also and I thought I should change the status quo for C plus make builds as cool as in Rust like cargo, but I didn't see how I could finance that without creating a startup without finding by this model. That will make it happen and that's the reason why I created my startup and I thought.

E

Okay, what I need is I, have a project I just need to to pitch it to investors, find funds and get started. That was not that easy, because uh investor don't understand anything about code, about trust about C plus plus they just want to see numbers. So I was like okay, I I learned so found the first customers very cool people uh on gave them or very nice little piece of code uh uh on the sadly the business made. It looks a bit other like when you need to deliver on features.

E

You need to to get the things working. Then you do some shortcuts that you don't necessarily like, but you have to do them and uh that's something that we've been fixing for for a long time now and we are getting to a way better quality and we could raise a successful president in 21 we raised around in 22, but it's a secret and uh and uh we uh we are now publishing and building up the team.

E

So if you have interest in uh in build systems in compilers in front-end development, you are like front-end in a sense, compiler front-ends, uh don't they say to speak to us. We also have some web front end. So it's also welcome uh uh and what we do is actually providing in the minute developer environment from anywhere um cloud-based automatic wheel.

E

Caching, for anything you do like, we think you don't need to take care of packaging binaries, because it's uh it's a it's such riskers over to put binaries in that are not equal to to the one of the sources, and so so we automatically cache everything that is built.

E

Do uh tracking of the sources so that we ensure safety of the binary result so that you can save time on CI workflow so that it not, as always rebuilt from sources but also rebuilt from binary directly, and we provide also a fast Cloud builds like big machines in the cloud to speed up your builds. uh Whenever you are using your machine, you would like to build on more than your 12 cores that you have on the M2. You want 128 cores, then you just say tp.j 128, and you get the course uh on this.

E

Also powered by your cache on all plays together, but you can use everything separately. So what is if I would summarize, it is building things faster instead of taking losing your time making things build, because in C plus plus world is not like cargo, uh you have to learn cmake, you have to deal with it. You have to love it on ate it. At the same time on the then, you spend a lot of time making things build and then, when you're done with that it will slowly.

E

So we thought okay. What could we do for us? Definitely not improve the experience with cargo. This is good. We thought. Okay, we could make remote build uh work faster and um so that we don't have to to deal with like oh I want to build for Windows, so I need to set up a Windows machine to test my build and so on, so which we take care of all of this thing uh under so that you can have CI workflow that get fast.

E

So actually the boring talk from Stefan will be very welcome, because uh what you like is building things not making this not setting up your Docker, and so what we try to bring to you is like the CI CD workflow uh directly at your fingertips. So what you used to wait for the CI to build?

E

We bring it to you whenever you type something we build it directly instantly and we try to provide as a optimization technique to speed up tests, speed up build and uh we automatically paralyzed test execution automatically paralyze build execution and we are trying to uh to bring that to rest as well, and that's if I will show you uh in a demo. That will just look like this uh here.

E

We have one piece of the code base uh from TP that is named Dev Checker, it's an open source project, um I I, don't know if everyone can read it's pretty yeah, it's pretty clear uh text and uh uh um what it does is actually does these cashbacks like for whenever you build something it automatically creates a cache entry, a snapshot, and these gets uh packed in a compressed format together, and we built that in Rust to learn uh better what we could bring to rest and also to bring something safe uh to C, plus plus build onto rest build like a way to build safely without binary packs that you can then trust like when you take these cashbacks.

E

You say: okay, I know it comes from this source file. There is no way someone puts on malicious code in between and uh if you will build it with cargo, uh then you would just do cargo build like usual, and then you have naturally uh the different crate with your dependence to build on to Lebanon. If you build it uh with TP, you can say.

F

E

I want to read it for Linux on the 828 cores. What happens is that we create a Linux machine in the cloud for you with Android 28 cores. We clone the sources the first time and once it's uh it's done.

E

We build with cargo as well, uh and if it happens that there is a cache entry for all the crates and so on, then we pull that and you don't have to wait because someone else already built that code and someone has built it for Linux, so you could just pull the cache out of it, and this is what we bring with TP, but also the fact that the first time you build it, it gets really fast because you have the you have the cloud you can also use it locally.

E

You remove one keyword and you can build it on your local machine as well on the or Cache format that is written in rest works like this. We take your sources that is in a workspace. We mirror them in some central location. So whenever we extract the cache, we don't need to unpack. We don't need to patch your files, because, where, if you have like um build files that have uh absolute path, then you you will need to patch that um that can be in format that you don't necessarily can patch safely.

E

So what we decided is okay, remove all the code from the user in a central location. So you don't need to patch the to patch the file you can just extract them, because all the builds up and always in the same place works also on Windows, it's in a CTP folder and then what we do is that we allow to switch pretty easily between the commit or the branches. So we have a cashback format that is based on that on that standard.

E

If you take 300 times, 10 gigabyte, Bill trees for llvm, we can compress them in 100 megabytes, where you have one pack of 100 megabyte on your 300 time llvm, and so, whenever you want to take, oh I want to test the code that my friends did or my colleague modified. You can just check it out and you get the cashback downloaded in its very short time and you can extract it and then you don't have anything to build. You can just try it out and that's what we do.

E

We upload the cache near to your code. uh We use techniques that doesn't cost you money. We handle that and then you can it's just one click away. If you want to release it to one of your customers, because the binary is the executable are also there. So if you start to distribute a binary with that, you are just one click away from releasing it and you can always track where it came from and you can always check check it. So it's very Alpha support for us.

E

uh It's uh not yet all released properly, but we are open source uh you can. You can get get to us pretty easily if you want to get support for the rest to get support for rest. uh We are uh in the stage where we are looking for Disney partner people that knows better or rustworks and so to to to like uh refine the product on. So so you can just register or go on GitHub and ask for it.

F

And we'll put.

E

Us in contact and at some point it's totally MIT or lgpl license. We have not decided yet, but uh that's TP, that's what we do and thank you very much for loving native software. We need that.

E

Okay, let's move to the actual talk.

B

Thank you Damian, so we'll see if everything holds up just unplug it yeah.

G

A

Then we try that.

G

We have the other side.

B

So with any luck, this.

A

B

Pick up soonish, Miller, okay,.

B

Power it works. Okay, looks.

A

Like it's chilling up yeah so.

B

Without further ado.

A

A

So hi everybody thanks for showing up it's more people than I expected honestly. So this talk will be about my journey to build an art, Linux kernel, to play around with Ross written modules and maybe first some stuff about me. I'm Rafael, that's told you. I live in robbersville, which is a city at the under other end of the Lego City I work at renewal and I'm, a founding member of hacker Makerspace in robbersville, which is the place where I play where I play.

A

Where program was the most actually yes, so a small disclaimer I use Ross since, like early 2015, so a bit earlier than the 1.0 release. So with the early battles I started playing around I have some experience with embedded rust, also from work from the previous workplace, but I'm not really a kernel developer. So yeah.

A

There are any errors, um please correct them afterwards and yeah. So the outline of my talk I will do like a quick start that you, if any of you, have your laptops with you that you could follow along and play with your kernel module event. Does anyone have a laptop with them and Arch links running on it.

H

A

At least some yeah.

A

So, let's get to that so I actually I just made an Arch Linux user package with a custom kernel building. It takes some time so make sure to adjust your make flags to actually use all your cores, like with the number of chops in your make package config, and then it should just work and then make sure to install it together with the header package, then reboot into your kernel should be ready.

A

If you start compiling now by the end of the talk, it's probably finished, yes and then you should just be able to clone the out of three module repository build it, and then it should just magically work. So maybe Stefan asked me to do actually live demo this to to prove that I'm, not all snake oil, so.

A

I move this to the oops screen. There doesn't work: okay,.

A

Well, try again try again.

A

Let's do it like this, so I will do a I. Think I need to do this with pseudo yeah I will soon shortly after.

A

Oh enemies type, my password.

A

Yes, so uh here you see in the D message: outputs that I just loaded and unloaded the the rust module a few seconds ago, um yeah so so much for the live demo uploading a kernel module, since it doesn't do much you just have to edit there kernel module print out all right. So let's get back to that.

A

All right so before we start about my journey to actually get to this point, uh I will just introduce you how people started the idea of writing Linux kernel modules in Rust, so the all the reference I found is actually from 2013, which is two years before the rust 1.0 release, which is kind of interesting. That people already then found it interesting to play around with the links, kernel and rust.

A

um I didn't try it out, but back then it hopefully worked. Yeah then, like the most. um The earliest reference, that's kind of like a bigger event, was at Linux security. Summit 2019 I didn't attend it, but I did watch The, Talk um yeah and there they introduce actually a plan to kind of bring it to Mainline. Linux support for current model certainly lost in Rust back.

A

Then it looked something like this, so you have like a structure where you hold all your data for your current module and init function that you need to implement, and then you have a magic macro that probably does a lot of stuff that I didn't really investigate to a registered account with the with the link with the kernel itself.

A

Yes, yeah, then plans for Mainline started on the Linux kernel mailing list at 2020.

A

done in July August. There was a session about that. Can you read it up in the references I put there if you're interested and then finally in March, 2021 initial rust support landed in Linux next, so that's the kernel tree. That's you supposed to be stuff that will soon be released.

A

Then in April 2021 there was an official RSC for rust support so for the kernel maintainers to discuss. How should we proceed? How should we do that then, at some point, Google starts started funding this project and then from July to September. So better half of a year or a few months at least the or the initial RSC was iterated.

A

And then in October 2022 actually merged for limit curl 6.1 in December. This kernel got released and then I started with ah let's play with this: it's never shiny. It's just got released. So let's what could possibly go wrong? It's probably an easy thing: now that's main line and everything so here my actual Journey starts to get this thing to run as I saw in the beginning. It was a bit naive, but so right, this can't be too hard right. Verify that error.org scroll actually has the configuration enabled I verified that okay seems legit.

A

They have this convex like that, install the kernel from the testing repository clone this modules compile and run. Then maybe this has an actual benefit to me or not.

A

um Yeah then I tried that and yeah that didn't work at all. So let's check the documentation and the documentation mentions something that yeah. If you have a distribution kernel, you need actually some build metadata that your you know how the kernel module needs to link against your actually running kernel. So I was like ah probably the Arch Linux maintain just forgot that that should be easy.

A

Let's just figure out then make a bug report and this will be easy to fix so I cloned, um the official, um the official that the Linux kernel from the roster Linux project um write the documentation, build a kernel uh compiled my module again status worked. Of course, I can't load that because I'm running a different kernel, so let's just um switch to the actual kernel sources of my kernel, that's running and compiled that again um yeah. So this first thing that failed was like my rust compiler is to new, and this is an error.

A

I never saw in Ross before my rosk compiler is too new like rust, is basically about backwards, compatibility a lot, and it's like okay, that's interesting and then, but we can fix that. We have rust up. We can just override it and install the versions that it wants. This is from straight from the readme from the um Ross per Linux project. So then I saw that oh I need to switch some config Flags, so yeah, maybe Let's do let's do that, compile the kernel.

A

This took a lot of time, but it did work and yeah like in hindsight. This was like an important step that I didn't really reflect that if I need to change the config, maybe the configuration of the Artemis kernel isn't uh uh all that complete. So.

A

um But it should work now right.

A

The first thing I saw is that the rust module actually doesn't compile because the curve, the kernel already changed in the meantime and the interface for us modules was already out of date for um the Linux six Linux. So the kernel out of three kernel module use newer bindings than the rust than the Linux 6.1 kernel used, so I had to patch up that module and yeah. That was just fixing some compiler errors. This is easy to rust, the compiler so helpful right.

A

So you just do whatever the compiler tells you make a pull request, switch then got merged and everything should work now we could compile it yes success so so this was like the stuff. I did in maybe a few days on the evenings, whatever when I had time- and this was pretty straightforward, actually and then I tried to test it to actually load it. And then it's uh says: okay, the version doesn't match: okay, yeah, let's just recompile it again with like the correct version Magic with from my current configuration whoops.

A

Were my notes, um so we can fix that and then the next thing I saw is unknown symbols and then I was like okay.

A

Maybe my kernel isn't really ready for us, despite the configuration saying so so we went along like maybe try to use the actual Arch Linux config that they use for their kernel and to compile with that again and then we got a very awfully familiar looking error at compile time. So basically, it's just like yeah those symbols aren't in the kernel, so I can't link against them and previously, just like a local build of Linux with different config, but I could link again, but then at runtime the symbols weren't there.

A

So we couldn't link so I just select the runtime errors from Dynamic linking of the current module at compile time, which then was like okay. Let's investigate how the configuration system of links really works so, basically, just because somebody says config have roster, yes doesn't mean that the kernel has a raster Port, because that kind of works automatically in the background that a few things are checked like is Ross available. It checks for a compiler that may or may not be true depending on you, have the compiler installed and then there's also like.

A

If you have certain other configuration options, enabled it will actually disable the um rust support, because it's not compatible with that, for example, the GCC plugins and since the art Linux Compass is Kernel with GCC the GCC plugins already enabled, and this disables the RAW support, even if they have the config have rust. So that's a yes. Basically, this flag is just to disable the support. If you would have like all the stuff, you need for Ross color modules. If you want to still disable it, then you can set this flag to no.

A

So the default is just yes for this flag. Well yeah. So my investigations in the beginning were a bit uh uh misleading yeah. So actually, what did we learn out of this short exercise for the RAW support in the Linux already to compiler with clang? This means you don't have all the architectures that Linux supports means. It basically means we have arm and the Intel architectures of 464 bits and yeah.

A

If you load an extra module that was compiled for a different kernel version, the kernel will sanity check that and you need a specific, stable, rust version, because the little kernel does a nasty hack. It sets the rust bootstrap variable to 1, which usually only the rust compiler itself uses to compile itself, because the rust compiler also uses unstable features of itself to dog food. These features to see if they are practical and so the leaves current looks the same.

A

So that means you need a specific, stable version and if you use a newer one, it may or may not work depending on if the onstable features actually changed. But we'll get back to this yeah and, of course, the module interface for um everything is unstable. Basically in the Linux kernel itself, because it's meant to have everything in one single repository out of Team modules. Aren't really that well supported right. So you need to kind of like update them constantly to match the current kernel.

A

Yes, I think I need a bit of water uh yeah and the kernel has automatic configuration variables which detect if they should be enabled automatically, which is a nice thing. If you know about them but surprising, if they just appear and actual RAW support doesn't only depend on config have rust.

A

A

So um one month later, I decided to like, um because it should be so easy. I decided to just combine my own kernel, which um has support for rust modules.

A

The motivation was yeah that artist doesn't support it and I just want to have like a most compatible um like they're, basically, the same kernel that I can boot on my laptop and everything and just have it with support for rust modules. I mean alternative for p to just use the kernel from the rust for lyrics project, but this is way more unstable, so so I decided to just uh build the original kernel yeah.

A

So I created the package which user, which might basically I just took the official package change the config iterated over it a few times until it actually worked.

A

um Yeah make sure you need to make sure that the correct trust version install is installed. You need to set some other config flags and you just yeah do voice thing. You can look at the package if you care about the details, let me try to actually build this kernel, reboot and finally, profit and yeah. This was first success. I could make this kernel load it. It showed up and did run so Victory.

A

Finally, I compiled and ran a kernel module in Rust, but if you watch closely- and probably you didn't have the time because I switched the slides pretty fast, I did I had a small little workaround I just passed the kernel module, my local Source tree of the Linux kernel.

A

So this means you basically need to have the source tree locally and to be able to compile out of three car modules, which is kind of like defeating the purpose, because you want to have an out of three module because you don't want to work in the Linux sources itself and yeah. If you just install such a kernel with your package manager, it will be kind of pointless because then you would still lead the local tree with the local artifacts. So if we remember a bit back, um it was this quote about the build metadata.

A

So let's actually try to package this correctly and yeah. The build metadata stuff goes into Linux roast headers package. Bytes called headers is a bit unfortunate, but that's just the way. um Rcs decided to name this package. Basically, in the beginning, it was probably just a c headers if you want to write a c module nowadays, it's a lot of other stuff that you need as metadata to build out of three kernel modules.

A

So my strategy was to try to build the out of three module check for errors and then add randomly files from the build output until this works, but more or less I did a bit more systematic but yeah.

A

Basically, I just did re- package the metadata package with different files according to the errors I got so the first error we had was that Target Jason, who knows how the Target or Jason is well I, have a few. So it's basically like you. Normally the roscompiler, you said yeah, let's compile for Linux, let's compile for Mac whatever, and this is like a Target triple the deposit like Linux, unknown or whatever.

A

And basically, if you have a not super, officially supported targets, then you need to specify it in detail in a Json file, and this is the target.json file. It just will tell the compiler what feature supports your CPU uh What. uh What is the calling convention and stuff like this? So, of course in the Linux kernel environment is it's a different environment, and so the compiler needs to know about that so yeah. Let's just add this. This is pretty easy.

A

Then I saw that okay yeah. We need to actually the whole standard libraries that the Linux kernel has like its own flavor of they basically took the core library from from rust, um which is more or less vanilla without many patches, but like the alloc library, for example, this is heavily patched for a lean kernel and so yeah. It needs to have those available.

A

So basically, in the end, after giving up of the few iterations of not getting the build to work, I just added everything from the rust stop directory from the kernel build, and it's still didn't work. Yes, and this was like, hmm so why does it complain that it's combined with a different Ros compiler, even if I set like the override locally, my kernel module folder?

A

So it turns out that, like if you're building out of three kernel module, the build itself will execute in the folder where the metadata is and just make files a reference to your directory. So you see it in the first few lines here like yeah.

A

So what do we do? We just overwrite it with a rust tool chain file in the variable metadata lines, so the trust up knows which compiler to use then in the end we could get pretty far.

A

But then rust, complete complained about like having that the core library is compiled for a different Target than my um out of three color module, and this yeah I mean we have a custom target.json, which is basically the same because it's just copied, but still um the compiler complaint that it's uh different and the targets triple hos, which is normally something like yeah.

A

Something like like this, it's just like a number, and it was pretty confused about that. But this is strange number we have a custom, Target Chase in any way, it's like the same file for both of them. Yes, what could possibly go wrong, but this at this point, I gave up for a few months and what did I learn? Building the workable kernel was surprisingly easy packaging.

A

The correct build metadata, not yeah I mean Jason is used to describe the kernel environments, the Ross compiler, the current use, it's basically its own standard Library, which makes sense it does the same for C and yeah out of three modules get executed in the concept of the build metadata folder, not in the folder you're residing on.

A

So um let's continue to the final part of this interesting journey to actually get a working kernel module in tune. I decided um basically because uh Stefan suggested that I give a talk at this at this Meetup to actually follow up on this.

A

And yeah in the meantime, I did update the kernel a few times just to play around with it ran it on my laptop for, like yeah, pretty stable, I and then I was like. Oh now, it's Linux 6.2, it's released. Maybe something changed in the build system and it actually works so yeah and also I had some nice interactions with um Miguel Ojeda.

A

Hopefully it's pranced like that from the Ross for links project. He was very helpful and yeah so like maybe let's um to explain why I was so resistant to um actually figure out. This bug is like what I learned, like growing up as a developer: I used to develop a lot of C plus as well and rust, and basically it's like there's this Golden Rule that you never really blame the compiler for an issue. You have it's mostly like that you didn't in C plus this you have undefined behavior and or something in Rust.

A

It's like you just understand what you're doing and but most of the time the compiler isn't the issue. So One does not simply blame the compiler. It's just like you usually don't do this, except for this time it was the compiler yeah for the custom, Target Json, the Ross compiler uses a hash. This was the magic number to assert that the target is the same but um decided for some reason, probably as a quick hack, to just look at it to run to use the path to the file instead of the content of the file.

A

So the magic number of the of the target Json is the path of the file which obviously changes if it's like from a local, build directory to to build metadata directory in user lip modules yeah and this block was actually fixed in rust's 1.63, but we kind of must compile the kernel with another version right, so ah dang yeah so close like, but maybe let's try and actually then it was surprisingly easily just use 1.63. You get a few warnings about that yeah.

A

This is not the version you should use it may or may not work. Then you get a few warnings about like the features that were previously unstable are now stable, so it's not necessary anymore to enable them, but it just worked so finally, something that works yeah then I actually also figured out what is like the minimal metadata that you need to build rust Uh current modules. It turns out it's just like the target description. Obviously the compiles the rest standard Library.

A

So this is like the uh the core Library, the Outlook library and some other libraries. The prick compiled procedural macros I, never really realized that procedural macro. Who knows what procedural macros are? Almost everybody that's like if you have derived blah blah blah or your stock? That's a procedural macro! So this if you have custom, whereas of that they get compiled in a shared library and the Ross compiler just loads. This shared Library I didn't realize this.

A

Until that point and yeah, you also need to tell the compiler, which version to use or rust, of which version to use and yeah. What did we learn? Sometimes sometimes it is the compiler that was my big learning and just upgrading one version of the rust compiler is probably safe, because if they would change a version, a feature that was unstable, they probably would make it unstable for another release and not yeah. If they stabilize it, it probably didn't change, so yeah should be very safe forward. So this was the end. I could compile it.

A

I could run it and then I was like oh I now just need to follow the Upstream version so to have the most recent up-to-date kernel and yeah I mean. Basically, this should be easy.

A

I mean we did it. We can compile out of three modules. We can just update and play around and actually do the thing I want to do originally play with those kernel modules and see what stuff I could do if in Rust, but right wrong, updating kernel and suddenly I've compiled errors again yeah.

A

So during compilation, Ross writes a lot of temporary files. We know that because the target directory grows to several gigabytes right, that's a well-known fact that the cross compiler is quite verbose in its outputs and suddenly we can't seem to write those files, so permission denied, probably because it's in the field metadata folder that wants to place something and yeah.

A

At this time, I had was already a bit in contact with Ross for Linux developers. So I just asked in the chat and yeah the solution was the rust compiler writes temporary files to the current directory, even if you specify where we want to add the actual output. So if the M8 flag, you say like I want my library there, because I need it there and Rustin just decides to yeah. The temporary files. I will still keep him here, because nobody will notice.

A

I will delete them afterwards, so we just need to pass the out to your flag to uh the compiler. This was a the patch was just a one-liner figuring out where the one line needs to be changed, yeah, probably they're. Luckily, for me there were competent people who actually know how the Linux build system works, and so, if we could edit there, so what did we learn so far? Nobody cares about current model so far, except for me, apparently, which is kind of like um it's still in the beginning.

A

If you want to do out of three stuff, which basically means yeah, why won't you do it Mainline uh that's! This is the way that kernel modules are developed. So if you need to update it anyway, all the time because the shaking changes will be frequent and breaking um yeah, you should probably develop it in in line of the kernel main line of girl yeah. But the rust for Linux Community is very helpful, so they um yeah once I finally got to know. I got to know the right people to ask.

A

If I have problems, then it was pretty pretty decent experience for me all right. So now some outlook on what stuff is happening. uh New York 6.4 got released like for auctionex. It was like a day before I think so, and the mainline hurdle was a week before or something um yeah I will update the package. Try out if it, if it again, Break Stuff they're landed a few interesting things in the uh like, for example, pin init API, which is a was a quite a hard problem to solve according to this post.

A

So who doesn't know, what's the PIN API does in Rust, okay, a few people, so the pin apis. Basically, if you have something that shouldn't move its memory address.

A

So if you have like just every box, if you, um if you switch a box it the if you switch the allocation, because you need to allocate a bigger thing- that's inside the box, then, if the allocator will allocate a new memory put the stuff there and yeah, your box will just point a new memory, but inside the kernel a lot of stuff shouldn't just move the address because they could see because it's a mixed, C and rust code base.

A

Obviously the they didn't decide yet to rewrite it completely in Rust, which is a shame, I guess, but uh quite obviously not really feasible.

A

um So a lot of some uh once you publish like a reference to to any structure which is basically just a pointer in C, then you shouldn't just move the location of that thing, so you need to pin it to a specific address, and since this needs to happen quite a lot and it's quite unsafe to pin something they need to build a safe abstraction. For that, and this was uh yeah I mean I.

A

Don't understand it well enough to explain it to you how it works, but I found it interesting to read it's a yeah and maybe I will actually start playing around with the car with current modules in Ross, because my uh but my initial Target lost was to have like a use. A Raspberry, Pi compiler leaves kernel for that and have some ice cream C device or something speaking with a kernel, module and yeah.

A

Maybe I will come to that in the next few months and, what's really nice is that Linux 6.5 will finally operate you in your compiler and they will change their policy of how they um upgrade the tool chain, because you we're using unstable features any air anyway, so it doesn't make sense. You have like an old kernel version, because then you make life just harder for people or just have the latest kernel version. So it's much easier to just have just follow the latest version.

A

So, with the next few kernel releases, we will just see them following the latest kernel as the latest rust compiler releases, which will then hopefully also fix um box uh earlier, so people won't notice them and yeah. So the the target is basically to just follow the latest stable release until all the onstage.

A

Now unstable features are stabilized, which could probably take some several years and then Define that as the minimum kernel, where, as the minimal rust version that the current requires and from that one, it will actually be a minimum version and the warning will hopefully disappear.

A

Yes, so of this talk you may think like.

A

Okay, do people actually use rust for Linux kernel modules if it's such a pain in the butt and people actually do very interesting stuff with it, I mean who has heard of like the Apple M1 gpusing, the written raster, almost everybody, because that's like a big news, I guess and yeah I mean this is quite an uh an amazing project, in my opinion, to reverse engineer a GPU and write a driver driver in a language that isn't yet really ready for that yeah and there are also other interesting stuff happening like the Ross double block drivers, so like the hard disk and stuff like this is always handling block uh device drivers in Linux, and so this is like kind of like the Domi driver for the for a device similar to death null um which then basically just lets.

A

You interact with the API in a safe way. So uh The copier some point where you can actually write block drivers in in Rust if this goes further and there's also a lot of abstractions for Network device drivers, also with Adobe driver at first but yeah. So it's kind of like on all fronts. There's kind of like progress in making more subsystems of the lens curl available for modules written in Rust.

A

Yes, so this was it already for me. I hope, I didn't set too high expectations with the description of the talk, but but yeah. If you have questions, that's uh now is the point.

A

I

uh Yeah, thank you for the talk I really enjoyed it. uh Could you go back to the final Slide, the one you had before this one so.

A

I

Was curious about these? What is the um output of this in terms of stuff that other rust developers can use? Are there crates coming out of this to allow it to make it easier to write drivers, or are we still forced to use the API that comes out of Linux.

A

um So you mean all those abstractions are being built. Yes, I mean there won't be crates on crates.io, but they will be like a Ros modules inside the Linux kernel tree, so they will build kind of abstractions over the um of the existing kernel apis.

A

So it will make those abstractions are heavily impacted with unsafe, of course, because they need to interact with c and getting those things right to expose in a safe, API is pretty hard. So the the target of a lot of the rust for Linux current developers is to actually have like it that it should be possible to write a device driver in complete, safe rust, which will make life a lot easier for kernel developers.

A

If you don't need to care about memory safety, while writing a device driver yeah but building those abstractions is kind of hard I guess uh over here in the front.

A

What happens is.

A

If you panic, I, think you just get a colonel oops message and can call it a day yeah and.

A

But actually the question is maybe more interesting in. If you look at allocations in Rust, if you have like a as just a whack, you can just push stuff to it right and we'll just allocate more memory, more memory, more memory, because it's basically infinite on a laptop right, um but not so for the kernel, so they needed to implement like a different allocation API that allows allocations to fail. So you have a lot of like text.

A

Also extended back the vector implementation has a lot of new functions like try push and try whatever to basically not panic. If, if the kernel runs out of memory, because you need to handle memory allocation failures in the kernel, you need to handle them in user space, you can just say yeah should. If it really runs out of memory, the out of memory killer will kill my application, then we'll restart or whatever, but yeah in the kernel. You need to actually care about us.

G

Yes, yeah so now the question uh you went on the llvm route, I guess yeah I think the other thing was to bring a rust compiler into GCC. If I remember correctly, it's like was that a no-go from the start, or why did you go with llbm that.

A

Was a no-go from the start for me because it basically you're just setting the Ross for Linux documentation. This is currently not supported, don't try it yeah. So I mean there is the rust GCC backend, but this is still work in progress and I. Think there's also GCC rust front end, which is also work in progress. I mean once those things are more mature it.

A

Hopefully you can also compile the modules in the Linux kernel with them, but until then uh yeah just use llvm clang, which on like on an Intel machine, it doesn't really make a difference. There's not a not many um kernel modules depend explicitly on GCC. There are a fews that just will get disabled but yeah at least on my system. It didn't cause any issues. Yeah yeah, exactly you see, actually see some automatic options disappear in your config file. Yeah.

B

So, regarding the memory allocation, do you know if they started the place memory allocation where you can specify the memory Arena or the allocator function for a specific allocation.

A

So just a custom, allocator or you mean something like placement, new value.

F

A

I think placement new in Rust itself is still unstable and maybe they use it in the kernel. I actually don't know I mean it would make a lot of sense, especially with the pin in its stuff, but there it looks like it just uses. I am heavily uses, macros, so I think they don't use that specific, but I knew ever need to look into it.

A

Okay, so that's a good answer, yeah. So the placement- you doesn't work reliably, so they just have a custom, allocator yeah.

B

A

B

B

So we don't have any online questions so we'll take a little break. uh Thank you again. Raphael you're welcome and if you give him a huge round of applause, he will maybe come back and while we re-plug all the things just grab a beer and then we'll continue in five ten minutes. Thank you. Okay,.

B

B

Hello Internet.

G

B

Right, everybody, let's continue I would like to introduce to you uh stream section it's not inception for you, but for everyone watching in the Stream. This is our fancy little website, it's a very wavy because we're at the lake and when we scroll down. Currently there is the live stream, but afterwards the talks will appear there. It's a normal playlist on YouTube, so no need for extra tools, and here is the the batch of the rust Fest from 2017 and we'll see we'll probably do another one, maybe next year, maybe the year after.

B

So if you want to hand in talks just watch out when there will be announcements cool so in 45 minutes, the pizzas will arrive fingers crossed, we might be a little bit early, but that's okay. So you can ask him a lot of questions. You can also ask other questions and if we're still bored and no one wants to stand up or talk to each other, we can play another game on the Beamer.

B

But I will ask you again, so you have 20 minutes half an hour to think about that. Holland stage is yours. Thank you.

J

So who here has heard or read about iron fuel? Not the HTTP framework, not an orm, it's completely off topic, it's an actual energy source. You can burn iron powder, you get heat and rust and I'm kind of excited about this. If you're like me, your heart is in school, you can burn iron, but there are actually people working on a project in the Netherlands there's a brewery, that's powered by iron powder and rust comes out of it and I think you should read it up.

J

So this is the I explored something and just want to share it kind of talk. I'm Roland I work as a PHP developer at dvf in Basel, and also as a rust freelancer for TP, I, co-organize, rust, bustle and puzzle hack at Ross parcel. We just had a full day workshop on Saturday for Ross beginners. Also thanks to Yasin and besides rust and hackathons I'm passionate about code tools, I think everybody should have great tools which.

J

Leads us to our use case, which is also about tools. In particular. In my day, job I have been working on a project that has extensive configuration, need I'm, talking, configurable business processes, working on configurable multi-dimensional data, and we built the configurators for the customer as a GUI wizard kind of thing, but somehow we ended up doing all the config work ourselves.

J

The same thing can happen if you build a website with a CMS and plan for the customer to configure stuff, you might also end up configuring much yourself. It has happened to me with cms2 and gosh. Do I hate these configuration, gui's, I'm really happy typing all day, but if there's a day of configuration clicking, my wrist hurts from using the mouse. So much and also my head.

J

When coding, we have it better, we also configure a lot. We configure systems, components, libraries and we have great tools for it. We have our Ides type systems, syntax, highlighting linting and refactoring tools and I should probably also mention co-pilot and the other AI tools and, in my opinion, the text medium for configuration is just a universally good solution.

J

The configuration is already serialized. You can share it over slack, you can Version Control it. You can attach it to support tickets, you name it you get so much free. If you go with text.

J

So why not give this to people to customers? Why don't we give them code-based tools? Maybe they like it better too, or at least if we end up having to configure systems ourselves, we know we will like it well. Non-Developers won't use an IDE because of setup and and context and setting up the ID does the smallest problem here before editing any config files.

J

You need Version Control and then you need to deploy when you're done editing it there's a lot if you're, if you're a non-coder and just configure and config and code, therefore, are often very distinct words. Sometimes you have to resync into your code base serialized config, sometimes there's still a file format for it, but maybe we can make this better with a low code. Format provide an in-app IDE that has context so the the right config object. You can call it file.

J

Even if it's not really a file, it's directly opened in the browser in there you can just import what makes sense and you can just call variables that are defined same as with the QE wizard, but you're writing code.

J

uh Sorry here, I go ranting again. Do you notice, when you have two windows open side by side, one for the staging server one for prot and you you copy the the configuration from from left to right by clicking through a GUI I'm, not sure who hears uh web developer in some way?

J

Okay, not so many um yeah! It's a problem uh yeah that could also have been text in a clipboard or in Version Control.

J

Anyways I think we already have enough Vision by now for this for for building or defining a POC, we want to bring the developer experience into a text field. For this. We need to set up the system end to end, and we need a good showcase for something you would want to do in a configurator.

J

These are our goals and I wanted to build a POC using language server protocol, because LSP is a great building block for for code tools. It provides an interface between languages and editors, so you don't have the combinatorial explosion from the ever-growing number of languages and every growing number of editors and have one by one Relationships by defining each language for each editor.

J

So a language can provide a language server and with that it is already supported by vs code IntelliJ, Vim Etc and each editor that can work with LSP protocol has support for a plethora of of languages.

J

The POC should just validate one possible system design, so we can improve from there. So in the browser we have ID the ID Monaco a strip down web version of vs code I was able to profit from from work of type folks. They open sourced a bundle of Monaco and and the retrofitted a lot of features to the strip down version, including LSP, and the actual LSP server runs on the API on an API service which is serving the actual app allowing for integration of other features of the API, such as an actual app around.

J

Our Gadget communication is done in Json RPC over web sockets, which makes communication inspectable by browser, which is always a great feature when debugging web apps.

J

So I built this really simple POC one on the 30 lines of rust code and some react. The language server is defined on the server side, with Tower LSP, so axome is the the framework and Tower LSP is one of the middlewares from the from the tower ecosystem that are supported by oxum and POC connects a Monaco instance in the front end with a language server in the back end.

J

Let's see it in action and inspect the communication over web sockets.

H

J

J

F so I have a front end and backhand process running. You can see here the front end in the browser. What do we see here? We have our text field plus plus for editing a config object in Json format. The text field is a Monaco instance.

J

You can see here the small preview that is typical for vs code and Monaco.

J

um The actual Json is something you would want to configure for a financial application. For instance, you define a row of numbers and it's in Json format. As of now the language server, for this is really really simple. It's just the end-to-end system design and it suggests a few variable names as autocompletes, for instance, if I type in here something with row, I get suggestions actually coming from the server.

J

And if I hover, it says, you're hovering big deal.

J

Now, let's inspect communication over web sockets, let me open my developer tools and reload the page.

J

Response yep for now, um as we as we type, you can see that requests and responses are being sent over web socket and they are human readable in Json. Rpc format can inspect them as I'm typing ahead. There is at least one request for every keystroke.

J

These are the text document, data change requests and actually the the language server keeps a replica of the currently edited document in state and syncsole changes atomically.

J

um If I'm typing something with sum again, we also can see the completion request um yeah, which we saw in action before yeah.

J

So this is what the POC can do now. Obviously the example LSP needs more good examples and an example for formula language, but the system works end-to-end, go back to presentation.

J

Are we where we left off yeah.

J

So have you reached our POC goals? Well, the the POC provides an enhanced developer-like experience in a text field for configuration check. However, the function is still minimal. The really cool features like linting are still missing, but the system is set up end to end, and since we already have oxum, we could start building an app around it.

J

Is that already a showcase for a configurator, probably not uh the the Showcase language, definitely needs more work even more important. Is this useful? Let's see in principle uh configurator user? They can directly open a configuration and edit in a code like fashion, but with the guard rails we are used to as as developers too and system providing this functionality has to provide their their own language server. The tower LSP implementation with rust requires us to write our own, but we could also plug in existing language.

J

Servers for Json, maybe add a Json shamer for for the config constraints we have- and this does make sense in my opinion, if we do a language server protocol on the server, we have the possibility to combine it with access control. For example, I could only suggest completions for certain variables to to administrators or people with a with an elevated role.

J

One issue is every keystroke sends a request to the API web. Sockets are relatively efficient compared to http requests, but still we send quite a number of requests for what could be a single post request.

J

If bandwidth is an issue or we prefer more Logic on the client side, we could also think of the alternative architecture of running the language server client side in a web worker that also works.

J

What have I learned with this experiment? First, it allowed me to get a little more Hands-On, quite popular right now with axum you define routes, assign them to handlers and Handler, basically Maps extractors, to results.

J

If you want to try XM here's tip number one, you absolutely need the debug Handler from Axel macros. We also discovered this in the workshop on Saturday. Otherwise, the error messages are not useful at all. It can be a simple thing like you, you did not derived share the deserialize for for an extractor model, but you get a three line: really abstract error message, not mentioning the model or Saturday at all.

J

So this helps when exploring undocumented functionality a flat plus debug printing to explore the structure is kind of of a workflow that works see what is being returned. What's wrapped in what and in general I, had no issues or blockers with with axum with this bill when building this Pock, but also I'm, just using one single websocket Handler at the moment. So no not too much to to learn from from this combination. I think um for me, it was really new to work with language server protocol and this is not trivial. You.

J

You really need to to know your protocols. Well, it's it's well documented, but it's a lot and uh also I had this initial expectation uh that you can just plug in Tower LSP middleware, since axon promises to yeah you can plug in any Tower middleware, but I I did not achieve this. Maybe it works for normal um handlers, but not for the websocket ones, or maybe it's it's not obvious. Instead, I am in instantiating LSP service instead and calling it from the Handler just as a as a function. So that's not.

J

The full integration I was expecting and as always, the the rust type system helps you a lot in not getting things wrong. They just won't build if you get them really wrong and you get Fearless refactoring as a superpower when using rust, yeah.

J

um What's next uh I'm really glad that I took the first step towards this architectural idea and got something even if it's modest that works and now I'm wondering about what should be the next steps. Maybe uh you can help me decide what further direction to take should I should I build a library, maybe uh that seems a bit too broad of a scope, since we have a client and server.

J

Maybe it's more of a template, app really already kind of is but I need to add more and better examples of what kind of configuration you could um configure with this or specific apps that build on the on the concept: CMS configurator, maybe something for coding, competitions or hackathons um or other ideas with existing for collaboration with existing projects.

J

It isn't yet obvious to to me where to move next. um If, if it's obvious to to to you, uh please let me know in the in the questions part which I guess we can uh start now.

C

So I um I I have been quite uh I thought of the same thing or I kind of had the same idea for work, but almost exact same steps as you, as you told, but I haven't built a pocket because there was no time there was deployment and stuff. Let's.

J

C

We can talk later awesome.

H

Okay, so uh have you thought about using webassembly, so it runs completely in the browser without an API and maybe making it a library.

J

um That would be the the architectural alternative I suggested to run it in in the in the client. But for me that's even more new land, so I.

D

Try to to stick.

J

With what I know it's even enough exploratory, but it would be available alternative to to do this uh in the in the client. I think some of the examples of type folks. They run an LSP in the in the in a web worker with and build it with Witham.

H

Yeah because I think a library to just start complete completion to whatever website you know would definitely be handy. I guess.

F

So what about a Super, Simple Solution without LSP, just using three sitter I, mean the examples you show seem pretty simple. It's just a bit of a you know: code completion. Maybe this could be achieved with three sitter just.

J

Locally three-seater and cold mirror, maybe yeah. They I mean this. Obviously, um the more you have stayed from the from the back end that you want to consider. You would want something on the back end, but also three-seater could run on the back end. I guess.

F

But yeah it would be consider you could probably just run locally I mean within the browser, probably even.

J

Github, does it I think yeah.

I

Cool um I don't want to rag on too much about the architecture which is apparently all the questions have been about so far, but I was really interested to know. um Do you have? Do you have examples in your experience where the need for such a like a heavy client is motivated by so it looks like Monaco is pretty heavy. The requested sends a very detailed looking at the Json RPC examples you showed and it can do a lot of stuff.

I

It can do syntax, highlighting and all of that, but so do you think that the kind of languages that configurators can use will need all that stuff? Are they are they? Are they complicated enough to justify all of this extra extra things? I mean the the auto completion looks very cool, but also the the all the other stuff that the code editor gives you. Thank you.

J

It's a good question: it goes hand in hand with the with the suggestion of using a three-seater to to go more simple, I guess um yeah, the the use case I have in mind for for this project at work where we have this configurator. This solution is, is uh more more complicated and we we could win something uh from this, and the colleague of mine actually implemented something uh more lightweight in the in the in the front end after I I did this poke um so uh yeah. There are really multiple approaches to this.

J

It was a an interesting experiment, I think and uh yeah. The the big question is: how uh could we build more more uh interesting things out of it.

K

K

um Did you so you said you did the puck at work? Did you actually test it out with uh with end users or if you want non-non-development users, and was there any kind of feedback in terms of adoption or willingness to I.

J

No I didn't build it at work. It's just.

K

J

Use case that come comes from work, so.

K

Then you said your colleague built one at work, a lightweight one in the browser, yes and.

J

Like but uh he just started, uh um completions there's also more Neco in this and uh he added completions um that need to I mean when you you have Team static for for the patreon.

D

Okay, thanks thanks very good. uh A topic. I I agree with you that a kind of visual only configuration has has a lot of limits. Like you said you know copy or Version Control, or any of that now you kind of switch to the Other Extreme to have you know something: that's completely text-based. Could you combine the two kind of for you know you have simpler cases that can be done visually and more complex configuration that can be done with the text.

D

Space like you know, in Excel put a few numbers there you want to have the sum you don't start with equal or you could, of course, but.

I

D

Know you could kind of click on the sum symbol and then it's there and then, if you want to continue, you would kind of continue there. So it's a kind of a Continuum of visual and you know kind of textual parts.

J

That would be even better I think many terminal. Apps are also going in this, this direction of free when Reinventing the the terminal. So it's not only text only but I also has visual blocks um yeah. It could be worthwhile I guess.

B

There seems to be no more questions. Thank you very much. A huge round of applause, please.

B

So maybe we'll do a workshop in September with you who knows, huh would you be interested in a workshop style thing? Yeah, maybe cool all right. So uh with that we close for tonight and I want to draw your attention to this table. There are stickers if you want some, as I said before, there will be a break in August just for vacation reasons and then in September we'll be back in the city somewhere and Damian. Would you like to say something or.

G

B

E

Thank you very much for attending, so we are going to have pizza in 20 minutes. We are on the way. uh um In the meantime, we can have a drink here and in 20 minutes we can go on the rooftop on The, discus rest and see if it's better or worse than C, plus plus I'm happy to to Counterpoint and to see how good you can make them into inter interoperate in the future. So.

G

E

Let's have fun and thank you for your attention.

E

Thank you sponsored.