Rust Programming Language Rust and Tell Berlin, 24 Oct 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Rust and Tell - Berlin - Aya: Extending the Linux Kernel with eBPF and Rust by Michal Rostecki

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

A

I work at Deep fence, where we use BPF to observe the network traffic and write firewalls and detect the malicious traffic I'm vadalovski on GitHub, Discord, Twitter and other social media and nowadays are mostly write. Rusts I think I started writing cross professionals two years ago, but I have huge background in seago and python before in my career uh and yeah.

A

uh Well, in the previous talk, there was a question like how many of you are system programmers, but I would like to see again like how many people all right uh so BPF stands for extended, Berkeley packet filter and at the beginning it was only about capturing Network traffic and looking at packets and being able to write very small C programs which get the packet and decide whether it goes for or not, uh but nowadays it's more than that and I would say that the most Fair definition uh to use today is that it's a virtual machine with its own instruction set, which is able to run sandbox programs, and it came from the Linux kernel, but nowadays it's also coming to Windows, and so there are.

A

There is also a blockchain project called Solana, which is using the technology for running smart contracts in that kind of virtual machine, but here, in the context of the code, I'm going to talk only about Linux kernel, so the definition for Linux kernel is that it's a virtual machine which executes the code, which is triggered by the events in the Linux kernel and in your code.

A

You can get the event and observe it or decide what's happening with it, and it's even even driven and usually, for example, when you write a BPF program which is receiving a packet. Your you receive the memory of the pointer to to that packet, and you can look into its con into its content.

A

You can modify it, you can drop it or let it go through and there are many events which you can observe in BTF right now, but the most important ones, of course, are the network ones, and there are two types of programs: xdpn classifier classifier is very general. You can observe both Ingress and Ingress traffic and XDP is faster. It's like very close xdpx on the level of the network device driver, but for uh for now it's LS only Ingress traffic filtering.

A

So there are some limitations, but uh also the other feature of BPF is that you can observe the function calls. You can, for example, write a program which is triggered every time when some function in the Linux kernel is called and receives other arguments or received the the return value of that function. So you can kind of like debug and Trace the kernel by writing your own small programs, and you can also do it for user space for you probes.

A

So if you have a binary which is exposing a symbol, you can also write a BPF program running in the kernel, which is attaching to that user space function receiving arguments, and so you can also use BPF for debugging your user space binaries. If you expose symbols and also quite fresh feature, but it's work. It's working on the newest Ubuntu, for example, it's got introduced in kernel 5.7. You can write BPF programs which are attaching to LSM hooks.

A

So, for example, if you are familiar with aparma or slnx now you can Implement that kind of security is. The security system is similar to those just using BPF and you can use the same API to it and many many more and well. The important difference between BPF programs and current normal kernel modules is that BPF has a verifier.

A

So it's it's a virtual machine as as I mentioned previously, so um in BPF, it's very important that the kernel and that the the verifier and the virtual machine are making sure that you are not accessing memory which you are not supposed to access and that the BPF program doesn't panic. The kernel. Does it crush the kernel, so it provides more security because when you are writing like a regular kernel module, it can crash.

A

um In theory, ebpf program cannot crash and when you are writing a BPF based projects, there are you usually splitted into Parts. There is ebpf program itself which is loaded into the kernel, but then there is a user space process which does that loading and manages the life cycle of the ppf program and also fits the BPF programs with data there's a concept of BPF maps and that's pretty much like a storage for BPF programs, which is shared between user space and the kernel space.

A

uh There are many kinds of them: there are almost like 20 kind of maps, but the most important ones are hash. Maps, arrays and ring perf buffers, so in case of hash Maps is, is just a KV star, which is shared between BPF programs and um user space processes which loaded those BPF programs and perf.

A

Buffers are buffers in which you can push an event from the BPF program to the user space process, and usually usually it works that um you define your own structure uh where everything is sized and you write the data into that structure and push that structure into perf buffer. So it's, for example, to push a notification to the usual space process that I received the packet and um to or that's uh yeah.

A

And examples of BPF projects, the most popular ones are celium, which is the stadium, is one of the most popular kubernetes networking plugins um and there is also Tracy for uh which is applications Racing, 2 and Falco, which is a security security tool, but yeah the the most of popular eppf projects are written in, go and see. I mean in the most of cases, especially psyllium.

A

The user space part, which is loading and managing the BPF programs, and also interacting with kubernetes, is written in C in go and BPF programs themselves are written in C, but um yeah in AI. You can do everything in Rust and there are already some projects which are doing everything in Rust I mean both BPF programs and user space part there is bpfd, which is a project made by red hat, which is like a demon for loading.

A

Different network created programs Network filters, so it's like a manager of different network policies based on XDP and there is a project called Pulsar, which is an observability tool for iot devices. So it's like a set of small BPF programs which are monitoring what's happening on Linux on iot devices and are generating metrics based on that and yeah the there are official libraries to ucppf and the first one. The BPF um is completely based on C, so both user space and ebpf part and written in C.

A

The later ones are introducing the other languages in the user space part. But the ppf programs themselves are still written in C, which is different in Aya in IR. Everything is written in Rust, even ebtf programs, so rust compiler is able to compile to BPF bytecode. That's because generating BPF bytecode is actually a feature of lvm.

A

So in theory you could try to make a bring a support of compiling any language which has an lvm front end into BPF bytecode, but so far the only front-end switch support that are Clank, 4C and yeah rust, C, thanks to which we can write BPF programs in Rust as well nowadays and now I'm, going to talk a bit about benefits of air and of Aya, and why, in my opinion, it's um it's nice to write BPF programs in rusts.

A

Instead of C, but before that, well there is a story in about trust in and Linux care. Now so very recently, uh Linda Starbucks, merch Depot requests of which introduces the support of writing kernel modules. In Rust it's going to be released in the kernel 6.1, which will come yeah.

A

It will be basically the next version of the Linux kernel, so starting from the next stable Linux kernel version, you'll be able to write kernel modules in Rust, but well since I mentioned that the difference between kernel modules and BPF is that BPF is more secure and you have the verifier, which takes care that you are not reading, not accessing the memory you shouldn't access and that yeah that it's it's not crashing.

A

There is also an initiative of rewriting some parts of Kernel in BPF itself uh and so far it's happening in C, but um yeah. The point of AI is that yeah you could do both like. If you are rewriting some kernel related fictionality in in ebpf, you can do it in Rust uh and well talking about benefits of uh of writing BPF and Russ and Aya in next slides, I'm going to talk about benefits, but people are often wondering if I write BPF programs in Rust.

A

They are probably more secure right in terms of memory because there is borrow Checker and unfortunately it's not the case. The most of BPF programs are receiving the pointers to the kernel memory or user space memory, so the most the most of the time you are dealing with the memory which isn't allocated by the borrower tracker by by rust, but yeah. You are receiving the memory given you by the kernel and accessing it is an ISO, unsafe operation. So, from my experience, half of your BPF code is going to be unsafe, so yeah.

A

So that's the remark that writing vpf programs in Rust has many better fits but uh yeah like um dealing with memory in BPF. You still need to rely on appf verifier and on the Kernel how they do that rust. Isn't saving you much with that, but the first benefit is type safety.

A

So when you write a BPF program in C, C isn't as strongly typed as rust. So here we have an example where we try to write an XDP program, which is the kind of program which lets you observe the packets on the network device driver level. But then we provide a wrong type of argument because, instead of providing the XDP context, we provide SK buff, which is related to networking, but it's. But you should use that argument in different types of networking programs.

A

So, instead of using xdpmd, we are using SK, buff and yeah when you compile it with Clank uh it compiles. There is no error whatsoever and you are going to see the error after you try to load that program actually but compiling. It is completely fine.

A

um But when you try to do the same mistake with rust, so here we have the XDP macro. But then we try to use Escape buff context um here. Rust compiler will complain because yeah macros included in IR strongly typed. So we make sure that you're using the proper types for a proper type of BPA programs.

A

The other thing that's super, that's very superficial about error. Handing in C and especially in BPF, is comparing integers of the time, and it's nice that when using air and thrusts, we can return a result. Of course. At the end, we need to wrap it in that kind of function where we match and return return the integer, because that's what kernel is expecting so at the end, kernel is expecting the ppf program to return some integer status, but yeah.

A

You can just wrap it and write your bigger logic in a function where you are actually returning the result, enum and handed the errors with quotation marks and Native frostway a very important thing. Currently, when writing BPF in Russia, you cannot Panic, we just don't allow it. So all the errors in your BPF programs have to be recoverable, so yeah. The principle is that if you write a BPF program in IR, you have a bigger function where you return result and you make sure that everything is is handled.

A

And rust to Chinese everything he needs when he is IR, so we are not wrapping Clank in any way. There is no need to install clunk or, like external VM, you just need to install Russ roof rust up, install few libraries and we are ready to go, and we have also a logic Library, which is a difference between writing BPF programs and CN trusts and see. There is the official way in a c to push it to to use.

A

Logging is the prepare print K macro, but it's not pushing those locks to the user space process, but it's used pushing it to the kernel Trace. So when you have multiple programs running there, the the logs are getting mixed there in IR. The approach we took is that our login library is using perv buffers to push the locks straight to the user space process, which is managing the lifecycle of that BPF program. So, for example, when you are writing some application, which is running all kubernetes, let's say, and this managing BPF programs.

A

You are going to get all the logs from the kernel space to that user space process and then, when you do for a cube cattle, lock your application name, we are going to receive them there. So yeah. We just decided that the architecture where you are pushing all the locks to the Aerospace process, which is owning the BPF program, is a better approach and yeah.

A

It looks like that when you so that's clone binary is a user space process which is loading a BPF program, observing the Clone Cisco in the Linux kernel and every time our BPF program is observing that there is a new process created. It's pushing the logs straight to the user space process.

A

uh We support a support async on the user space site so yeah. It is a space part of the project and Bs and Chronos. We mostly support Tokyo, but there's also a support of async STD, although less commonly used um and yeah the steps to start with ir. First of all, installing nightly that's because uh BPF Target is still tier, free, Target and rust C. So it's not stabilized.

A

We are working on doing that. But for now you need to you need trust nicely to compile the BPF programs. Then you need to install BPF Linker, uh which is our custom Linker, to link together uh the BPF BPF different BPF objects into the final BPF object. It's not done in lld. Yet there is, there are going patches to be able to do it with the official lvm Linker, but so far yeah we have to.

A

We had to come up with the custom Linker and then the last thing you need is Cargo generate to use the template of our project because we provide the template.

A

So it's easy to start and when you try to generate the new Air related projects, you are prompted for what kind of program you would like to create and what project name and yeah then, depending on the program program type you choose, you might be prompted with some more questions, but at the end you receive a simple example of the of the program which you can start modifying and the project layout yeah I mentioned before that. Usually BPF related projects are divided into the user space part and ebpf part.

A

Here we have, we add, also the common crate. So the purpose of common crate is to store some. If, if you need to have some types or structures which you share in between BPF part and user space part, it's very convenient for providing types use in BPF Max for maps, for example, then you are supposed to put them in the common crates and yeah. The firework rate is the use of space crate, which is managing the life cycle of BPF program. Firewall BPF is the BPF program itself and it has the rust tool.

A

Trade term file, which is enforcing usage of rust, night leader and TI for convenience. We provide some x-task comments for building the project and yeah. Those issa's comments are.

A

um There is a extract, scrum comment which does everything so it compiles all the parts of the projects and uh finally executes it, and we have an x-task command for uh building the ebpf crate, because otherwise, we've had this excess commands. You would need to write cargo plus nightly and provide all the arguments that wouldn't be so convenient, so yeah by default in IO template. We just provide extra comment for that.

A

There is one huge thing missing in Ayah, so in BPF there is a concept of compile once run everywhere, which is a mechanism to handle the differences in kernel structures in between kernel versions. So, basically, BPF has its own debug type format called BTF, which stands for BPF type format it's generated from dwarf, so basically, what lvm is usually doing is that it's converting the dwarf debug info into BTF, and one of the purposes of that is to be able to relocate the kernel structures when they differ in between kernel versions.

A

That's not supporting an IIs. We need some more work there, but yeah. Hopefully in some time like let's say half a year, we'll be there uh so yeah one remark: uh Aya is I would say very ready to use when it comes to networking programs, because in networking programs kernel structures, don't change much. There is no chance that the definition of Ip, header or ethernet header is going to change anyhow in the next 20 years.

A

But if you are using something some kernel features or some kind of structures which change in between recent kernel versions, then yeah so far you still need to do stuff and see, and but hopefully we will get there.

A

Okay enough talking and let's see some simple firewall written in Ayah in action. So here I have a quick demo.

A

So yeah, let's try to access the IP address 101. For now we can do it and we have a small BPF project which acts as in firewall and is supposed to block all the egress traffic going to 111.

A

and as we see it's working, so we have a lock showing that the traffic going to 111 got blocked.

A

And to show that's not a fake stuff, we are, we can try to change the IP address to 8888.

A

And see that it's that filtering is still working.

A

A

And some a bit of cold example: maybe yeah I still have a few minutes. So that's how the BPF crates looks like so we have a. We have a definition of hashmap, which is basically our hash map of IP addresses, which we want to block. We store uh all right, yeah, so with it's ipv4.

A

Only so yeah for convenience, I'm, just using 032 for storing ipv4 addresses I, have a helper function for getting the packet headers from because well here we received the XDP context, which is like a which contains the whole packet, and then we are using that PTR at helper to firstly get the ethernet header, then get the IP header and so yeah get the value of the source IP address from IP header as as other fields, and then we are using the block IP function to check whether our IP address is in the block list, whether we are supposed to block it.

A

If, yes, we do XDP drop. If we allow it, we do XDP pass, and here we have the user space process. We initiated logging, and here we are loading the bytes of the BPF program. So the BPF program is like at the end kind of embedded into the user space binary.

A

We initialize the logging of BPF, we get our program and loaded and attach it to the network, interface and yeah. Here we have the club option for the network interface, which we want to listen on. Oh.

A

And yeah, then we access the block list map, oh my God, and at our IP address, which we want to block to that block list. We insert it and then yeah everything is done and we are just waiting for Ctrl, C and uh well here. The life cycle of the program ends with the scope of the main function. So once we press Ctrl C and the program exits, the lifecycle of that ppf program ends and it stops working.

A

That's why, on the demo, you could see that when I interrupted the user space process, the blocking didn't work anymore, it was allowed. There are options to extend the life cycle. If you want, you can get the get the link to the program and pin it in the BPF file system. There are ways to not make the um lifestyle figure out the program dependent on the scope of your function, but by default yeah your program is dropped. When you go out of scope, okay, I think that's everything I wanted to talk about.

A

So do you have any questions.

B

Wow lots of questions. Does this work um I think he was first.

B

Okay, thank you for a talk was very interesting, I wonder about the restrictions of the evpf part of the program because it runs in kernels. So obviously you need to use no STD, I guess yeah.

A

B

There will probably be other restrictions like: can you even use external crates and the eppf verifier? Is it true that it needs to prove that the loops terminate at some point and and what about the size? If you include like lots of template heavy code like? Can you talk a bit about that.

A

Yes, so actually you can use crates without any pro external crates without any problems, as long as they are no STD so and then no STD crate can be a dependency of ebpf crate. There is limitation of a number of instructions, so one BPF program cannot have more than one million of instructions in BPF assembly.

A

uh If your BPF programs are too complicated, you can split them and do the so-called tail calls. So you can do up to 32 take holes, so you can, at the end of one program, called the other program uh to keep doing the stuff. So in theory like if you plan it properly, your limit is 32 million of his instructions, but then you need to do those State course logic and um and what else um yeah?

A

You cannot do too much Loops of course. So you need to make sure that when you make any Loops that they are limited by some constant value which the BPA verifier knows about, if it's some Dynamic value, which BPF verifier is not sure how big is it, it will of course complain because it will assume that's yeah like there are infinite number of instructions to be done so yeah.

A

You need to be really careful that if you put any Loops that you bounded to some constant value, which is not too big and same also with accessing some memory, so you really need to make sure that you have some constants with the max size of stuff you're accessing.

B

B

We already have a few.

C

And then we can maybe fit a few in yeah yeah.

D

um My question was um you, you say it was forbidden to those panicked and unlit um panic and you, oh, how did you manage that in in your library, dude yeah.

A

For now we just forbid to panic, okay, so, oh dear yeah, so so when you write a ppf program or the yeah, all the errors need to be recoverable. You you cannot unwrap. You cannot Panic directly yeah well,.

B

What happens is compiled error.

A

uh Not compile error but definitely verifier is going to complain perfect.

C

And uh questions.

A

But for for compilation, I think that in examples we are explicitly doing.

A

Yeah here we have our Panic hander ownership, but yeah. If you wouldn't have that then yeah. It would fail later in the scope of running it and trying to run it in the kernel.

C

The first questioner had the brunt of my question and the answer was really interesting, but I'm also curious when the ebpf verifier runs.

A

uh So it's France uh when loading the program, so it's it's basically in the kernel. It's um part of this kernel cheat so before karna actually starts executing the program. It looks into the bytecode and has a set of rows. What to look at.

C

And I'm just curious if it doesn't rely on any runtime Behavior Could It, Be Moved as into a compile time step.

A

um I didn't think about this, um maybe if it could yeah, because it's basically yeah it's operating on the BPF bytecode, which is generated by llvm So in theory, it could be maybe moved uh to the compilation step before but yeah for now it's the part of loading in the kernel, yeah I'm wondering.

A

B

Prevent you from you can load anything that you're into your calendar. You still cannot crash the.

A

Car I mean yeah in that Vision I would say that the verifier stays in the kernel, but we also have some step in llvm to get compile time error, because that's what you meant to have the step in where LVN call already complains. That's an interesting idea, I think but yeah it's it's just not done. I! Think.

E

E

We have time for two more YouTube.

E

Yeah, so my question was also related to panicking. So one of the main reasons why Linus wasn't really convinced of having rust in the kernel is that raspberry fall tends to panic rather than giving undefined Behavior. So my question is what, if you? Actually? You know, you don't call in a rap, but you could still Panic somehow right if you fail to allocate memory for your data structure or something rust, can't panic and that's not quite a compilation time, but the I guess the verifier right.

A

um I mean yeah. If you fail with allocation um on the stack because well in BPF you, you can only allocate on the stack and the stacks and the stack size is 512 bytes. Otherwise, if you want to allocate something bigger, you're supposed to use BPF maps and then yeah, basically kernels responsible for allocating the memory, so it's yeah very common case that you put some bigger buffers in maps.

A

If you, if you want to store kilobytes of data, let's say um yeah I, don't know what to say: I never encountered such a problem that we would panic, because of rust, being unable having some issue with allocating something on on the stack memory. I'm.

E

Not sure if that's still the case but I I was aware that that's the main reason why rust wasn't allowed in the kernel, because it doesn't have like kind of failable alternatives to what it provides as the default yeah.

A

Right I think rust for Linux came with their own replacement for the airlock library because of that yeah yeah, so exactly yeah, so they are not using the standard analog. They have their own uh and yeah. Here we are using this standard analog, Library yeah. We never encountered such an issue, but.

E

It maybe it doesn't happen because you just restrict what the program can do right. So you just don't allow to panic. Yeah.

A

Also, on the other hand like when you write BPF programs in practice, you don't use that stack memory. Much so like especially here we, we start just Primitives and variables, so we have like let H Proto, which is like q16 like win in BPF programs. In practice, you never store bigger data than Primitives. So, like you allocate pretty much Primitives on the stack and when you have something bigger, it's like either coming from the kernel, then you have like a big chunk of memory coming on or when you want to allocate something bigger.

A

Then you create a map. You create your own structure with a buffer with like some u8 array and then yeah. It's on the Kernel tool.

E

Panic, after all,.

A

um It's cute well, but it's then not for us doing it because then you are when you are creating a buffer in the uh in the BPF map in the kernel, then you have an unsafe operation of like uh instantiating and map, but it's then we currently doing that. So it's not Russ itself, like money, King.

E

B

E

Few index into something that's not found.

B

But I guess click people catch that if you forget a slice indexes.

A

So out of bounces code by verifier already so it's BPI verifier caching, like out of Bounce error,.

F

There was one more question and then.

F

Sorry, just curiosity about limitations of ebpf as a platform about loops I understand the verifier cannot be very smart, but at the cost of being not ergonomic, is it possible to just make a loop that is obviously bounded by a constant and actually managing earlier returns early like the track by yourself, so the code of the loop is less ergonomic for the developer, but in practice the verifier says that is not going to exceed a constant period.

A

I mean yeah. If you define some constant, which is fine for verifier, you can already return yeah.

F

So you still have break as a statement which.

A

Means yeah that.

F

You can, and at that point is there a limit on the number of the iterations.

A

Yes, so if you have a break inside a for Loop verifier will still assume the number of instructions which will.

F

Be done without respiration is the number of counted instruction as a time, so every Loop counts for the length of the loop times the number of iterations. This is the 1 million yeah.

A

Okay, I mean the whole program is supposed to have fun at uh at Max, 1 million. So if your Loop is in the middle of the longer program code, then yeah, but but assuming yeah, if we assume that your whole program would be kind of wrapped into that for Loop function, you have a bigger, you have some break inside it, but after the break, if you, after that break, you still have some code which is executed.

A

When the condition for braking doesn't happen, then the verifier is going to assume all those operations happening after break because it will assume their worst case scenario. That's the break condition never happens, so it will assume all yeah all those instructions and times.

F

And anyway, it is possible to create user data types like structs and put them as values in maps. Yes, yes, share them between the BPF and the user. Space part.

A

Exactly exactly so, I'll quickly show one example: it's on my friends, GitHub.

A

um Sorry, I cannot type on Mac keywords.

A

So, as I mentioned, like the common case in BPF, is that if you want to store, is store buffers longer than bigger than 512 bytes yeah, you can put the capacity, and then you have a structure buffer, for example, when you store bytes and that buffer is stored in the map, but also the other example Maybe.

A

There is an old version of the firewall program, I just shot, which didn't use ilog Library, yet so for pushing out the events about some packet being blocked or accepted. It was actually using a common structure and it was using the events perf event array, so it was pushing like the pocket, lock structure and the structure is defined in.

A

um My upcoming in the common crates yeah here it is yeah, so you can find I mean as long as that structure contains Primitives, which are like fine foresee and for the kernels yeah it's possible to do that.

C

Anything to the Meetup. Thank you. Mika thank.

D