Cloud Native Computing Foundation Cloud Native eBPF Day EU 2022, 19 May 2022

Previous Meeting

⏯

youtube image

►

From YouTube: eBPF? Safety First! - Kemal Akkoyun, Polar Signals & Dave Tucker, Red Hat

Description

eBPF? Safety First! - Kemal Akkoyun, Polar Signals & Dave Tucker, Red Hat

eBPF being a promising technology is no news. And C is the defacto choice for writing eBPF programs. The act of writing C programs in an error-prone process. Even the eBPF verifier makes life a lot easier; it is still possible to write unsafe programs and make trivial mistakes that elude the compiler but are detected by the verifier in the load time, which are preventable with compile-time checks. It is where Rust comes in. Rust is a language designed for safety. Recently the Rust compiler gained the ability to compile to the eBPF virtual machine, and Rust became an official language for Linux. We discover more and more use cases where eBPF can be helpful. We find more efficient ways to build safe eBPF programs that are parallel to these developments. We will demonstrate how we made applications combined with Rust in the data plane for more safety and Go in the control plane for a higher development pace to target Kubernetes for security, observability and performance tuning.

A

All right uh welcome everyone. Surprisingly we're going to talk about ebpf. Today, specifically, we want to mention how we actually build a program that uses rust in the kernel space and go in the user space. So, let's first start with interactions.

B

Yeah so hi, I'm dave. I work at red hat as a software engineer. My background is in software defined networking uh and most recently ebpf and uh a lot of rust.

B

You know it's quite often that you can forget to update little bits of your code and then spend hours debugging, something which was really trivial, which was oh, actually, I'm sending the wrong thing. On the the perfect end, it'll nag at me and tell me dave you're not allowed to do that and that's a lot of what iyer is able to offer for you. If you choose to use that in the kernel space.

A

All right, uh let's also talk about uh like how actually go, helps uh with the okay. We actually heard a lot of nice things about trust, but like how actually go comes in the equation. First of all, writing go. Applications are like super easy, so you can actually uh faster and you also have like ability to produce static binaries, and we will see in a minute that you can actually embed the ebpf programs in the binaries binaries themselves as well.

A

But uh for me I guess uh the like uh most important part like uh a pro for goal. Community is like there's a lot of uh development going on around cncf and we have a lot of packages that we can just utilize. In our case, it's gone again. We have. We are like talking to the container runtimes. We are talking to the kubernetes api and for all those uh work.

A

We can actually find some sdks uh to just like embed in our user space code and uh discover the c groups uh rather easily and then attach our profiler programs.

A

I will keep doing this yeah.

B

All right can you progress on. I forget what we're going to talk about here now ah right, yeah, how it actually works. Yeah. Now, that's a question so the way that we're actually able to write our rust programs or rust bpf programs is all really thanks to llvm and the work of alessandro descena. Who is my co-conspirator on ire? He was the one that actually added support for bpf as a target in russ c.

B

So rust knows how to compile down to bpf byte code, which is amazing, and we are just you know, building on top of that using rust. So we were able to use rust macro system to make things really easy from a developer standpoint. You just annotate your code and say hey. This is an xdp program and it will keep you on the right path to writing that.

A

Okay in this section, I'm going to talk about the bpf core, which we, which is the library that we use in the user space uh thanks to aya being like super nice and compatible with the bpf api. We can just like use the bpf goal, loader in user space and, like just run the programs, and we can actually, there are certain advantages to this. We can just statically bpf and like we.

A

We just have corey support uh for our user space program as well, and if you want to change any implementation, it's kind of in parka in my company, we are like true believers of open standards and lib epf kind of an open standard for us. So keeping in that uh in place like how uh gives us the flexibility to just jump between, for example, we just rewrite our whole, our like bpf programs in rust, and it just worked.

A

So, yes, uh I'm just like skipping forward, sorry for the agony, so in the in our example, I was gonna just like demonstrate like what we do actually with this program, which, in the user space we just like, discover the c groups, and then we just like attach uh the the bp programs to those c groups and we collect data and aggregate them into the ebpf maps and in the user space we just like convert them into ppro format, which is another open source uh format and we send them over to the storage layer.

A

So the program is actually rather a simple one, uh because most of the apis are just provided uh with ebpf and lib bpf. We just get the stack ids and the stack traces from the kernel, and then we have a map that actually counts the unique stacks and those aggregated values then sent to the user space.

A

So I will just like try to walk through if this continues to work uh on the ras program. So rus programs are uh for the ebpf, they are rather simple, so they don't have any standard library or they they don't have an entry point like a main program right. So your your program is starts with, like importing couple of libraries from uh rust, create that io provides uh which are which is specifically for the kernel space.

A

If you are using an old kernel, you need to actually put a license section in there. Otherwise, like compiler will just rant about it, but it's not the case anymore. I guess, if you are using something uh recent 5.2 plus, I guess.

B

Yeah and to be fair, this is not indicative of the nice sort of ap that we like to provide you in ire. It is a hack to work around all kernels, so don't read into this one too much.

A

And then like, we just create a ebpf map to actually count the stack traces, and this is a struck representation of a key that we put in a hash map and it in that hash. Like the important part.

A

I guess in this section of the program is like having a map macro, that where you can actually define the ebpf map, ebpf maps and having this key, it needs to be represented as a seastrack, uh since we are just like also sending this uh thing to the user space, and uh we need to be like sure that there is. There is nothing uh rather than the data itself like padding or whatnot yeah.

B

And type safety look, so here is our map and it's a hash map of stat count keys to u64s. So you know we, we can't make any mistakes with that right.

A

Exactly yeah, so what else like this is how we actually attach to an event in this case it's a perf event, and this is another feature for rust. We can use actually results for that. This is a rather like common pattern. In io programs we ever have an event handler, but that event handler actually calling and helper function, which returns an result, and we can just like uh in a high level, handle all the events, basically rather successful, or failure. Yeah.

B

So that that's important, because there's a lot of fallible operations that can happen inside an ebpf program, but we can't panic inside ebpf because we're not allowed to it's. It's not a permitted instruction. So this actually gives us a relatively nice way to handle errors, um and we also have a logging library as well, which is kind of like bpf, trace print k on steroids.

B

So you can actually leave it there in production as well for your logging and that that will help. You know try and debug some of these errors, if and when they do occur.

A

And this is the actual program we actually update the maps by like also getting the stakes uh stacks from the kernel itself, and then we just count them and uh update the maps. I will start being a little fast here. I don't want to take so much of your time, which we already had, so you can also panic in the ebpf program. So you need to add a line like that uh to actually handle the panics.

B

Yeah this this is literally just making the rust compiler happy that you actually can't panic uh so that we we just have to have it there. Otherwise rust gets upset exactly.

A

So this is actually how you build. uh We still need to have a nightly tool set uh for me to make bpf linker and there's als already a task in there for you to just generate that program.

A

And, let's quickly check out the go program that we have like after you compiled that ebpf program go, has a facility to just embed them into the binaries. You just say that, okay, this is where that object, file, actually stays, and just like embed that to the library we actually mentioned that in the rust uh as well like we need to match the keys uh so that we can read them in the user space as well.

A

uh In this case, like kind of the memory layout is the same with the c program and the go program, and we can just like uh deserialize it in the user space. This is how actually you use the bpf, go to load that module and the lower the object, and then this is actually how you start listening to perf events, and then you bring everything together and you attach your program to that specific perp event and the data starts flowing.

A

This is you need to just like uh keep a reference to the maps that you have and you this you can just like get that maps using the bpfco, and then there there's another like kind of nice api that the bpf exposes you can read them in batch.

A

You can do your magic in your user space. This is just a snippet to how to build it. uh The bpf core user program slides are available. You can check it. The whole program is in a single pr. You can actually see the difference, how we get from the c program to a rs program, even with how we actually updated our build tools. What not like github actions uh just check this pr out if you want to go down that path cool.

A

So what is next the whole idea behind this? We are planning to write rather complicated profilers uh that runs on the linux kernel to just support, for example, dynamic languages or the vms like java. That's why we just like pick uh go down this pad uh at this point. We didn't have a lot of cpc programs uh written for bppf, so that also helped us um like. We are planning to add them in parker soon for the eye habits, yeah.

B

Lots of work to do in ira um plenty of program types still not implemented. It's an easy thing to do. We're just looking for users. So if you're interested come and grab me later and I'll, be happy to point you in the right direction,.

A

And like most importantly, we really want to pitch our both of our projects. We need help. uh We need contributors if you're interested just like, go and find us. We have links everywhere. You can parka as a discord channel. I has a discord channel, we are rather active. You can just like come and ask for help and we can uh actually help you to get into the ebpf land as well.

A

So thank you for listening and thank you for bearing the agony with us. It was rather such a mess. Sorry for that, and these are the links yeah hit them. Also, a quick shout out to the bpf and aqua security. They really helped us as well yeah. They are also looking for contributors, so yeah. Please thanks.

C

All right, yes well done, for dealing with all those technical issues. uh I see one question already we'll just take a few questions.

D

So, first of all, thank you for your patience in dealing with all the issues. We really appreciated the talk um when dealing when writing, ebpf and rust. You have to make a lot of unsafe calls when you call unsafe, you turn off a lot of the compiler protections, borrower, checking, etc.

D

What are some ideas or plans that you have to move away from that, so you can actually gain the full power of the rust compiler.

B

That's a really good question, um so a lot of it we're trying to abstract away inside the library itself. So you, as a user, don't have to do the unsafe work we will for you. So, like you just call hashmap insert and, like all of the unsafe stuff happens in the background, we can't get away from it, because, ultimately, it's forum, function, interface and changing stuff into raw pointers. To pass into a ce program is inherently unsafe and working with raw pointers is unsafe, so yeah, the real way is just abstract that away from users.

B

I think that's the plan.

C

More questions.

C

Anyone, yes, I see another one.

E

So first, thank you for the great presentation enjoyed it. The question is: what is the performance overhead of rust against riding in c in the kernel? Of course,.

B

um None not that I've observed anyway, I mean ultimately, it compiles down to bpf instructions, and the program that you would write in c is basically the same rust.

B

Around some of the lvm stuff, like mem, set and mem copy, which we're just working through but for the most part, um they're identical performance-wise.

C

All right one more question: if there is one, if there is one, am I missing anybody all right, so, let's give one more giant round of applause for all those technical difficulties.