Cloud Native Computing Foundation Cloud Native eBPF Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Portable BPF with CO:RE - Grant Seltzer Richman, Aqua Security

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Portable BPF with CO:RE - Grant Seltzer Richman, Aqua Security

eBPF is the future of Linux, and CO:RE is the future of eBPF. There are as many features of eBPF as there are challenges to using it, and one such challenge is distribution of your eBPF project. With so many different kernel versions out in the wild it seems like an impossible task to compile your eBPF program against all of them to ensure compatibility. By using CO:RE, a feature of libbpf, this gets quite a bit easier. In this presentation Grant will talk about one of the biggest challenges in eBPF development, portability and distribution. Grant will talk about how these challenges have prompted my team, which develops Tracee, to come up with creative solutions for cross-compilation. Grant will then show how CO:RE changes everything and talk about how you can get involved to drive the future of it!

A

Hello, everyone, uh you could all hear me: okay, okay, uh so yeah. So the title of my talk is portable bpf with core.

A

uh I am on the open source engineering team at aqua, I'm from brooklyn new york he's my pronouns. You might see me on the ebpf slack with uh this little crab guy.

A

So here's an outline of my talk, followed by this slide. We're going to be talking about my experience as the one of the maintainers of tracy, we'll talk about what tracy is our goal for distributing it and how cre helps us achieve that goal and we're going to look towards the future of next steps in ebpf development, especially as it relates to portability and distribution and how cre fits into that.

A

So the primary project I work on is tracy.

A

It's a ebpf project that is, it handles runtime detection and forensics using ebpf and the way that works is it uses hundreds of different events, things like system calls trace points network, tc, hooks lsm, hooks all things that you've learned about earlier today and if you're looking for a simple tool to use to gain a ton of visibility into your system, tracy is the kind of drop-in tool that that can let you do that and it also lets you apply policy on top of that, and in this example, we could see uh an exec ve system call that's being traced, so we're we're specifying exactly e to be traced uh and where we have all this information about it.

A

So we have the actual command line that caused this exact v to trigger.

A

We have the time stamp. We have process id parent process id all of this additional context. So it's a lot of reading of memory in the kernel. That's essentially the primary thing that tracy does.

A

So the problem that we've observed, while developing tracy, is that from kernel version to kernel version there is there becomes differences in uh data structures as we're as we're reading them.

A

So in this example here we could see that there is a different field that we're reading in the task struct for kernels before 419 and after that, we're reading something different. So our initial approach to handling these differences is to have at run time, this check to see what kernel version we're dealing with and read from it accordingly.

A

This, of course, has flaws, one being that sometimes fields are renamed or completely removed, so we could run into errors where fields don't exist, so the code won't compile properly against the headers that we're compiling with also it's just a real pain in the butt to deal with when we want to support all these different distributions and all the versions of those distributions and if you're not paying attention to every mailing list and looking at every diff and every patch, it's hard to keep track of all the all the differences in versions uh not to mention.

A

If we have somebody that's running a custom kernel, you know I'm not going to have a check saying like if we're running on grant's custom, gen 2 kernel then read from this field.

A

So this is one approach that we've taken and clearly has. It has its flaws in addition to that, what we were doing is doing the same thing. That bcc does, which is build on the kernel that we're deploying our project to so as we deploy or as a user picks up our picks up tracy, and they run it for the first time it would look for their kernel, headers, build against those um and then and then run from there and cache it somewhere.

A

But this is, of course, an issue because you know we would get bug reports about tracy not being able to be built because, let's say the kernel, headers are missing or you know. Perhaps we just aren't keeping track of all the different kernel, header versions, for every major distro like we would get. You know we don't want to have to support scientific linux sex or something like that like something very specific.

A

So what do we want? We want to be able to write bpf code without worrying about kernel version differences. We want to be able to distribute the bpf object where we can just host a single bpf object. Have people download it and have tracy run it without any issue?

A

We don't want to have to rely on a special kernel feature to do that and for the most part we don't need our users to know about that.

A

So when do we want it? Well, we have it now. So this is what core compile once run everywhere enables us to do so.

A

Core is a concept that's enabled by the bpf, and these are essentially exactly what it gives us so right, so compile once run everywhere or core is a concept enabled by lib bpf.

A

There are helper functions within lib bpf that will take into account kernel, debug information to try and relocate kernel fields, as they have changed from kernel version to version and just to clarify that lib bpf is a user space library that is in the kernel, repository.

A

And it's used for interacting with the bpf subsystem, so loading bpf programs and attaching them to the various hooks.

A

So let's go into how this is actually working, so the first thing I want to talk about another another acronym for us to to learn about is btf. So bps btf stands for bpf type format. It's a special, a simple space, efficient, uh debugging symbol format. So if you've ever worked with dwarf, it's basically a just a compressed paired down version of that. So, if you think about it, the linux kernel is a compiled program, just like any other, and it has all the symbol. Information or you can have all the symbol information for it.

A

But since you know, the binary itself is is massive. All the debug information is even bigger or makes it even bigger, so btf, because it's compressed and because you know engineers have found a way to get all that same information in much less space enables us to efficiently do all these relocations and by relocations.

A

I mean you know if we were to uh let's say: read a specific field in a test struct on a particular version, and that changes in another version btf enables us or enables really lib bpf, to read that debug information see where that field has changed, like the actual memory offset uh find it for us and read from there, instead of where we initially uh specified it to be.

A

So here's a diagram that will hopefully help explain that so here you see, we have on this this right box or this box on the right is tracy itself. So we have the user space side. We have lib epf that it's using to interact with the bpf programs and the bpf system.

A

We have the tracy bpf object itself, and then we have the kernel btf, the btf for the kernel that it's running on that btf can be found either in sys kernel, btfvm, linux, so vm linux is a special file.

A

It's the compiled kernel stuffed inside of an elf binary that will have a vtf section if the kernel was compiled with a specific configuration option, config debug info btf, otherwise lib epf also has an option for us to specify an external btf file, and that brings us to another cool project that my team has worked on called btf hub.

A

So if you are trying to support in your bpf project, if you're trying to support a particular distribution or a particular kernel, where this can this kernel, compile config option hasn't been specified, um you normally wouldn't be able to run a program that has core enabled. So you would have to go back to that uh previous solution that I that I outlined that we were previously working with, but instead you can pre-compile on your own.

A

The actual kernel for that version include that debug information extract that btf information and supply it as an external file, which is what we've done in this github repository, where you could just instead of doing that, compilation yourself download this, these btf files and load it with your bpf program.

A

So now we want to talk about uh code changes that we had to make to tracy, uh and really this is more like code simplifications, it it really uh honestly cleaned up our code base.

A

The first thing is vmlinix.h, so in the same way that we have that vm linux file, which is the compiled kernel, vmlinix.h, is another generated file from that. So it goes into that debug information with all the different types and it produces a header file that has all of their type definitions.

A

So if you take a a quick look at this, you could see the definition for underscore underscore u32. So if your bpf program is relying on that data type, you can import it and use it accordingly. Using vmlinix.h, I wrote a blog post about this file. There's there's more to it. That is very cool. You could check it out the url there on my personal blog, so you can see here that the changes that we made in our imports we actually are currently currently have both of these.

A

We have a compile time variable that we define, but we're moving towards being able to remove this um this. This left side here, but on the left, we have all these user apis and internal headers that we're importing getting those definitions of the data types that we're trying to read from in kernel on the right. You see that we have we're just importing vmlinux.h as well as this additional file that I want to talk about.

A

So that's a missing definitions file. So while we do have the convenience of being able to just import a single header file, the vmlinux.h to get all of our type definitions that does lack all the macros and all the functions that we otherwise would be using in those kernel, headers, so macros, certainly aren't put in the type information or aren't put in the debug information and functions aren't aren't included either. So I had a conversation on the mailing list about the best way of handling it and right now.

A

The best solution that anyone was able to give me advice for is to just redefine it. So, theoretically, if there is a macro that changes from version to version, we would have to be manually updating that, so we still have. We still have work to do there uh and essentially the last change is just using the correct header file.

A

uh I'm sorry using the correct helper functions, so you could see here we have this internal macro in tracy called read kern, where we zero out memory before doing a kernel read, and the only difference here is that, instead of using bpf probe, read we're using bpf core read, the arguments are exactly the same. It works in the same way. It's just under the hood using btf.

A

And here's exactly that in action, reading a amount namespace id.

A

So the most exciting way that we've been able to change tracy and the real purpose of this talk has been distribution. So previously, if we wanted to publish binaries of tracy, where we want to support all those distributions that I had on that slide earlier, we would have to say you know here is tracy for ubuntu 2004 or here is tracy4204 or or 2104 or for specific kernel versions.

A

Now what we can do is we compile a single bpf object that has btf enabled and we embed it in our user space go program. So at run time, when we start up tracy it unloads that from memory and loads it- and you know we can compile that go program statically. So it's just a single artifact that, where that we can ship and run on any system.

A

So this is, uh you know again uh this this same slide, but all these simple changes enable enable us to not get bug reports about tracy having garbled info on a particular distribution or someone's custom, gentoo based kernel, and certainly not on production, centos servers.

A

We now have the confidence to distribute tracy to a wide array of environments and can focus on developing the great new features that we want to add and less on the changes in different kernel versions and, more broadly, I would say that this is the future of bpf development, that this isn't going to be something that people are going to ask like is: is this bpf project cre enabled you know as new kernels are or major distribution? Kernels are shipping with btf enabled by default.

A

You won't really have to worry about about that at all, so you can have your bpf program written in confidence to not have to worry about if cre is enabled or not, and for those custom, kernels and those older kernels. That's why we have btf hub, so I actually want to show a demo of exactly this.

A

I'm going to quickly mirror displays I'll make it easier.

A

So here on this linux vm- I have a uh I'm sorry is that is that big enough? Okay, uh so here on this um uh linux, vm I'm running fedora uh 34. This has a 5.12 kernel and I'm just going to run a version of tracy, so you could actually see what it does.

A

Just uh look at the first 20 lines. So here with this debug flag turned on, we could see that I unpacked the cre bpf object into memory and I'm loading that and and running with that- and this is actually a static binary, and you can see that it's running properly, we have all these different commands that are being run. We have the events that are actually triggering these to be printed, so this is an lsm hook of a file being opened.

A

This is a proc info. We have all the arguments that are being passed back into user space and being interpreted correctly, meaning that it's the we're relying on the proper definitions and if we're not relying on the proper definitions, cre is handling it for us to to find the where they've been relocated to so here I also have another vm, and this is running ubuntu.

A

I believe it's 2104. This is running a 5.11 kernel, I'll run the same command.

A

uh And yeah look at that. I have a security file open for starting it as well, uh and it it did the same exact thing where it unpacked the embedded bpf object from memory.

A

And loaded that into the kernel and ran it so again, all the arguments are properly properly displayed and the kicker to all of this is that.

A

If we look at the sha.

A

It's exactly the same file, that's where you're supposed to all clap and go.

A

uh Okay, so uh right, so that's what cr cre gives us uh next steps things that we want to work on uh working upstream to find a better solution that than that missing definitions, header. uh So you know, one of the suggestions was having a um like a in claim have a like built-in attribute for checking if a type is already defined so that we can still import the headers that we want, but not worry about them, having a collision with the names in vmlinux.h, uh better verify our output. So that's certainly not something.

A

That's related to cre or even lib vpf, that's across the board, but we certainly ran into issues where we had verifier output that we just could not make sense of during this switch documentation. So a recent effort that I've been working on is launching the lib bpf read the docs site. So if you haven't seen that it is a full api documentation for the bpf, you can go check it out and contribute it's sort of if you're familiar with, go and go doc. It's exactly that.

A

So, as we document more of the api that documentation will get better and certainly we're going to keep working on, btf hub, so go check out that repo as well and- and that's my talk. Thank you very much.