Cloud Native Computing Foundation Cloud Native eBPF Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: LSM BPF Change Everything - Leonardo Di Donato, Elastic & KP Singh, Google

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

LSM BPF Change Everything - Leonardo Di Donato, Elastic & KP Singh, Google

Much is being said on security recently. Almost as much was said about tracing the syscalls happening in the Linux Kernel with BPF. Aside from all the buzz, we need to appraise some gaps in the current narrative. We need to fill in the gaps in the actual syscall execution flow to avoid attackers using them. Here enters the game the general security hooks for Linux, namely LSMs, and their integration with BPF. LSMs via BPF will change everything. They're still relatively unexplored, so this talk aims at giving a pragmatic overview of LSMs via BPF. Join me to discover why I believe their integration with BPF is paramount in the security context and how to effectively use them.

A

I'm kp singh and I work for google security in zurich and today, leonardo and I are going to talk about how lsm and bpf change everything we're going to talk about. Why did we go about writing a vp of lsm? What can we do using it and then leonardo is going to show an awesome demo showing the vp of lsm in action?

A

So why did we go about writing a vp of lsm?

A

I work for a group called detection and response. We try to keep an eye on google's internal systems and detect when unusual things happen. From a security perspective, google has loads of linux systems all the way from servers that run key services down to the very laptop that I'm using right now.

A

These end points are constantly piping security, relevant telemetry data to our software pipeline, which is doing some clever stuff. On top of this data, this information is then used by security analysts who use this data for responding to incidents.

A

My team builds the agent that runs on these internal google owned machines at the very beginning of the pipeline.

A

So what is the background? What was going wrong? Why did we make the bp of lsm, so the traditional ways of doing this would be using audit d and audit d was not very flexible. I remember when we wanted to audit environment variables and the amount of work that needed to be done on audit was quite high. We needed to patch the kernel and the audit user space.

A

We tried using kernel modules and the kernel modules are a pain to maintain even with regular tracing and evpf programs. The number of places where one can attach to is so high that one doesn't really know. What's the best place to observe a particular behavior, and this will be very relevant once leo shows his demo and, of course, policy enforcement was out of the question using any of these systems.

A

Security can be broadly classified into two approaches which work hand in hand, monitoring what is happening on a system and enforcing which is taking action based on the monetary monitored data prior to bpf. One could monitor using linux audit or use perf or the performance, api or trace points, build custom, kernel modules or use trace probes and the enforcement was completely separate from all these systems.

A

One could enforce or do mac policy enforcement using security modules like app armor se, linux or one could do some sandboxing using sitecomp, which is based on the syscall leg.

A

The bpf lsm, which we also call ksi, sometimes sits somewhere comfortably in the middle and provides a unified and flexible way to do both enforcement and monitoring. Now there are about 200 lsm hooks.

A

These lsm hooks are a layer of abstraction higher than the system called api. For example, one can execute a process using both the exec, ve and exec ve. Add system call, but they both call the same set of lsm hooks.

A

The lsm hooks correspond to changes in the kernel objects and are placed strategically in the life cycle of the object and the ongoing operation.

A

This strategic placement ensures that an attacker cannot use attacks like time of check to time of use to trick the security logic into believing something that it's not actually doing. Leonardo is going to show the importance of this in his demo.

A

The bp of lsm can happily coexist with the other lsn's, like se, linux and app armor, and does not do anything by default.

A

One can load an ls and ebpf program to implement their own custom logic for the lsm hook. In this example, the vp of lsm for the process execution gets invoked after all the other lsms and, of course, this order can be changed in the config lsm kconfig parameter or the kernel command line.

A

The lsn hook can then write audit data back to the user space using ring buffers and maps and can also make policy decisions. As we talked about.

A

Lsm programs are compiled once and run everywhere. They have verified pointer accesses, which is made possible by the btf information present or compiled with the kernel. We also added local storage or security blobs for inodes task, structs and sockets.

A

These blobs live and die with the kernel object, making the memory management much easier and can be used to tag security metadata. These blobs are especially useful in conjunction with the with the atomics api we added to bpf and and especially for using or generating custom unique identifiers, for example, namespace, ids or container ids.

A

Think of implementing a custom logic for container ids and sending a patch to the kernel. People have tried that tried to convince people for multiple years on what a container is and has not succeeded.

A

Lsm programs also can also sleep. This is especially useful if you want to get them to read data from user space, for example, command line arguments. The kernel leads to page in memory which requires the program to be put to sleep for the memory management unit to page in memory.

A

We also added helpers to get the ima hashes of files, which can be useful for binary fingerprinting and, of course, we have all the other dpf goodness like the bpf ring buffer and maps which help maintain state and send data back to the user space. So what's happening next. My colleague flora is working on dns auditing, which has some interesting challenges, and we would like to do a presentation about it at some point.

A

We also want to add helpers for mac policy enforcement and really make ppf lsm powerful enough to be a major lsm or be able to implement a major lsm in the kernel.

A

We also want to provide more places that ppf lsm programs can attach to especially useful for auditing events like nmap or kernel module loads, and we also want to make the ppf lsm users to use it more easily by providing an api like ppf phrase or something that is more abstract from the kernel internals right now. You still need to understand that dprm check security means a process is being executed, which is not ideal.

A

I'm now going to hand over to leonardo who's going to show an awesome demo of the bpf lsm in action.

B

Hello, everybody I'm leo. I guess it's my tour now. First of all, I want to thank kp for all the wonderful explanations and for introducing me and now to explain you why I believe lsm bpf will change everything and for the better. Let me first tell you a story. Some months ago, while I was still working on a daily basis on falco, a security researcher contacted me presenting a very cool, bypassing technique, as some of you may already know, falcor just like many other runtime security tools out there does its job thanks to ebpf.

B

Specifically in its case, thanks to the cisco trace points, the falco bpf probe basically attaches to the kernel cisco trace points with little abpf programs that are responsible for any cisco, being called on your machine to extract its argument plus other data and put all of them in a ring buffer.

B

Then all rules assert conditions against those data. For example, the default rule set contains rules to detect crypto mining activities on euros by looking at the outgoing connections happening on your machine.

B

Such rules simply trace saint to send message or the connect, syscalls and assert conditions on their arguments electing if the destination ip, for example or the destination domain match those of known mining pools out there. Just to give you a sense of what I'm talking about. Let's take a look at those rules.

B

Let's take a look at this netminer pull macro used by the rule, detect.band connections to common minor pull parts, as you can see here.

B

This macro basically asserts the conditions on the send to and send message schools and on these other macros, which are defined above here. If we inspect them, we can see that the miner pool ladder, macro checks for destination, part and domain name to be in lists of well-known mining pools out there. For example, these minor domains list contains things like this okay, so in this case falco when detects specific ciscos with specific arguments containing these values alerts the user for outbound connection to manage boots.

B

Also, it's worth taking quickly a look at this pretty generic and pretty useful outbound macro that a lot of falco rules use as you can see. Basically, it checks for the event, so the cisco being executed on your machine to be either a connect or a send or send message and for the the connection being in progress and not towards private networks, okay or towards local addresses, and to be epv4 or epv6 all right so far, so good.

B

The problem is that the code sofia x, thank you. My friend presented me was able to reliable bypass those falco rules by exploiting a tacto in the corner. 100 of the time I was like, oh my. How is that even possible? Well, such code was exploiting every window between the moment. The bpf three spawned read the argument, for example the destination ap. At the moment, the current actually executes the cisco with its arguments.

B

The core issue is that cisco arguments are at a two different point in time. The delta between those two moments represents the time window into which the attack replaces the malicious argument, with a good looking one that the bpf trace point will eventually read it's important to note, anyways that the beauty of this bypass was that it worked again. Any tool using bpfc is called trees point as a mean of detecting malicious activities and threats, not tolerance, falco, which I'm using because having worked a lot on it, I'm very familiar with it.

B

Let me say this aloud before people. Excuse me, I will not explain the bypassing technique here in detail, because it's not the goal of this tool, but I strongly recommend you to go. Watch selfie rex talk at deathcon for more details on it. It's worth it trust me so long story short. This attack connects to ip1.1.1.1, which we are considering malicious just for the sake of the this demo, while the falco bpf probe detects another ip, and for this reason it doesn't trigger any others at all.

B

But let's see it in action, I feel pragmatic right now. First thing: we need a falco rule that detects connections to 1.1.1.1 I've already prepped one. Let's take a look at it. Okay, as you can see in this little jumble 5, I basically coped the outbound micro I shown before, and I wrote a folk rule called unexpected outbound connection destination that basically used this macro together with the check on the destination e3 to not be 1.1.1.1.

B

So if this is an oddband connection and the destination ap is 1.1.1.1, falco should alert. Okay, let's check it. Falco starts now, simply let's call 1.1.1 and it works. Look as you can see photo detects outgoing connection to the targeted p in normal situations. But what if I run the bypass? I was mentioning before notice that for this bypassed working glass to be vulnerable, but it needs to have a privilege user fault of the activated. Let's check its cctv knob, it's one!

B

Okay, in this case we are vulnerable, but first let me run tcp down, because why not- and now let me run the attack as you can see, the attack succeeded connecting to the 1.1.1.1 ip. This is its intelligent representation and tcp dump catches it while farco doesn't iaibpf cisco, tracebind.

B

Okay in falcon091, I implemented an initial workaround binding support for tracing the user. Full cisco, which is the attack driver that surface code, is using to swap the cisco arguments under the hood. In fact, if we run falco 029.1 with the full true side that contains a rule for it and the attack, we can see that falcon is a critical art about user fault to fd getting cold.

B

But aside from the fact that there are also other means that I will not present here, you know to me the same precondition that this bypass requires as it is at the moment. My patch just makes falco scream about user fault if this is called getting cold, but does not provide any detail or deep visibility on what it really happening after it. Furthermore, we laid a modification. This attack called work also for other uh cisco's, like renamed unlinked creators and two, you name it.

B

So what will be an actual and truly meaningful fix or more there's even a way to avoid this to happen at all. This is the question. In my opinion, a certain condition against the actual arguments getting used by the kennel would be the best way to go for detection tools, but first of all how to grab them and to do so, we need to first understand a little bit better. The cisco's flow in the linux kernel.

B

Let's take a look at this whole image of the flow of the open cisco that I have on my laptop the flow for other cisco's in here and is very similar and and sophie also provided a bypass for the open. So when a processing user space calls the open cisco on a file path, the cisco is dispatched and the path string is used to contain a kernel file, object and an unknown object.

B

If the parameters are incorrect, error is returned as usual, but then the normal dsc descript access, control, file, permission checks are checked and if they are satisfied, the lesson framework enters the game. It tags for reach of the file lsm aux for hold, enabled hell, sams, also the vpf ones. If there are and finally or if all those security checks pass, the file is opened for the user space process and the new file descriptor is returned to it.

B

It's important to note how deep in the cisco flow the lsm hooks are, which is not the case for bpfc school trees points not convinced. Yet, when I take a look at the kernel code for the connect c school and just one of the various lsm network, cooks, okay, you want don't blame me well, taking a look at the current. It's always the right thing to do so here in the lsn ox depths either you can find all the hooks. The lsm framework provides, for example, this one at line.

B

208 is the one we are interested in, or blocking sophie to connect it to 1.1.1.1. Isn't it in fact, if we go take a look at the definition of the connect c school in the socket.c file here, cisco defines disconnect disconnect move address to kernel, sees connect file. If the arguments move successfully to the kernel and sysconnect file calls this function, security socket connect. If we don't inspect what this function is and does we suddenly realize it is the function that calls the socket connect lsm hawk.

B

So is this a ringing bell for you, as we have seen, alexa moocs are very low in the cisco's flow and happens after arguments have been moved to the kernel so to prevent our friend sophie or any problem using this fantastic technique to connect to 1.1.1.1 to provide a real security layer. We may want to use this lesson hook and attach a bpf program to it, and this isn't anything but just be aware that this feature is new and it requires a kernel greater than 5.7.

B

Furthermore, ensure your camera is configured with the options: config bpf lsn and config debug info btf. Also, you need to have lsm bpf activated to check it. Take a look at the lesson: kernel boot parameters under the config and the square lsm configuration so basically like this okay, we can write some lsn bpf time to get our hands dirty with some lsn bpf. Finally, please meet restrict connect, please little bpf program that I've written to attach to the socket connector lessen mock.

B

As you can see it's simpler than anything with the sec helper macro, I'm marking the following function to handle the lsm socket connect lsm hook then by using bpf prog macro. I am gaining the benefit of not having to mess around with the registers, and I can simply declare the signature of the function to contain the same parameter as the socket connect to that we've inspected before in the kernel notice. Also that I'm appending a red parameter that I check in the function.

B

In fact, since various programs can be attached to this lsm hook, I have to respect the cannot override the denial rule, meaning that if this function is not the first one attached to the lsm socket connect 2 and the previous one returned a value different than 0, then I have to exit from it and respect the decision of the previous program attached studio. The rest is pretty straight forward. I know non-epv4 protocols, because this is just the example.

B

I obtained the destination ip and I check it against a value, the value that I want to block a value that I've articulated here for the sake of simplicity, but here we could also look up into the bpf map. Please notice! This is the same number that the attack connect outputs when it succeeds, so it's the ip representation of 1.1.1.1.

B

Let's run it now,.

B

A lot of relocations, let's attach to the trace pipe, let's first simply test it by fishing. This command called connect, whoa, look, okay, but let's be serious and try it against the bypass error connecting operational permitted lessened blocking.

B

Lsm blocking oops.

B

Operation not permitted connection to 1.201 block. Sorry, my friend, this is good. Isn't it in the procedure I'm going to share the end of turk. You'll also find the c code to attach restrict connect thanks to the bpf to the lessonbook, actually make files to compile everything and stuff, but now that we are deep into this amazon rabbit tool, let's keep the focus. What if we cannot want to block connection to 1.212.1, but we want to receive the correct cisco arguments, so we want to have deep visibility about this nation.

B

Ip address we've seen, bpf, cisco, transparent, are not that reliable in our use case. So let's make a step further and let's try to answer this question first idea that I come up with is: let's attach a kernel probe to a key probe to the security socket connect function that calls the socket connect lsm hook that we've seen before in the car. Do you remember s8 cisco arguments at this stage should have been already moved to the kernel, so we should be able to see 1.1.1.1 and not another p, but we need to try.

B

I need to try. I'm not sure I wrote this other little bpa primer called key probe connect yeah the name, it's a bit misleading, but who cares? Let's take a look at it also, in this case, I use the sec macro. Please notice that instead of the bpf prog helper, I'm using the bpf key probe one here. It serves the same purpose as bpf prog, but this is specifically for key probes.

B

Thanks to it, I can declare a signature like this and easily having the arguments of the security socket connect function without getting crazy in a way very similar to the previous bpf lsm program. This key probe bpf program checks for the protocol.

B

Okay, I'm only caring about epv4 at the moment and then uh in this case I restrict the outputs in the trace part of this program, depending on the program causing the connection attack, connect. Okay, the rest is simple. I cast the address to an ipv4 basically and I bring it up.

B

Let's run it, the attack ready set, go all right, fine, correct destination: hey, please wait a minute, let's not get carried away so easy. Are we really sure this approach with key probes works? 100 of time? Are we sure it's reliable, but it gets easily followed by the attack on net minor, just like the bpfc school response, I'm short on time and that probably write some tests to assess the reliability of various approach soon, in the meantime, take a note of the repo it's oral.

B

Is this one you can find it on my github here you'll find also other bpf programs to play with and understand how much a secure and connect team for they give to you now. Let me also tell you why I wanted to show you various experiments and do some considerations.

B

The main reasoning was to prove why a lesson bpf are here to stay and the other way to go. They were reshaping improving it. The way that we currently do security, buff on prevention and on detection side, but they are a fairly new feature that is not available in all the kernels.

B

So I told that showing you a key probe approach, given they exist on all the kernels too, was worth a try, even though pause a second to consider that attaching a key probe to the function, calling an smoke makes it some more dependent on this new feature: that's not available in older kernels, so maybe it's better to experiment with the key probes on something else like tcp v4 connect function, but this is another exercise for the audience at all. Furthermore, what does it take to use a lesson?

B

Bpf programs for auditing, it's just a matter of changing a bit the restrict connect program we've seen before by making it always returning zero, so not blocking connections to specific aps and taking note of the detected destination ep, maybe putting them in a bpf ring buffer or in a vpf map. I prep this audit connect example that basically implements what I've just said. Let's check it out, as you can see, I'm always returning zero.

B

Okay, I'm just clamping the program depending on the binary making the connections and I'm outputting the destination the trace pipe, as you may notice, it's very similar to the restrict connect previously shown bpf program and it's an lsm one. It's time to test it, let me run tcp dub to see where packets go attack, time.

B

Okay, the bypass is not able to use the race window and it's connected to 13.107, etc, and the last same, it's catching it correctly. Let's try it again. Oh the attack is connecting to the malicious b, as you may see, also from tcp dump and from its output and the lsm is getting the actual destination. Ap. The malicious one try again not able to use the race window not able to use this window, it's working, easy, peasy, lemon, squeezy, isn't it so not telling using lsm bpf for auditing is possible.

B

It's also, in my opinion, the recommended and future proof way to go. Another and work would be to investigate whether bpf product type tracing bpf programs. A new kind of tracing problem are vulnerable to this family of talk to attack or investigate the network. Trace points in the repository I'll put an example, for you whole to try out now go right and try them, and please ping me back with the results of your experimentations, I'm so curious. My twitter vms are always super recap time.

B

Today, we've learned various things first, that by providing linux with a standard api for policy enforcement, lsm ensures to enable the widespread deployment of security hardness systems. The protections provided by lsms do help protect your system from being hacked when an attacker uses flows in one of the running programs. Lsms are definitely an important layer in any serious defense in depth.

B

Strategy on linux machines we've also learned that bpf cisco trace points are not that reliable, as we may have hoped, and also that lsn hooks happens at a very, very deep level in the cisco's flow and they can be used for buff prevention and auditing with little to zero effort. So I'd say it's all for today. Thank you for joining us ciao.