Cloud Native Computing Foundation Cloud Native eBPF Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Falco & eBPF: Is the only Limit the Sky? - Federico Di Pierro & Andrea Terzolo

Description

Falco & eBPF: Is the only Limit the Sky? - Federico Di Pierro, Sysdig & Andrea Terzolo, Politecnico di Torino

eBPF is a powerful technology, but how could it be used in critical scenarios with throughputs higher than billions events per second? If this question doesn't make you sleep at night, this is the right talk for you. Building a large scale tracing tool is hard... Falco uses BPF to trace syscalls, page-faults, context-switches, and many other system events; but collecting all of those inevitably leads to data losses. The situation becomes even more complex when we consider the portability issue: supporting lots kernel versions, even very old ones, means not being able to take advantages of all modern BPF concepts like ring-buffer, global variables, and other advanced tracing features. Starting from these assumptions, Andrea and Federico will drive you through the actual BPF probe architecture, its strengths, and the criticalities that must be faced every day. More precisely, they will outline some possible mitigations to actual problems and what they are planning for the future to improve the situation, exploiting, when possible, modern BPF tracing features.

A

Hello, everyone and welcome to this talk. I am federico pierro, a manager of faculty security working for csd. In the last few months, I've been quite busy in improving the falco bbf probe today. Here with me, there is andrea.

B

Hi all nice to meet you, I'm andrette, also a research fellow at polytechnic of theory and right now, I'm working full-time on ebpf technology philly. Would you like to show? Where are you.

A

Yeah sure, let's go to our agenda.

A

First of all, we will explain the idea behind the falco project and the reason why we can call it a peculiar bpf use case. Then we will have a look at how falco uses dppf technology under the hood analyzing the most relevant aspects. Finally, we will focus on the main issues faced with our current instrumentation.

A

More precisely for each pitfall that we will point out, we will share possible mitigations and some future announcements.

A

So, if you're ready to start, let's see what the fico project consists of, as you can see from this fun animation, falco is an open source project that basically sneaks all your system searching from a level's action to perform this inspection. It uses two main methods: an epf probe and a kernel module.

A

But first, let's try to understand the benefits that bpf technology brings on the table in regards to the kernel module. As you can see, there is no risk of kernel crashes due to a wrong instrumentation, because bpf verifier ensures that the bat code is safe before injecting it into the cargo.

A

Moreover, the kernel becomes really easy to instrument adding new features on the fly. We have just to inject an epf program without caring about the internal state. Last but not least, we can solve all the painful portability problems that we also touch in our discussion with modern video features like compile ones and everywhere approach. The so said, query approach, okay, but why falco is a particular use case?

A

Basically, it wants to exploit all these advantages, even in unconventional scenarios. On one side, it needs to capture almost all the system, events without degrading performances, this means capturing cisco's, content, switches, page faults and so on, generating up to billions of events per second to the. On the other hand, fico requires high portability.

A

The support matrix is battery is pretty huge, going from kernel, 4.14 and clank 5.0 onwards. Therefore, therefore, dpf probe must be successfully compiled for all the elements of this matrix, taking into account the problem related to current optimizations and pdf verifier requirements.

A

You can see that these are the two main reasons why we consider farco and if we have peculiar use case, we will investigate these two topics more in detail, but before doing that, we must dive into our pbf. Instrumentation really works andrea ball to you.

A

Thank you freddie. I will try to explain how our pub works without messing around with too many details. Let's start.

B

Our journey as people said farco catches all the most important system event, but for doing that, we need to estimate some kind of look. A kernel look is a particular point into the camera code, where we can attach our bpf programs in particular to deeper investigator system. We need different hooks, let's seal them, one by one.

B

As a first thing, we touch the entire flow of cisco, starting with the center book and terminating with this exit one after it, we have page fault, user and page for camera hook whose job is created, they catch all page fault emails in the link below we have the sketch rich hook, which catches content switches.

B

Next to it, there is the persex hook that collects alternated process and, last but not least, we capture all the system signal to the signal delivered book. These are all the seven books that are involved in our instrumentation.

B

As the first thing, we will focus our attention on cisco hook, because the behavior is a little bit different from the others.

B

More precisely. We will consider the center hook, but, as you might expect, the exit one exactly the same. Behavior, to better understand the real logic. Let's imagine that an open cisco is executed on the system. The bpf program attached to the sender hook is called and we usually refer to it as a dispatcher.

B

You might ask the reason behind this name, but I think it is quite intuitive in a nutshell, this program has to understand which cisco has been called and what is the corresponding vpf program that we have to trigger in response? Yeah, you understood correctly. We have a specific vpf program for every cisco.

B

This specific program had the task to collect some security relevant data for regarding the cisco, but before looking at a concrete example, here we have to clarify how to perform this dump from the dispatcher to the specific program. We can surely use the telecoil mechanism, but what is a tail call in the bpf context?

B

Attack call can be seen as a mechanism that allow bpf program to call another one without returning back to the old program. Moreover, this call has a minimal valet because it is implemented as a long jump, using the same stack frame so keeping in mind on this information. Let's see a complete flow when the open arrives, the dispatcher recognizes the cisco id in this case true, and it performs a tail call to the right vpf program.

B

The specific program collects some information about the cisco, for example, the path name modes flag of the file being opened after collecting all the desired parameter. We are able to finally push data to user space.

B

Now, if we consider a different hook from the cisco one, for example, the page fault user, we can easily understand that the flow is simpler than before. We don't need anymore dispatching phase and we only have to correct to manage the specific capital logic here. For example, we collect some data regarding the pitch fault like the address and the error, and we send them to user space, as you might expect, all the other rooks that are not cisco related have exactly the same behavior.

B

To finally recap: we can see in this slide all our programs more precisely. We can notice that cisgender and nc's exit books generates way more even than the others, because they are triggered every time a new cisco is executed.

B

This isn't the instrumentation part, but from the previous slide it should be clear that the number even for wobbly trigger space can also grow dramatically, and this could become a serious problem for farc fede. Would you like to provide us a little bit of context.

A

About it, yeah sure, as you said, there are some situations where this exhaustive instrumentation can generate troubles and we have to consider alternative solutions.

A

The first two problems are added in this slide, so the consumer issue, an instrumentation issue- are mainly related to our instrumentation, while the last one refers to the concept of white portability that we have introduced at the beginning of this presentation. But let's proceed in order and start with the first problem.

A

We call it the consumer issue. This term means that the user space is unable to fully consume all events generated by our pbf instrumentation right now, we have identified two main reasons behind this phenomenon: the first and most intuitive one is that in very staggered environments with lots of lots of cpus, the number of protested events is huge up to tens of billions per second. Of course, our single targeted user space is unable to analyze them all. This space process is single threaded, because we have to guarantee the even chronological order to.

B

Maintain a consistent system state.

A

Moreover, a multithreaded consumer would force us to adopt synchronization mechanisms further, producing ciscos that, in the end, would produce a backache on the kernel. So this is the first reason we don't solve events against a single throttle user space.

A

The second one is less obvious: it depends on the cpu time granted to user space and obviously on the raw system load. Given that the second point is harder to grasp, you realize it a concrete stress test.

A

As a test environment, we use the local machine with 12 cores while as an event generator, we used the sng tool. Sas ng is a convenient tool that produces an enormous quantity of ciscos, filling all the cpu time and degrading system performances we let it run for 10 seconds and we obtained these results.

A

The total number of events produced by the kernel is 34. Millions of these only 4 millions have been successfully captured by user space, while the others have been dropping. Basically, the whole capture was lost. This means that user space has conquered the cpu for very short times, consuming only a few millions events.

A

Of course, this is an extreme case tailored to our needs, but in the end it fully allows the issue. Now you could ask us: how could this kind of problems be mitigated?

A

The answer is pretty simple indeed, given that all we want is to provide more time to use your space, let's reduce its work by producing fewer events, but the question now is: how can we reduce the number of events forwarded user space here is where the so-called simple consumer mod comes into play. Running fast, simple, consumer mode means turning on the kind of side capture pattern that denies events that are not so interesting for the system state to reach user space.

A

This filtering logic has two main benefits on one end: fewer events are produced. On the other hand, the instrumentation time is reduced, improving the overall system performance.

A

Now, let's check what really happens with simple customer mode enabled considering always the same c center. Let's imagine that, as is called that ppi id is executed, how the special program will start the execution, but this time the flow gets stopped, since the simple customer logic detected that this is called is interesting for our system.

A

All these events will be completely rejected and not sent to user space.

A

As a last thing, we can observe the results we obtained are running our stress test. With this new filtering pattern, the simple consumer is as effective as we expected cutting down the number of drops from 90 to 60 percent. In this specific test, the capital is still partial. However, by applying the simple consumer to real cases from the faculty is 0.31, we have seen notable improvements.

A

The future is still brighter. In fact, we are thinking to further reduce the number of cisco traces in simple consumer mode. As you can see from this slide, we started with a pool of 300 trends, 32 syscalls.

A

After our first analysis, we chose to keep only half of them, and this is the actual state of the simple consumer. In the last weeks, we are thinking to further improve this logic. Obtaining a final set of 84 cs goals, with what we can call an enacted, simple consumer anyway, you have to consider that the analysis behind these choices are quite complex, since every event can have unexpected impacts on the system. Therefore, it is always good to move carefully through these stormy paths.

A

This ends up the discussion on the consumer issue. Andrea, would you like to drive us to the next pitfall? Yeah sure, let's talk about the instrumentation issue, the first starting with the discussion.

B

I would like to thank jaisalton, a member of the farco community for the amazing research he carried out about this topic. The data collected by giles raised a series of reflections that will summarize in these slides. You can find both the giant research in the discussion meet in the presentation we have uploaded on sked.com. You have just right clicked there.

B

Now, let's go back to our topics to understand what the instrumentation issue means. We have to consider that in some use cases we only want to capture specific scores and not all of them as seen before.

B

But what is the cost of capitaling just a bunch of cisco without instrumentation? If.

A

You remember well.

B

Every time a new cisco is executed on the system, the dispatcher is tricked and it will take all a specific bpf program. This approach perform well, if you want to test cisco, but how would it behave if you only want to capture a subset of them, for example, if you're only interested in 20c score, what are the performance?

B

Unfortunately, the dispatcher would call all bpf programs even those related to cisco debt. We are not interested in increasing the current execution time and slowing down the entire system. This is what we usually call. The instrumentation issue giles in his paper proposed an alternative instrumentation to address this overhead ccpf offers the possibility to attach programs directly to cisco specific hooks. Why we want to export these instead of using the generic center hook. If you want to twist just 20 cisco, we have only to instrument the right hooks.

B

For example, this is entered right center, read center fork and all the others conceptually. The advantages think seems clear, but let's see it in time of performance, as we said before, we assume we are all interested in 20 c score to measure how much the current is bounded by a certain bpf instrumentation rather than by another. We consider as a reference the execution time of one of the cisco.

B

The more the system will be overloaded. The longer the c score will take time to execute. In this case, we choose the right as a reference cisco. The first column represents the value, the variety execution time without any kind of ebpf instrumentation, so on a vanilla current. This is the ideal time to which we want to get close with our instrumentation.

B

The second column shows what is the cost of using our actual instrumentation, so with this is ender and this active book. Finally, the last column represents alternative instrumentation, seen in the previous slide, with only the 20 hooks necessary for capturing the required cisco, as expected, trying to capture only a bench of cisco, with our current instrumentation introduce a significant kernel of red compared to the alternative instrumentation.

B

However, if we think about all we have said so far, we can notice that the simple consumer approach could then pass to mitigate this situation. Let's see how.

B

The simple consumer interrupts the dispatcher execution flow, stopping instrumentation before recording the specific ebpf programs. This seems to be the solution we are looking for, however, measuring the right execution time. We can notice that we don't have a great improvement, but why does it happen with the simple consumer? You don't call anymore or the specific bpf program. So where is the problem.

A

If we consider the filtering.

B

Step again, we can notice that it doesn't happen immediately at the beginning of the dispatcher. In fact, before calling the logic, it is necessary to read some data inside bdf maps to the well-known bpf helpers.

B

Although these calls are not particularly expensive, they are repeated for every cisco, even for those that will be filtered out by the super consumer approach. This is why we still have this overhead, so in this case we can say that the simple consumer is not the great mitigation we are looking for.

B

Before. Jumping to the next topic. We can see what we are planning for the next future. We have to consider two main alternatives. On one side, we have the solution put at the plate by giles team, so the one with the cisco specific book.

B

The only disadvantage of this kind of solution is that we have to maintain two different upf instrumentation and enable the right one, according to the falcon use case, it's for nothing that this alternative instrumentation bring advantages only if we are interested in tracing a small number of cisco, while the main parkour purpose is to trace the entire system. So, in the general use case, there is no particular benefit from this approach.

B

The alternative solution to maintain a single instrumentation for the use case could be to adopt the simple consumer approach in a slightly different way. Recently, we are thinking of moving the filter in logic near the beginning of the inspection. Hopefully, this will stamp. The number of epf calls as much as possible, making the instrumentation cost for all filters, cisco, almost none one possible way. We have to move. This logic is using modern bpf filters like global variables and btf enabled tracing program.

B

Btf stands for bpf type format, so the bpf debugging, but without going into too much detail, we can just say that on one side, global variables allow us to read and write data inside a map without using bpf helpers, while on the other one more addressing programs allow us to directly access the kernel memory without having to issue the well-known bpf. Probably so explain these new features, we can surely dump down the number of bpf helpers that we call in our problem solving the financial problem.

B

Moreover, reducing the number of helpers, we could also obtain a substantial improvement on the overall instrumentation cost, since almost all our programs widely use them anyway. Just to summarize, we can say that even if we don't have a working prototype, yet we hope to get similar performance to the solution on the left. Since instrumenting, an almost empty ppf program seem to have a pretty low overhead.

B

This end of discussion on the instrumentation issue. Now it's your turn fade.

A

That was quite long. The last point that we have to address is slightly different from the others, as between an only console performance, but only portability, as you already stated, fico requires like portability.

A

The support matrix is pretty huge, going from kernel 4.14 onwards and from client 5.0 onwards, so the same bf probe must be successfully compiled for the elements of this metrics.

A

In addition to the obvious issue that internal current structures can change from one version to another, we have a much more serious problem. Each clang version enables different optimizations on the code generating a different byte code. Moreover, this same bad code could be accepted by some versions of the current verifier and not by others.

A

Here you can see an example of an issue we visited some time ago, but it is not alone. We have plenty of them. In our repository, you can find the link on this issue in the presentation of bloodnoundscan.com. We have just to right click on this image.

A

The code on the left was correctly accepted by the current verifier when compiled with all clank versions, except when built with clank 11 and kernel version. 5.8, this was pretty strange, so inspecting the generated bad code. We noticed that the client organizations introduced the possibility of a backache in the code. The catchers, of course, are not aligned by the bpf verifier.

A

B

A

This big catch was unreachable, the kernel rejected will probe. The fix is pretty weird at the first look by removing the black construct. Clank is the night to introduce the unexpected the catch.

A

Of course, once you solve such an issue, you must double check on all other metrics channels. Then no issues appear because of the fix.

A

In the end, maintaining such a code can become a great mess, giving all the hidden actors out of our control, checking out the support metrics every time a change is committed. Is time, costly and slows down the piece of development.

A

Unfortunately, we currently have no real mitigations for these problems, but good thing is: we are already working for future improvements. In the last few years, an ebpf concept was born. The compile ones run everywhere, approach, as its name suggests, the idea is having a single bpf probe that is portable among different kernels.

A

This means that the bytecode is actually relocatable based on the kernel where the probe is being interaction. To achieve this, we need the key requirement over the others using dbpf library, to add our probe all the logic to perform the relocation between our code and actually canada version is necessary inside this library.

A

Even if this requirement might seem easy to achieve today, falco does not use the bpf as a loader. Instead, he directly uses the ppfc school to perform. The value's actions remember that falco was born before korea was even a thing. Therefore, its code architecture follows low level patterns that are not easily portable to the dbf logic, without rewriting all the existing code.

A

Anyway, decorary approach will allow us to compile our probe a single time with just one clang version, and this is exactly what we are looking for. We can forget about all optimization issues related to the different clan versions. Basically, we will remove the clang axis from our support. Metrics yeah with this. Finally good news, we have ended our last topic. I live to you andrea for a final recap.

B

Thank you fed just to conclude, we can say that the main idea behind this talk was to provide you with an overview of how falco uses the ppf technology to trace the type system and which are the issue that we often face in production due to the huge quantity of data that it generates. We hope that you were able to follow us and that you almost understood all the different problems and possible mitigation that we can put in place.

B

If you have any doubt or any curiosity, please feel free to reach out on the slack channel, hoping to see you soon bye and thank you for your attention.