Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond the Hype: Cloud Native eBPF - Frederic Branczyk, Polar Signals

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Beyond the Hype: Cloud Native eBPF - Frederic Branczyk, Polar Signals

eBPF being a promising technology is no news, but for a long time eBPF has struggled with portability, it needed to be compiled for each kernel, or a compiler and kernel headers needed to be shipped executing effectively arbitrary code. The eBPF community acknowledged this and started the CO:RE (compile once-run everywhere) initiative, which is young but quickly maturing in the form of libbpf. Parallel to it, we are discovering more and more use cases where eBPF can be useful, especially when combined with Go and Kubernetes, from security, over observability to performance tuning. In this talk, Frederic will explain what libbpf does in order to achieve compile once-run everywhere, how it can be used in Go, and lastly how to bridge the low-level Kernel features to higher-level Kubernetes concepts like Pods, and demonstrate all of it with real-life examples to help measure and improve performance systematically.

A

I'm frederick I'm the founder of polar signals, and today we want to talk about cloud native ebpf beyond the hype, as you can probably imagine, we use ebpf at polar signals, and that's essentially why I'm here, but the idea of this talk is to kind of take this technology. This hit, that has been hyped over the last couple of years and see how we can kind of connect it in a meaningful way to the cloud native ecosystem to create some useful things.

A

So, hopefully, you'll walk out of this talk, understanding, ebpf a little bit more understanding uh kind of the the state of development with ebpf a little bit more and lastly, um how to connect potential, ebpf programs with the kubernetes or cloud native ecosystem. More broadly and I'll show an example of what that could potentially look like.

A

So without further ado. Let's have a really quick introduction to ebpf now this won't be completely exhaustive of an explanation, but I just want to lay a little bit of kind of information down so that as we progress in the talk, we have a foundation that we can work off of. So essentially, what is ebpf ebpf allows us to run certain programs in kernel space, and we can attach these programs to particular events or hooks, so some things that people may be familiar with are k-probes or u-probes.

A

Essentially, what you're saying is you wanna run this program every time a particular thing happens in the kernel. For example, a syscall gets executed.

A

And the reason why.

A

We do this is so that we don't have this really expensive context switch when we, whenever we go from kernel space to user space. This is a like widely known fact that this transition is like something really expensive and the reason why we're doing this as well ebpf or the kernel and the operating system has traditionally always been a really great place to do. Observability, security and networking.

A

But uh the reason why ebpf was necessary here was that with ebpf we can now have a really flexible development model where we hot load this this code into the kernel and execute it as part of kernel space without having to you know, load a kernel module which we had to compile to a wide array, wide array of architectures and kernels and so on, um and even even though it used to be really hard.

A

Some companies even still did that right, like cystic started that way, um and um only eventually when ebpf came around uh adopted ebpf as a technology, so sometimes it was even then already worth it, but with ebpf we're kind of um supercharging. This possibility and.

A

I just want to give a really quick uh and it's really high level, but a really quick overview of how this actually works. So ebpf, as I said, is kind of code that we can hot load into the kernel and um it gets executed on hooks and, if you're, uh maybe new to this, uh this sounds kind of dangerous right, like we're running something highly privileged in like with kernel privileges effectively um in inside the kernel and so something that was created in order to allow this is a spec special just in time compiler.

A

uh That is part of the kernel um that uh kind of transforms evpf code um like ebpf bytecode, to um actually executable code on that host and before we even do that before we even kind of load this program, we verify that it will actually halt, and in order to do this effectively, we restrict what this code can do.

A

If you maybe remember from computer science classes or if you've kind, of seen seen this particular class of problem called the whole halting problem.

A

It's actually kind of an unsolvable problem without diving too much into it, and the way that evpf kind of makes this a solvable problem is by reducing the things that the pro program can do, and so things like endless loop, potentially endless, loops or stuff like that are not allowed or it's everything in an ebpf program essentially needs to be restricted in a way to make sure that it will actually hot and it will only use a def predefined amount of memory.

A

Now we can still do bad enough things that will crash a kernel um or we'll crash a an operating system. But um the point is that we um can't essentially escape um the the security boundaries and, as we all know, uh and as we probably some of you may have heard this already- even that's not entirely safe. There have been bugs in the verifier. They have been bugs in various pieces of this, so we're not totally safe.

A

But I guess that's the nature of all programs, but nevertheless this is still really really powerful and it's only getting more traction.

A

um But uh going back to this once we've actually loaded our ebpf programs and once they're executing on these hooks the way that we then communicate with our ebpf program or from our ebpf program to user space, where our like typical processes are, is through bpf maps, and these are essentially think of it as just kind of shared um spaces of memory that both the user space program, as well as the ebpf program program, can write and read from write to and read from, and that's essentially all the moving pieces on again a really high level.

A

But this is also where the problem kind of started in the early days of evpf.

A

Even though we are compiling our ebpf programs to this generic bytecode, there have been portability problems, because, even though this is just c code that we're compiling to ebpf um bytecode, we still had to bring all the dependencies or we had to bring the kernel headers that we were going to access, because at the end of the day, we're doing this to do something in the kernel right. So most likely we're interested in something in the kernel.

A

So we needed that those type of informations, and so the problem that this kind of created is that we either had to compile our ebpf programs for all the possible kernels that we could think of that. We would want to run our ebpf program on or we needed to ship a c compiler with our user space program.

A

That would compile the c code at startup and and then essentially require the the kernel headers to be present on that host, so that we could be sure that um kind of what we compiled is actually compatible with that host. Now that kind of brings multiple problems with it. We actually require the kernel headers to be present on that host and, ultimately, that results in really large artifacts.

A

If we, for example, put all of this into a container image, that means that not only are we shipping our user space program, we're also shipping, our c compiler, so llvm, for example, and then not only that we're also.

A

We also require to actually compile our program every time at startup, right or every time we create that container and, aside from all of this being really expensive, it's kind of potentially dangerous right, because what we're doing is we're taking some arbitrary string that we're then compiling um and then putting into kernel space to be executed right. So you can imagine like, even aside from potential vulnerabilities in the kernel about this, uh now we're adding a bunch of potential attack surface in our program.

A

So this is obviously something that the the community has worked on and I'm going to try to explain to you how the community is has essentially solved this um again. I'm I'm a user of this. I didn't create these mechanisms, so I'm going to try to explain them in the best possible way from my experience of using this and researching it and so on.

A

But essentially what the community has come up with is the a combination of this bpf type format and an overall initiative of that's called compile ones, run everywhere, kind of the mantra of what we just talked about in terms of portability right. The the goal of all of this is that we can just compile this once on. You know your my machine and then essentially run it on any kernel. That's that's the goal, and so what the btf, um the bpf type format essentially is.

A

Is it's a highly compressed version of the debug information, so essentially the all of the information that we need from kernel headers um and it puts that by default into the kernel, and so essentially, what we're kind of removing in terms of requirements now is that the kernel headers theoret interior, at least all of the information, is present on every kernel um granted that they actually have uh our new enough kernel to support this right and the way that we can think of this is essentially that btf is kind of an abstraction on top of our actual data structures right and that's kind of one piece of the puzzle and then the other piece of the puzzle is this new library um called lib bpf.

A

So, if you've heard of ebpf before maybe you've heard of bcc the bpf compiler collection, this is essentially the next evolution of this, and it works really closely with btf and effectively what it does is when we load a an ebpf program on a kernel that has btf enabled and the ebpf program was actually compiled with btf support, then it kind of rearranges and replaces the kernel structs that the ebpf program was compiled with with the ones that are actually available on the host right and there's a lot of really complicated like relocating stuff and kind of backward compatibility.

A

Stuff, that's going on here, um but essentially what we can. um The the thing that the thing that is important for us as users about this is that now, if we use all of this, we have actually portable ebpf programs that we do not need to recompile when we start um start our our user space program. But we can truly um compile once and run it everywhere.

A

So this is really amazing and this is essentially possible um on. I think the the earliest things about btf landed in the 4.4.18 kernel but effectively.

A

If you actually want to make use of it, the recommendation are distributions of these versions or higher, and essentially what we do is when we compile our ebpf program. We generate this stub. Essentially this vm linux.h. These are kind of our headers that we're compiling our ebpf program on. You know our machines, with with ac compiler, with target equals bpf and when we then load it via lib bpf, then all of this gets rearranged and it actually works on that kernel.

A

So this is pretty magic and I'm thankful that all of this work has been done by the community.

A

Now we're at kubecon and pretty much everything in in the cloud native ecosystem, or at least predominantly things are written and go. So we at polar signals exclusively do go as well, and so we went on the lookout for if anyone has written a library like that- and thankfully there was a start written uh by aqua security.

A

um Now uh this when we started uh working with the akrasek people, not everything that we needed was necessarily implemented, but they had a really awesome start. So we contributed and a lot and it's been a really great um collaboration and we hope to do a lot more together with them on this library. But let me give you a really quick intro.

A

Essentially, lip bpf go is you know, just like many other c wrappers? It's just a really thin wrapper around the c bindings, and then you know a thin layer of gold that essentially makes using all of this go native and then uh to get kind of the entire end-to-end experience.

A

We, uh you know, as we said before, we pre-compile our ebpf program using btf, um and uh we embed that into our resulting go binary using goes 1.16 built and embedding functionality, and if we now take all of this and statically link lip bpf into our resulting binary, we truly have an actually portable by ebpf, like user space program right and um our ebpf program embedded into this, so that we can then load it on any kernel.

A

So now we've achieved true um kind of portability, of both our user space programs, as well as our ebpf programs right. So this is. This is really awesome just for the portability aspect, but uh we have multiple other uh kind of advantages from this strategy. Of course, we don't need to ship a um c compiler anymore. We don't have to have you know the extra kernel header packages installed on our hosts and maybe most importantly, we have the comfort and safety of go so that we can.

A

You know, create programs that are memory safe around our ebpf programs, which, by the way the verifier has been tremendously helpful for us as well to make sure that the things that we're doing in c are actually safe. So this is a really cool combination.

A

So now I want to walk you through an actual example that we do at polar signals, so at polar signals. We do continuous profiling and I won't go into too much about this, but essentially you can think of it as just normal, cpu profiling, and so what is profiling? Essentially, profiling is finding out what our program is doing and we do that essentially by measuring where cpu memory or io resources are being spent, and this works by kind of capturing the stack traces um that our program is.

A

You know currently in a number of times per second, so let's say 100 times per second we're taking these member or cpu profiles, and if 10 of those observations were in one particular function. Well, then, we know, um statistically speaking, 10 of our time was spent in this function and so effectively that's how like sampling profiling of cpus works, and then we typically visualize them using something like flame graphs or icicle graphs.

A

That's what we call them when they are upside down, essentially or other visualizations, but this is just a really super quick intro to profiling in general, and there is prior art for profiling in linux.

A

So um there's it's not entirely new, and I think this is also something really important to understand about ebpf. A lot of evpf is not necessarily something that's entirely new. As I said earlier, there have been kernel modules around for similar things that we're doing with ibpf today, but just kind of um like the hooks have always been there like k-probes have been there. You probes have been there.

A

A lot of these things have already been there, but ebpf allows us to write really specific things that now are also thanks to compile ones, run everywhere, portable to every kernel, right and so in linux.

A

There has for a while been the perf subsystem, um there's kind of a lot of things uh that are all called perf, so there's the actual linux subsystem, perf event and there's the user space tool to interact with the subsystem essentially, and do you know useful profiling things and then there's the actual syscall called perf event open, which you can which essentially the perf user space tool, uses and interacts with to do all these useful things, um and this is something that has existed.

A

I believe, since the since the linux 2.6 something so this has been around for a while, and this is one of the kind of tools and then another format that we really love at polar signals is pprof. Pbrof is a standard of how you can represent stack traces and, as I said, essentially, profiles are nothing but observations of stack, traces and prof is a format in protobuf that represents stack prices in an optimized way, essentially um with all of its metadata that you actually need to. You know make make sense of the data ever afterwards.

A

So how do we actually build an ebpf profiler?

A

And maybe even in go right because, uh as I said at polar signals, everything we do is go so we use the bpf go, and this is actually one of those things that we implemented in libby pf go when we started the aquasec folks didn't really have a use case for profiling, so they had never implemented this particular syscall. That was required.

A

Sorry, not cisco. This particular function in libpf called attach perf event because they just had no need for it. But kind of this is the awesome thing about a community. They had implemented large other parts of the bpf go wrapper and we just had to go in and do a couple of these contributions, and you know everybody kind of profited from that, but effectively on a high level. How our ebpf profiler works is that we have a c group because remember containers essentially are nothing other than c groups and name spaces.

A

These are two kind of mechanisms of of the linux kernel c groups allow us to to limit how much resources um a set of processes can use and namespaces are essentially to limit how much they can see, and so together these essentially make up what we usually refer to as containers, and so, if we want to profile a container, we need to just understand which c group we want to profile, and so we essentially assume in this case that we already have our c group and we open a perf event on this c group and we tell it the frequency so, as I said earlier, let's say 100 times per second, we want to get an event from this right and what we then do is we get a file.

A

Descriptor back from this perf event, and we pass that file descriptor into attaching our ebpf program to this event right so now what we've essentially done with these two things is that we've created a perv event that triggers 100 times per second, let's say, and we have told it that it should run our ebpf program every time, this event, fires, and so now, all that we essentially need to do in our ebpf program.

A

Is um we grab the stack traces at that point in time right? We see, uh let's walk through it, for one of those events triggering right. The first thing we do is we read the process id that the stack trace is from then we retrieve the user space stack. So essentially, what is happening in the user space program?

A

Then what is the kernel space stack and then we take all of this and we put it into our ebpf maps and we have two maps in this case: one map just to identify our stack traces, so there's an id for our stack, traces and they're. The actual memory addresses that you know make up this.

A

uh The stack trace, because in the case of native binaries, so like compiled languages like go rust, c plus it's truly that representation of the memory addresses that map back to our actual binary and that's how we can at the end, make sense of what is being executed.

A

So that's our first ebpf map and then our second eppf map is essentially that triple of process id user space stack and kernel stack. That is kind of the unique identifier for this stack trace right. It's the process, the stack within that process and the stack within the kernel, and then we say how often have we seen this particular.

A

This particular combination- and this is all the information that we need to build a cpu profile, and so all our users, space program at the top here needs to do every 10 seconds it just takes all of the data from these ebpf maps puts them into a format that our tools understand. In this case we chose to do pprov and then we can just throw away all of the data and do it again in 10 seconds after you know, the avpf program has filled up our ebpf maps again.

A

So let's have a look at a little bit simplified version of this ebpf program. So at the very top we see this key that I was talking about right. We have our process id. We have our user space stack id our kernel space stack id, then what we're doing is we're defining our two ebpf maps.

A

Now there are some helper functions that I omitted here, but essentially, what we're doing is we're defining a map where we have our key from above as the key and then u and 64.. So, just literally our count that we are going to keep incrementing and then we have a map of stack traces, and this is kind of a built-in type in ebpf, and we just say we have stack traces in this map and our stack traces are allowed to be.

A

This max stack addresses long because, as I said in the beginning, ebpf cares a lot about everything being limited to some to some degree right, so that we can be sure within these parameters.

A

The ebpf program will succeed and will halt, and then the function that we have defined here is literally what's going to be executed. Whenever that that event triggers so first we retrieve our process id. We put that process id as the first field into our key. We then retrieve our stack, our user space stack id. Then our kernel space stack id, and then we take this key. We look it up in our counts map right if it exists, we we just take the count that we already have.

A

If not we initialize it, and then we do an atomic increment on that on that number, and that's really it right. Like that's all we're doing, I I simplified this a little bit but effectively. This is what we do now. I said earlier and I simplified that all we needed was the c group right, because all that containers are are c groups and namespaces.

A

Now, how do we actually find the c group right? We we, you may know uh in kubernetes we have the concept of pods and there can be multiple containers in a pod, and so we have all of this metadata in the kubernetes api.

A

So it must be possible to get the right c group for the right container right and it turns out when we have the process id. This is actually fairly simple. The procfss actually has all of this information and can tell us all of this information, and when I was researching this, I was like wait a minute. There is a tool that abstracts container runtimes and I'm sure this tool would be able to help me with this.

A

However, the problem is even though it's called container runtime interface cri.

A

It's not actually specific to running linux containers.

A

What it does is it abstracts away sandboxing, and so the reason why this was done is so that we can have hypervisors like virtual machines instead of containers, and so that's why there's actually no first class support for process ids because we're not actually a hundred percent certain that there is going to be a process id involved here right.

A

So this left us in this somewhat unfortunate situation, where the only thing that the cri has specified is this kind of additional information map of strings to strings where the container runtime is recommended to put the process id into.

A

Now this is really precisely specified right. Well, no, it's really really not and the unfortunately, because it's so vaguely specified uh that's kind of what it resulted in in container runtimes as well, and so, even though there's this recommendation, unfortunately, each every cri that we've worked with so far behaves very subtly different um and so just a couple of examples here, uh cryo uh a container runtime that was specifically designed for the cri standard specification has the string key info in the info field.

A

That is a json object that has pid as a field that then actually contains the process. Id docker actually asks you to use the docker api to to retrieve this and the docker api has this because docker is actually specific to linux containers right. So this is kind of cool.

A

However, it's not the cri, and then container d is kind of similar to cryo, except that there isn't an entire json object in there there's just a string that is an integer and you need to parse it to be the process id and then this is essentially what you need to do to find the process id for each individual container.

A

In kubernetes- and this was a little bit painful, but you know after implementing all of it, it does work, um although it would have been nice if there was something a little bit more specified.

A

So if we now put all of this together, we can retrieve the containers that are running on a host from the kubernetes api.

A

We can figure out uh through the cri and through all of what we just talked about. We can figure out the actual c group from that container. We can attach our ebpf program to that c group, and then we can go through what we talked about earlier. We wait 10 seconds for our bpf program to fill up the maps. We can then read all of this information.

A

We can transform it to the ppra format that we know and love and then at this point we can send this data away to a remote location um or we can save it to a file or something right. But um then we can just clear all of this data, and you know until eternity do this um again: wait for 10 seconds, transform it to p profit and so on, and so now what we've essentially implemented is a really low overhead, cpu profiler in ebpf, and the reason why this is really cool is using ebpf.

A

We can do this with extremely low overhead because we can already save this data in our maps in a way that is useful in our ppr format, and that way we can create something that can just always be on and always be profiling all of our containers, and this is exactly what we did at polar signals and open sourced as a an open source project called parka agent as part of the wider parka project, which essentially is a storage for profiling data.

A

It's also open source, and so um I encourage you to try out this project, but um even even if not, I hope you enjoyed this.

A

This talk, and I hope you learned something about how you can make use of ebpf how you can create portable programs with ebpf how you can make use of it from go and ultimately how you could maybe put some of these things together and even connect it to the wider cloud native ecosystem if you're interested in checking out some of this check out the website or, if you're, interested in the profiling storage, we have a talk by my colleagues, matthias and kamal as well about this, and if you have any questions, feel free to reach out via email or twitter and if you're interested at all in any of these topics, we are hiring so feel free to reach out for that as well.

A

Thank you. So much.