Cloud Native Computing Foundation PrometheusDay NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Achieving Zero-Instrumentation Monitoring with eBPF - Kemal Akkoyun, Polar Signals

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Achieving Zero-Instrumentation Monitoring with eBPF - Kemal Akkoyun, Polar Signals

Metrics are powerful tools that exist in the cloud-native ecosystem using Prometheus. However, using facilities to enable monitoring requires instrumenting the code. Everyone wants observability, but nobody wants to go the extra mile to instrument their clusters or applications. This is where eBPF comes in. eBPF, a promising technology for observability tooling, is not news. To observe the infrastructure and applications, eBPF-based system-wide agents can help us to capture events without requiring recompilation or redeployment of applications. In this talk, attendees will discover alternative ways to collect metrics from applications and infrastructure using system-wide eBPF agents. The presenters will demonstrate what level of observability could be accomplished without instrumentation.

A

ah There we go all right, so get your stickers, okay, nobody at the table anymore. You can get more stickers uh after the next talk um by the way, I'm Matthias I'm on Prometheus team. Just so you know um working there with Richie on Prometheus team and others, um and next up uh we have Kamal, actually colleague of mine. So next talk won't be that weird to introduce my co-worker coming from a Prometheus perspective, um but yeah Kemal is talking about achieving zero instrumentation, monitor using ebpf, so yeah. Please take it away.

B

Okay, hello, everyone, as mati has mentioned, we are today going to talk about uh achieving zero, instrumentation monitoring.

B

So a bit of a background before uh this is kind of relevant, uh because why I'm talking on this so I've been working on observability, tooling, since 2019, uh and if it's not obvious I'm working for Polar signals, uh we are relatively new startup and we are building continuous profiling tools and I'm, maintaining a couple of Open Source projects or I'm, helping to maintain them, uh tunnels, Prometheus client, golang, and uh for the relevance of it parka, which is an open source, continuous profiling project, which is using ebpf and, as I already mentioned, I'm also an internal of client goal anx or instrumentation.

B

So that's my passion lies. You can check out the slides under these links afterwards, so why I am doing this uh so I've been working on Prometheus, related observability tools for the cup for like four years back and when I'm like uh started to work on ebpf tooling I suddenly realized that there are like a lot of things that actually intersects here and we can achieve maybe some cool stuff just using ebpf and exposing some metrics right.

B

But of course like as all my ideas, it's not original and people already discovered these areas, and there are a lot of Open Source tooling around and today I'm going to talk about them. So first instrumentation or in other words, white box monitoring. uh If you are familiar with Goal code and if you ever use client golang. This is actually how you can instrument an HTTP service.

B

You need to register a bunch of methods and you need to know about your histogram bucket distribution, and maybe you can have some cool middlewares and add to your application. And then you can register to your Prometheus HTTP server and Export sub metrics.

B

But like there's a lot of things involved to make it right and we actually with bartek talk about like about these things, the best practices of instrumentation uh using client calling a lot you can check out on the internet and like, but when I first like start working on ebpfix the same question. Maybe can we just like just get rid of this tooling and just have something generic and so that we don't need to get into the code for every time? And add some code to just monitor or instrument our application.

B

So for that like we should have some goals right. So when, when I talk about zero instrumentation or like black box monitoring, that means we are. We want to achieve this without modifying any code for our piece of application or not adding any proxies or like sidecars those type of facilities as well.

B

So how we can do this uh today, I will try to explain that so, first of all, like uh I'm planning to just enlight your way for about ebpf how you can actually craft and compile evpf applications and with using these examples. Maybe this could be a framework about like how you can start a journey with your own instrumentation or observability for Prometheus ecosystem, and we will see a couple of like real open source projects that you can just use today.

B

So let's first talk about ebpf. If you don't already know, maybe you are living under Iraq I, don't know because this has been a hype for the past couple of years in the cncf environment ecosystem. So BPF actually stands for back the package filtering, but it's like and the E part is extended version of it, but like this is not true anymore, like it's a concept at its own and like it doesn't represent anything anymore, but first when they added this to the Linux kernel, it was uh for filtering some Network packages.

B

So right now you uh what ebpf is it's a virtual machine embedded in the Linux kernel and you can write code and compile that to the for it that targeting that virtual machine and execute that in kernel Space by using that, like you, can do a lot of manipulation around networking you can detect security breaches or, like you, can have a lot of observability tooling by using ebpf. There are already a lot of tools in the cncf ecosystem and there are a lot of sdks for different various programming languages.

B

So how does it work? uh Let's like this is just an example to actually an EB, simple ebpf program that hooks into a Cisco it's written in C right now. You can use C and clunk to compile it or there's an alternative way to use rust for that rust. Compiler. Actually, support partially supports it. It's not like covering all the use cases yet, but there are some Works ongoing.

B

So this is actually you just like change your target with Clank give it a c program. Of course you need to be using special evpf headers, which we'll see next, and then you can load this e by byte code to the kernel, ebpf virtual machine you could be asking. Is it actually safe right? We are writing some C programs and it's running in the Linux kernel.

B

uh That's actually that's like why the Linux kernel developers like uh give a lot of thought on this and they come up with a way to actually verify the programs which is the ebpf verifier. This runs in the load time of your user space program and we passed this compiled byte code to the verifier and verify verifier. Make sure that, like the ebpf program is safe, there are certain restrictions like there's an instruction limit. It was a lot uh fee earlier, but it's like you right now.

B

You have a million instructions or so so you can write the relatively larger programs and the verifiers make sure that you are not doing anything harmful. It kind of make sure that your program actually stops like this famous uh computer science program, halting problem, it's actually unfoldable, but with some heuristics uh that verifier actually make sure that it's uh kind of stops at a certain point so and then from that byte code, it just just in time compiles and executes that.

B

So how do you communicate with the kernel program and with your user space program? This is where, like the ebpf maps comes in right and there are. These are special data structures that you can manipulate in your evpf programs. You can write data in them and using user space libraries which the most popular only BPF- we're gonna, see here in that in upcoming slides, and you can read this data back and do whatever you like with that data.

B

So one of the things that we need to talk about is the hooks, so lyrics are already exposes couple of hooks that you can actually attach your ebpf program. These are ciscals. Some kernel functions some tracing points. Also, you can have some custom hooks uh like j-props or u-props or perf events, and then you and you can, when you write your ebpf program with special syntax. You can say that, okay, this program, whenever this event happens, just run this function in the program that you define.

B

So this is relatively new thing: compiling your ebpr program and running everywhere. Before this we were using BCC, which stands for like BPF compiler collection like GCC, and what it does is like you just package that tooling, with your application and in the runtime you add, you actually compile your C programs to the bike called and then you load pad. So you don't need to be targeting any cross compilation issues, but, as you can imagine, it's like challenging, because it's a lot of tooling that you need to pack with your host environment.

B

If you are running in a containerized environment, it's like it means large containers. So recently this is sold with uh this new paradigm called compiled ones run everywhere. Sometimes it is like pronounced as Corey, and the goal of that is just omit all the compiler facilities from your runtime and do everything in the compile time and make sure that you can access all the related kernel facilities with your program.

B

How to do this is this is this is we are doing this uh thanks to the thing called BTF, which is uh type definitions for the kernel? What we do is when we compiling our program in the development environment. We use this our own BTS information and compiler byte code, but then, when we are loading the program into the kernel, we use the BTF information from the kernel and we do some relocations and make sure that the BPF program that we run in the kernel has the correct access, for example, a struct field.

B

So by this this actually makes our life a lot easier to actually compile and load programs to any Linux environment. Of course, to be able to support this, you need to be targeting certain kernel versions, but this is kind of implementation detail, so how it how it does that, like, as I quickly mentioned like loader, do some relocation, magic and then compile the program, so it it makes sure that it actually supports your program thanks to Corey. It makes our program super portable, and because of that, we can actually have something like that.

B

We have a manifest. We have a container or a program, we can just keep CTL apply and we can just run this evpf program in our cluster and we can collect any cluster related metrics or any other observability data that we need. One thing we should mention like to be able to run these programs. You need to access to the kernel, so this these are privileged containers or pods.

B

So, like you need to make sure that, like what are you loading in your cluster so before we continue to The Matrix part, we need to make this distinction between system metrics and application metrics in Prometheus ecosystem. We, with the help of like node exporter or C advisor. We can have a lot of system metrics available to us right and on top of that, we can have application metrics, which we get from instrumentation so but, like these tools are actually using uh Linux, observability or monitoring facilities.

B

So they are mostly Contours and you just get whatever you. You can collect from the Linux kernel itself so for the application metrics, as we've seen before, you need to instrument your application so, but we can actually solve these problems using some evpf tooling already available.

B

The one of the most popular ones is ebpf exporter from cloudflare and what ebpf exporter does it to be able to solve certain problems that we have with node exporter or C advisor it's using ebpf, Trace programs and collecting some data and like, rather than collecting, just counters? With the help of these Trace events, you can have histograms and more granular metrics. We will see some examples uh about that, but I mentioned some tracing too.

B

So so what are those again like I briefly mentioned about BCC, about compiling programs and I mentioned that it's the old way, but in the BCC also, we have already enabled BCC to uh BCC Trace tooling. So, with the help of these tools, you can have a lot of uh you can reach out a lot of observability facilities from the Linux kernel, most of it.

B

I don't even understand like this is this requires like a lot of in-depth knowledge, so you need to individually check out the tools and if you want to reach out like certain I o usage, for example, you can find the correct programs in this tool set again like the community is moving from the BCC to the BPF based tooling and again be all the equivalent programs are written with EBP, BPF, apis and they're readily available. uh Under this link, you can check them out.

B

So what ebpf exporter does like using the recently BPF tools? It's exposes like more granular metrics, for example, about our systems, so how it does it like? You have a configuration format with that yaml, you can configure a metric and then you can have a corresponding ebpf program to actually populate that metric with all the necessary labels, and you can load this to the ebpf exporter and you will get all the metrics that you need already in ebpf exporter. There are some implemented examples and you can use these for certain system.

B

Metrics like this is a socket example. It's like trying to find out like the socket acceptance latency seconds and if I'm, not it's an a histogram, and it's called like this, and it's exposes this Matrix and then you can visualize it in a grafana dashboard using these metrics and you will have more granular information about your runtime system. If you want to dig deeper and other examples, for example, you can find out about the um kills, because you can still ex like for each C group.

B

You can attach these programs and access to this information and you can have this metric as well. So there there was a talk in 2018 if I'm not mistaken. In the prom con and the creators of the ebpf exporter, they dive deep and they explain. Why did they do this in in detail? One keyword about that is at this stage.

B

They were using BCC so and they were mentioning about the BCC challenges or all the programs were using BCC apis right now, the repo is updated to use the BPF, so some examples wouldn't apply right now, so I suggest you to check the GitHub repo. If you are interested so one other talk I would like to mention is from bartek.

B

He also explored how we can use ebpf exporter and how we can actually create custom programs to be able to monitor some Network application in the user space right, because ebpf exporter also can help you to attach your custom programs again.

B

These examples in this talk is also a bit outdated compared to what ebpf exporter has because these are also based on VCC, but the tooling moved on to the BPF. So some of the challenges that mentioned in this talk is not valid anymore, because the bpos thanks to the BPF, the programs are more portable, but the just an example. You can just add this example, for example your cluster and all of a sudden, you have all this red monitoring metrics for all all online systems that you are actually looking for.

B

um One other example to achieve similar things is Hubble from psyllium psyllium project. This is a network network based ebpf tool, and they already give you the all the uh necessary metrics. If you want, if you drop the Hubble in your clusters and just works magically one other tool, I would like to mention in the same area. Is the pixie it's another cncf project?

B

What pixie does they hook a lot of ebpf programs into your cluster and they call like a lot of uh data and new. They store that for you and then you can actually query with their own language or you can write custom even scripts for that and all of like the out of the box Pixi can give you automatic red, metrics and database query profiling. Already they have implemented a lot of facilities around that. There is only one downside of it to be able to expose some Prometheus metrics out of it.

B

You need to be using open Telemetry, because pixie project doesn't like export any Prometheus, formatted Matrix out of the box, so there are also some more experimental tools. One of them is, this is a little bit out of date, but the repos and examples are there and there is a talk attached to that. What they do they do is there is a project called Cube, CTL Trace, where you can actually execute all BCC tooling against your cluster and in this repos they try to like discover using this tool and an operator What.

B

Can we achieve and collect and expose them as a Prometheus, formatted Matrix? And there is actually a really nice talk about it. You can check it out if you want to learn, but these projects are a little bit outdated. That's why I'm saying it's this experimental. It's been three or four years since they've committed anything and I. Don't think they are production ready, but the idea was like really nice to discover to just like expand the our Horizon and going outside the Prometheus ecosystem.

B

There are lots of other things that you can do with the BPF one of the things. This is actually a really a fancy project like in the uh they are calling this open, telemetrical instrumentation.

B

What it does is it's actually attaching ebpf programs to the U return props, which you can do, and this is actually how popular rolling debugger Dal actually works as well, and then you can actually uh manipulate what you have in the function call so what they do, they just inject the tracing ID, and so that you can have this manipulate with the manipulated context you you can have traces, Auto magically this using these type of methods. You can actually have similar metrics as well.

B

You can like yes, this is kind of you can actually break how your go. Runtime program is works because your return probes are actually important for the go runtime and then you need to Super. You need to be super careful about it, but this is achievable. You can achieve similar things to you to expose metrics as well for Auto instrumenting.

B

On top of that those experimental stuff you can. We mentioned that one of the use cases is the security. You can use a tool like Tracy to actually check out what's going on in in your cluster or in your running environment, about security, wise and you can alert on that and Shameless plug and like this is how all these things started. I'm working on this project called Park agent, which is an ebpf based profiling tool.

B

What we do is uh we like attach all the profiling application to the processes, that's running on a host, and we just give you a granular picture of what your program is doing as the CPU. The first version of the agent was using Prometheus service Discovery. So, like any for service Discovery, we had in Prometheus itself. We were using that what we were doing is we were attaching to the kubernetes API, getting all the c groups for the ports, and we were attaching programs for that.

B

By doing that, we will also be able to get all the metadata for that pod and we we can attach those labels to the our profiles at a certain point.

B

Actually like one of our products, is actually converted those profiles to the actual metrics and we just carried over all those labels and we just like store them in the in a tunnels cluster and so that we, if we were able to give a broader picture of what's going on uh in your cluster using metrics, and you just can drill down from those metrics to the actual profiles.

B

But right now, after that, like we faced couple of issues so right now we built our own storage uh and we are moved on from that stage but like while we were working on that. uh We realized that, like there are a lot of things that could have happened in the Prometheus ecosystem, so the gist of it achieving uh zero instrumentation monitoring is possible, but it's a lot of work thanks to ebpf. These are achievable, but like for each every each and every language runtime.

B

You need to specifically Implement certain programs and probably when the program runtime changes with the versions. You need to update that if you want to automatically collect some runtime metrics, for example, it's not magical like, but it's like. There are a lot of hard work, so maybe we can find a middle ground to actually automate this and there are a lot of startups out there or products that are actually doing this.

B

There are some APM tools that are actually using ebpf and you just install that to your cluster and you get all these metrics yeah and ebpf is stable, like there are a lot of hype around it. Lib BPF is like it has a lot of developers.

B

People are, Community, is Converge on that and we are building a lot of Tools around it and Someday I'm, hoping to have like not to be able to use client goaling, but just have some magical container attached to our kubernetes cluster, and we got everything that we need with that thanks for listening. This is all I've got and if you have any questions, just go for it.