Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Auto-instrumentation of Prometheus for RED Monitorin... Bartlomiej Plotka & Harshitha Chowdary Thota

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Auto-instrumentation of Prometheus for RED Monitoring with eBPF - Bartlomiej Plotka & Harshitha Chowdary Thota, Red Hat

Prometheus is known for robust client libraries in many languages allowing you to instrument your application with useful metrics. With those capabilities, many solid patterns emerged in order to get unified monitoring views for a cluster of applications. One of those is called RED which tells us from what points we should monitor our application (rate, errors, duration). While it’s easy to instrument such RED view for one application, it can be challenging for heterogeneous workloads, written in many different programming languages or in the close source. What if we could remove the instrumentation step from the equation, similar to what service meshes give, but without added complexity and overhead? In this talk, Harshitha and Bartek will go through different approaches of “auto” instrumenting your workload using eBPF, a way of safe and fast execution of code in the Linux kernel. You will learn how to leverage eBPF to get essential data in uniform format into Prometheus for RED monitoring. Join to see a demo showcasing these capabilities using open-source software.

A

Hello, everyone and welcome to our talk. uh We are super excited here to walk you through a concept of auto instrumentation of your metrics. uh You collect your own cloud infrastructure cloud native applications and beyond, and you know we can distinguish three main goals behind this talk and those were motivation why we are doing it.

A

First of all, you know recently we hear a lot about ebpf and and for a good reason, because it's quite powerful, so we want to explain how ebpf actually works and how it can be leveraged in the cloud environments for monitoring purposes uh with promote use. Since you know there might be high potential. Secondly, technology is often you know um over hyped, so we need to be careful. We want to look on with pragmatic eye on how this concept is really feasible. On modern clusters uh like communities- and you know.

B

A

It ready last, but not the least. We would love to share some practical tips and give starting point for those who really want to start using ebpf right now for observability purposes. We hope it will get you you know started faster, and so we can get. We can together, make a space of ebpf uh in clouds uh more nature, especially for observability um purposes, but before that, let's start with the short introduction, I'm here with harshita hi.

B

This is hersheda, uh I'm a software development engineer at amazon and I was a ex-thanos d-soc. Mendee and bi-tech was my mentor.

A

My name is bartek, full name is bartuami, but you can call me bartek, I'm prince principal software engineer at red hat, I'm from use maintainer. I also maintain thanos project and I am cncf attack. Observability tech lead and I also write a book with already called efficient go.

A

So, as you can see, we are not necessarily ebpf experts, although we spent quite quite much time debugging and programming in bbpf last week. Yet technically we are observability experts um looking into low-level ebpf programming as a way to improve observability in cloud-native ecosystem.

A

That's why we will be looking um on this concept right now in this talk from usability perspective, um especially on how realistic is this to be applied today in the cloud native computing foundation um space.

A

So the flow of the talk is following. First of all, we will explain red method. What exactly this is how it is useful, we'll talk about. You know practical instrumentation nowadays for implementing great method and challenges into obtaining and maintaining those we will talk about ebpf and how it can improve the situation.

A

We'll show a quick demo and talk about practical. You know adoption concerns or potential.

A

Harshita, do you want to tell us more about red metals.

B

There are several methods of monitoring when it comes to managing workloads in production on the cloud, for example, learning the health and status of the system to alert when something is not expected to efficiently do that. We need to look at the service from many perspectives, especially if it's a web service, and there are methodologies and patterns people invented that help us to identify what things can be good to monitor and which might be unnecessary.

B

So red method is one example. In the beginning, there was something like golden signals that we could read from google sorry book. There is used method by brendan grick and fairly recently. Red method was invented, I think by tom wilkie. So so what is the red method? It is essentially a method to monitor, request based service rate, which is, which is the number of requests per.

B

Second, your service is serving how much traffic you have and errors number of fail requests per second duration, distributions of the amount of time each request takes how long things are actually performing. Example may be nice traffic not to not too much and no errors, but requests are taking too long, so red methods. Indeed pretty amazing. We can build automation around it, reuse it. For example, we can build dashboards to show the common views of the system held for one service.

B

We can build slo views alerts and recording rules all the prometheus guris, but if any other services using red method, we can easily reuse save resources, so dashboards alerts, recording and so on. No need for magic, templating or reproducing things from scratch for every application.

B

All developers need to think is. How do I satisfy those signals described by red method and style? Devops tooling, will be compatible with it, but how we? How can we instrument application in order to satisfy red method.

B

Let's take the simplest example: ever we have prometheus server, it is a web server that serves some http requests. For example, it responds to promptgear queries, but it also collects metrics and allows monitoring. So let's use prometheus to collect metrics about itself to show how we can monitor prometheus in red method fashion.

B

We could then reproduce the same pattern for any other application. Then prompt you, let's go as we mentioned, prometheus collects the metrics, but it is also a server, so it exposes some metrics about its web serving one of the most important metric about.

B

It is prometheus http, request total, which is a counter of all requests by method and status code, so how it was done well, in fact, we could see this metrics, so instrumentation was implemented by han and promise code, and in this graph we can see that it overall gives us the rate of request served by prometheus.

B

So it satisfies the rate item in our red method. Great, but that's not all of it. It also satisfies the red error error site, as it has a response code metrics label.

B

So we know if this request was successful or if it failed, then we calculate the rate of errors percentage of those and satisfy error element on red method.

B

What about duration? We can look into the histogram prometheus exposes promises, http, request, duration seconds. It allows us to know the latencies of those requests and again this is the last item on our red method. So with just two metrics, we satisfy all of items on red method. We can kind of sure that we have a basic and essential monitoring enable for prometheus.

B

We measure and collect enough metrics to use red method, dashboards alerts, recording rules and so on. So that's the red method.

B

Let's now talk about how we can implement the red method, I mentioned that the way prometheus implements red method is kind of handcrafted, which means that atheist maintainers, like biotech, wrote the code from scratch literally imported the client of prometheus and go defined and registered required metrics and incremented. The counter everywhere web handler returned the response. This works great, but this has its own limitations.

B

So there are a few challenges. Firstly, we have to implement it, but what if we don't have access to the goal of the application? Maybe it's closed source. Maybe you don't have enough knowledge, comparation, tooling, or dependencies since you're implementing from scratch. You can make mistakes and it's harder to maintain.

B

There will be inconsistencies when one team crashed their instrumentation, they might have different metric labels, but different behavior different listener position. From the other hand, for the same metrics bringing in inconsistencies- yes, some of it can be fixed with templates, but it is not so easy.

B

So that's why handcrafted method is not the best it's it's not what we would recommend.

B

The other thing you can do is to use a well-known library like some gi, pc or http middlewares. For example. Biotic is a maintainer of thanos and I was a thanos mentee and in thanos we used certain x hdb package that we wrote, which is a library anyone can use. It can be hooked into any http server that will immediately instrument all the roots and http handlers.

B

The same can be done in grpc, using go grpc, middleware project that biotech maintains similar to http middlewares for focus on gi pc in form of interceptors, thanks to those reusable libraries. As long as those are red method, friendly, well tested and used, they are much better choices. Thanks to consistency, if you use the same library in all your applications, those helps with maintainability and avoiding mistakes, but still we have problems. For example, we still might not have access to the code.

B

Applications can be written in many languages, so you would consistency across libraries in different languages and those middleware and libraries might not be existing in all languages. We know it works and go, but maybe not for other languages. Like java, ruby, python, etc, last but not least, if people do instrument their example, hjp applications using something like a service mesh, for example istio, link3d or maybe just proxies where you don't have to add a full service mesh. Just this sidecar proxy to application like nginx or envoy.

B

Those proxies are part of every http request. So these can collaborate metric, read, method, friendly statistics about the pro about the traffic. What are the pros? Those are tested and reliable. You don't need to write your own stuff, so you can avoid making mistakes.

B

The there is solid consistency, because every application has the same side current this service mesh and it will have the same metrics. But there are problems, is much more complex to operate such system because service mesh is just another processes and other side curtains and containers. There are a lot of things that can go wrong. It's hard to operate on scale for sam application. There is also introducing significant latencies making it, so it won't work for everybody. There is some extra cost. It uses extra compute power like cpu or so.

B

This is this can translate to real money on scale. So at the end, this might not be an ideal solution.

A

Great, thank you harshita. So, as hashtag explained, um we have quite many choices when it comes to collecting, promotes metrics for red method, uh monitoring of our http requests, but they all have had its own traders pros and cons, and you know none of those can be described as automatic right.

A

So, let's explore if this ebpa thing can get us there, but exactly where, like what could? What would be the dream world that we can call auto instrumentation of our metrics so to us, auto instrumentation is when I can create a new application from scratch in any programming language I want, or I can take some application from somewhere and when I put that into the container, deploy on my communities and on many clusters and point my point, my promote servers around those applications.

A

I will have a consistent metrics about what http traffic is happening, so I can use red method to measure it consistently. Using my dashboards alerts, I can create my run books consistently, slos and all of those goodies that we care about all kind of consistency consistently.

A

That's beautiful right! That's that's! Amazing! um Let's see if ebpf can get us, there can help us here, so what dbpf is first of all. Nowadays it is the whole ecosystem, actually, with its own foundation, called non-surprisingly ebpa foundation.

A

This foundation is aimed to help ebpf projects ebbf-related projects to to try to be maintained, but also ensures that the core ebpf is maintained, and you know it's still open source because you never know um big players are in this foundation, so it looks pretty serious. um There is google, facebook, microsoft, clouther and more so um it it looks very, uh very ambitious. So what is the ebpf car?

A

The core is essentially a technology that allows uh allows this hooking into kernel functions, for example, syscalls and by executing small, save mall snippet of code every time this function is invoked right and everything in the kernel space, so you know very close to the kernel and um quickly and safely. This is kind of the the underlying technology of evpf. The feature was itself added in kernel 3.15, which is like quite old kernels. So nowadays we have like five something.

A

um So the technology is quite mature, so you can also store some data through ebpf program. It can store insert in in a in stack of course in here, but also in ebpf map that can be accessed by the user space, so some application. That is, loading this and managing this program and in our diagram. It is this go library that is doing this.

A

For example, ebpf program can gather how many pcp, packets, your linux, socket process, for example, and and and and then a user space program can access that information through by looking on this ebpf map. So this goliath library will would know about that, so how we can practically practically use such data in prometheus with prometheus.

A

It's quite simple in theory right we could develop an exporter which, by the way, is already developed by cloudflare, which is super amazing that allows us to inject our ebpf program and do certain mapping between ebpf map data and the prometeus metrics with what labels we want and, and then this ebpf exporter will periodically check ebp map or actually on every scrape to expose them. The the data in the metric form in a time series form.

A

So then it's as easy as pointing prometheus to this exporter, so kind of scrape it every whatever interval you want and collect this data periodically and expose it for the prom, ql and further integrations sounds very easy right and indeed it is actually matching. You know um lots of items from uh from elements from our goal.

A

You can actually you know, implement one ebpf application that will hook into places you want in order to gather those hdp requests statistics and then, if you have that it will actually whatever application you will, you will write in whatever language in whatever um you know, technology, as as long as it uses like normal http requests through kernel, you know we can.

A

We can measure that in a consistent way, so there's no custom code involved. I don't change anything in the application, I don't add any sidecar, so there's no additional cost, no significant additional cost or operational burden, and there is consistency because, whatever application I use, you know this ebpf exporter will provide a metric with the exactly the same name and behavior. So it's kind of powerful, but you know as everything you know. This obviously has some trade-offs, like everything has trade-offs and there are some catches. This code is from our demo ebpf program.

A

We wrote and you can see it's written in the c and it's kind of worse than that, because it's limited see. So you cannot import standard libraries, you can you have. You can only use few dedicated helpers and that really depends on the kernel version you are using. You have only 55 12 bytes for your stack, so you cannot allocate lots of a huge variable to to store. I don't know some some buffer or whatever um and like that even expects you to understand what is the stack, which is not normal.

A

Knowledge of devop engine develops, engineer or sres really not required. So that's already a challenge um and you can still kill performance of your kernel so of the coal machine because you can't create deadlock still, you can have major slowdowns, uh so so it is serious and also certain errors are um we know um some. You cannot hook into certain functions, even though they are in the kernel code, because they are in line, for example, so uh which is kind of a compiler optimization.

A

So there are lots of um the the the development in in this uh is is quite brittle. I would say- and you know that there is more like um the error handling is, is just painful like, and I guess this is just c uh so lots of mistling errors like, for example. This is unexpected type name, something, but actually this function, this macro bpf per cpu, is just not existing. Why? It doesn't tell me that this is just not existing.

A

I was you know, super confused and also like this is pointing to line 25, but indeed it was. It wasn't light quantified. It was some some other you know line, because everything is shifted because of those micro. So you can rely on that and, like it's quite painful in comparison to you, know beautiful programming in go in python, so you need to use to that um second catch. Second, major catch is that you have to know linux very well.

A

You need to be able to understand the tools and helpers which are available through this bcc, so bpf compiler collection. You need to understand what is available, what's not in your kernel, in a specific version of the corner and your operating systems and how to exactly check the signatures of those functions and cisco's, because you know you need to know those signatures in order to use sometimes to even hook into it and then use those arguments of those functions for your tracing or or um or monitoring right.

A

So it's kind of kind of hard, and you know there's also this also mean that uh this, this kind of being specific to the kernel version. Is that really none of the tutorials I I saw work for me, for example, because I have like pretty new 5.11 kernel version, so I had to dig everything from scratch. My function names were totally different and it just was well. I have to build things from from the bottom up, so expect your code to be not portable across operating systems and kernel versions.

A

um So there's a lot of effort to to maintain such programs on the scale right.

A

um Like imagine, I don't know like as an example, I couldn't use key probes, uh which is essentially a certain way into hooking, uh creating a hook or into some function on in the kernel on the accept for cisco, uh which is what typically you use to kind of trace tcp connections. I had to use trace point, which is like a different mechanism.

A

It was kind of fine like at the end. I could use that, but it just took me four hours to figure it out so great. It is a bit hard. um So let's go to demo like what what what we want to show what's the architecture. So in this demo you will see one way of getting two of the redmi method, monitoring signals using ebpf rate and errors, and we have cloud for ebpf exporter running everything in the containers, because we are cloud native and we have a promotion server.

A

As harshita mentioned, we collect use this prompt used to collect metric, but also we want to use this rec method. We want to monitor from use itself, so we want to compare the handcrafted metric with this ebpf based metric and the critical part of this demo is, you know the actual ebpf program, so there are kind of three ways I found how we can get the http uh request kind of metric statistics.

A

You could trace your program right, so there is something like you probes, your probes, which allows you to essentially hook into function, which is not in the kernel, but actually in your application.

A

So it's kind of similar how you do that in debugger right- um and um this is great you- we know that in golang we use http, uh slash net http packages and there's a certain function that is invoked when we have a request response and we could hook into that and every time we hit this into our code. uh Our bbf program will be involved, so we can make statistics from it. But, as you can imagine, this is really code dependent because suddenly the library changed, the semantic of the version changes every time code changes.

A

We need to update our ebpf program, not fun like we're kind of breaking our goal here and also it doesn't work with java applications, for example, because all of those are running in jvm, which is a virtual machine of java, so everything is compiled in time. It's just hard. um Second thing: I've seen is, you can attach your program to the socket, and this is useful for filtering packets or filtering http request.

A

Unfortunately, I couldn't use that, but maybe useful stuff and the last item I I chosen to do- is kind of a neat neat way into hooking into important, accept four, which is uh required to um you can use other stuff, but like this is typically used. What, when you want to obtain a tcp connection and the file description to the descriptor to the connection? And then you can read from that connection or write to that connection.

A

So we do that as an in the server side, we read requests and the right response right, and then we close this connection. So if we hook, we have kind of visibility into what's happening. The the tricky part is that you need to. You know read and write it's used for any, I o so any file. Reading writing. So we need to be able to really filter out the tcp connections only and ideally only http request and ideally only from our container.

A

So this is a lot of work. You will see that and I have no idea how to do it with the tls. I hope yeah. Someone will show me at some point.

A

So let's go um so in order to show that I we wrote some kind of golang test which is kind of end-to-end test using using go test library, and um so we what we have. We have here, a docker environment where we start prometheus and we start I obtain some kind of hacky in hacky way a prometus process id, because this is how I will I I will filter all those requests in ebpa program and I can start abpf exporter in the container as well.

A

So you can see it is a kind of privileged container here that um that I have to build my own ebpf kind of exporter version. I have to patch it a little bit, uh but you need to provide a lots of stuff right. um You need to provide headers, you need to provide access to the to the prods resources and and some debug uh trace point um kind of file path, but you overall start the you know: kind of containerized environment exporter in the containers environment in the privilege manner.

A

So then I have to configure exporter and x, ebay. Exporter has a nice configuration file, I kind of have it typed here, but at the end it's the yaml. I specify a program I specify metrics. I choose to have started connection, closed connection and overall requests, and I tell which bbpf map I use kind of it's called table here um and then I choose where I want to hook my program.

A

So I have four functions I want to hook into when, when the when we enter the accept for function, six call actually, when I exit that and when I enter right and I when I enter close and then I parse um this program- and I kind of put it here in this yamo, but I also substitute some pit, because I want to also filter by certain process id all right.

A

So we do that and then we just start everything make sure prometus knows about the cppf exporters and let's start it up and we open some kind of browsers. That's it so when this starts um both ebp of exporter and promotes should be running and it should open our browser and we should see after some time that uh we obtain some metrics. So the first metric is what hashtag shown it's like a prompt use metric for http requests.

A

We have apparently like around 30 of those, and this is exactly the same. Well, it's kind of the same. We aim for this to be the same information http request, but from ebps directly we have like similar value um around 30..

A

um There might be like some different delays on on where we check this data, so so that's why maybe the numbers are not exactly the same, or maybe I have back in my bp program, but um in in ideal world those you provide the same. You get the same information from from both methods, just ebpf just more. It has its own benefits as we mentioned, and we also have like you. You can see that process started nine connections and didn't close any of those.

A

Yet so um you know you have some additional information, so let's test it out. So, for example, let's um let's go to my code and when I kill this, I should open a browser with like a wrong query. So I do an api call. That was bad, so it should return with 400 status code. So we can see that uh prometheus handcrafted metric it you know found it really nicely.

A

Yes, we have just one metric, which was which was uh which was found uh as a 400 and then, um let's see, if vba program found it yep, it found that there was a one bat request happening in the system, but there is one more thing which I want to show. So when we go to the code and click to another step and we'd make a different call, which is a wrong, uh essentially not found error. But you will see something interesting where handcrafted metric doesn't show that we have only like one error.

A

However, ebpf metric shows that you can see there is increase and there are two um two errors with 400 code. So this is pretty neat, because what is happening is that we in prometheus- and I maintain prometheus- we probably forgot to- or maybe yeah, we didn't plan to put put this metric uh instrumentation for all the paths in prometheus server. So we missed this path of kind of root and some broken um broken path and we didn't instrument. We wouldn't know about that.

A

If someone would be spamming there and maybe kill our service because of ddos of those requests right, we would miss that because our handcrafted implementation was maybe is missing. Something but ebpa program knows about all calls so immediately. We know more than than just handcrafted metric right. We know that there are some and if you make more of those you know- and you know- maybe someone spamming in this request- we'll know about that from the ebpf implementation uh right.

A

So this is the whole demo.

A

um I can also show you the code of um ebpf program itself, but you know um it takes some time to explain all of this um and we can do it at some point, um but it- and maybe I will provide like a blog post about that, but essentially yeah lots of lots of work were in there and kind of the most tricky part is to how to ensure we measure um really http requests only and only the rights that gives, because you know, when you write response, you you do lots of write cisco's, potentially not only one, because you maybe write like some response and and and not only http headers right.

A

So so we want to also measure only that so there's a lot of lots of complexity, but once once you do that properly, it is rewarding right. So, just to finish, um we have kind of um you. You know you can do it, but let's understand this feasibility right. um The technology is major, but we see um that it need. It requires some kind of low level programming skills. You need to know about kernel. You need to be aware that there is literally zero portability here.

A

uh You need to be very aware of your kernel versions, so you need to deploy, for example, when you deploy that into kubernetes, you need to probably be in not probably you have to be in control of your current version of all of the nodes and operating system which, usually you have so that's possible.

A

um You need to fix your containerization access, so it has to be privileged container and so on still doable things and uh and mapping challenges. So all ebpa programs usually have is like say, group c group id, which is like number of e-node and then thread process id and there are ways to map it. I have to create a patch on ebp exporter to decode that, but you know I think, with some work it is totally doable. So um thank you. I think I hope you learned something. I hope it will get.

A

You started if you want to go and and and try to make something production ready with ebpf for observability needs and check our demo and sources in this repo and follow us on twitter and ask any questions if you want.

A