Cloud Native Computing Foundation Cloud Native eBPF Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Extending Cilium with eBPF for Exposing HTTP/1 Golden Metrics - Aidan Carson, Solo.io

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Extending Cilium with eBPF for Exposing HTTP/1 Golden Metrics - Aidan Carson, Solo.io

Metrics are table stakes for any modern service mesh, and because of the placement of proxies in a traditional sidecar-based topology, fairly straightforward to retrieve. However, as use cases evolve and the space matures, we have seen an evolution beyond that of traditional sidecar-based implementations. In this talk, Aidan will technically deep dive into the mechanisms through which we may retrieve the most valuable metrics — request count, response code, and latency (the so-called "golden metrics") — in a sidecarless environment with the help of eBPF. Aidan will discuss the hardships of implementing such a solution, and a few tips when operating at a higher protocol layer. All of these metrics will be exportable in Prometheus so that you may integrate a solution like this with toolchains in your existing infrastructure.

A

Thanks everyone uh sounds like we're live yeah thanks for coming uh like she said, I'm going to be talking about extending psyllium with ebpf to expose HTTP golden metrics, so before I dive in I just want to give you a little bit of background on me. I work at solo for those of you that aren't aware of solo they're, a startup that essentially do a big part of them is a managed istio. So they do kind of istio for larger deployments.

A

Multiple clusters, that kind of thing- and as you can imagine, there are a lot of networking challenges as a part of that before I was at solo, I was at nginx working on their service. Best solution, writing modules to help service their use case um and and I kind of jumped at the opportunity to be able to work nxtp to essentially service.

A

This UDP redirection mechanism and I kind of took that I, discovered kind of a passion for BPF started thinking about how we might be able to do the same for TCP redirection and offload some of the responsibility off of what the proxy was doing at the time and ended up enjoying joining solo, and they had some interesting kind of work with what was kind of going on, with, with kind of as the the process is evolving. The the role of ebpf is kind of really interesting.

A

There and I've been looking at what we might be able to do with ebpf in terms of servicing metrics.

A

So before I begin, I just wanted to give a quick shout out. um This document was really helpful, getting started and has kind of served as the Holy Grail in a way for for BPF and XDP, especially, and it's just really awesome to see the community evolve in terms of you know, documentation and support and interest, and and all of that, and it's it's really cool modern, to be presenting in front of in front of you all today.

A

So the agenda I have for. You is essentially just talking about a background on on kind of what we're trying to achieve, as well as what's kind of been going on. What's what's the current state of the world uh for for Golden, metrics, HTTP metrics, using BPF uh I'm, going to talk about the concept and then the execution of attempting to expose these metrics purely in in BPF?

A

So the background, so most of you are probably familiar with with these golden metrics Google defines it as if you can only focus on on a certain set of metrics. You would want to kind of focus on these. The the traffic number of requests, errors in terms of Response, Code bucketed by 2xx, 3xx and then latency, which is, uh will end up being the most complex to support which I'll talk about.

A

It's a really interesting problem to solve and then saturation which I'm not really going to be talking about too much today, just because it's kind of outside the scope of what I find interesting in these golden metrics in terms of kind of collecting.

A

um You know data as as networks or or packets are flowing through the network, and you can get it pretty easily through the kubernetes metrics server or or, however, you might get CPU utilization memory, utilization off of the Linux machine and then I'm going to just show quickly how you might be able to export something if you're using this architecture to to you, know export it using Prometheus and integrate it with with existing tooling.

A

So when we were looking at psyllium, we noticed um for our use cases at least that there were a couple gaps in terms of, for instance, like not being able to get like the number of bytes that were sent um or or kind of L7 traffic classification.

A

We noticed that psyllium did have L7 metrics exposed, but you had to have a policy enforcement using that Envoy proxy in order to get it, and so we started thinking about how else we could kind of get these get these metrics out of out of psyllium or or you know, in a psyllium ecosystem kind of get these metrics. So pixie is doing really interesting work. It's it's really cool what they what they have going.

A

They use you know, as people have been talking about K probes, um they're, they're tracing these, these ciscalls on entry and exit, collecting information, classifying the the request and the response, as well as determining latency.

A

um One interesting thing, though, is that they kind of uh they parse the ethernet frame. That's that's getting sent and they they send it to user space for for processing and that's where they do their latency calculation, um and it makes a lot of sense to be able to to do that because you have a lot less restriction in terms of you know what you can do. The verifier places a lot of restrictions there, as well as it's outside the the data paths. You're.

A

Not adding extra latency from that calculation, so it makes sense, but there are potential like drawbacks of you know, if you're sending a bunch of events through a map to user space, you have potential like permissions.

A

um If, if a bad actor were to kind of read the data on that map, um they could potentially get access to that data. Of course, there's you know when you're thinking about permissions. If they had permissions to read that map, they could probably kind of do their own tracing. So um you know there's an argument to be made there, but also when you're sending so many events to user space, and you have the potential for kind of overflowing that that centralized, aggregator and potentially dropping events.

A

So it's just kind of interesting to think about what we might be able to do in ebpf itself, instead of kind of sending a bunch of data to to user space.

A

um So in terms of psyllium, we were looking at psyllium again, it's a it's a really cool product and- and it's really interesting how they kind of own the entire networking ecosystem inside of a cluster. You can imagine traffic coming outside of outside of the cluster itself or from another node or another local pod, on the Node you're wanting to reference.

A

All of those have different nuances that you have to kind of deal with um in terms of tracking the network flowing through a system and psyllium is really cool in that it manages all of this, and it provides a a essentially an attachment point where you can run your own custom, BPF programs to add in your own custom functionality. So that's what we're going to be talking about this custom data path, I'm, gonna, and it- and it does this through the use of BPF tail calls.

A

So in terms of the concept. What we're trying to do? You guys are familiar with BPF you're, familiar with with what traditionally has been kind of done with it and when I was kind of looking at it um it the way, I kind of see it is when you're operating at these lower levels.

A

um When you're, when you're trying to look like the IP header, TCP header, there's a there's a you know a kernel structure to be able to reference, and then you can just kind of get the offset in uh into that packet and expose whatever information you're wanting to user space. It also enables really interesting use cases, say you're wanting to like alter the destination of traffic to a sidecar. That's wanting to operate differently based on the different protocols or something it enables really interesting use cases.

A

But as soon as you get to kind of HTTP, this is more traditionally, you know dealt with by user space applications. There's not a you know, there's no respective kernel structure that you can kind of reference in order to be able to operate on this on this data. So we have to fall back to kind of the RFC. You know, there's still a structure sure but you're operating now on characters you have to kind of iterate through in order to get that information, rather than like a bit wise.

A

You know something efficient or something easily accessible like a kernel structure and there's an element of complexity that comes with with trying to deal with the latency of the latency aspect of golden metrics. um So this isn't a like a simple example where you know t0 a client sends a request.

A

The server receives it calls any dependent, backends does any processing, and then it sends it back, and we can say that the latency here is it's three units, whatever you're wanting to talk about, but a more useful example of latency is this, for example, I I know with nginx they send the response headers in the body separately.

A

If you have a large response body, it's going to be fragmented, so you're not going to have one response per request: you're going to have to have a bit more of a you know: you're gonna have to track the state as multiple responses or packets are kind of sent in response to that request. So it's going to add some complexity and how we're going to end up having to you know, support this is by essentially tracking the content links.

A

That's that's sent in that response, header and then tracking the receive content as it's coming in to be able to ensure that it's equal to that content link specified just how HTTP kind of operates in terms of how this can be executed.

A

So again, going back if you're looking wanting to look at something like the number of connections, this is really straightforward in in ebpf, you can. Just kind of you know operate on if it's a classified program on the socket buffer um and then just kind of submit something user space, but there's not the same kind of it's not as straightforward and HTTP.

A

um So that's kind of what we're gonna we're gonna, try and solve here. So the idea here is that we're going to place two different BPF programs to help service the use case both on the on the if we're per client, whatever we're trying to get metrics for we would attach on the egress side, as well as the Ingress side, to to.

A

First on the on the send side, the egress side, we're going to catch HTTP requests to submit that to get that first kind of element of the golden metrics, the the HTTP requests, the number of requests and then we're going to begin latency capture and then on the response side. We're going to catch that HTTP response to bucket the number the in the response code in a 2xx 3xx that kind of thing uh and then what we're going to do is we're going to determine the content length.

A

That's in that HTTP header and then track response track. The response so that we can ensure that what we've received is equal to what is specified by by the what's the server sending and then we're going to log that latency.

A

So again, these are the the golden metrics that we're going to kind of looking for traffic number of requests, error rate in in response time. So, as it turns out, the number of requests is, is really straightforward. If you're just operating on on the request line of an HTTP request, you just look at get, you can see in the the RFC.

A

um You can just support these, these different kind of methods and then what's really nice is it's at the very beginning of the user data. So there's no complexity in being able to get this. You just essentially, you know, check the the character array at zero index zero, one. Two, um it's really straightforward to get um and then submitting to user space you could just key it what's what's interesting to call out here is that, when you're operating within like a silly, uh the psyllium system, um the destination address I've found is not necessarily reliable.

A

For instance, if they're trying to redirect to an L7 proxy for policy enforcement, that destination address might get changed, so I had to key it based off of server identity, which is going to be it's going to be consistent when you're dealing with when you're dealing with an egress program here.

A

So that's the that's the kind of key that we use and then we use the client address and then we have to choose a BPF map, so kind of iterating through um I, started with a with a hash map which which makes sense whenever you're kind of collecting metrics, but that is lock implications uh because all the CPUs are going to be kind of um kind of. You know trying to add to that hash map at the same time.

A

So there's the introduction of the CPU, the per CPU hash, but that has disadvantages in terms of the back end gather that you have to do, which is you know, pretty straightforward, but still something to be aware of. There's also kind of a Nuance.

A

That's interesting in that, whenever a pod gets kind of restarted and there's a new pod that comes up with a different IP, because that key is is using the identity and psyllium maintains that identity, um essentially uh you're gonna, have requests that were attributed to that last pod be be attributed to the the new pod. That comes up. So it's it's just kind of an interesting thing.

A

If, if your granularity is on a workload level, that might not be too too big of a deal or like a service level, but if you're wanting to kind of get an accurate gauge of which requests to go to which pod that's going to be a problem, and that can be alleviated by the use of something like a like a ring buff where you're just sending events uh as they happen to some sort of collector I kind of like how pixie does it um so errors is going to be very similar.

A

This is now operating on the response code. Again, it's very straightforward because you have the user data and at the very beginning of the user data, if it's a HTTP response, you're going to have this, you know standard HTTP response line, you can just check.

A

You can essentially check for the HTTP string and then check for the the number so the 2 and 200 and then and then bucket it that way, and you can see a little bit of the code on the right, how simple it is to kind of check for that string and then some code for for submitting it to a map. So both of those are are really straightforward to support what What's, um oh and then you can see kind of the something similar not exactly the same because we're operating on Ingress.

A

Now it's now the source identity rather than the servers, but but it's still talking about like kind of the remote identity and then that destination address is the client. So latency is where a lot of the complexity actually comes in and supporting a solution like this of being able to do that calculation in ebpf itself.

A

um So the complexity comes from on that received side, getting the content, lengths, header, which is anywhere in that response, header itself and then tracking the response body, as as these packets start flowing in. So essentially, let's see, there's something that we're going to have something like this, where we have a state, that's maintained throughout the the connection, as the requests are going in, starting with with essentially this.

A

This start time, that is, that is indicated by the egress program and then, when we receive that first response, we're going to look for that content length, um the string inside that header and then, as the receive content, is flowing in we're going to essentially match it until we equal the content length and then we're going to submit an event to to user space.

A

So the reason this is so complex is because there is restrictions placed by the verifier to ensure that you know the BPF program is, is safe to run and in order to do that, it needs to evaluate instructions, and these instructions grow um quite quickly when you're operating when you're trying to do something like this as I'll kind of get into, but essentially the complexity of trying to get that content length and that that end of the header, so that we know when the receive content starts say that the response, header and the response body share a packet.

A

You need to know that boundary, so you can accurately track that that received content. That's coming in it adds some complexity and then I was kind of thinking as I was creating the solution. I didn't you know, want to have any artificial restrictions. One benefit of passing to user space. Is that you don't really have any restrictions? You can you know you don't have that verifier complexity limit. So, even though you know this is only like a 200 header, 200 character, header response, header I wanted to be able to support more than that.

A

So I have the code on the right. If in case, anyone wants to try and read it, but that's just in case. You want to try and parse it for later.

A

Essentially, this is a first pass iteration of trying to check for headers that, for the header that was coming in the different lines, essentially look for the content length string and then, when we find that match we're we're just going to essentially look for the the numbers associated with that content link so that we can add it into our state and be able to kind of accurately account for that.

A

um You know pretty complex but but you'll kind of see how that worked out. So, as I was trying to load this into the the you know, load this into the kernel. um I found that at 15 loop, iterations I had 131 000 instructions that were you know uh at the verifier, the states, essentially that the verifier verified and then at 17 iterations.

A

It was a 570 000, so a Forex increased and then a 19 I hit that million instruction limit um and, as you guys are probably aware, uh at a million instructions, the the verifier just kind of gives up and says that it's not going to continue working. So this clearly isn't um a solution that that works. There's no header, that's going to be. You know useful that is 19 characters.

A

Long, um so I had to kind of figure out in depth of what was going on and try and get some efficiencies out of this code to be able to parse longer, headers um and and the tools that I worked with were that BPF Prague load, essentially the devote the the verbose um verifier log was really helpful, as well as getting a debuggable kernel to be able to check um the various States.

A

The verifier was in uh in order uh at various execution points um and then, as you'll see, uh making use of BPF tail calls.

A

So the reason that this um verifier kind of balloon to complexity was because, as you guys know, if statements or branches and the verifier, so the verifier needs to evaluate both of these branches whenever they occur, to make sure both are safe to run and that adds to complexity, especially given the context of a loop. So these branches are going to be exacerbated every time you Loop, and it's just going to balloon in terms of the number of instructions they need to get evaluated.

A

State pruning does help in this scenario, where essentially it'll prune a state if at a certain instruction, it's seen the registers before the register values before, but again you saw how much the instruction the number of instructions balloon. So it's not going to save us.

A

So this is a reprised architecture that I came up with in order to be able to support this in in BPF, essentially starting from the top left. If an egress program creates our start time and and maintains some sort of header State, because we're going to have to maintain State across these different blocks, we're going to have a base program that essentially parses the status, but then calls these other blocks and a Saving Grace in all of this was the the idea of using BPF tail calls to essentially reset the verifier state.

A

For those of you that don't know tail calls when you, when you make it essentially resets, it's essentially essentially an independent program.

A

So that's going to reset the state that the verifier is in and and as you go kind of go through this, you have a header manager that calls, and the idea is to kind of take away as much complexity as possible away from these parsers, because that's where most of the kind of ballooning happens, you have tail calls then to the header, parser and then the content length parser um in order to get that content, length, string or content length number into our state.

A

So the header parser is going to find the end of the header, which is going to create a Bound for uh for where to look for that content, link string and then the content link header is going to be found inside of that.

A

So the BPF tail calls helped a lot, um but another efficiency was just kind of the using the the vervose verifier output uh at. So this is kind of a reprised bit of code that had less branches, but you could still see at 30 iterations.

A

So we've made some improvements that we had 514 000 instructions, um and so it just kind of took some time to iterate through and look at why this was happening so I bolded register three and, as you can see, register three correlates to this match variable and it was saying that this match could be up to here.

A

It was 67, even though the purpose of this match variable is to make sure or is to check that the essentially the the length of the match, string and we're looking for the content link string is if it's equal to 14. We know that we've we've found our found. Our match, so the fact that it's 67 means that the verifier isn't aware that this is, you know, is in a state that we can actually get to. So.

A

This is a good example of us needing to add explicit boundaries in order to prevent the verifier from kind of ballooning that so after I added a boundary here that that verifier complexity went down to 18 000 instructions, so in 97 percent decrease in in terms of the size.

A

So using this reprised architecture and and finding some of those those that that increased complexity um or and and finding some of the finding some of those unbounded variables. um I was able to get a Content length, parser of 600 characters per program and then a header parser of 1024 characters um that was just limited by memsat. As soon as you go above that that built-in men's set wasn't wasn't working, I'm sure you could get it more.

A

um But my thinking is that, because these are now independent programs, um tail call you can actually tail call itself. So this 600 character can just keep iterating as long as, if you have a massive header and be able to actually service any kind of use case, you could see if you had like a jumbo frame. It would take like 15 tail calls. I have a slide after this that I that I didn't actually include just because of time. That goes into kind of the latency implications.

A

The the you know, performance of these tail calls and- and it's been improving throughout the kernels, but that that is a consideration. But with this architecture you should be able to support it, um and- and this is kind of just a a review of what we're kind of doing here so I've blocked the content length.

A

We we get this content links header as the packets are going in and then, as the received content is flowing in, we just match it against that content length, that's the responsibility of that receive base prog and then, as soon as it's it's matched, we just send send an event to user space.

A

So then, now we have um exported metrics. So if you guys want to check it out, um bumblebee is a really cool tool for being able to work with with these kinds of programs and be able to actually visualize these metrics and Prometheus. It just takes data that is existing in maps um and and essentially export them, so they're, they're viewable. You can see the number of requests the response codes, as well as the the latency that gets exported right now.

A

The keys are the client Adder that 10.0.0.97, which you can see at the in the middle boxes, correlates to this L7 metrics pod, which I was just running a curl request from and then that server identity which which psyllium maintains a state of so you can imagine kind of translating these Keys into whatever granularity you want um and and kind of using psyllium's kind of built-in mechanism, cylinder's built-in state to be able to kind of translate it to whatever you you'd like to do so. There are some limitations to this approach.

A

It doesn't right now support HTTP pipelining, it's it's! You know it. It uses the fact that there's one response per request, which I'm sure could be improved, it's just not something that this architecture supports, it's not supporting transferring coding, chunked um and, and it doesn't work with with TOS, since the data is all encrypted. Pixie has a really cool mechanism and the previous talk was talking about. You know you probes and being able to hook in there in order to support encrypted data.

A

That's that's really cool and pixies doing some really cool work there in order to support that.

A

But uh in conclusion um there are quite a few considerations whenever you're trying to kind of move responsibility away from user space into BPF, and it's very difficult to do, especially with the complexity limits that the verifier places but um being able to check out that verbose verifier as well as utilize, BPF tail calls.

A

We are able to move Solutions into the kernel and be able to essentially like service more complex use cases which is exciting. If we're wanting to kind of do things more, you know dynamically, um and but there are you know, there's there's both pros and cons to this you're not having to iterate the entire packet in order to support this, um but you are adding kind of latency to the data path by by doing this, um which is you know, non-zero, and something definitely to consider.

A

But the result is: is golden metrics done purely in in ebpf and then exported to user space? Just for um you know uh just for Prometheus, essentially integrating with Prometheus tooling, but that is it thanks. So much everyone um yeah appreciate it.