Cloud Native Computing Foundation Cloud Native eBPF Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Exposing the Revolution: GRPC Observability with eBPF on K8s - Ori Shussman, Groundcover

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Exposing the Revolution: GRPC Observability with eBPF on K8s - Ori Shussman, Groundcover

eBPF is becoming the go-to technology for K8s observability, pushed forward by projects like Pixie and Cilium, and expected to shed light on a lot of data left unseen today. gRPC, the most widely-used protocol to transfer data between containers, is difficult to monitor, when compared to more modern methods using eBPF. Join a journey through an eBPF solution that monitors the gRPC library used in python, C++, C#, PHP, & more. Gain understanding of why gRPC and other compressed or encrypted protocols are harder to monitor, and see how eBPF’s strengths can be utilized to trace these hard-to-reach cases. Talk is cheap - that’s why you’ll get the tools to try it out yourself — Experience a demo of this research and technology, which are now part of the Pixie CNCF project that recently started supporting these advanced monitoring methods. These methods will become important stepping stones on the way to reaching the ultimate goal of the monitoring world: full, yet effortless observability.

A

Hi, um thanks to amid it was a great talk and I'm here to talk about observing the grpc protocol, but first I want us to take a look at the bigger picture, because this is about much more than grpc observability into our clusters will undergo an enormous revolution in the coming years, thanks to ebpf when I, say observability I'm talking about our ability, as developers and devops, to see everything that's going on in our cluster.

A

This includes all the interactions between our services, all the metadata that accompanies them. Various events such as program exceptions on protocol errors and determining exactly when all of this occurs and to which processes it belongs. This is what I call full observability.

A

Today, your cluster is most likely in the dark when a bug occurs, you don't really know what happens so you have to try and reproduce it in your development environment and that's only if you care enough, but if we are able to reach full observability, this will totally change I'm. Already a founding engineer at ground cover I'm very happy to be to be here today.

A

um I've been implementing observability solutions for many different use cases for quite some time and I I fell in love with the magic that is also known as ebpf.

A

What I want to share with you today is the most important Insight that I found throughout my experiments that with evpf we can reach full observability.

A

The best way to understand that is by examples- and this is where grpc comes into the picture.

A

Grpc- is quickly becoming the preferred tool to connect microservices replacing a lot of the uses of HTTP in the modern era, and it is also quite hard to observe- and this is exactly why it makes the best example for our Revolution.

A

I will not dive into the technicalities of grpc here, but there is some basic stuff that we should know.

A

When the client wants to request a resource from the server using grpc, they say they get like request with the resource name as a header, the path header, the headers, are then cached to minimize the total amount of data sent over the connection, because headers usually repeat themselves.

A

This basically means that if the same resource is requested later on, it is basically given an ID at the first request and the ID is used instead of the resource name.

A

In our example, the id7 is requested instead of the resource name only.

A

We also need to know that grpc supports multiple streams.

A

This means that the client May request more than one resource at the same time, and if it does so in request, for example, two resources almost at the same time, the server may even respond to the first request. Only after responding to the second one.

A

Sorry grpc, like every other protocol we know of, is implemented in different libraries. The main library that I will discuss today is the grpcc library. It is used by a lot of languages, including python.

A

So today, here we are gonna Implement observability for the grpcc library, but first, let's consider what do we really want to observe?

A

Let's take a simple example: the most simple one I could think of of two containers: a client and a server communicating over grpc the client periodically every 10 seconds sends a request to the server requesting for resource only it's quite a common use case of a client that needs to know when the resource is updated.

A

And in that communication, what do we really want to observe? Well, we have to see the grpc data. This is the most basic part of the protocol. In our case, it's thus it's just the resource being downloaded. We also need to see the headers, especially the path header, that we talked about the resource name.

A

We also need to know when the stream has been closed, that will indicate to us that the resource has been downloaded or that there was some other error and, lastly, for our simple example: we don't really need to know the stream ID, but if the client were to ask for more than one resource at the same time, we would need a stream ID to differentiate the responses.

A

Grpc connections can live for a very long time. These requests every 10 seconds could go on forever, and this means that we are most likely going to start observing the traffic when the resource name alongside other headers is not transmitted anymore.

A

So the first solution that evpf brings to the table is using K probes, kernel probes. They can be attached to Kernel functions, including system calls and there we have access to the arguments to these functions.

A

So, for example, we could attach probes to the send and receive system calls and have access to their arguments, and this will basically provide to us everything going in or out of the container, and this is really powerful. It is used this way. Evpf is used to monitor a lot of different protocols like http, and it also looks kind of promising right everything we talked about. All the four items have to be there right coming in or out of the container.

A

Well, the problem is that K probes, just don't cut it. In our case, the header compression mechanism we talked about that replaces the resource name with an ID makes the grpc protocol stateful.

A

If we used K probes from the beginning of the connection, we would observe the first the first request, and we would see that 7 represents the resource only, but we might just start observing when the connection is already alive, and we also want to do that without harming the connection in any way with zero downtime.

A

And if we do that, we are going to miss the resource name. So what can we do? Well? Evpf luckily, also introduces your probes user mode probes and they can be attached to user mode functions, including.

A

um Sorry, including Library functions using your probes, we can just like, with K probes, see the arguments to the functions that we probe so probing inside the grpcc Library can provide to us information from the library from the memory of the library in the middle of the connection too, the library, the grpcc library, obviously knows that the number 7 represents the resource only otherwise it just wouldn't have worked. If the server didn't know what 7 was it wouldn't know to respond to a request for resource 7..

A

So if there was a grpc receive function that receives all the information we seek as arguments, we would probe it and just see all the incoming information into the container. We would just need another another probe for the complementary grpc send function, and we would be done. We would have full observability when you think about it. This kind of solution works for all types of stateful connections, including even encrypted traffic.

A

The encryption libraries, for instance, leave us a cell, obviously know what the unencrypted data is. After all, they are used to translate encrypted to unencrypted data, so using u-probes in the correct location inside these libraries will provide full observability into encrypted traffic.

A

We are here: I I've got a feeling that we are going to hear more about that today later in a talk by Dom, hi I'm. Definitely looking forward to that one. But for now let's focus on our grpc use case.

A

A similar concept that uh Omid talked about briefly is usdt user levels. Statically defined tracing usdt functions are basically functions like the grpc receive function we just saw. They are functions that are designed to make monitoring the library easier. They receive all the information you need to monitor as arguments monitoring. A library that has usdt functions is the most simple and reliable. You probing solution you could have.

A

Our problem is that the grpcc library does not have usdt support.

A

We can add usdt support. After all, this library is an open source and a Community Driven Library.

A

Hopefully the library developers will like the idea and allow us to merge this code into their pull request, and hopefully they will also deem it important enough to be turned on by default at newer versions, and that is great because when the new version that has usdt functions is rolled out and everybody updates the library version they are using, which will take I'm guessing about three years, then we can have full observability to the library's traffic using usdtu probes.

A

But what can we do right now.

A

Well, there ought to be send and receive functions inside the Library and the headers, and the stream ID must also be somewhere in the library's memory too. So the solution to our problem may not be as simple as using two neat usdt probes on Library functions, but the concept is similar.

A

We are looking for functions that receive everything we seek as arguments.

A

Grpcc is an open source project and that makes our task way more feasible. We can just look directly at the library's code to search for these functions.

A

So, let's see what you probe, hacking looks like.

A

The first thing we're going to search for is the grpc data. It is the most basic part of the protocol, so let's look for grpc receive and grpc send functions, and hopefully we will find functions that not only have access to the data, but also to the stream ID and to the headers, providing us with everything we need in just two probes. This is our Dream right now. This is what we are hoping we can find.

A

But how can we even start to search for these functions in the hundreds of thousands of lines of code inside the grpcc, Library and trust me I've counted? Well, we can search for them in three manners. The first one is a bottom up, search where we start by looking at the kernel system, calls send and receive. We've established that the data we search for goes through there, and then we can ask ourselves who uses these functions.

A

The data must be there too, and we ask the same question again and again working our way up through the flow of the data receiving and data sending uh sequences, and hopefully we will find our perfect functions.

A

The second way is a top-down search where we start by looking at the API of the library. These are the functions that the user process calls in order to communicate using the grpc protocol using the library, and then we start working our way downwards use again working through the data sending and data receiving flows. And again, hopefully, we will stumble upon our perfect functions now. The Third Way is a middle outsearch, where we start right at the middle between the API functions and the kernel system calls.

A

We can do that by, for example, searching for a string say, send and if the functions in the library are named properly, we will land somewhere in the middle of the flow, and then we can start expanding outwards in both directions.

A

So, starting with a bottom-up search, I found the flush data function, it is used to send data and it also has access to a stream struct that has stored within it, the stream ID and the headers, and that is amazing, because all the data we were searching for is just in one place.

A

But well it wouldn't be that simple I wouldn't be here. If it were that simple this function is compiled in line. This basically means that the compiler, for whatever performance reasons it knows better than I, do decided that it's as if this function doesn't even exist. It's just as if the code that's written in this function was instead written directly in inside the function. That calls it and that's bad news for us, because we can't probe a function that doesn't exist.

A

So, let's look at who uses this function, which is by the way bottom-up searching.

A

And that is the begin write function. What it does is look through all the different active grpc streams and sending the data for each of them. That is ready to be sent using a set of three functions.

A

All of the three are compiled in line, and that makes the begin write function, be quite a a huge bloated function.

A

Theoretically, we do know that the flush data logic is embedded directly there in the middle of begin right, so we could probe it in the middle. It is a power that ebpf allows us, however, doing that is way harder and even more hard to maintain. So, let's look for something else.

A

Instead, we found a pop writable stream function, which is used by beginright to iterate through the different active grpc streams.

A

Probing at the end of this function will retrieve to us as the return value the stream struct the same one that is later on passed to flash data.

A

Distract has stored within it, the stream ID and the headers, and from that specific context, we can also use it to access the data and that, just as we dreamed is everything we searched for in just one probe.

A

If you thought this was hard enough to see incoming data, we needed three probes, one for the data and two for the headers.

A

So summing, up with five probes on five Library functions, one for the outgoing data, three for incoming data and a last one to see when the stream is closed. Everything works and that is really exciting to me. So I want to show you this solution put to action.

A

In this demonstration in the bottom right, you can see the ebpf code in the top right is the grpc server that is now starting, and the first thing that's going to happen is the ebpf code. Attaching five you probes.

A

The client at the left will then start sending periodical requests to the server and also printing, all the information from the connection and, as you can see, with zero code changes to the server or to the client. The ebpf code magically has access to all the grpc information from the connection.

A

You Pro packing was also used in the Pixi open source project to monitor the golang grpc library, foreign, with collaboration. Pixie, now supports tracing both libraries, grpcc and grpc go and all the languages that use them. You can see the code to both Solutions in GitHub.

A

There are still some languages left unsupported. You can see three examples in the diagram.

A

A

This was rough.

A

We can repeat this process for every Library. We wish to observe.

A

But why bother, as I said before, using usdt for short-term solution for observability purposes is futile. It would take three years in the best case until we can reap the fruits but long term wise as one of ebpf's forefathers Brendan said, usdt is the solution.

A

The main reason is that you Pro packing has its problems. They all stem from the fact that we use the magic of ebpf to access information in the library that is not made visible, To Us by the library Developers.

A

It's not such a big of a deal right, I mean we saw this process it's hard, but it's not impossible. We can do it just once per Library. We want to observe well. The real issue is that this kind of solution may be hard to maintain.

A

So let's explore three reasons that this specific solution may require more effort in the future. All of the three would not exist if we used usdt probes.

A

The first is that Library developers refrain from changing the API of the definition of API functions in the library. This is important because they wouldn't want that when someone updates the library version, their code just stops working to encounter this. Whenever you you probe hack, and if you do that be cautious, probe, API functions, they usually have all the data that you need.

A

In our case, it wasn't a possibility, because, when peep installs the library for python usage, it does something quite different.

A

So what we did is we looked back. We looked two years back and the functions that you saw that we chose are functions that don't change. They didn't change in that time period, but this is just a way to minimize the risk because they can change in the future and if they do, our probing solution will need to be.

A

Accord will need to be changed accordingly.

A

The second is that the stream ID and the headers, and sometimes even the data that we found, is all in one huge stream struct that has thousands of bytes and it changes almost every Library version.

A

If, in a new version, a new field is added to destruct, even if it has nothing to do with the information that we seek. It might, for example, shift the stream ID forward by four bytes, and this means that for our probes to work, they need to be aware of the library version. They are probing.

A

In our case, we had no easy way to find the library version, so what we had to do is download a bunch of Library versions and hash them all. And now, when we encounter a new library, we can hash it. Therefore, knowing its version.

A

The third is that new versions of the grpcc library, this specific Library, are sometimes compiled without symbols.

A

This means that we don't know the addresses to the functions that we are probing and if we don't know the addresses we can't probe them. The functions we chose are functions that have unique strings inside them. Like the one you see on the screen, these strings can be used to find the address very easily using reverse engineering, but it still needs to be done for every Library version.

A

The work you heard of here you Pro packing, is for observability right now. Your pro packing is a necessity and a necessity that I enjoy very much to be honest, but ebpf can be made easier. It shouldn't be that hard, modern libraries should add usdt support to make this easier, because this is the future we already added usdt support to the grpcc, open source library and, if I've convinced you that this is a worthy cause, please upvote our pull request.

A

Meanwhile, we pave the way using your probe hacking for observability to grow, giving it everything we've got so that in a few years we will be able to leverage the true power of ebpf, full and effortless on demand. Observability.

A

So you experience the power of ebpf in changing the world of observability. Grpc is a very interesting and complete and complicated use case. This work depicts only the first steps of the revolution.

A

I invite you to stay tuned in the coming years and see how it all turns out or even join me and be a part of these two. You could, for example, add usdt support to a library that you use or even maintain, if you have one of these, if you do, even if you don't please message me, I would love to be of help.

A

So thank you very much for having me today, I hope to see you the rest of the week and I'd love to hear any questions. Thanks. Sorry,.