Cloud Native Computing Foundation Cloud Native eBPF Day NA 2022, 2 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tracing SSL/TLS Encrypted Microservices with eBPF - Dom Del Nano , Twitter

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tracing SSL/TLS Encrypted Microservices with eBPF - Dom Del Nano , Twitter

SSL/TLS adoption in the Cloud Native environments is growing rapidly. While great for security, the encryption in such environments pose a unique challenge for observability tools. Many traffic sniffing tools can only collect the encrypted data, which is of limited value to the application developer. Important attributes like the operation, the endpoint and the payload are undecipherable. To truly help in the troubleshooting process, application developers need to be able to see these messages and their contents. In this talk, we present how eBPF can be used to tracing SSL/TLS connections. The method we present is used by tools like BCC’s sslsniff and Pixie’s protocol tracer. Specifically, we cover how eBPF uprobes can be attached to popular SSL/TLS libraries, including OpenSSL, BoringSSL and goTLS. We show how eBPF enables us to collect clear text data directly from the TLS library, while discussing the challenges of tracing dynamically vs statically linked TLS libraries. Finally, we also present how this feature could help with improving application observability at some of the largest engineering organizations without disrupting their production environment.

A

Hi everyone thank you for joining me today and I'm excited to talk about TLS tracing encrypted microservices with evpf, so before we get started. I wanted to give a brief background about myself.

A

My name's Dom I'm, an SRE at Twitter working in observability across metrics and logging prior to Twitter I was a production engineer at Yelp, where our team was responsible for keeping the website and mobile apps working and that's where I got a deep appreciation for observability tooling.

A

Let's go over our agenda for today, we'll start off with a background and motivation for why Twitter was interested in this work from there we'll do an overview of protocol tracing techniques, then we'll do a deep dive into the Twitter TLS tracing work. We implemented we'll see a live demo that and then we will generalize the criteria for how to support new TLS use cases.

A

Moving on to the background, so at Twitter scale we rely heavily on right time, aggregated metrics.

A

This is great for monitoring service health, but it really limits your ability to slice and dice your data, and so when you're investigating outliers or doing other real-time debugging, it becomes a real challenge.

A

Another thing is that our collection and ingestion for our observability data has largely remained the same. We saw that ebpf had promise with auto instrumentation and we saw the Hubble and pixie projects doing exciting things, and so we wanted to explore this further and see if the protocol tracing could help aid. This real-time debugging use case.

A

Another important detail is that services at Twitter use mtls by default. This is fantastic for security, but renders many observability tools useless. So whatever ebpf instrumentation we would use supporting encrypted connections is a necessity, otherwise it wouldn't provide any insight.

A

And, lastly, the tracing needs to be extendable for the protocols used at Twitter.

A

The majority of our microservices are built on top of this project that we open source called finagle, which is an RPC framework for communicating between services and while it supports protocols like MySQL, postgres, Thrift mux is the most widely used protocol, which that's Apache Thrift over a mux session layer protocol.

A

With that background, out of the way, let's dive into how protocol tracing can be done, we'll first discuss how it works for the plain text use case and then dig into encrypted use cases after that, so for tracing plain text traffic, we can attach a BPF program to the read and write sys calls through a k-probe.

A

This curl command that you see here is dumping out. The networking sys calls when we are curling google.com and the arguments that we see to the syscall would be equivalent to the data that we would be able to access through the k-probe.

A

So you can see in the send to syscall five is the file descriptor, and the second argument is the raw protocol data. So from here what a tool like pixiq does is it sets these probes on those syscalls? It pulls out the socket file descriptor and the raw protocol data. It sends it to the pem, which is their user space Daemon for doing more processing and from there it stitches together the requests and response Pairs and adds more metadata so that you get rich information about your traffic.

A

Let's move on to the encrypted case, so we are unable to hook into things at the same layer for encrypted traffic Because by the time it gets the syscall it's already encrypted, so we need to access it just before it's decrypted or after it has been decrypted and where this occurs is in the TLs Library.

A

In this example, here we can see the SSL right and SSL. Read functions exist within the open, SSL Library. So this is a common point of instrumentation.

A

It's worth mentioning that, while openssl is popular, the there are many other TLS libraries and so, for instance, golang has its own crypto implementation. So, while it it has a similar layer, these SSL write and SL read functions. Aren't necessarily it don't necessarily exist. In that case, it's worth mentioning that the popular BCC tools project has a tool that implements tracing in this way called SSL sniff.

A

So it sets you probes on these two functions and it's able to dump out the unencrypted traffic.

A

Pixie also takes a similar approach, and so it puts the probes on those two functions and from that point forward it sends the data to the user space process and the request and response request and response pairing occurs, just as it did in the plain text case.

A

However, there's some additional challenges here in this diagram: you see that there are three requests going on service. B is making a HTTP call to service a while service. C is making two HTTP calls to service a.

A

We need to be able to understand what data is associated with which connection, and so, while we can access the protocol data from those functions. That's just part of the story.

A

We need to have some type of connection identity, which, as I mentioned in the plain text use case, is done through the socket file descriptor, unfortunately, with the SSL right and SSL read functions, the socket file descriptor isn't passed in as its own argument, and so we need to work out how to determine that.

A

So you can see the Prototype of the SSL rate function. The second argument is the plain Text data that we can use, just as in the plain text case.

A

The first argument of interest is the SSL struct, and so what you will see from the other struck definitions here is that the SSL struct has been embedded, rbio member, which is of type bio, which is open, ssl's basic input, output, abstraction within that struct there is a num member, and that is where the socket file descriptor is actually stored.

A

So, in order to get this connection, identity working, we need our BPF program to walk this structure and pull out that num field. So, let's see how that works.

A

On the right hand, side you'll see a diagram of how this exists in memory. The function will be given that SSL pointer, which will go to the beginning of the SSL struct from there. We need to determine two offsets. We need the offset between the beginning of the SSL struct and the rbio member.

A

We can then follow that pointer, and then we need the additional offset of the bio struct to the num member, which contains the socket file descriptor.

A

One thing to mention here- and this was discussed earlier in the grpc talk- is that these offsets aren't guaranteed to be the same across open, SSL versions, and so what we assume is that for a given open, SSL version, those offsets will be stable.

A

The way pixie's user space process determines this is through DL, opening the openssl library and calling the open SSL Version num function before it attaches the u-probe, so that those so that the offset calculation can be done when the function is triggered.

A

Now that we've understood the background and some tactics for doing this protocol tracing, let's move on to Twitter's use case, so we built out our TLS tracing on top of the pixie project. The reason why we were interested in Pixi is because it provides granular observability data and it's at the edge model scales for real-time debugging.

A

I mentioned I work on the observability team, and many of the tools that we provide are for service owners to understand their systems, and the data that pixie provides is very applicable to those customers and what they really want to do is understand service to service interactions, which is data that protocol tracing provides.

A

And finally, Twitter's core instrumentation focuses on internal use cases. What this means is that open source integration typically lags behind our internally developed projects.

A

So while we were hoping that pixie would give us better instrumentation for our finagle and our you know most widely used projects, we were hoping it would raise the bar for the open source use cases that we didn't have enough time to fully build out Integrations with all right.

A

Now that we've seen the that background, here's the familiar TLS tracing diagram, so finagle uses a project called Neti, which is a Java networking Library and that Library, underneath the hood uses a project called Neti, TC native, which provides its own shared Library that fulfills the same role as the open, SSL shared Library.

A

So in the context of this, we needed to trace libnetti TC native and it's unlike openssl. Typically, it's statically linked with boring SSL.

A

Another note about Neti is that it's pretty widely used across many Java projects. So when we were embarking on this, we were hoping that if we could solve this TLS tracing problem, this wouldn't only solve it for Twitter's use case, but it would implement it for Cassandra for elasticsearch Java grpc. There's a large variety of use cases that all use this library and so I felt that there was a lot of potential here.

A

Let's talk about the challenges that we had to overcome in order to get this tracing working, so the first is that pixie's original openssl version detection did not work for finagle libnetti, TC native.

A

It does contain the open, SSL Version num symbol, but it's local, which means it exists in the Sim tab section of the elf file, rather than the dynamic symbol table and so Pixies call to DL Sim would fail to find the function.

A

We can see that here in these two commands, so the first one shows all of the dynamic symbols and greps for that symbol, and we see it's not present, whereas the second command shows all of the local and dynamic symbols, and we see that it does show up.

A

So this was one area we needed to have a solution for.

A

The second is that the socket file descriptor was not populated in the SSL right and SSL read SSL struct, so that was another area we knew we were going to have to solve this connection, identity problem and finally boring SSL was used rather than openssl.

A

The most obvious difference here is that one of them is statically linked and the other is typically dynamically linked and we weren't really sure exactly what needed to be done here, but we were pretty confident that there would be something to work out starting from the first one. Let's talk about how we determined the finagle TLS version, so as I mentioned, that openssl Version num function is in the elf file, so one way to solve this is that we can actually calculate its raw address and call the function with a raw function pointer.

A

The code you see on this slide is an Abridged version of the code that exists within Pixi, that does this called The Raw function, pointer manager.

A

So you can see the kind of high level ideas is that we have an elf parser. We also DL open the library, so it's accessible within the user space daemon's virtual memory.

A

We then use the elf parser to determine the symbol offset and we add that to where the library is loaded in virtual memory, and that gives us the function pointer that we need so from there. We were able to determine the TLs version and that problem was out of the way.

A

Next up is addressing the missing socket file descriptor, and we solved this by Upstream upstreaming changes to Neti and Neti TC native to populate this on connection creation.

A

This pull request here is the one to Neti, TC native, which is the portion of Neti that does the actual TLS and native code.

A

If you look at the top part of this pull request, you see that it's handling a bioc set FD case and what it's doing there is it's making sure that the num field on that bio struct is correctly populated.

A

The part of the pull request to the bottom is the change needed to export this functionality to be accessible by the Java runtime. So what it's doing is it's defining a new method called bio set FD, so that Java can call this in order to take advantage of this functionality.

A

The second piece to this was making. This was actually using this in the Neti project.

A

Nettie has a concept called a channel which is a one-to-one mapping between a connection, and it also provides some hooks into connection life cycles, and so we there is a thing called Channel active, which is called when a connection is created, and so all we needed to do was call the function there and pass the channel in so that it was passed into the native code problem. Two was then solved and then the final piece.

A

The differences between boring, SSL and openssl I mentioned that the boring SSL is statically linked, and so what this means is you actually have to set u-probes on the application binary rather than the shared Library.

A

So if we take an example like Envoy, you would actually put the BPF Probe on the envoy binary itself rather than binary or envoys TLS shared Library finagle's case is a little different, because Java is not native code, it has a runtime, so it is still put on the shared Library, but the shared Library doesn't have Dynamic. Linking second thing is boring.

A

Ssl is Source compatible with openssl, but that doesn't guarantee that the offsets that we use to find the socket file descriptor are the same, and what we found out after investigating this is that they are actually different and that boring SSL is written in C plus, and so the layout in memory is very different from the opensslc implementation, and so the way we solve that is. We have different version, offsets for boring, SSL and different ones for openssl for each version. That's supported.

A

All right with that I realized I called this a live demo before, but it is pre-recorded. Let's see it in action.

B

I went ahead and installed pixie on a local Mini Cube cluster.

B

Running in the pixie docks there is this Pixi CLI you can use to deploy to the cluster, so I went ahead and done a px deploy in order to visualize.

A

B

Enable traffic setting up two applications- one of them, will be generating.

A

Planes verify that the network traffic.

B

Has plain Text data then set up an encrypted example and use T utus there as well to verify that the traffic is truly encrypted, and then we will visualize.

A

That in the pixel UI.

B

So pixie comes the demos out of the box.

A

One of them being the.

B

A

Demo, which generates thriftbox traffic at this time, the finagle demo does not work with generating encrypted traffic, so we will use that for the plaintext.

B

B

B

B

We set up the encrypted case. I've ran some containers on the cluster manually since, as I mentioned, the demo does not support.

C

B

Yet, let's dig into things so in this terminal window, on the left hand, side I have the TCP dump output from the plain text case and the.

A

Logs from the enabled we can see that the client is constantly generating this hello, ding traffic, which is viewable in the TCP dump output, because it's plain text on.

B

The right hand side we have the encrypted depth.

A

It is constantly receiving a string string response from the server and if we look at its TCP dump output, we can see that the string string text does not show up validates that this case is using end-to-end encrypted traffic.

B

With that, let's take a look at the Pixi UI and see how things look so the Pixi, with a number of scripts out of the box which allow you to visualize your clusters, data.

A

There is a upstreamed mux data script that supports viewing this mux protocol traffic I've went ahead and made some slight changes so that we can encrypted case versus the plain text case, but otherwise it's the same as the the script.

B

So what we can see here is the mux protocol is a session layer protocol. So, in addition to sending requests and responses with the tissue has a.

B

Looking at these individual entries, we can see there are two different sources.

A

One of them is PX finagle, which is the name of the the full name of the Pod that the demo is running.

A

C is able to understand who metadata and so that this localhost case is actually our encrypted demo. So we can see that it doesn't have that rich data to attach to it since I'm running this cluster itself. But we can, we can see that both of them are generating traffic constantly. So we can see that the plain.

B

Text and encrypted cases cases are both working out of.

A

Them all right, hopefully, you get a good view of how all that works end to end.

A

Let me get back to my presentation.

B

A

All right, I knew this was going to give me trouble.

A

Foreign there we go all right, so in conclusion, uh I wanted to give an overview of the different dimensions of things you need to be aware of in order to support a TLS Library. So, to begin with, we'll be just describing the differences between nginx and node.js, and so for the SSL Library. They both use openssl.

A

They are dynamically linked. The SSL right and SSL read: symbols are accessible and in the dynsim, which means that we can set a u-probe with the symbol for the version detection so that we know which offsets to apply. We can use the we can use DL Sim to call that openssl Version num function in nginx case the socket file descriptor is populated, which results in pixie supporting that use case for node.js. The socket file descriptor is not populated and pixie has additional u-probes to capture that data, which results in node.js being supported.

A

Moving on to Neti so for finagle we know the library is boring. Ssl the linking is static. We are the SSL right and so read. Symbols were able to attach it with the symbol itself. In for the version detection, we are using the raw function, pointers and the socket file. Descriptor is populated if you're using a TC native version greater than or equal to 2.0.5.3.

A

On the other use cases of Neti, all of the other details are the same. We believe that it should be supported, but it's yet to be tested and, as I mentioned before, this is something that I think will be really powerful to for this. One use case to support many popular projects and finally, thinking about Envoy Envoy is another use case that uses boring SSL and one of the ones that the pixie team was actively working on its linking is obviously static.

A

The SSL write and SSL read: symbols aren't present, but we can do that raw address calculation, so you have to set your uprobe with the raw address.

A

The version detection since the last time I checked in on this I believe it was to be determined and Envoy actually has apis that hide the socket file descriptor. So that's not used, and so, as indicated here, it's a work in progress and and just to note, this is for istio istio also uses Envoy internally.

A

Looking at vanilla Envoy, though the SSL right and SSL read, symbols are actually stripped from the binary entirely, which means that we have no way of attaching the u-probe, which ultimately mean means that, unfortunately, Envoy can't be supported through this type of tracing.

A

So, in recap, to talk about the current status and the future work finagle TLS tracing is one minor bug fix from being functional. The rest of the functionality that we saw is already in pixie's, latest release and upstreamed.

A

The building blocks introduced to support, boring SSL will be shared by Future Integrations. So I mentioned that istio use case. It's piggybacking off of those changes and then looking to the future, I mentioned that Neti TLS tracing has the potential to support many other projects. This could include Java, grpc, elasticsearch, Cassandra and I think it would be extremely powerful to have fully end-to-end tracing all the way through to your data store and so I'm excited to eventually pursue what that would look like.

A

That's all I have so. Thank you all for your time and I'm happy to take any questions. You have.

D

Well, we've got a question over here.

E

Maybe, while we're asking questions, Mauricio could uh okay.

F

E

Sorry presentation ready.

G

Dueling mics, so through any of the work with any of the library or code maintainers, was there ever any discussion about them, modifying uh structures or emitting data or doing something to make it simpler for ebpf? You know to observe sort of a meta question so.

A

As I dug into this problem, I was once we identified that we thought. Pixie was a good first step into doing this, so that we didn't have to build everything out ourselves. We were working a lot with what already existed there, and so, as I mentioned in kind of the protocol tracing overview.

A

Pixies method is, you know, plucking that socket file descriptor out, so we were kind of just continuing down to see what all would need to be done to unify that, because we wanted the solution to work, ideally in the same manner, to minimize like one-off use cases I think now that we've like explored it fully like there could potentially be other things. You know like thinking about ways to get that data with other methods and.

G

I was just thinking because there's some Twitter specific, not not necessarily specific but lead projects in here, but thank you.

D

Other question: oh we've got another one, fantastic and then one here: okay,.

H

Hey cool work, uh my question is: have you experimented with the kernel mtls? That would also allow you to see uh the encrypted traffic or have you any comments on an adoption of that.

A

I know some of that's possible, but I, don't I haven't dug into it myself. That's definitely something I think. That's worth exploring and understanding more.

E

Hey thanks for the talk. This is a little lower than I, usually know, but I feel, like I learned a lot, um so it seems like you're kind of at the mercy, I guess of like the the server, the the networking framework that you use right to be able to expose the things that you need to do the tracing. So I guess. Is there almost uh like a wider Call to Arms for different Frameworks to open up this access up so that we can have this kind of tracing throughout?

E

A

Exactly what Ori was talking about in his talk earlier today about you know, this is using you probes and fortunately, for this use case. Ssl, write and SSL read are more of a defined contract, because boring SSL provides Source compatible support for it, so it's less unstable in that sense, but I do think that doubling down on like implementing ustts in popular libraries, is the right way to go to get stable interfaces for getting the data out of this.

D

So fine Dory is the answer.

E

F

D

F

Hi for Java based applications with the jit compiler. Sometimes the code that is generated is native code. For example, a you call a function, one time or ends and many times so it generates a native code. So when you do the u-probes or even on the SSL or some other like and Java as its own SSL version. So how do you cope with the digit compiler?

F

A

I mentioned before Oh I, don't have my slides in the um Nettie has kind of a sub project inside of it. That's the Neti TC native part, that's all Native code, so the jit does not have any control over that I wouldn't consider myself a Java, runtime expert but I believe it's it's separate in that sense,.

D

D

What yeah hang on I'm coming.

F

Sorry I'm asking when you have the jit compiler working and it's not out of it. So how do you cope with it or if there is no solution, you have to go higher levels.

A

For other type of instrumentation, yeah, so I believe there is a project to add. You can do usdts in Java code I when I initially worked on this I thought we were going to have to explore that. But then I was relieved that there was native code that I could attach to. So that's something. I would like to explore more, but I barely scratched the surface.

D

They still have more questions. This is great.

C

uh Do you do you use this instrumentation and only for debugging when you have a problem or you can have it all the time running,.

A

We are not using it today, we're currently understanding where our first investment will be. We aren't running a very small number of our use cases or on kubernetes, so because pixie is a kubernetes observability tool, we're understanding how we can run it outside of kubernetes and extract the data from it, um and we are anticipating that we want it on all the time.

C

My question was: do you have any, do you foresee or have any performance impact when running this instrumentation so.

A

There is, there should be a slight performance impact, because the u-probe will cause a contact switch when that function is called from the issues that I've seen at Twitter of us not having good capabilities for slicing and dicing or investigating real-time issues. I think that that's going to be worth it, but it's not been measured yet.