Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: eBPF Powered Distributed Kubernetes Performance Analysis - Lorenzo Fontana, InfluxData

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

eBPF Powered Distributed Kubernetes Performance Analysis - Lorenzo Fontana, InfluxData

Since the Linux kernel 4.x series, a lot of enhancements reached mainline to the eBPF ecosystem giving us the capability to do a lot more than just network stuff. The purpose of this talk is to give an initial understanding on what eBPF programs are and how to hook them to programs running inside Kubernetes clusters in order to answer targeted questions at cluster level but about very specific fine-grained situations happening in our programs and systems, like: - Had that function in my program been called ? - For a given function which arguments have been passed to it? And what it did return? - Which TCP packets are being retransmitted? - What are the queries running slow? - Insights on programming language events/gc - Had that file been opened? Imagine a programmable Kubernetes performance analysis tool that runs at cluster level without performance implications how would you it to be?

To learn more: https://sched.co/GrYw

A

Hello. Everyone thanks for coming to listen to my talk about the vpf. How excited about to you argue about this topic? Oh yeah! Thank you. So the first time I met these may be f. King I will present myself later I like to present myself at the end of those lights, so that you listen to what you like first so the first time, I've seen like this term popping out on the internet like two years ago, I was like what is and my brain was like.

A

Do you really want to learn that it's something that we should go together and I was yeah I want to do that brain. So please, let's Google, something that's understand. What is that, and it turns out that this thing is nothing really new I mean it's just Berkeley packet filters that I didn't know anything about so even better, and but they are extended. So there's something more. So the basic meaning of work with packet filter is that they are some utility that lets you, as the name say, to filter packets and I said well.

A

So that's just for networking and it turns out it's not. So if you were confused about that, it's not just for networking is for particulate, it's basically for everything, but we will get. We will dig that in a bit in that a bit more. So what I like to say is that BPF is actually a tracing framework and tracing framework is basically something that you can use instrument to actually construct information from a system right. It's actually not just that in a sense that you can actually create entire programs with vpf.

A

So you can basically not just distract the informations but also write programs to do actual stuff like, for example, there's a framework built on BPF. That's called xdp I've explained that that may be a bit of that later. That actually allows you'd like to drop packets that are into your network stack even before the actual colonel knows about it. A mind-blowing is that and vpf is actually use to access colonel tracing backhands.

A

What that means is that BPF is not the actual team that extract information from the colonel. It actually leverages other tools to extract to extract the informations other kernel functions. Those are actually static, trace point: it's subtle, trace points are just some actual points in the kernel where you can actually extract information from and they have been coded into the kernel directly, so they are just there right.

A

You can see. How do you are by starting that information from this file here and for each one? You have like a set of information. You can extract from it like, for example, for TCP. You can extract reservations for virtual file system, which is they actually place where everything every file is where it's actual function. That is called every time. I find this written.

A

You can actually start information about like how many bytes is file contained or how many times this one has been caught, and then there are like two dynamic trace functionalities, which are like the most interesting because they are the actual ones that you can use to start information from your programs. Let's say you have like kubernetes right and you want to know how many time the main of kubernetes or some functional kubernetes is called and which, with what with what arguments or what it returned, you can actually use this thing in the dynamic trace finalities.

A

We can actually distinguish three different sources. There's the you probes, the user pros. That's the actual probes that you can write on your program, so you have a binary. You can actually sort information from it like if it was like running gdb right and keep Rob's Colonel probes. We're like you, just go to the actual kernel functions.

A

It's like static, trace points, but set current points have been coded there, while key probes are actually functions, implemented in the kernel, and you can actually distract information from them and that has expressed that a path that allows you to actually load your EPF programs inside your network cards so that you can program the network card to drop packets to understand a little bit the packets before it actually reaches the can, which turns to be very convenient.

A

If you want to drop the packets earlier, like let's say that you want to ban an IEP or you want to ban a might be class, you can just well it is this IP class XDP drop but, as I said, what I like the BPF for is to actually trace events and I also think that is better, sometimes to actually have a place to allocate your events even before they actually each other system, so that you like.

A

Let's say that you want to know how many time a function is called right or with like the sum of the bytes written to a file. You don't want to actually send every byte of being written to the back end and then some like in in Fox TV.

A

You want to actually have the Sun like at the kernel level, which tends to be very, very performant and for those of you that were like asking yourself well, this is that right, I, don't know how everyone in views from your with the kernel, but actually the ways you have to interact with the kernel are like tree. So you can actually write the kernel module very convenient for everyone right, so everyone wants to do that.

A

I maintain that or you can actually leverage the kernel Cisco's, but it turns out that you cannot like do whatever you want to it like there's, not a Cisco for anything and also like writing. The zip code is complicated. If you want to do that, and you can also use the filter Now link, but it just for networking so yeah.

A

So then there's another cool thing in town: that's ebps, so VD PA, PB PFS. You can actually load your specific programs. Evp F is a virtual machine, but nothing like the classical sense or beautiful machines like it's not like.

A

Kvm virtual machine is, is just an implementation of an instruction set and of a machine where this instruction set works, that the kernel used to load your program. So when you write your user program, you actually like you want to write like it'll, be if your program you're compiling it to it like a compiler that can actually emit the BBF byte code right. So now takes a c6 64, but a BBF by like Lang. Does that and that bytecode is then sent to the kernel with the BPF Cisco.

A

We will see that later. There are like tools that simplifies this right. I just explained the actual picture, and when you load this program, the kernels receives it in kernel, space and analyzes that program and after analyzing it like white analyzer I should analyze it like just to make sure that you don't kernel panic this machine right and when that's progress is analyzed. Then it's loaded in the BPF VM, and it turns out that, after doing that, you can actually send the formation back to user space using vpf maps.

A

They're, like any sort of any kind of map like you, can have like cache maps least whatever, and in that way you can actually stream out informations from the kernel. I thought to be very cool, and you can also like take the strings back and like to other things with the streams. So you can actually like use the strings to basically instrument you're polite, but this is the list of five PS.

A

I want to block you put those map and maybe, with the same, if your program, you found new IPs to block, and you write them in a map again, so that they're blocked next time but hey.

A

What are those used for like you're saying of these stuff to us and things that we never met that just now to not be actually actually true, because you use TCP down right and who uses it, raise your hands cool, pretty popular right and this if we dump actually emits PPF bytecode when you actually want to analyze your packets, so you're, actually using ppff like every week at least or every month. You just have to pass to disappear down this parameter, and it will show you the actual instructions that it sends to the vpf p.m.

A

to to actually dump out your pockets.

A

I'm not like a JD of the PPF bytecode, just like not OEM for x86 64, because they're compilers but I've been trying to write down notes on the actual bytecode I got from these TCP dump instruction, and it turns out that they wanted the IP, the IP stack and then TCP protocol and on port 80, which turns out to be HTTP, usually, and basically what I got is this instructional says this is an internet on a TPP for packet?

A

Yes, so that when instruction passes to go to the next one and is the sir seaport the source port on 80, yes and the destination? Yes. So by looking at this thing, you can actually understand if you wrote to write the BPA, the right, TCP dump line in or disappear now, so that you can actually see what it does. There's more like you, use, kubernetes right, use, containers and everything and turns out that one of the features that allows containers to actually be there is second right.

A

Second apparmor and all the lingo security module you may want to use. Seccomp is used in kubernetes in docker, container or containers or whatever engine you want to use to actually filter the Siskel's. Your container can do to the machine so that the container basically cannot become the network administrator and, by extension, the route under the container right.

A

It turns out that in succumb to the field to the filter- and you just like you can do like basic filtering like I- just want to block the Cisco, but you can also write actual BPF programs that actually have logic inside so that you can basically define your logic like if the process is called Lorenzo, just drop it I, don't want Lorenzo processes there, these kind of things, it's very cool, more practical examples.

A

You can actually use it to trace file opens by filename, for example, not trying all those just one I think I'm ready for like extract your runtime events. That's what I do a lot. I do a lot for character to restart JVM events. You actually want to know what the JVM is doing.

A

You can actually, let's say that you have a database whatever database you have and you want to actually know all the query is executed in that database and maybe how much the query talk to execute if try to use a new probe to actually go there and take the query and then, like analyze, the time span between the pretty walls entering the function and the pretty was executed right and you don't even need to actually tell the the actual developer of the database that you are doing that and it doesn't seem a big deal not to do or live in open-source.

A

But you really want to modify like the influx, the be sort just to get out that information order, my sequel source or whatever it is, and it's also big deal for those of you that might use like as close a source database, because you can actually or what was the source software whatever, because you can actually go. There understand what the programmer was.

A

Hey yeah, this was intensive right I've seen many people die on Excel, but there are high-level api's out there, so you can actually just forget whatever they said, nothing joking, you can actually just take whatever I said and put that in your knowledge base, so that you understand what's going on on the road. But then you actually use something. That's easier.

A

There's some group in the kernel that coli or Weiser that actually makes I a level tool for all of us to use so that we don't all have to understand the stuff, and that group made the thing called io, bytes or BCC.

A

If you have a compiler collection, that's actually a repo that contains a lot of examples, documentation and whatnot programs that you can actually just take and run, and it also has a lot of abstractions on local resources so that you can actually have something that already loads a program for you or something that already takes care for you for your Maps, and it turns out that it has go BPF, which is actually like a library to use busy single that you can use to like convert the map into a channel so that you have something the synchronous that sends you the matrix extracted from the kernel.

A

This program here may be a bit confusing, but in the slides and when I showed the slides, it's gonna be easier in this case, for example, what is doing is that there's some main function, I hope everyone can read it and this main function is loading is bash. Read line file here.

A

If that is this, one and after some water play to actually treat Kristin sheet, vbf Bob, EPF and everything what it does is that it attaches to this get return value here, and it attaches that function that might be a program to the red line function.

A

The colonel in that way, I can actually dump out every single batch common. That's done in this machine where I'm running this and I receive all of that in channel here in a go channel that they can then consume to do whatever I want and after having that, like I'm, just printing that tindy in the console, but I could send it to like database or whatever, like an influx to be to actually analyze that and maybe understand, if there's maybe any malicious program running in my machine.

A

Someone like took my machine and start running bash commands. Take I, see that right and I didn't even I didn't have to actually modify the bash. The bathroom acceptable to do that they just slow this thing. Some of you might wonder, and what's the difference between this engine, actually, yes trace or running a debugger right. Those are the DES trace is a good thing, but it actually add symbols to your program and like it actually adds how to roll them.

A

It actually our trace points, your program that makes it slower or make it a bit unstable and even worse, the debugger and this thing is completely or another level.

A

This thing is very: it's like a lot more, a lot, less intrusive for your programs and actually it's very like a lot very performant, because it actually runs and separately in the kernel in another place where it has access to everything and then there's a visor BPF trace, which I'm really excited about, because it's an actual DSL a domain specific language for BPF programs lets you actually write your BPF programs in a way that you don't alight, understand like see or go or like is this I think that loads may be a programs?

A

Is this a client compiler is is bcc, so you just actually fall vpf trays and you like here, as we seen before, actually a touch a trace point to a specific to the syscalls point to this specific circle. So you actually you're see all the time this is exit. Read this call and you actually can print out the arguments of the cisco.

A

It turns out that nip your face can also produce instagrams or do counters these kind of things.

A

This image, you I'm not saying the image itself because takes time, but if you want like to actually know like where and when things happen in the kernel, just so that you can actually understand where thought where it's best for you to attach, you can look at new career at this image that comes from the I advise of EPF trees, repository so like since I have a demon and that SK be later I will just like, for example, when you load the xdp programs that I was saying before you actually are before all of this okay, the actual packet hadn't yet reach the kernel.

A

So it's not yet reached with the net ASCII be skb struct. The net escapee struct is instructed your packets received when they reach the kernel, so they basically are enriched with this struct. That adds a lot of burden that packet, because the packet is ready like when you see when you have a a very traffic intensive application, it might be difficult to actually analyze packets, this layer, so you want to analyze them in the network are and that's where use xtp.

A

While, if you wanted to use vpf here, you will have to use traffic control, okay, yeah, what about kubernetes right? Go! It's a cute car, so you're saying all this angle of the media. Why turns out that, while working at the influx status, which we were cloud offer, we had a lot of performance program problems in under? We are working on new version, so everything is new. We are like writing in your storage, layer and whatnot, and it turned out that it was like.

A

Yes, we are building an observability tool and while building it, it was a bit difficult to observe it and it turned out that it's still difficult because you don't know in advance what you want to observe right and vpf gives you a way to actually observe what you're interested in now right.

A

So it's not like you can use BPF programs to actually like decide in advance what you want to observe and the survey for the rest of your life, but it turns out that you can know so like damper, there's some vpf programs that you want to run now so that you understand, like you, recognize the UNIX philosophy here, like you, have your Linux machine or what what the UNIX machine you have and you just want to run like a top or vmstat or MB, stat or I/o top or whatever.

A

It is so that you can understand what's going on and it was not there's nothing like that for Humanity's clusters.

A

So I made this think of QC TL Trace. That lets you actually run, maybe of trace programs on your kubernetes clusters and the way it works is that it's a QC gel plugin that you just download and put in your pack and once you have it, it actually use the same cubic config file so that once it can't connect your kubernetes cluster, you just go. There load the BPF programs in your kernels for the specific machinima. So, in this case, I'm running the key probe on these function of the kernel and I'm printing.

A

What to get as argument, 1 and I'm doing that on this specific machine right- and this is just for attaching something that might be interesting- is like actually extracting the Rinne Siskel's so like actually seeing the distribution of the read of the files in your machine in this case, so I just notice it that most of the files I read are between 4 and 8 kilobytes, and it turns out that's a very specific number and that's the number that most like TCP and UDP connection use. So if you then analyze that.

A

Right, Cisco, you will see that it actually on it actually is, unlike the UNIX socket or the TCP socket or whatever there's a more complete example. Here you can like pipe cue, CTL trace, like visit data, so that you can get a different way of graphing your your output, like with visit a that you can create actual complex graphs more than like, if you have traced us so I just wanted to do to add that here and there's this time for my name, I should so.

A

So I have this. Everyone can see me turn me off. Yes, cool.

A

So I have this little cornutus cluster locally, that does just a DNS and some cats on it. So yeah cats are cool right, so I just wanted to actually go to the cats and see them, and these little application just shows you how many time it has been called with the counter, and there are free replicas.

A

So you will see three different countries right and actually want to extract that specific counter information with BBF, instead of like having like to send the that information to influx CB or like to have to change my program to send that information, I just want to extract that information with BBFS. How do I do that? I wrote on one line you're here, so basically, what you need to do is to run this QC gel program that I run now.

A

This program here attaches I. You read probe and not told you about this right. You read probe is a new probe that actually gives you also the return value of the function so that you can get whatever the function returned that there's the same thing for surkhi probe, cable probe right attach the unit probe to proc process.

A

Id executable I'm doing like this, because I didn't yet implemented the way into CTL, trace to actually go there and understand this, but the but I'm doing I'm using this part instead of like the actual binary part, because it turns out that your containers are container, and so the only way you have to get the binary from the host is well. You can go like to dock redeem or whatever use understand that, but the best way is to actually go to the process.

A

Id and just get the executable deterrent- and this is this main counter values the actual function in my go program that actually gives you the counter value. So every time that function is called I receive a notification it might be have trace program. I can actually dump you. The this read literal value here as an integer, and this is locals because I'm running the kubernetes cluster locally weighted because I didn't actually trust the internet here and I just actually started the programming. My cluster, so I can.

A

Do is get your CTL trace, get shows you all the beefier trace program. You have been only in the default namespace in this case and I could like attach to a twin kill city of trace attached, but it turns out that I am already attached to this program because of this parameter. So I can just can't you'll see.

A

It doesn't work because I didn't actually called it so.

A

I'm calling it through some thanks and it actually is dumping the counters I'm, attaching it totally to one of the Caturday. So we will see only one of three and there's other. This one was 27. That is precisely the last value ago. This one was 28 and 28 and was 29. We can't go on for the world conference. If you want you want it know, you have better things to do and then it just stopped it.

A

It delete all of them and it turns out that it doesn't need you to actually have anything more than what you actually have in your cluster, because it actually uses resources there a lot it uses basically jobs and it creates them in space, you're saying it to create them, and so you can actually go there, inspect your content, QCT locks or whatever you want, there's also QVC TRS logs program just as a convenience, but commanders the convenience, but turns out they're just playing to registry. So you can understand.

A

Last thing I want to say to you: is that maybe like there's a lot of lack of resources out there?

A

Basically BPF is just documented in man and in the kernel sources and there's a lot of good work around from like see Liam and I advisor and I'm. Trying to do my mind. I have a BP, f, dot, SH domain. That turns out to be very good and service, your sister, but there are these two people that are actually writing a book.

A

That I would like to recommend because I'm actually in touch with them, like to the reviews and to actually see how it's going on and I just want to say that a book is coming soon. So if your interest in this topic, there will be more material in the next.

A

The next couple of days and I just want to add some more references of like things that I think are important in this BPF landscape now, so that you can actually go there and, like maybe shoot a photo and understand that there's some work going on I.

A

Think it's time just to tell you who I am because I didn't tell you in the first place, so my name is Lorenzo I work at the influx data they create or stuff in flux to be before you know it as an SRE.

A

We are actually managing a lot of traffic because we ingest all the points in our cloud offer and it turned out that there was a bit upset on the fact that they couldn't really expect my infrastructure, so I just trained doing this stuff and I started understanding it and they just wanted to give a talk so that more people can understand that and the next time it's always a BPF people doesn't look at me, be the scary face. Thank you for coming. Listen.

A

If anyone wants to contact me you're welcome my work. Him is Lorenzo at the influx a to become my personal email or low at PDF daughter's age.