Cloud Native Computing Foundation Kubernetes Community Days (KCD) Chennai 2022, 30 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Why Everyone Should Care About eBPF by Neependra Khare

Description

eBPF - Game changer technology to program the Linux Kernel dynamically. Using eBPF, we can now program Linux Kernel dynamically and get insights, which were not possible earlier or were very expensive . It is changing the game in the system's side the way JavaScript helped us bring everything on the browser. There are now many emerging use-cases of this technology. In this talk we'll discuss about its background, uses cases in security/tracing/networking/observability with some hands-on demo.

A

My name is nipah and I am founder of cloud before starting cloud. Around seven years back, I have worked over a decade in different companies, but primarily with red hat, though I have experience as system administration, support engineering file system, kernel and perfect engineering.

A

In 2015, I wrote a book on docker and then published a course on edx for the cncf on kubernetes, which has been taken by more than 200 thousands users across the globe.

A

At cloud we provide trainings in the space of cloud native and have many clients. Now, let's get back to our topic, eppf at a heart of it is the way to program the kernel dynamically and get insights from it.

A

It is changing the game in the system side. The way javascript has enabled us to bring everything on the browser and because of that many interesting use. Cases are coming up for this.

A

Even the windows kernel is going to support the ipvf in some time. Let us now look at brief history about ebbf.

A

So before ebbf there was bpf, which is berkeley packet filter, as the name suggests it is used to filter the network packets in any os. We have two primarily things: one is user space and the kernel space in the user space we have programs like browser editors and so on, while on the kernel side, we have programs using which we can interact with the devices like network card disk and so on.

A

Before bpf, where the network packets used to come to our system, we used to send them to the user space for filtering, which is very expensive affair, but with bpf we have enabled that functionality in the kernel by creating a virtual machine inside the kernel itself.

A

So when the packets come at the network layer, we could easily filter them out, and that made things very faster and I believe you would have used the software like tcp dump, which use this bpf and helped us debug the network issues.

A

Let us now look at epbf, which is extended berkeley packet filter, but it is not meant for just the network anymore, though the name is a bit confusing, but that's how it is with epbf.

A

We have implemented journal event, processing and filtering mechanism.

A

We have a global data store as well, which we call them as maps to store the state between the events and share it with the user space as well.

A

The core of eppbf is lies in enhancing or extending the capabilities of the linux kernel without changing the kernel code or loading a module, and this makes things very similar. So if you have been part of the linux kernel community, you know that to bring up a linux feature in the kernel may take up two years, which is just too much and by the time the future has arrived. We may have moved on as well, but by dynamically programming the kernel with epbf.

A

It changes the game significantly and we can now come up with n number of use cases for that, and we will look them in few minutes. We can attach eppbf to different programs like any linux kernel. uh Anything in the network stack any user space program and so on. So we can attach this program to any of these events and whenever those events happen, we can perform something.

A

There are many use cases for epbf now now, if you look at in the cloud world, we'll talk about kubernetes, which runs multiple containers, and these containers in most cases, are just programs and if we can attach the epbf at the color level, we can actually look at what's happening in every container and so on. So rather than putting something at per container level like a side car, we would filter them out or we can inspect them at the kernel level. So things would become very simple.

A

Now, there's one more use case in the networking space as well.

A

So if you look at in the kubernetes ecosystem, we communicate to the parts throughout the commodities cluster and generally we do it with the help of q proxy. So with q proxy, we create these firewall rules and that help us communicate with different parts.

A

If we have lot of parts, then that can be significantly slow because of the way ip tables would work. But again if we have lot of parts, but on a similar case, if we move to the eppbf based uh cni plugins, like cilia, things are very fast because we are not going anywhere in the ip tables route. We just go from the color layer itself, and that makes things very fast.

A

Ebpf has been well tested for some of the good balancing projects like cat run from facebook unimog from cloudflare, and there is an active word going on to implement service mess with eppdf so good times ahead.

A

As with epbf, we can look at everything what is happening inside our system. There are very good use cases for both security and observability as well.

A

In the security terms, we can have the tooling built around networking and the runtime security as well fall code, tracy and the policies are some of the example of security uh things there. Similarly, for observability, uh we have different tools like pixi and so on, using which we can observe in the system what's happening through and throughout before we jump into the architecture of eppf.

A

Let us look at some very basic examples and see what all possible with just single liners. You can also trigger the lab and then perform the hands-on lab along with me. So now, here, I'm going to first install a tool called bpf trace, which is a high-level testing language for epbf programs.

A

So let's install that first.

A

Once those will get installed, we are going to run a very simple program to trace all the system calls and then we print them by counts, so our bpf trace has been installed. Let us now look at our first example in which we are going to get system call counts by the programs.

A

So if I run this script, it is going to collect the system calls called by every program. Now, if I just press ctrl c, you can see that there have been few programs which are running on system and they have called different number system calls like. The node program has called 41 system calls, and so on now, with the next example, we are going to list down all the best commands which have run throughout the system. So I run this particular program and let's say if I open up a new terminal and run few commands.

A

Let's say I run a few commands like this.

A

Now, if I go back, I can see that these programs are listed here, so we have run the date and the cal command and they are listed here so again, a very powerful example. If you want to kind of look at what commands people are running across the system, let me do a ctrl c here.

A

Now, in the next example, we are going to trace new processes via the exact calls. Let me run this program here, so the program is now running. Let us now go and let's say, do few more things here. Let's say if we run date, command do a bash here and so on, and let's go back here and check now we can see here um what are the new processes we have called here, and these are being listed here.

A

Now, in the next program, I want to trace all the active tcp connections, so I'm just going to again just run a very simple script.

A

And this particular script is now running. Let us now go and try to make some connections. So let's say if we do a curl on cloud website.

A

Let me just add the minus l here yeah something has come up and if I kind of go back here, you can see here now it is listing all the tcp connections. What we are making here.

A

Now on the screen, we can see different bpf trace types which you can attach at different places in the system. So, as you can see, we can attach them with respect to devices memory, cpu user space programs and so on. So we can actually put or have these scripts for each in individual subsystem. What we have here and we can then see what's happening exactly on our system at any point of time now we have seen the basic examples. Now, let's see, how does ebpf actually works?

A

So here is a high level diagram, which is going to explain you how things are happening behind the scene, so we need to load our ebbf programs by the bpf byte code, which we can write manually but, of course, writing manually by codes. It's just going to be too much complicated and that's why we have different tool chain which have been built using which we can easily write these um ebbf programs. So there are different tool links. We have see the example of bpf trace already. Then there is bpf, compiler collection. We call this bcbc.

A

You can even write now the programs in rust language and so on. So once we have kind of have generated the by byte code.

A

So now, once we have the byte code either by writing it manually or from any one of these tools, we can load that byte code to our kernel, with the bpf system call and first we hit the verifier, which would verify whether the code which you are loading is safe to run on the kernel or not. If it is not safe, then verifier would fail it and not run it post verification we are going to use just in type compilation, step to convert our instructions to the respective architecture.

A

What we have, and once we have done, that we are going to attach the program to different events. What we have and those events, as I said earlier, can be any kernel, call can be any user space programs any trace points or, and many more so once we have attached to any of these events and if those events occur, we are going to do the data collection and the data collection.

A

We are going to do with with the help of maps and using this maps, we are going to share those stats in the user space which we can visualize or see it in the text of whatever it is now. There are different kind of maps we can create to collect the state of the events and so on.

A

Now here is a very simple program which we would write with bcc and python and try to uh hook it with an event. So whenever that event happens, we get some data out and we'll see that so again, there are different front ends, as I have mentioned earlier. So we are going to use the bcbc library with uh the python program to run the particular example. So only we just added the front end now. So let's see how this is going to happen.

A

So for that purpose we are going to first load a few packages to perform those steps so we'll just load the bpfcc tools and the linux headers package.

A

Once you have done that here we have the python program and, as you can see with the python, we are first importing the respective uh module and then we have this bpf program which we just written in like in the c language here, and we are going to load that particular uh program with bpa func with bpf call here.

A

So if you look at in this particular program, we are simply saying that whenever this particular function is being called run, uh the print hello, kts and clarity community, now what we are doing is we are loading, this particular bpf program and attaching it to the system called clone. So whenever we are going to fork, we are going to run this function and print out the output um uh like hello, kts and the clogged community. So let's try to run this particular program now.

A

So we are now running this program. Here is all there now, let's go ahead and do some fork so just going to kind of spin up few more terminals here, maybe right date, some other commands here and now, as you can see here, as we kind of perform any of the clone functionality by any means in the system, we are getting this particular output on our screen.

A

So with apbf, we are just bound by our imaginations with eppf. We can program the color dynamically and to the end user or developer, we're just going to have an api call using which they can program the kernel for their own choice and that just brings enormous flexibility to the developers to bring up many and many use cases.

A

So in the government future we are going to see lot of use cases coming up with this technology, so I highly recommend you to keep an eye on this technology and see what our use cases are coming here. Thank you.