StackRox Office Hours, 13 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: StackRox Community Office Hours E2 eBPF 101 — Implementing Security & Monitoring Kubernetes

Description

Streaming live on openshift.tv: eBPF is the behind-the-scenes subsystem of the Linux kernel that enables new and simpler methods of profiling, networking, and security for Kubernetes without compromising speed and safety.

Join our monthly Office Hours to get your eBPF questions answered. We’ll start with an overview of the capabilities that eBPF provides.

A

A

A

Good morning, good afternoon, good evening, wherever you're hailing from welcome to another stackrocks community office hours, this is episode. Two we're gonna be talking about ebpf 101 today and I'm joined by the one and only robbie cochran. How are you robbie? I'm great.

B

A

B

A

You want to introduce yourself to the audience just to let them know who you are, what you do around these parts.

B

Yeah absolutely so um I am an engineer at red hat formerly uh was an engineer at stackrocks and have been working in kubernetes security for a few years now, um and in particular, I've taken a a focus and interest in kubernetes security and evpf together and uh yeah.

A

Cool, so we have um a doc with some questions in it that I'm trying to pull up uh first question: what are the use cases for ebpf.

B

So ebpf, you can think about ebpf as a better way to extend the kernel um and the use cases are are pretty wide open, yeah, observability security networking features.

B

Ultimately, you can think about. You know I like to use the analogy, and it's I know this has been used before from the perspective of a web. Browser without javascript is kind of like a linux kernel without evpf, um and you, uh if you, if you, if you think about all the things you can build with javascript in a web browser um and the the way that linux kernel development used to go, is that you would have to. If you wanted some new feature in the linux kernel.

B

You'd have to go talk to kernel, maintainers or write your own kernel module, and what ebpf provides is a way to get data from the kernel. The linux kernel, as well as take actions on events, um and that is a seem may seem, like kind of a.

A

You know not that big of a deal but.

B

uh It allows for a whole other layer of um tools.

A

B

Techniques and things to be built on top of the kernel um and.

A

B

To think about the use cases of ebpf, are you know, I don't know your persona or you know, what is your role and you know what problem you're trying to solve um the the kind of the first way and I'm going to give a demo in a few minutes um that I think about it is in terms of just debugging. So I want to know is this. You know, like think about a break point in your code. I want to know what's happening um when.

A

B

Within the kernel is being called, uh or um you know how many times it's being called that sort of thing, so performance counters things like that there have been sub-systems built into the linux kernel, but ultimately, you run into you know. Well, uh this feature isn't available in the subsystem like the perf subsystem, the audit subsystem.

B

These things are uh not extensible and what ebpf allows you to do is attach small programs on these events and then report data back into uh user space. So you can have a user space program that understands what's going on. In addition, you can actually write new uh networking stacks within the kernel in ebpf and then, if you need to update them or change them or configure them you're able to to do that.

A

Nice, so you know if I were a new person using evpf inside my cluster, like what would that look like from a pod perspective.

B

Yeah so there's there's several ways. So let's say we have a kate's cluster up and running openshift cluster and we want to um understand what's going on within the cluster, so we have a performance issue, uh for example, um and maybe it's related to disk activity. You could use evpf scripts and tools and piece together kind of uh an idea of what's going on by just extracting the data and getting performance metrics. So this is, you know observable from the observability standpoint.

B

Another use case is if you want to actually use ebpf to help with security, so visibility less less about getting sampling and aggregate information from the kernel. But more. I want to see an audit of all of these events and have it correlated with additional pieces of data like the container id or the network connection that a process made or something like that, um and so that's that's one way to get uh uc bpf in your cluster.

B

um I think from the individual perspective, the debugging is kind of the debugging and performance analysis is the first entry point um it you know it actually requires just like building a large web application on javascript. A lot of effort building a platform that's built on ebpf, is you know, a significant effort, and so there's a lot of existing platforms and tools and things that are that are available.

B

So, for example, um the stackrocks product now acs is uh uses ebpf to extract process executions as well as network connections from the kernel and what's great about ebpf. Is it allows us to operate on any kernel version kind of going back to that challenge of uh um you know any recent kernel version that challenge of uh you know we're not able to add stackrock's specific features into the linux kernel. We don't want the customer to have to have a special kernel.

B

um We can just insert an edpf program within the kernel and then get this data out.

A

That sounds way easier than running custom, kernels everywhere, yeah.

B

Totally um yeah yeah.

A

Cool so, like I think, of the ebpf, and I think of like tracing and kind of the same vein right, you said debugging right, like there's nothing better than running an s trace on a process right really seeing where it's breaking right does ebpf. Give us more insight, less insight, I mean what is that? What is it yes,.

B

Like yeah so s, s trace if for folks aren't familiar s, trace, is kind of an old school system, call tracing approach and it uses uh it's kind of ping pong's back in between user space and kernel space.

B

So you you can print every uh system call that an application is running and that's super useful for determining you know what actually is happening when I'm reading a file- or you know, um and uh so the the problem is that s trace has a huge performance overhead, oh because it's essentially um you know adding an extra round trip between user space and kernel space. uh The um the the cool thing about ebpf is that you can. You know, of course, evpf can introduce overhead because you're doing work.

B

You know some sort of additional compute within the kernel, but you can be selective about it. You can say uh you know. I want to record all the system calls and then uh report them out that sort of thing uh so yeah, eb. You could.

A

B

Build an s trace on top of eb.

A

Yeah um and getting those system calls like very telling very helpful.

B

A

How would you install it right like how would you get ebpf in your cluster, so I.

B

Will let me do a demo um and uh let me show you guys: some, let's see share my screen and uh before I do that, I'm gonna go through just a little intro on some slides here.

B

Is that visible, chris, yes, cool so kind of? As I was saying, you know, we have the linux kernel which is hard to update, and then we have our application stack on the right. That's just a set of processes, and you know these could be running within containers in our cluster. But ultimately the processes are sending information on the network, reading, writing files or launching other processes- and you know, since the the the unix api, the postx apis were created.

B

It's it's made a uh um it's been a stable place to run programs on linux, so you can run exec. You can run open, um it doesn't.

A

Matter? Which version.

B

Of linux you're on or you know which flavor, however, within the linux kernel, there's all these different subsystems- the memory manager process, scheduler disk network.

B

If you want to get data about those internal components, it's difficult, so you can either write a kernel, module or actually submit a patch to the to the kernel maintainers, and that takes a lot of effort.

B

So we have this stable api for our user space programs, but we don't necessarily have a good way to extend the kernel without you know a lot of effort. So what ebpf lets us do is add hooks within the linux kernel and you know whatever we can add the hooks at the syscall level. So that's thinking about s trace where we want to see all the interactions between a given process and the kernel um via syscalls, or we can attach ebpf uh programs to any any function really within the linux kernel.

B

um Of course, the syscall layer is going to be more stable, so I think the you know the easiest way to get up and running. If you have a kubernetes cluster um and run some uvbf programs is to use a a really nice tool, called coop cuddle trace and that's what I was going to give you the demo of here. So this is a available here at this github url and the prerequisites are really just. You need a kubernetes cluster up and running.

B

So let's see I'm going to share my terminal.

B

So I have a cluster a case. Cluster running um and I'll show the uh nodes. So, let's see, we've got make sure we can connect to the cluster there. We.

A

Go so we got three nodes in this cluster.

B

um And in fact, let's look at what what the the um host os is.

B

So these are running ubuntu 20.4, linux, kernel 5.4. um So that's uh you know you're! Normally, when you have a case cluster you're, not really thinking about the node level. Nodes are cattle, they come and go, and um uh what ebpf allows us to do is um kind of get into the the linux kernel on the host, so we're kind of getting underneath our containers and seeing what's happening um right now. This is a completely empty cluster.

B

A

B

Of course, if I uh check the cube system, namespace.

B

We'll see the.

B

We'll see that um there are.

A

There we go there, we go so.

B

We've got the you.

A

B

Base um uh containers that are actually running in kubernetes so, even though there's no applications running in this cluster there's still stuff going on. um So what we can do is use coop, cuddle, trace and coop. Cuddle trace is a coupe cuddle. Plugin. um There is uh essentially um it allows us to still use the coop couple command, but then it calls out to another process, and so we can run coop cuddle.

B

We can install crew cuddle trace using crew, which is a coupe cuddle, plugin manager or just downloading the binaries directly and coop cuddle trace, allows us to run bpf programs, and particularly a particular type of bpf program, called a bpf trace program, which is a higher level scripting language, that's built on top of bpf and, as I said, this is, I think, the easiest way to get started with running ebpf in your in your case clusters. So what we can do is, let's, let's see what processes are running on a particular node.

B

um So, of course, I could exec into the node um and do look at you know, there's a lot of different ways to get that information, but let's see how we would do that with ebpf, so I'm going to get nodes again and then I'm going to run a bpf trace program.

B

Let's start with exec.

B

So what this is going to do is it's going to run a kate's pod on this node g4 x2 is the the particular node and it's going to run this execs new program. So we start that it creates a pod, and this pod has to be configured a certain way, so we can actually look at it.

B

And let's describe this pod actually edit it, so we can see all the different configurations.

B

So, what's key about this is it's running a container that is actually just cuddle trace bp of trace and it. The other kind of key thing here is that it's running um in with admin privileges, so the security context here is privileged and if we're going to execute bpf programs within a container, we need to be privileged because uh the bpf system call, which is you know, we're monitoring system calls or observing system calls. We also use a system call to insert the the program um it requires.

A

B

Privileges, so I've run this trace program and I want to see what what's what uh what's actually running, so we can tail a the pod itself, which is actually going to be just printing. The data.

A

It's getting from.

B

B

Bbf program, and so this is just tracking the it's adding a bpf program to the exec system, call looking.

A

B

The arguments and sending them back to user space and so from a single container. We can see all of the process- executions that are happening within on the host system within each container and on the host system itself, and so this is.

A

Actually, kind of.

B

A cool way to see: what's you know, what's going on what kind of activities happening on a on a kate's cluster there's a lot of just you know: version checks and sanity checks. uh Some health checks that sort of thing.

A

That's cool, though,.

B

Yeah, so let's so we ran this program called uh exec snoop. So let's look at that program.

B

And this is kind of when I was saying earlier about how you know one of the first places I think people you end up using ebpf is to diagnose performance or other kind of issues in their cluster. So you know you might want to see what's running in a pod or that sort of thing, um and this bpf program is very simple. I mean here's the here's the code right here.

B

It attaches a um the hook that we're attaching to is the syscall for exact the e, and we are just um combining the arguments and then printing them out. So an exact system call has a path associated with it and we're also printing the process id okay.

B

These programs can be run using on any linux system. I think all distributions have these tools available in their package repositories. um This is I'm talking like if you're, if you're, not on kubernetes customers on the linux box, yum install bcc tools or yum apt-get pcc tools um you'll actually be provided with a lot of these programs.

B

A

B

There are oh, this is just exact. We could also look at networking. um Let me delete this trace.

A

And look at hygiene, your wonderful pristine cluster! I.

B

Know right, this cluster has uh has never seen any any high cpu all the cpus are just ready to go. So what I'm going to do here is.

B

Let's see what we want to do actually sure what do you.

A

B

A

What do you got? I mean well, there's a directory of bpf trace programs, I'm curious! What's in there, yeah.

B

So this is it, this is the kind of default programs and, of course all of these are you know, I would say, not it's not too difficult, not too much effort to extend them for what you need, um and this the snoop programs are particularly interesting, um and you can imagine.

A

B

Couple different models of getting data out of the kernel. You can just stream all the events or you can also aggregate within the kernel and then uh uh so.

A

B

Just uh let's do connections so.

A

Yeah yeah: what's something that's actually gonna give us a bunch of results.

B

um So uh so we'll see, let's get the logs for that now. This is probably gonna, be really active. There's gonna be a lot of connections and this will show us the.

B

Just stream this into a.

B

Cool okay, so this is just a snapshot of connections that were being streamed uh via this ebpf program and um we're getting the the pid the command of the program. So this is the the command is going to be. You know, stored in the kernel and just reading that data look at the source address the destination address, and so we can see all of the um connections that are being made and which program is making those connections.

B

Let's say we there's a you know: curl, that's probably a health check, just checking for readiness, we've got kublets talking to the api server, most likely and so on.

A

A

B

A

You talk about performance, metrics right like, and can you show us anything around that sure.

B

Yeah, let's take a look at the performance.

A

Of a trace that'd be super meta.

B

Yeah, you know, and I think it's that's worth mentioning um you know if you know, what's the overhead of running ebpf, anything based on ebpf within your infrastructure and, um uh like I think s trace, is a nice example because you know we were describing s trace for every system call you're, adding an additional round trip because it has to go in and out of the kernel, um and uh so you want to make sure that whatever events you're attaching to aren't too frequent and that the data you're extracting um is reasonable.

B

Of course you know you could you could attach to every function in the kernel and just completely, you know dos your system or you can have you know, kind of targeted and um events that you want to get information about, um and there are a lot of ways to um aggregate data. So ebpf provides different techniques for for getting data of the kernel.

B

One one approach is histograms and maps, and so on um you know I don't know if there's a good way to do that with uh um these programs oftentimes, you kind of want to, like you know, build a flame graph or some kind of uh um more visual uh view of this data.

B

um This is kind of the raw data coming out, but let's say um just for example, you might want to see, like I think. What's really, nice is looking at uh tcp um uh issues like uh drops, or you know, tcp lifecycle. That can be very helpful for networking debugging.

A

And everything yeah yep yep, so question came up in chat. Where did these programs come from.

B

Yeah, that's a great question, so these are bcc programs. So let me go back to my slides and I'll kind of give you a better idea of um the ecosystem of ebpf.

A

B

So let me actually go to.

B

Switch down a few slides.

B

Okay, so this is a this is from the edpf dot io website. This is a nice excellent resource for edpf and it's recently forming you know, kind of an open source uh consortium of uh developers and companies and everyone that's interested in ebpf, and these.

B

What what I just showed you with coop cuddle trace is um a bcc program. Those are bcc programs, so this is the logo for bcc, um and these are available on. uh You know github, so you can download these and, of course, there's our there's a lot of um uh oops.

A

Sorry about that no worries.

B

A

Yeah there's a I was.

B

Going to try to show you the the website, but I'll just um tell you what search for so.

A

B

Bcc tools are um what those scripts were b and dpf trace programs, and then we were running those uh using the coupe couple trace it's a lot of bpf and trace and tools, and that sort of thing um you know, I think that it can be overwhelming when you're, looking at all the different ways that bpf can interact with all these different kind of uh subsystems.

B

The the key thing is that you know the use cases of networking, security and observability and tracing are kind of where you you want to start with. I would say: bcc tools are ideal for observability and tracing, um and these are historically based on the uh d-trace approach um and solaris.

A

uh Yeah, so is iowa visor bcc the right reaper to get these from yeah yeah cool I'll drop. That.

B

A

B

Yeah, um the uh uh bcc, compiler collection or bpf compiler collection is what these are toolkits and wrappers for writing. Ebpf scripts. You still need to write in kind of a pseudo-c and then the bpf trace programs, which I showed you guys are built on top of that.

A

A

Cube cuddle trace, says from chat, uh cube, cuddle trees makes things easier. How do we use it to generate flame graphs so earlier.

B

Yeah, I think that um the the best way, probably the one of the the most full-featured um toolkits for generating uh data like that is the vector, subsystem or vector system from a performance monitoring system. That's uh netflix has open source um and also uh you know these. You can generate flame graphs in different ways.

B

You can just have you know, text histograms, but I think the vector uh um performance management system is ideal for flame graphs.

A

A

I'll find that for folks drop them a link. um Sorry, I'm gonna see if that see. If web pages right now so.

A

What has been done to harden the ebpf subsystem right like how do I know this? Isn't a back door right.

B

Like, like, is this something.

A

I need to watch right, like are people making ebpf things out of my uh yeah.

B

Yeah, so um you know the kind of going back to the you know. What are the alternatives to running? um You know you're this, this container, that um is inserting these bpf programs is running and is privileged. So um of course you want to make sure that your user space code, that's managing and running your ebpf programs is, is secure and managed properly the um verifier.

B

You know there's there's kind of two angles to that question. So one is: what's this you know. Is this going to crash my kernel? Is this going to crash my host um and then the other angles you know is this? uh Is this secure and I would say that you know there have been um some uh there haven't been an edpf attack, but there has been some data leaks and things that enabled with ebpf, but really the any user of ebpf has super user privileges, um and so that's that's something to watch out for now.

B

Now, of course, the uh the verifier essentially um that your program must pass is built into the linux kernel and it will. uh The verifier will um ensure that it, the the code, terminates and isn't going to get into any infinite loops. In fact, unbounded loops aren't allowed in bpf programs. The bpf instruction set is actually quite quite constrained in what you can do and that actually, that helps a lot of security um null d references, a.

A

B

A

B

Kind of uh you know, caveats that come with writing in c that you would have if you're writing a kernel module are, are not there with bpf, because it's got more safety features and it's more constrained language. Okay,.

A

So very tightly scoped, that's good! Very! You know.

B

A

For lack of a better term, you know.

B

A

B

Would say that you know if you're, if you're using evpf um the uh you know it's it's essentially a sandbox within the kernel, um and so you get the security guarantees of that.

A

Nice awesome! So what about.

A

Are there any downsides to ebpf right, like we talked about overhead a little bit? What other downsides are there potentially.

B

You know, I think, the the there really are no downsides, because it's and you know, of course, that's a very blanket statement, but uh you know the maybe what are the you know? What are the issues that it can bring? I guess is maybe a different way to think about it, and um so the um I think one one challenge is uh making sure that you're running on a kernel that has you know you really need to be on a more recent kernel.

B

There's been a lot of uh effort and um changes that have been added to the linux kernel recently.

A

And so, if you're running.

B

On in the cloud you're going to be using a more recent kernel, but if you're you know, if you're in an air-gapped environment or running older distribution, you're not going to have all the most recent ebpo features.

B

Rel red hat has back ported a lot of ebpf to older versions of rel, and so usually I think, if you're on rel you're going to have all the features you need, but making sure that your host can support evpf is is key. um The other challenge is so. I mentioned that we're.

A

We're essentially.

B

Extending the linux kernel, and in order to do that, you need to know what is the layout of a particular struct that I want to get data from and across linux kernel versions that those struct layouts will change, and so in the coop cuddle example demo. I just showed the um uh um what actually is happening is we need the linux kernel headers in order to run the the bpf program.

B

So when we compiled that script into bpf bytecode before it's inserted into the kernel, um the headers are needed and uh that can be very challenging because, um especially if you're trying to uh support you know you're developing software, that you want to run on a like a variety of different linux kernels. You know and coming from someone who's worked on a security product that you want to run on a lot of linux. Kernels! That's a challenge!

B

I mean that's something that the community, the dpf community, is very well aware of and they've been adding features to export the struct formats and the data types within the kernel, so that you can just subscribe to that. It's called btf or bpf type format and there's a another initiative called that uses.

B

Btf called core or compile ones run everywhere, and that's kind of the the ideal in state for these tools is that you're able to write a bpf program and and run it anywhere, but right now that can be challenging, because uh only the most like recent kernels uh support that you know, depending on, if you're, if you're on a cloud, if you're in the cloud, it's it's going to be supported, but um otherwise, maybe not that's just something you have to look into yeah. um You know the other.

B

The other challenge I would say with bpf is it's. um uh You know still relatively new and you're running uh code in the kernel, and even though this is like a great tool to get information out of the kernel, you said something earlier about profiling. The.

A

B

Meta of bpf right.

A

B

A

B

Difficult, you know any time, you're you're, adding overhead within the kernel. If your application's slowing down- maybe that's that- can be difficult to diagnose. um There are tools and techniques being kind of developed for performance counters and measuring okay. I want to see you know how much time is being taken every time.

B

For that example, I was only attaching to the exact system call or the tcp connect system call, but um if you've got a lot of places, you're attaching you're going to want to profile these and see make sure that, especially in production, if you're running uh evp in production.

A

Nice good call um is there anything else you want to just show off with the ebps? Is there anything you think people's minds would just be blown by it? You know.

B

um Yeah, let me let me go back to this um this slide here and I you know. I think that um so we have this bottom layer of the linux kernel, and you know this is the. I would consider this slide from uh the kind of the road map of edpf um there's you know, ebpf is being added to linux. Kernel. uh Microsoft has been adding an ebpf layer to windows.

B

There are several sdks for writing evpf, because not only do you have to write bpf code, you also need the framework to help you inject the program, kind of manage it and get the data out of the kernel, and so there's python and go and um c, plus, plus and rust uh libraries for doing so, and then on top of those, are these these tools, tools and platforms that are being built, and I think that's probably the most interesting thing going on with ebpf.

B

Is you know it's good, just like it's good to know about assembly code, but no one's actually going to be writing assembly right, usually.

A

B

And so it's it's really cool to see what people are building on top of ebpf.

B

From the perspective of security, the coolest thing is be to be able to have a way to not only get data from the kernel but actually take action within ebpf, and so that's where I think, when you start writing when people start writing- and this has already begun- writing ebpf programs that actually can take action within the kernel, then there's a lot of really interesting tools and platforms that can be built. So, from the perspective of security, um there is a new uh linux security module um based on ebpf called. In fact, I'm not sure.

B

If I get this right, but kernel security, it's ksri, um okay, kernel or maybe I got the snr swapped kernel, runtime security instrumentation, there you go and what what that allows you to do is is actually have checks. So you know, is this process authorized to take this action and you can just deny it, and so those types of rules can be updated and tied into the higher application or authorization frameworks. um And, what's great about that, is it's.

B

You know anytime you're, taking action within the kernel, it's performant because it's not requiring you to uh go back and forth between user space and kernel space. The the other place. That's really interesting, is um just observability. So there's a lot of observability frameworks. um uh Pixy is is one uh observability framework and I think in general, if you're doing any kind of application, tracing or monitoring, it's gonna ultimately be based on ebpf, and I haven't really specified this or kind of gone into detail about it.

B

But not only are you able to trace events in kernel space? You can also do that in user space, so you can attach to essentially use ebpf to check break points in user space or to look at events in user space and basically.

A

You can imagine that just.

B

Opens up a whole world of observability and you know just flame graphs are kind of like the beginning of all the different ways you can like right dive down into what's what's happening um so security in actually doing enforcement. That's one area, that's, I think super cool, um the observability frameworks that are being built on top of ebpf and then finally, service meshes and networking. So.

A

B

And uh psyllium is a kate's cni plug-in, so it's an ebpf based um networking plug-in for kubernetes and I'm not an expert on psyllium. But my my kind of my.

A

B

Of what the benefits.

A

B

Provides is again you're able to not have to use ip tables to manage the connections between deployments or even between clusters. This can all be configured within ebpf, and you know you. There are um there's a component of the linux kernel called xdp the express data path uh that is used often by um uh hardware manufacturers to tie you know as a as a way to integrate their nics with the linux kernel, but then also you can have essentially a software layer built in there, um and so that's that's what uh what psyllium is?

B

um A a kind of software uh ebpf based uh networking plug-in contract, ktran or k-a-t-r-a-n is a load bouncer based on ebpf and even even calico has has added ebpf features, um so the I think you know moving forward if you're running a service mesh, it's also going to kind of probably be built on ebpf.

A

Yeah it'll have some bpf components to it. It sounds like for sure um so quasi questions slash, you might not know the answer to.

B

A

Wouldn't be bb ebpf yeah, wouldn't ebpf effectively introduce a stable kernel interface and permit proprietary programs for the kernel.

B

Yes, so um like.

A

They're yeah, I'm sure that's a contentious thing so.

B

So I'm actually not sure I completely understand the uh what's.

B

What the questioner is asking in terms of where they think where they're going is that, of course, if you are writing a bpf program and it's inserted into kernel, there actually is a gpl license check in the verifier, um and so, uh as you know, I'm I'm not a lawyer, but you still have to you know. If you're going to be interacting with a um gpl api within the kernel, then your bpf code uh needs to be open source as well. um So you know, I think, that's an interesting idea.

B

I'm thinking from the perspective of hardware, like a you, know, a device driver or something like that. I would say that you know it's. This is already feasible with kernel modules and you know other other things. The kernel is.

A

Already extensible.

B

This is just an.

A

B

And safer way to do it.

A

I think safer is kind of the right like yeah. Easiness aside, this is just safer to do than like building kernel modules left and right. That's right, yeah, um yeah, and, to be honest with you, I don't know where eighth doctor was going with the question. But if that didn't answer your question 8th doctor, please let us know um khalid raid says. The most interesting thing to me is how you can control network traffic on layer, seven with evpf, and that that is super awesome.

A

uh The fact there's a load balancer that uses the bpf behind the scenes. I don't know how you could extract any more performance out of the kernel than that. um I mean yeah.

B

I've never used a tran.

A

Before but it's got to be fast and performant and all kinds of different ways.

B

A

B

I mean that is, you know, get essentially you're giving these kind of core data paths context and you you know, user space can provide the um the context through evf and then it's um you're able to build these performant networking tools. um You know whether you're doing packet, inspection or um you know, yeah taking uh taking action on things. You know I haven't seen anyone do like an ids and ebpf, but those types of things that would be cool, yeah yeah. You know, I think, and maybe going back to the challenges.

B

Yeah you're, you are limited to how complex these programs can be um right.

A

B

Off the top of my head, how many there's there's an instruction limit on how many instructions can be in the program? And um so you know, I think, when you're thinking, if you're thinking of small snippets and like you know, you're you're building together, uh you know for psyllium falco these these projects have large ebpf programs, but they're really pushing the limits of uh um the the size of what you can do. So you you can't build a you know.

B

You can't have a machine learning engine running in your kernel in uh using uh ppf right. So you you are. You are limited to also the just the the limitations of the ebpf verifier. As I was saying earlier about, you know, bounded loops and what memory you can access and that sort of thing.

A

Yeah yeah detective conan kudo. uh He basically the issue here is that proprietary programs for the kernel that are tied to specific kernel versions bring us back to the problem that prevent people from upgrading kernels in the first place. I don't think ebpf does that here you have that instruction set limit it. It doesn't live in the kernel. It calls the kernel right like so it's one of those things where it's like like. If people are concerned about this causing you to like stick to a certain kernel version, I don't think that's the case.

A

This is going to be persistent in kernels, going forward, you're able to run the same. You know traces as before. um Right.

B

Yeah I mean it's important to like.

A

B

We're not hot patching the colonel.

A

B

We're we're kind of uh we're it's event, driven hooks that we're taking action when that event happens, and so that does have certain you know kind of limitations.

B

You can't necessarily um uh you know uh of like change code paths or that sort of thing you're, adding on additional uh you're extracting data or um now, for example, if you do the thing so I described from the perspective of attaching to system calls but um cilium, which is attaching to the xdp subsystem you're, extending that so, rather than going back and forth between user space you're going between your ebf program and that subsystem within the kernel, if that makes sense, um yeah.

A

B

The same with linux security modules, you.

A

B

You're, not necessarily uh you need to have some code within the kernel that allows you to do those actions, and so I think it does from you know, from the perspective of linux, kernel features being added, knowing that you can build a subsystem with the idea of users using evpf to then make it what they need to be. You know like to extend it the way they would like. um You know the the linux kernel.

B

If you're previously without ebpf, you had to be very opinionated on what things were, how things were going to operate, but now it can have a you can have the framework, and then users can write their security rules or all the rules, for example, for the kernel, security, instrumentation, runtime and determination, um and uh you know, use them as they need as they see fit.

B

A

Cool, so we've got about 15 minutes left. Is there anything that we haven't touched on that you want to?

A

um We talked about the cons. We talked about. Oh, what has been done to harden the ebpf subsystem. We touched on that a little bit yeah.

B

So you know limits.

A

And stuff like that, instruction.

B

Call limits um the you know, I think, from the from the beginning. uh You know, ebpf is uh one of those acronyms that doesn't really have any meaning anymore, because it kind of we're we're talking about the whole ebbs sub system and all these platforms and things when we say uvpf, but ultimately it's extended berkeley, packet filters um and so berkeley packet filters and ip tables have been around for a while, um and so uh you know it's I think it's.

B

It is secure through the simplicity um in that, in that you can, uh you need to have a constrained programming, language or instruction set, and once once you have that, then it allows for the verifier and static analysis to be much more effective because we're not exploring you know a million different paths.

B

um All all the different paths in your ebpf program are essentially walked to look for nullity references to make sure that you're. You know it's there's no way that you're going to dereference a null pointer and also the kernel, provides a lot of evpf helper functions and that's usually what's used to get data. So you're using the helper functions to then interact with the rest of the kernel within edpf.

A

That's cool okay, yeah.

B

A

um Folks have already asked about psyllium. I've dropped the link to the psyllium page. For that.

B

uh So I I would definitely want to mention that the ebpf uh dot io organization- um just actually, I think it just finished like an hour ago- had a summit yesterday and today is the second summit. They had it uh last year for the for the first time um and uh for people that want to know more particularly about you know the lower level details of epf, but also just different use cases, definitely check out the ebpf summit. uh It's free. All the videos are online.

B

That sort of thing um there's a lot more information there um and it's find a link, but it's.

A

B

um And you know, there's an active community and kind of like I said before that you know, depending on what persona or what role you have in your organization. Ebpf may your interaction with ebpf may be either debugging or you're just using a platform that happens to be built on ebpf.

A

Right, yeah, that's a good point: the I'm looking at some of this schedule. It's intense, um but you know typical suspects, liz rice, janna, dugan, all the fun people, tabitha shale, sable, um yeah, all the low-level folks. I know that know everything about syscalls are speaking at ebpf summit, so that's cool.

A

um We also have or we're a diamond sponsor for the cloud native security conference coming up at kubecon. Are you part of that or.

B

A

I'm not playing in any way.

B

I'm not planning on presenting.

A

B

Yeah, I will I'm sure I'll be involved, one way or another um yeah.

A

There's just to read a little bit: there's a hosted uh capture the flag, learning experience that will run in parallel to the event for attendees to learn.

B

A

Practical attack, defenses and techniques.

B

Yeah, I think those are those capture the flags and those hands-on labs and demos are such awesome ways to learn.

A

Yeah, the the capture flag demo that was done during kubecon eu was just fascinating to me um and, like the second, we found the problem because we were running out of time, and I say we because it felt like we were all working on it together in the live stream. um It was just like. Oh there's, the problem, that's the naughty bits right there, let's get rid of those and off we went but yeah. It was a nice yeah.

B

You have a lot of those aha moments where.

A

Yeah, like you, can really get an eye-opening kind of mind-blowing experience from some of those. Sometimes.

B

A

Yeah you can easily stick something on a lxbox and make it do bad things. Yeah.

B

A

You know, I think, there's something.

B

Really exciting and cool to be able to uh you know, get information from the linux kernel and get you know get into these low-level details. It really kind of opens up your eyes into like what's going on so yeah those those are great great learning tools.

A

Absolutely uh anything else for the audience they should. You think they should know.

B

um So I I can share some resources. uh Yeah that'd be great, um so you know.

A

B

A

Kind of folks I'll give you a link.

B

Yeah dude no need to get your pen and paper out for this one. um The uh I kind of broke down some of these resources based on you know, if you're going to want to do ebpf development, um I think ebpef development is, is, is going to be.

B

You know most likely associated with some sort of platform right, you're working using epf within psyllium or that sort of thing, um but uh for scripting and debugging, and that's that sort of thing the um coop cuddle trace is is a great place to start, uh and then there are books uh brendan gregg is the resident uh expert at netflix on uh uh ebpf he's got a great book on performance and observability with ebpf and uh yeah.

B

There's a lot of platforms, um uh red hat, advanced cluster security for kubernetes we use evpf, falco um aqua has some ebpf tools, uh psyllium and so on.

A

Yeah now there's a wealth of.

B

Information available online and and, like I said before, I think it's it can be really overwhelming when you're getting started like looking at these things- and you know just to start with something simple like bpf trace and I think that's a good uh good starting point.

A

Yeah um I'm looking at questions here. Are there any chances? Red hat will become a member of the ebpf foundation? I do not know. uh I have no idea. uh Have you seen any discussion on additional stack, unwind support for ebpf.

B

A

B

Haven't, but that is always a pain point when looking at you know performance, performance, counters and things like that, where you want to be able to. You know, diagnose issues. I know that pixie has done some work on performance. You know kind of continual performance, monitoring with evpf and um I believe they've opened. I don't think they've yet open sourced their it's. They have a community edition, but um they have some interesting blogs about uh uh performance for profiling with ebpf. Okay. I can. I can share that after after the talk yeah.

A

Yeah that'd be great um yeah and that's so you know conan kudos says I hope they don't do too much more for ebpf. If the goal is to not be turing complete, then it needs to be fairly restrictive.

A

I mean: do you know of anything on the horizon for ewf? That would give folks reason for concern. So.

B

No, I mean, I think that language is gonna, stay relatively stable. um The instruction said I mean there. There are, I think, there's been recent work to add atomics um and also additional, like linux features to get data out of the kernel, and so there's there's several different ways to to you know you can insert a bpf program just using the bpf system call, but then to get data out. The program can write to a perf buffer or also there's bpf ring buffers which have um performant uh characteristics, and so I think, really the language.

B

The instruction set will stay relatively stable, but then it's all these kind of tools and other ways to interact with it that are gonna um change.

A

That makes sense, yeah and grow awesome. Well, cool. I don't have anything else for us unless you do or the audience might toss in something here at the last second, who knows, but this has been awesome. This has been very insightful, I'm glad you came on today. Thank you so much. uh Thank you to the audience for your questions and participation. This was awesome. um What's next I mean you know. Where do we go from here and you know.

B

um I think you're just gonna see some more more products and and tools powered by ebpf. You know.

A

That yeah and I dropped the projects link, and I noticed that, like some of those are newer projects um yeah that have come along in the past few months, so I expect to see a lot uh going on in this space as more people become aware of it right like having the second bpf yeah.

B

That's a it did, and it just really fits well within, like you know the microservices architecture and like where.

A

B

Have we have all these ephemeral containers and clusters in our in our environments and you you need a kind of a way to slice through you know not not horizontally, but vertically. What's the user space program doing what's the kernel doing um and uh ebpf is kind of like the the the best way to do that right now and uh so the the I think, like I said, there's going to be more and more things built on bpf.

A

Awesome well robbie. It was great having you on. uh Thank you. Yeah.

B

Thanks for having me chris.

A

Hey no problem, um and we do these- how I forget how often we do these hang on a second. Let me double check. uh Please, like obviously, please come back for the next one, but the next one is.

A

I forget. um I believe these are monthly, so.

A

That is incorrect, but uh so here's a better idea subscribe to our calendar and you'll, get all the fun links to all the programs that are coming up on the show or the channel and uh yeah highly encourage you to do that. So you can keep up to date with all the latest and greatest cloud native fun things we're doing here at red, hat, um yeah, so robbie without further ado. Thank you very much and y'all stay safe out. There see you soon.

A