Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tutorial: Using BPF in Cloud Native environments - Alban Crequy & Marga Manterola, Kinvolk

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tutorial: Using BPF in Cloud Native environments - Alban Crequy & Marga Manterola, Kinvolk

Kubernetes provides a high-level abstraction layer that makes it easy to deploy distributed computing resources without knowing what’s happening in the kernel and applications. But when debugging, Kubernetes does not provide any help in inspecting these low-level details. We showcase the following tools specifically designed for running on Kubernetes: - Inspektor Gadget, built on BPF Compiler Collection (BCC) and traceloop - kubectl-trace, built on bpftrace These allow devops teams to answer debugging questions such as: - What were the last system calls executed before the crash? - Was this function called? With which arguments and return value? - Which TCP packets were retransmitted? - Which queries run slow? - Was this file opened? You'll need access to a Linux machine and a Minikube installation. Please follow the instructions at https://github.com/kinvolk/cloud-native-bpf-workshop

https://sched.co/ZejN

A

Hello: everybody uh thanks for coming to this tutorial about using bpf in cloud native environment.

A

My name is albon, I'm a co-founder and director at kinfolk labs. We do consulting and engineering around open source projects related to linux and communities and at control clubs. We are a team, dedicated team focused on innovation in our open source projects and also collaborating with other software company and upside projects.

B

Hi, I'm marga. I also work at kim folk, I'm an engineer in the flatcar team. Flatcar is a container optimized os which can be used to run inside kubernetes if you want, but I'm also interested in other things. So I joined alban today for this workshop on bpf, because I found bpf a very interesting tool for debugging applications.

A

Thanks um so I mentioned keyfront labs before atkinfolk we had to build 100 percent, um open enterprise cloud native stack. It includes a flat card container linux or linux disk software containers, as maga mentioned on locomotive or kubernetes distro, but everything we do uh show in this uh talk is about um things you can do um with other linux distribution and other kubernetes distribution. So it's uh generic technology is not tied to.

A

B

Yeah, so uh this is a hands-on tutorial, it's a workshop where we want you to follow along and do the the exercises, the demos that we will show on your computer. So uh we want you to please now clone this repo and and get started if you haven't already.

B

Hopefully you have done this already, but if you haven't, uh we will cross our fingers that everything works out of the box and then you can get ready. The workshop uses a mini cube. You can also do these things with other kubernetes distributions, but the we've optimized everything so that everything works out of the box or hopefully works out of the box. So we recommend that you follow along using the tools that we provided, for example, for minicube.

B

We are providing an image that already includes the kernel headers, because for using these tools you need to have the kernel headers, so um yeah and and then you have to install the rest of the tools. We will do this along with you uh all the different steps, but we want you to do this on your computer.

B

You can also do it on a vm. If you want to use a vm, there's a few caveats that you have to take into account, but you can do it on a vm all right, so we will start with our first hands-on task, which is to get mini, cube, running install it if you haven't installed it, and for that I will share my screen.

B

Okay, so we have already cloned the repo in this cloud native bpf workshop directory.

B

Hopefully you have cloned it by now and we will go into the getting started directory that has a startminiqueue.sh script. Let's have a quick look at the script uh to see what it does before executing stuff. It's usually a good idea to check what things do so this script, downloads mini cube, then verifies the hash sums and then it will ask you for confirmation, because it will delete any mini cube that you have running.

B

If you have it on your machine and then uh there's two options: either you pass this use, drive or non parameter, which is only if you are running on a inside a virtual machine or you don't and then it it runs with virtualization.

B

Ideally, you want to run with virtualization it's a better idea, but if you are running this inside a vm, you want to use the use driver non-parameter, all right. We will now run this start mini cube.

B

So this is now downloading the mini cube image that we prepared for you with the uh with the headers, and now it downloaded the image, and now it will start mini cube, and this can take a while. So while this is running, uh we will go back to the slides and go back to uh explain a little bit. Why we're here and why why this workshop even is necessary?

B

uh Someone asked for the git clone url again, so I will go back quickly to the url. The this is the git clone url, and one thing I wanted to mention on this slide was that we also have a slack channel for this tutorial. The slack channel is in the kubecon. Slack is 2-cubecon tutorials bpf, so in the channel uh feel free to ask questions or like answer each other uh inside that slack channel all right, so while mini cube is installing. Why are we even here? What's what's all this about?

B

So this is about debugging applications and in particular, debugging applications inside kubernetes, debugging in general is hard, but debugging inside kubernetes is even harder because, like everything is running somewhere and we don't know, the pods are communicating with each other and how do we even figure out what's going on the bpf tracing tools, are really good to help us figure out what's going on, but the problem is that ppf tracing tools that already exist out there were thought to be run on a machine and investigate what's going on inside the machine.

B

So the tools that we will demonstrate today, inspector gadget and cube ctl trace, are plugging this gap of like how do we run this super useful bpf tools inside kubernetes or with kubernetes, and to explain a little bit more about that. Alvin will tell you more about bpf and how bpf works.

A

A

So bpf stands for berkeley package filter. Maybe you know about that? If you are on this talk, but I will give a bit more context, um so it takes for quite a while. Initially it was for tcp dump that was used to uh filter network traffic on a road socket so that tcp dom can be performant.

A

And can you can specify filters on to say only want to get http traffic, for example, or only dns traffic or arbitrary expression like that and to make it performant they use bpf to execute the filter actually inside the kernel.

A

So instead of tcp, then getting all the network traffic in user space in the discipline process and then filtering there instead of that um it create a bpa filter, put it in the kernel and the kernel will only wake up the dissipating process only resists necessary for the packets you want.

A

That was the initial use case in meanwhile.

A

Recently, it developed to a lot of more use case, their use case for networking where a different, socket filter or you can do traffic shipping use case for security, where you can have a site comp where to filter the system calls you want to do or more recently on linux security modules and, uh lastly, there are use cases for tracing. So that's more. What we are focusing today, where we can uh use bpf for guns to uh hook into trace, prints or k-pops or eupops, and so on.

A

So I can explain a bit more into details. What eppf is this extended version of vpf? So the workflow looks like that. At the beginning, you write a bpa program, usually in a language like a c, you can compare that with c local lvm into your bpf by code, and then this bpfpi code can be uploaded in the kernel with the bpf system call and then the first thing the kernel will do.

A

Will it will verify the bpa program to see if it is safe, but it means it, it will make sure it's not possible for the bpf program to access any random memory in the kernel if it's not safe or corrupt your data or anything. So it's not like your name modules. Where is dangerous using bpf program? Doesn't risk crouching your channel, so this ppf system call will return a file descriptor. That represents your bpf program, but so far it doesn't do anything until this bpf program is attached to something in a kernel.

A

It can be attached with k ports to some kind of function as to on a network stack, a different layer, and this ppf program can make use of bpf helper functions to execute some things outside of the close sandbox.

A

There is another concept called bpfmaps bpf maps are kind of like a global variable that are accessible from your bpf program and accessible from user space applications. So that's a mechanism for ppf for grants on your user space application to communicate together through this bpf.

A

Maps, so there are varieties of tracing tools on linux that use bpf. I listed some of them here. On the left side, there is bpf trace, which has a custom language to write your bpf programs. There are bcc tools. Bbcc is a really useful repository with a lot of examples how to use bpf.

A

So that's a good static plan to learn, bpf, to see the collection of tools and try to execute them or to read how it's done by reading the source. There are other tools like trace loop. We will talk about it today and other one, but most of those tools are more designed to run on one computer.

A

So it's not really cloud native with something you can do on the kubernetes level. For that there are tools like cube, ctl trace, that's a capacitive plugin you can use on kubernetes and internally. It will use bpf trace on the work or not, and there is inspector gadget that will use tools from bcc from traslip and so on, to be able to use those tools at the kubernetes level as well.

B

Right so now we will move on with the next hands-on task, which is installing inspector gadget.

B

Hopefully your mini cube installation is done by now uh and if, if it, if you have run into any problems, remember asking on the slack channel, it's kubecon tutorials bpf, two kubecon tutorials vpf, a few people are already asking questions there remember to ask questions there: okay, so the next handsome thing that we will do is uh there's another script, which is called get inspector gadget, which first checks that you have qctl installed and that it's working and then checks whether you have it already installed like you could already have inspector gadget on your machine, and so it doesn't do anything in that case, it then uh tries to install it with crew if you have crew installed, because that's the recommended way.

B

But if you don't have crew install it downloads, the latest release for you and installs it in the same path as cube ctls.

B

So let's now execute that in this machine it was already installed. So it didn't do anything, but in your machine, it's probably now downloading and installing the gadget. But this is now just installing the plugin, and what we want to do is to now install a pod in our cluster.

B

So first we will check that it actually works by running cube, ctl gadget help, and we, when we get the help, uh we see that it has a bunch of sub commands that we can use, and we will show a few of those subcommands in this workshop uh and now that we know that it's running, we need to deploy a gadget. So how do we deploy a gadget?

B

There's this very handy, cubectl gadget deploy command that we will first inspect before we run it. This generates a yaml that says all the different uh service accounts, cluster role, binding and demon set. That will need to be created to run our gadget pod in our cluster. So now we will actually apply it doing. Q ctl apply dash f dash, okay, so this created the three resources and we can check.

B

It's container creating so it's creating. It will take a little bit until it's done with everything we can check, maybe now yeah. So now it's running so we have the pod and it's deployed uh in our cluster and we can maybe check the logs cube ctl logs cube system.

B

And then this is kubernetes up gadget.

B

And yeah we see that uh it launched and everything was successful, which is great news all right, so we have inspector gadget in our cluster. What can we do with it? Let's go back to the slides.

B

The first gadget that we're going to look at is called network policy advisor. It's uh the use case for this gadget is when you shine a project, and it turns out your project has a lot of boots a lot of services. They talk to each other, they transmit data, but there's not network policy, so everything is open and if something malicious, what's going on, there's like no security whatsoever- and this is a bad idea and you want to deploy a network policy, but you have like no idea of the architecture you're new to the project.

B

So how do you even start thinking about this architecture?

B

And if you care about this idea of like it's, not a good idea to start putting the security at the end, alvin actually has a talk later today, almost like right after this workshop uh called pod security as an afterthought, where he will cover these issues and more, and so this is the problem, and this is what a network policy can help us with, and we will see what this means with an example.

B

In particular, we will use an example that was generated by google, which is a microservices demo with a bunch of different services for like a an e-commerce site, so it has a database. It has checkout service shipping service like it. They just like. They are all dummy services, but it can help you give you an idea of what a micro service application looks like in kubernetes, and we will use this example to try out the network policy advisor all right. So with that, I will share my screen again.

B

And we will change to the zero one directory in this directory. We have a yaml file and a readme in the yaml file, there's a bunch of different things, so we can, for example, grab for the different kinds of resources that are there. We see that there's a bunch of deployment a bunch of services if we try to make sense out of this, it would take a long time. So, instead of doing that, we are going to use the network policy advisor.

B

So we will do cute ctl gadget network policy monitor because we first put it in monitor mode and then we will say name spaces, and this is important. It has to be plural, otherwise it doesn't work and then uh we will output this into a file called networktrace.log.

B

So we are starting a monitor in the demo namespace and we are saving this in the networktrace.log file.

B

Now I will split the screen and in this other side of the screen, let's change again to this directory, I will actually start the deployment. So first I will create the namespace create namespace demo, and then I will apply the yaml file.

B

Okay, all these uh pods are now getting created. It will take a while for all the pots to get created. uh We will now see that it says container creating for most of them. We can check that our network trace.log.

B

Should be getting input soon, but maybe not yet because all the containers are creating, but okay, it's getting input. This is good. Now we will let this run and in the meantime, I'll stop sharing and go back to the slides and I'll. Tell you a little more about what the network policy advisor does so first, these were the comments and and remember they are all documented in the github repo.

B

So you can go to the github repo to to check them out and check a bit more explanations about them, and then uh this is a diagram of what's going on with inspector gadget, so inspector gadget is a cube, ctl plugin. So the first thing that happens when you type cube ctl space gadget is that it turns into a cube ctl dash gadget, which is the the executable and it only talks to the kubernetes api. It doesn't talk to the node where the the pods are running it, everything it does.

B

It does through the kubernetes api. So the first thing we did a little bit earlier was to deploy the gadget part, and that was done through the kubernetes api, and then we asked that gadget pod to install a vpf program in this case. In the network policy advisor case, it was the the vpf program to monitor what was going on on the network, uh and all of that is done through the kubernetes api inside the node. Our gadget port is the one that talks to the kernel.

B

It creates the vpf program necessary for whatever action we are asking of it, and then it feeds this program into the kernel and, as we later explain more about what's going on in that stage, but this is the basic idea of what's happening of what happened right now when I type network network policy monitor all right- and this is the log that we are generating- I show a little bit of what the log was looking like when I started generating it, and uh it has basically two types of entries accept and connect, and these are like the tcp actions of accept and connect and it stores uh the ips that were involved.

B

The ports, the pods and all this information, of course, this log is very verbose and it has a lot of information and it wouldn't be nice to parse it uh as a human like it's hard to make sense of all of these information. So of course, there's a subcommand that helps us make sense of this information, so the the report, sub command, will parse this log and generate a yaml file, which is a basic network policy file it.

B

It's not the idea that you just use this yaml file and deploy it in your cluster and forget about it. You should look at it and think if you want to add things or remove things, it's it's just a basis, um but it's very helpful to have it to not have to think of all these policies on your own.

B

Alright. So, hopefully our pods have deployed by now, and we have.

B

We can generate the report.

B

Let's stop this tale and yeah there's one that crash hoops, because uh the recommendation service crash loops because it requires like a service account and we don't have any service accounts, but the others are all running. So with that we can actually stop the monitor.

B

With control z- and we can instead call the report and for this we will pass dash input and the file we generated, which was networktrace.log and then redirect this to network policy dot yaml. So this will generate a yaml file with the network policy, let's close the other window, and we can now look at the yaml file and this is going to be complex because the service is complex.

B

But basically you can see that there's ingress and egress for each of the different services involved and the labels that they match the ports that are used, etc. So, as I said, this is basically a basis of what you want to then customize for your actual service, all right uh that was it for the network policy and now alban is going to show you a few more interesting gadgets.

A

Thank you so now I will talk about trace loop, which is the next gadget using bpf that I will talk about. um So to summarize, I have this sentence. It's about tracing calls in c groups using bpf on overwatable wing buffer. That sounds quite complicated, so I will go through that and show what it does so. The use case I want to show there is something similar to strays.

A

So I really like to use trace as a developer to debug my applications, but when I use it on the kubernetes level, trace has some problems is um he can be slow? uh It means it's not possible to use it for all my parts in production all the time, because that will make everything too slow and although sometimes I have the use case where I have some unexpected crashes and then when something has crashed it's too late to use trace because the process is not there anymore. The part is not there anymore.

A

So there is nothing to use stress on so here the come. The idea to use something like a flight recorder so to use bpf to record the system, calls used by applications, but instead of printing them all the time. I just store them in a ring buffer, so there will be a buffer for every pod uh that constantly record the events and it uses a writable ring buffer. That means that when the buffer is full, you just override the older standards and so on.

A

In this way, we can use that to debug later.

A

So, if I compare stress on trace loop, that's quite different stress use the system called ptress to be able to capture the data twice loop use bpf on transparence with stress you can capture the data on one or several specific processes and on treslope it's uh is done to capture events on the at the level of c groups on stress. um It can be slow because for every events it need to have some context switch between the tracy and the transfer for every system calls and twist loop doesn't work like that.

A

It has a ring buffer that is only written and so they're just copying something in memory without actually having any context switch between processes. But stress is uh as a good points too. It's it is cycle loose and reliable. It cannot lose events, so you, when you see the logs from stress, you necessarily see everything.

A

It doesn't forget, parameter or something press loop work in a synchronous way and can lose event in some case. So but it's good enough to debug most of the case so event, 2 is not. It doesn't guarantee that you get all the parameters everything.

A

So if I go into a bit more detail um twice loop, a similar interface to stress it, gives information about system calls, but it uses ppf program on the transferences enter. That means that every time a system call is executed on your system from any process, it will execute this transport and execute the bpf program, and the first thing this bpf program will do we'll check which container it is about.

A

So it can, in this example, use the groups to look at the secret id to know which container it is or it can either use the name space id and then, depending on the container, it can redirect the execution to another bpf program, one for each different container that has been configured and it will output. The data about the system call into a ring buffer for each container, and this ring buffer is written on overwritten in loop.

A

But it's never read unless the user explicitly asks for it, for example, if they want to debug the application, if it does crash, for example, and that in this case the data is retrieved by the daemon set where trace loop is running and the user can see it with a custodial gadget.

A

So now it's time for another hands-on task. I will share my screen again.

A

Here it is, and I will go back to the parent directory with that workshop directory.

A

From here there is with me that you can read from github or I will run commands directly there. So we have this cube. Ctl gadget right slope come on what it does is.

A

Add some sub command on one of the subcommand is to list the trace that is recording, so I will capacitor get that transfer list and it it's a bit like a cl. You need to specify the chain uh default for by default, to use distance space or touch the packets a to see all the name spaces from here. I see quite a lot of parts that markers executed in a previous example from the demo namespace.

A

Let me go to the top and we can see that uh those trace is from a part that has terminated four minutes ago, for example, for this one, but it is a stress: loop still have a record from them, so it is a ring buffer with the last system calls that can look at them to see what's happening.

A

So I will take the trace id here and I will do a cup still gadget twist loop show oops like this.

A

A

Show- and here I get the last few system calls that were executed by this part before it has been clicked, so I can go a bit higher and you can see a lot of critex commands and some other system calls from the java process.

A

Okay. The second thing I can show here is: um I will run.

A

Example, we have one pod that I will start until it is executing multiplication and then saving the result into this file and then attempting to read this file, but because there is a bug in that script, uh it might not work as expected. So let's see that I'll copy paste this from guitar.

A

So here I get error message: it cannot read that file, so in this case I will use press loop again to attempt to debug that and to create the result.

A

Gadget press loop, this see that even if my pad has terminated, I still have the trace.

A

So that's this multiplication part that turned on can even delete it.

A

And when it's deleted, I still have press not remain here and I can show um this trace.

A

um So here it is so maybe I can ask the audience to see something. Do you see what's happening.

A

A

Okay, so we will go through that um here it say it's opening attempting to opening this file, but this file doesn't exist and if I look a bit above, I should see another.

A

Here on this line, but this is a different file, that's why, um when I write with this file, I try to read a different file. That's why it doesn't work.

A

I can. We can also see here this read system call on the white system column. That's the bc program that receive on a standard input the multiplication to perform and to write the answer. So in this case the answer is 42..

A

Okay, so everything that was here is in this repository. So if you did not have time to do everything, you can do it yourself later, I will stop showing my screen.

A

Go to the next slide, so here there are the commands that are executed, cuba, ctl gadget, dress up list, and then this one I don't show if the pod is still running you can actually, although see the last system calls at that time.

A

But if the pod has been destroyed, then the ap api server doesn't know about it anymore, but you have to use this show command to be able to see that okay.

A

And there are more gadgets in inspector gadgets and a lot of them are reusing tools that are inside the bcc repository, so inspector gadget is kind of relying on the shoulder of bcc to do a lot of things. There are a lot of examples here.

A

Buying snoop, for example, come from bcc, exact snoop as well, and so on.

A

There are snoop tools that are detecting when there is a new exact system called detecting when there is a new open system called to open a new file and so on.

A

um I will show a bit later, some example with exact snoop, um so on this um bcc tools like exact snoop and investment, um usually the usual experience we want is to trace parts in general. We user don't really care about specific process id.

A

When we work at the kubernetes level, we don't want to know about the pid or exactly which node it is running on. We want to use kubernetes concepts like the pod or kubernetes levels or namespace, and so on other, when we I could get information from different process from different parts, and we want to upgrade them aggregate them using kubernetes levels.

A

In summary, we want a user experience like cube ctl, where developers don't need to use a section or not, and they don't need to know about the internal details about pids and so on.

A

So to come up with that, we have this user experience with, for example, the exact snoop gadget where we specify which port to trace using one of those filters. You can filter by kubernetes levels like this or using namespace or the pod name or the node, or if you have um pods with several containers inside one part you can, although select, which container you want to use all those labels, you can use one or several of them. You can make combination or just one. If you want to.

A

We'll go to next slide, um so this kind of filtering is really complicated to do, because, um when you uh use cubesat gadget, uh exact snoop, for example, there might be zero or one or more part to trace, and that's how this set of parts to twist uh change with time, because part can be created or destroyed over time, because it's a dynamic system and so different tracer can have different part to twist and that's charge of your time. So, let's make it difficult to implement.

A

In this example, uh it uses oc gadget. This can also come from openshift. That's the cli tool for openshift. So if you use openshift, you can also use inspector gadget as a plugin for oc. uh If you use scriptctl, you can use cubettl gadget as well. Both works the same way.

A

So how does it work here? It works with a component that runs in a daemon set called the gadget, tracer manager, and this demand running on all nodes, and it has a simple grpc api with four meters.

A

So the methods are to inform the gadget faster manager of new containers. Every time a new container is started, it knows about it, because our container is called every time. There is a new twister um exact snow, for example. That is started. It knows by this little call the the add container remove container methods are called by the ocr hooks. So I will explain a bit what is horse here hooks um when you have a kubernetes cluster. The parts are started by the curbide through the container runtime on the container on time.

A

In the end, my, for example, execute run c and run c and other container runtime of this ocr hooks where you can execute arbitrary scripts or commands every time it starts a new container or every time it stop it.

A

Through that mechanism, we execute a command that will uh call the grpc method to the gadget source manager so that it knows about all the containers on their kubernetes levels.

A

On the other side, we have a cube, ctl gadget when it executes a gadget, it will run it on on the gadget pod and it will use this bcc rapper shell script. That will tell the catcher tracer manager about the tracer, so it knows which kind of uh node or label or it wants to select.

A

So the gadget tracer manager knows about the container. He do knows about the tracer and can use this information together to know which tracer should trace, which pole the next thing it does. It will update bpf apps with that information, so there is one bpf map for every tracer that will contain the list of parts that need to be traced by this tracer, and if that information change, it will automatically update the ppf map. Accordingly,.

A

On later, when this bcc repair script executes, for example, execs loop from the bcc project, it will use a bpf program, the in this case installer caprob, and it will use the bpf map in order to know whether it should trace this container or not.

A

Okay, so now it's time to.

A

Share my screen again.

A

So I will go to the next.

A

A

And as usual, there is a real me, a file that you can read on github as well, and I will show two things here. The first thing is, I will execute this exact snoop gadget to filter to get the information on one specific pod.

A

A

A

Okay, so here I executed exact, move for all the pods in the default space with this label. But if I look at that, there is no currently.

A

No parts with that level, so it doesn't print anything next thing. I will start a new pod with that label and I will use this command curl, something by bash, which is a maybe a bit of an anti-pattern but useful for this demo.

A

So I will run in this part into in the interactive mode and show what it does here.

A

So first it has to download the fedora image, because I use the fedora docker image here and once it executed, it run some script, but at the bottom, thanks to exact snoop I can see which command it runs so mkd here, rpm, rm and so on. So just finish to execute it, but exact, smooth, captured everything here at the top.

A

Rpm with this european package and so on.

A

I can't stop that and the second.

A

A

I want to show on this one: maybe you can do together at the same time just deploy uh ng-mix application. So let me show you what it contains.

A

You can edit a configmap with the files for a website in front of my batteries, and it contains a coughing map with the configuration for nginx. So you see nginx configuration and then a deployment file which um has three replica of nginx and it will use the configuration from um that's specified on the website uh in the coffin gap that is displayed with okay.

A

So, let's deploy that.

A

So it creates the configmaps the deployment on the services, so you can access this engine x from outside of the kubernetes cluster through the services. In this case we use minicube. So I will use this midi cube service command in order to.

A

A

Directory so I use this minicube command in order to proxy the request through this url. So if I call this, hopefully I get something you can either use firefox or another browser. If you want here, I get a 404, but then I want to either use this yellow.txt file that I put in a coffee, mac and hope to get it.

A

Unfortunately, I get the 404, because there is a bug in my deployment general and here I want to use um inspector gadget open smooth in order to debug that and see what's happening. So I will give one minute for you to try.

A

A

It's happening continue that and I will use.

A

I will use, um I will show how to use inspector gadget open scope to debug this, so ctl get pod here see. There are three deployments. uh Sorry, three parts from a deployment. It means that every time I do a curl uh command, I'm not quite sure which part it will reach.

A

But if I do show labels, I see that all of them has this app equal nginx level. So I can use this filter in in factor gadget to filter on the parts with that level.

A

And snoop, I will say only the namespace default and I will select on the label app equal nginx.

A

You can ignore the running at the beginning and look at the output.

A

Let's see what happened when I do curl here I got a 404 and I see that at the same time I see that nginx attempted to open this path. Slash, tc, slash, nginx, html, dot, txt and it fell. It returned the error minus one, which means it failed to open this file, um so that gives a hint what's happening it attempt to with this file, but we can check uh if we when we installed this cropping map.

A

Nginx data feedback contained this file, but maybe we mounted it at the wrong place, so that gave her a hint okay. This coffee map is not, in this volume the henchmix dodge volume.

A

Let's see it's mounted in slash bar www, but engineering's expected to have it in this path. So that's a mistake.

A

It doesn't match the configuration of nginx and where I mounted the volume, so I can use cryptitel edit.

A

I ng mixed implement.

A

And find this month on three: let's add with okay.

A

Okay, so now it should hopefully restart the nginx parts. You can get powder.

A

See that kubernetes is stopping the whole parts and stamping new one, and let's see what happens if I do a call command again this time I get this uh could reply.

A

You can see it it's getting. I don't get a 404 anymore, okay, so let me stop the chair and we'll go to the next slide on pass like too manga.

B

All right, so, up until now, we've been looking into inspector gadget, which was one of the tools that we said we would look into, and now we will look into another one which is called cube. Ctl trace, cube, ctl trace is a wrapper around bpf trace. Mbpf trace is a command that lets. You create bpf expressions using a domain specific language so that it's easier so that you don't have to write the c program and compile it and install it.

B

So all of that is done by the bpf tool, and the thing you need to write are the bpf expressions. They are a bit complex, so I'm going to show a few vpf expressions and it you might get a little bit scared, but once you see a few examples, it becomes easier to understand what's going on.

B

So this is what the vpf syntax looks like. uh This is just the bpf trace command and we will talk about cube ctl trace after we understand a little bit. What vpf trace expressions look like so as this slide is showing vpf expressions are divided into three parts: the probe, the filter and the action. The probe is the thing we want to trace so in this example, here we are saying we want to trace the kernel function called do nanoslip, so anytime that kernel function is called. Our tracer will get activated the filter.

B

As the name says, it basically allows us to filter the tracing events for some criteria. In this case, the process id needs to be higher than 100, and the action is what we want to do whenever this probe after the filter gets hit. So this is the most complex part of the syntax.

B

This add sign with square brackets means that it's creating a hashmap and the key in that hashtag hashmap called com is basically the command name that was executed, so the command ended up. Calling this do nanoslip kernel function will be there, it could be bash, it could be ls. I know whatever is calling this kernel function in the end, that would be the command and the plus plus.

B

Is that, basically, we want to count an increment, and so, at the end, when this is finished running, we will have a table with all the commands that were executed and how many times they called do nano slip on the next slide. We will see a couple more one liners like this.

B

Instead of using the kernel function that we saw in the previous slide, these ones are using trace points. Trace points are points that kernel, maintainers and module writers have identified.

B

There are good places for debugging and there's a lot of trace points, and you can use bpf tool to inspect what trace points are available in your kernel, and the good thing about trace points is that they tend to be more stable, there's no guarantee that a kernel function will not change its name or the parameters it takes or the returns values, uh but trace points are kind of stable.

B

They could still change but less, and so, when there's a trace point for something that you want to trace, it's a good idea to go for that, and in this case we are both. Both examples are showing syscalls. So this is the trace bonds for a syscall is starting.

B

In the first example, we are using a similar hash table that is just counting how many times each command executed at different cisco in the second one, it's using a little bit more complex, it's counting all of the ciscos that were called, regardless of which command called them and then once per second, it's printing that number and clearing the the stack so that it in the end like what will happen is that once per second, it will print how many syscalls were called that.

B

Second, um and so these are just a couple of examples in the website that is pointed there, and it's also in the github repo uh there's a lot more examples, there's a huge reference that you can consult and in in our exercises. We will see a few more examples, uh but of course like this is super flexible and you can do a lot of things, uh but you don't need to understand all of it just to start trying some things out.

B

So if you get an idea of what's going on, then then you can start using it and you can like mix and match things from different examples, and so that was ppf trace. What about cube, ctl trace so cube, ctl trace is the way of running ppf trace commands inside kubernetes.

B

It doesn't have this handy, namespace, selection or app label selection. You need to tell it either the node or the pod and the pod needs to be like the specific part. No, it can be just like pods that start with this. You have to give it a specific part for it to run, and you can give it either either an inline expression or a file. So if your ppf script turns out to be very complex, you probably don't want to have like a three line: wine liner.

B

uh Then it makes sense to have a file where you can add comments and and make it a little bit more readable.

B

Okay and now we will actually see this in action, as we mentioned earlier. It's a good idea to follow, along with the the github repo, because everything is documented there. If, if you are not sure what we are executing check out the comments that we documented there all right uh is this still running, I think so right.

B

Yeah, so let's stop this nginx deployment.

B

Yeah and let's now switch to zero four.

B

B

Yeah all right, the first thing we will do is uh create an alias.

B

So the tricky thing about both inspector gadget and cube ctl trays is that they need to have access to the kernel headers, and this is why we gave you a mini, cube iso that has the headers and and that why we are using this like special things, it was trying to like simplify getting the headers, because if you don't have the kernel headers, then you can trace and so qctl trace doesn't have the compression the decompression mechanism to access the kernel headers in our minicube instance, and that's why we need to give it a different container where it will have this xc executable, and so I'm going to copy and paste you if you are following along, you should copy and paste from the github repo this alias.

B

So what I'm doing here is I'm copying the I'm creating an alias that is called qctl trace, run that basically calls qctl trace, run with this long parameters and then every time I will call qctl trace run. It will pass this parameters so that it can access the kernel headers.

B

Okay, so we have the alias, and now what we need to do uh is know the node name. In our case the node name is minicube, but if you are running uh on a different platform, if you're running inside a vm, uh it might have a different name. So we need to know that now name where we want to run the expression, and so now I'm going to copy and paste this expression.

B

So uh what I'm saying here is, I want to run this trace in my note, called mini cube and it's going to be a cisco trace, and what I'm doing here is a little bit different than the one I showed in the slide. What it's doing is it's tracing all the syscalls and then it's creating a hash table with the probe, which means the cisco as the entry and then it's counting how many times it's called okay.

B

So we are going to create a hash table that has all the ciscos that were called and how many times those ciscos were called.

B

Oh sorry, I forgot to actually install qctl trace, so here there's this getcubectltrace script, which looks very much like the inspector gadget script that will install the plugin for me. Of course, you can install it uh yourself. uh This is just a handy way to save time and it's installing yep. Now it's done now. Let's try the other command again.

B

Okay, so it says that it created a trace and we can actually list whichever traces it has right now. It will have only one trace, uh but as we try things out, this list will get longer with all the different traces that qctl trace is collecting and to actually interact with the trace we will attach to it. I will copy and paste this one.

B

uh I could also copy and paste this one, but it's annoying because it's mixed together with the no name so, okay, I'm attached to the trace and now the kind of trace that I executed only prints something at the end. So in order for me to see anything, I need to tell it to finish, and then it will print something. So I will press ctrl, z and okay.

B

So if this happens, it means it didn't work uh if it says signal killed, it means that it got killed before it managed to print the table and we need to try again so I'm going to try again.

B

So this is the trace I want to run. It says it's created and now I want to attach to.

B

It and now I'm going to press ctrl c again.

B

And this time it printed the table, I'm not sure what the other thing is clearly a bug, but I'm not sure what the bug is. Let's not worry about that right. Now, anyway, it printed a table. We see that there's a bunch of different ciscos that are being called.

B

uh We see yeah a lot of differences, calls uh I'm interested in the right cisco, like what is being written like I stopped everything, there's nothing really going on in my cluster. So why is anybody writing anything? So I want to see what why why there's a right call so for that I will.

B

I will edit my previous trace and instead of having a star here, I will check the right syscall and instead of counting per probe, because this will only be right, I will count per command, so I will see which commands are writing and to avoid this having to like start the trace and then attach myself I'll use the attach command, and this will just let me stay attached when I run it so now the trace is created, it attached, one probe to the kernel.

B

So it's now gathering data of which programs are writing stuff, and I will now stop it and it should print a table yeah. So we have a table of a bunch of different things that we're printing cubelet. Writing. Cubelet is the one. That's writing. The most hcd is also writing a lot uh and yeah. We could inspect why? Any of those are writing, but I mean yeah. They are writing stuff, uh but let's check, for example, out of those uh let's check one of them so to check one of them. Remember.

B

I said that there was this filter part like up to now. The examples I showed had no filters so now I would add a filter and in the filter I will say I want to check if the command is core dns.

B

So if the command is core dns, then this will match, and now I don't want to count stuff. I know that it's writing and then it's it's core dns. So what will I count? What I wanted to actually see? What is it writing? So to do that, I will do a printf.

B

And then to the printf, I'm going to pass the arguments of the trace point and from the arguments I'm going to select the buffer.

B

So with this, what this will do is it will print every time coordinates, writes anything it will print what it is that it's running that writing.

B

Okay, and so we see that it's responding to http requests and to most of these requests, it responds response 404 not found, so there is some service, probably a liveness check that is trying to check if it's working and it's responding for a four note file, which is kind of interesting. But anyway we see that this is what car dns was writing all right, so that was like the basic usage of qctl trace.

B

We are now moving on to the next example, which will use one of the files that we have here: the bash redline. So in this bash redline file, we have a script and, as I mentioned, because it's longer we have comments and- and we can format it so that it's easier to read, and so this. This is an example that traces bash commands, and here what we see is that it has this special variable called container pid.

B

That is the is replaced eventually. After a few calls, it gets replaced by cube ctl trace for the pid of the container. That is executing this, uh the probe that we want to match. Why does it need to get replaced? Because, here the trace that we are using, the probe is called you read: probe it's a user space function, and so it needs to access the executable, the bash executable that we are going to trace here. It needs to actually access the file and to access the file.

B

It needs to have the right inode and to be able to find the symbol for the redline function, and so that's why we have to put this container bit here and then cube ctl trace will replace it. Okay, let's, let's see it in action. We are closely running out of time, so I need to hurry up a little bit.

B

So the first thing I'll do is I'll split the screen and in this side of the screen, I'm going to start a test pod that will just have a bash command in it with an ubuntu image, yeah, okay and it's working. I have an ubuntu image and I have a bash command in it and now on the other port and the other part of the screen I'm going to attach to that pod and execute the bash redline.

B

B

So I'm going to do a cube, ctl trace, run and I'm going to pass the file that I showed and then instead of the node, like I showed before I'm going to attach to the pod and keep it running so my trace, it's created is waiting for tracing bash command and then on the other side of the screen, I will try things whatever I type here will get printed so and even if I type a command that doesn't exist, it will get printed, because this is basically what we're doing we're tracing what the redline function returns.

B

Okay, you might ask yourself: okay, this doesn't seem particularly useful. Why would I want to trace what bash commands are getting executed? Maybe it's useful for someone trying to hack my system, but not for me to debug anything, so, let's now uh actually debug or well see how we could debug in an actual kubernetes application.

B

So for that I will stop this trace.

B

And I the one of the other things that we have in this directory is this caterday yaml that has a deployment of a catterday application, and I will apply this card to the yaml.

B

Okay, it applied, and now uh this is an application that serves a website with uh cats in it, and so uh I will toward in order to use it. I need to use a mini cube service redirection, so I.