Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 50 Shades of System Calls [I] - Jorge Salamero Sanz, Sysdig

Description

50 Shades of System Calls [I] - Jorge Salamero Sanz, Sysdig

Transaction tracing is typically thought of something that only developers do when they need to troubleshoot a piece of their software. And lately, it’s also been used for tracing microservice-based transactions too.

These are really useful capabilities, but what if you could profile everything? Yes everything - software functions, microservice calls, file access, network requests, even bash scripts. How would this change your view on your systems? How would this enable you to better understand what your software is actually doing?

In this talk I’ll show you how to trace everything using Sysdig, an open source system visibility tool. We’ll cover:
How to trace everything from a method in your software, a service call, a network request, a shell command execution, a script, and more
What effective tracing in containerized environments requires
How to report on your traces to make the most sense of the data
Use real-world examples of tracing that show its benefits
When you leave this talk, it’s very likely that you’ll skip the next one so that you can go trace something!

About Jorge Salamero Sanz
Jorge enjoys monitoring all the things, from his Kubernetes clusters to writing sensors plugins and DIY projects with Raspberry PI and ESP8266. Currently he is part of the Sysdig team, and in the past was one of the promoters of HumanOps. When he is away from computers, you will find him walking with his 2 dogs across the countryside.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Well, thanks Harry want her coming. My name is Jorge I do bunch of different things, it's a snake. How many people know says like here: please raise your hands Oh, so quite a fee for the ones who know Cystic we do or we have a few different projects. This is my expense. Here we have a different, a few different projects. We, the most important one, is an open-source travel shooting tool that works very well with containers and kubernetes.

A

We have a monitoring tool for the docker and kubernetes and other orchestration these systems, but today I'm not going to talk about any of those I'm going to talk about the next kind of experimental project, a pet project. Actually, we release it. A few months ago, it's fully open source seeking code to get, have and see how that has lyric. How does it work? It's based on our system called tracing technology and I'm, going to use some a slides to show you basically how it works and then I'll try to focus mostly on demo.

A

So please, if something fails, this gives me, because this is really dangerous. So our tracing or system called tracing, it's different from open tracing or other different tracing tools.

A

It's like more useful, probably for subjects and when, when like building our container troubleshooting so a tool cystic, we saw that previously our system means they were using all these kind of tools for the internal cheating and when we, when we really assess take, we saw that they really liked the tool, because basically, they could ditch all these tools that they were not working with namespaces with containers and they could start using cystic.

A

But then with containers like how can you look at what they call doing like hell? Oh, we. We have all these changes with containers namespaces, so on so there there are quite a bunch of different tracing tools. All the open tracing paste same kind of Prayer. There are a bunch of them. There are other commercial tracing tools, there are quite nice and then there is people who they don't want to use neither of both and they do some in-house actions. So they start using a stat CT to report matrix on the performance on their call.

A

They do you just print. They do some login, although might have no all the information. So we thought what if we have something open, sirs, everyone can use it, but it's really simple. We don't want to do funky stuff, it's like like uptime, like the basic troubleshooting tools we shall be for and and something that is very easy to use, know like EBP F base tools like be CCC, something lightweight that didn't have a huge performance impact and that could work everywhere, pretty much or like cystic containers, no containers everywhere.

A

So we thought we have the technology. We have cystic open, sores that it does native container troubleshooting. We also have the monitoring tool not going to talk about that.

A

What if we use the technology today, some kind of system called tracing for the ones who knows a snake. You can ignore me over the next couple of minutes, but I'm going to explain. Basically how says the technology works for the ones that you don't know. Basically, what we did is instead of instrumenting call the services we lake at the kernel level, so we have ups running directly on the house. We might have containers basically what we did. We know the kernel module, this current module interest.

A

Soare captures all the system calls we put them in a share, ring memory buffer, and then we send it to the user space where we do all kind of formatting report in some kind of magic we can dump into of one. So it's like pretty much like TCP dump, but for system calls.

A

So we can capture all the system calls that. Actually we transform them to events, because we keep like some relationship within them. We can apply filters, we can run scripts. I'll show you some examples of these scripts, so we call them chisels you can dump into a file. You can run our analysis afterward because we work at the system call level. It doesn't really matter. If your application, your code, is running inside a container or not. We have full visibility across and we have a command line and courses interface internally, Cystic.

A

What it does is basically this we have like an event stream. Where we read all the system calls open rate close connect whatever, and then we have these different actions them to disk filter or analyze what we did or how we build system. Assisting tracers was okay, so we have this event stream. What we can do is to insert markers here and basically, a marker allows us to the limit. What we call a span a span can be everything we can have function. Calls inside Network requests any piece of code.

A

You can use this from any language because, basically, how we mark as pumps or we do these marks. It's like writing into that new, because at the beginning we thought about using a special slash, depth device third mu, and we wanted this to work into containers. We thought, mmm like definitely is available everywhere. So let's use that because we see all the system calls and it has very low overhead, so you have the same event for we do this right into that null. So we insert one entry and we do another right to the limit.

A

The exit of the tracer so inside you can have system calls called everything I'm going to show you, you can nest different spans, so you can delimit different pieces of cold or calls that you want to measure and G can write manually. There's no use in this format, simple, very simple form of plain text. You can use JSON as well, but it's like more complicated and the fun thing of days like this is like super weird jorge. What the hell have you done here? Well, these allows to do tracing in partial scripts, for example.

A

So this is a piece of bar script. We just need to do an echo into their node to the limit, the entry of the spam and another echo to the limit D exit. So if you look at here these symbol, basically the limits, if it's an entry or an exit, fun things, you can did it with all this. Well, we can measure latencies in your code. You can save tracers and filter traces with cystic. You can analyze with CSIS date, which is the courses UI.

A

He can see the system activity inside of a span, so I in the previous example. I could see the system calls doing inside I can do treasure where lock monitoring. So if I have some code, that's doing login but I want to see only the login inside that is spam. I could do that and we can export these metrics or tracers. They have a metric associated the latency, for example, and we can export them in to start see and get some graphs. We probably going to see that this is everything.

A

I have a miss lights, it's like less than 10 minutes, so I'm going to spend the next 20 minutes, or so we'll leave some minutes for questions or begin and playing with this. So when you walk out from this talk, you are already familiar with again: it's been ten years from here onwards.

A

So please excuse me if something fails, so the first thing I want to show you is this example: I showed before I have a script here: Marcia scrape that it's basically pinging to Google or loading Google, so I'm going to run this script and then I'm going to send this to the back, and this is are the things I wanted. I want to show you, but let me open CSS day since the stick is the courses interface.

A

This is pretty much like a CH top, but on asteroids, because if you click here and B, you have different views like we have cover Nittis or the registration systems, but that's something different today. I want to focus on tracers. So if we click here, we have an Instagram of how these are these tracers that we are running in the vasa scrape is performing. So if I stop this, you see this is going to stop or maybe have something running. Let me double check.

A

No Tina stop. Okay, now I did stop okay, so I can basically measure the latency of loading the google web site. But imagine I want to compare this with something else. I was looking for a fast lolene site and I found the router, so I'm going to run this in parallel and I can see the difference so, depending like we have here. The latency from nanoseconds in two seconds or I can see here. The router is pretty fast while Google Website is a bead slower.

A

I can like this is just a B of Instagram, but if I go here to tracer at least I can see them with the latency, but I can also get some reporting.

A

Okay, so I get average sigh minimal time maximal time very simple watch is Krypton.

A

We thought this is cool and primarily people want to do more complicated things with it. I said you can do a course or write into dev no, but we also created our wrapper for pythons community members. They erode our wrappers for no guess for a goal and I'm going to show you a Python example hello world. So this is probably more probably better, so you just import our wrapper and we can use either with statements in Python. You can also use decorators.

A

I'll show you later on to basically talk the pieces of code you want to have in this are spots. So in this case, what I'm going to do is to the inside this tracer that I call it hello world I'm going to do hello world, but then outside I have this other hello world. So if I run this okay, nothing special but commands I can use now the common line interface to do the following yeah. So let me first explain the syntax a little bit here. We call cystic Kamala interface and we apply a filter.

A

We want to get the Evanses stream and from the events, history I want to look this pants and talks and I just want to share the ones that they are inside the hello world and with all those events, I want to pipe them into a script that we call chisels, and this one for Jesus are written in Lua, very simple to you to write your own and this one's. Basically, what it does is to echo everything that's going into a file descriptor.

A

So if I run this I see that a Python process is writing in PPS this string. Okay. So this is very simple way of individually analyzing or the system phone I can do all the things we I'm going to show you another example of T cells. So if, instead of doing a print ID an HTTP request and.

A

I use a different cheese: oh yeah, there we go this one instead of going to put this at the top yeah this one. Instead of doing an echo of the file descriptors, what we do is to we format, HTTP requests. So what I get here is the I get the response code, the latency the size of the payload? You can write your own thing, okay, so it's kind of neat.

A

Today, all these are tracing with very little effort and lightweight, but we can do like more complicated things and I'm going to show you how to travel are kind of like dumb or a stupid API.

A

So for that, let me see you didn't forget. Now there we go so I created plenty of points in here server there we go. This is a Python server that it has a bunch of different API endpoints, one returns of evil. Nutty number one downloads, a file- it's not really important, but I want you to see what I want you to see here is how we use the the Python robbery. So you can use a decorator and decorators.

A

Actually, we can capture the arguments, so I can see the entry and the exit of X function, or we can years with statements, as we saw before so going to show you this piece of code because probably we'll be looking at this later. I can individually talk the handler and inside of the handler.

A

I can do nesting and do the hitters and probably I know that's going to be very fast, because right in the hitters is fast, but, for example, I'm interested in this I'm just writing a random file, and we can look at how this performs so I. Have this server running in a docker container looker file, server, nothing special and also I, have a client that it does random requests to this API, so I'm going to run this. Oh sorry went to this run. This foreground.

A

So we see the connections run in here now, I can run, say: Cystic, life and I can see. For example, they containers yeah I got this occasion. I'll show you that later on, but I can see my two containers. Click here and I see you can see the Python and the process, and here I can see the system calls and AH they call the scriptures it's connecting a lot of data. Maybe it's a lot for my laptop, don't just of this, but if we go into tracers I can see all the tracers going on here.

A

All the information I can see the spectrogram I show you before and now this is like a lot of different things. I see some slow requests, some some very fast and probably like. If we are other Belapur or we are some operation guy running somewhere else cause and I'm trying to figure out I, don't understand this code. What's making these API endpoints going this low here, they're like from these in today's it's a big difference, so I can start using, says the common line to apply different kind of filters and find out what's going slow.

A

So I have done this previously into a file, so I didn't have to capture wait for some time, so, basically I dump everything into a file demo. This cup, which is like TCP, dump rom foil, but with all the system calls. So if I run this, it was the first sorry I can start applying different filters. So one very interesting one is a span. Duration I want to see what takes more than half a second to run so I run this and suddenly I get all the system calls or now we ever have system.

A

Well, here's I have the entry on the exit of the different tracers, and now you can see the tax so gives. This gives me an idea of what kind of things are going to slow I see like Fibonacci couple of thumbs, but it's mostly the download handler. So probably there is some function over there going a bit the slow. Let's keep digging and see what we can do more.

A

So if it's, the download handler I saw that a piece of code was doing some writing so I'm curious to see what kind of the next one is. First, what kind of input-output activity are doing each of the sponge, so I can use again a chisel which is called ft bytes by and now I'm going to use the T cell and I want this chisel to aggregate code based on the different tax.

A

So, if I run, this I get like a little report with different tracers names or the tax and the amount of I/o activity associated with them cell I can confirm that from all these do the download right it's.

A

It has a very heavy input/output activity, but I'm wondering what kind of input/output activities this could be network could be file system. What we have more more filters, so, let's see first network, will train I just need to have another filter very similar to CP des that is file, descriptor type IP before I run this and I say oh half of it is network.

A

Let me see how much is filesystem I can do the same. So I see this guy is doing a lot of file system and almost the same amount. It goes into the network so I to see exactly what kind of files I am writing. So what I can do again? It's apply further filtering, so I can use this filter. F divides by with a different parameter.

A

The name of the file I'm using and I want to filter out something so I want only files, not network and I, want only the system, codes associated with the download handler and no others and I want to exclude writing into the standard output because it's going to be dirt so I see there are a bunch of different files and one of them is quite bad. So my assumption what I can see is that this piece of code is reading a big foil and then is writing that into the network.

A

That leads me to think that that's the reason why the some calls to the API are as low.

A

Where else I wanted to show you this.

A

Last thing I've been: basically, you send chisels to aggregate and to format and report all the system calls, but if I'm interested in seen exactly all the system calls happening inside of these responds, I just need to go, meet the chisel, here's the same filter and I get all the system calls.

A

This is, if you see we can go very, very low with all this tracing, but we can also do like a little bit higher, and so we have another chisel that takes all the tracers and we export them into a stop, see, stop see it's been automatically collected by this is the agent we got here. So if I run this and I go here, this is Cystic.

A

It's going to take less or around ten seconds, we'll see that they start seeing matrix will start appearing here. Let me know this.

A

My server is not connected, no ogre compose up, says the page and I want it. In the background.

A

This is sending on a shoot, see various in my laptop in here, and the idea is that this is sending into our into our product, but any stats. The collector that you got can be used to collect these metrics and basically, our latency measuring.

A

Something had to file, or at least be as low so.

A

Second, there we go delete it. This is my laptop and we see that automatically we get here. I, don't know. If you can see this in the corner, all the different metrics and in a few seconds we should be able to see.

A

The metrics coming in.

A

All right questions in the meanwhile yeah.

A

To get the data it's here, it's coming as I said before we talk an entry point in the event stream, and then we talk the exit point and because we keep all this, a stream of system calls. We measure what's happening in between.

A

We, what we we have, it's a kernel, module that captures all this and then it's our own technology says take that measures was happening inside, so we are not using anything like open tracing or things like that. I know. If you have answered your question more or less.

B

A

Are not using that we are considering are moving into vpf and things that are until now we are Navy PF. We didn't have everything we needed to get all this information, so, for example, I remember one of the limitation was that the buffer Andy BPF only gave us like fork.

A

4,000 bytes of every buffer of the system call correct me if I'm wrong, but now that has been extended, so there are a few things that it was preventing as to you see, BPF, probably this is but we change in the future, because the like the real magic, it's not like the current module, which is a very tiny thing. The real magic happens in the user space where you are Craig ate all the system bills think there was another question of.

B

A

The performance over have it's very little at the at the kernel level, because all we do is in the case of tracers est. Do your own echo or are right into them, no Sochi right at the end, almost nothing it can have more overhead when we move into the user space depending basically on how complex are the filter in the GD. And how big is your capture these captures? They tend to grow very very fast, so let me show you just uh actually I'm going to do one, so this is just.

A

So let's leave this like 1 2, 3, 4 5 seconds.

A

This is 31 mix of chi containers running our HTTP requests as fast as they can. So, if you run this on a production server, we have customers that just a couple of seconds files system calls capture files can grow to 100 Meg's, even in gigabytes. So if then, you try to analyze one of those big files. It has an overhead because this report, in this aggregation of the system calls we do, requires computing power.

A

If you did that in on a tiny file, it was so depending on the amount of data you want to do the aggregation on that one and then.

A

Yes, yes, so we you can filter your pardon process. You can filter child processes. I think you can go your grandfather too, so I, don't know how many nesting levels we support at the moment, but at least your father and your grandfather, you can do and that's basically the semi column there. There is one of one of the format's of the d-string. You capture, you can follow your peed or your peepee.

A

Actually, this is what we call that we have not only system calls, but we have events, because we don't in this in this FM stream. We just don't have the system calls, because that would be I, don't know a stress. For example, we have some relationships between the system calls and some metadata information that allows us to do the different aggregation. That's the that's! The nice part of this.

C

A

The moment we don't have support for that, this is restricted to one single host. We are thinking how we should do this. Like there are few option. We can merge different capture files into a single file that then you can analyze our commercial tool. It does something similar. We don't have all the system calls we have.

A

We can get some metrics based on the system calls and that allows our commercial tool allows to monitor a distributed system and measure given metrics like service latency, and things like that. But at the moment there is no oscillation working out of the box for cross host tracing. That's why this is like a pet project. It's something we just stayed in a couple of weeks months, based on the previous technology. We are trying to figure out what we want to do with this. It's something this hopefully opens her.

A

So maybe it's a break for someone did you say a I want to merge, says the captures together into one file and then analyze them it's a possibility later on. We have plenty of time.

A

Yes, well, yes, and no, we have some kind of support. So, for example, probably the most higher-level thing we can do is attacks so or there are the decorators where we can capture the entry arguments and the exit arguments of a function, but we don't have like let we don't have like a back trace for a language or things like that. This is we work in a lower level.

A

It's very different I, don't think there is an overlap like we work at the lower level order tracing tools, they work at a higher level, different use cases.

A

We have the Fed support, but if I remember the war, some use cases that we were not covering I don't have the full information we we have go wrappers. So it's very simple to this very similar to this Python.

A

What was probably that we didn't have filtering support for threats, I, don't know, I would have to check you get a t-shirt. Oh that's a question. I didn't know how to answer.

A

More questions feedback comments. Yes,.

A

Yes, so we require- or this requires root permissions. Basically, we go because we need to do an in smoke to load the kernel module if ye or during this time, or been running this on the host level. But, for example, the.

A

The assisting agent that was report in the system calls in to our commercial product I was showing you here.

A

This is running inside the container so because all the capturing happens on the system level and then we send all the system calls or we collect all the system calls from the user space. It doesn't really matter. If that user sprays process is either on the host or inside of a container, you can run Cystic from a container and you would have visibility outside of the container and into the host and across all the other containers running in the same house.

A

If you say like this is the kind of matrix.

A

We get automatically here.

A

No more questions! Well, if you have any other question, you will find me in our booth. You can come and say hello, you liked it. You didn't like it whatever thanks very much for coming. I hope you enjoy it and oh.

A

Some further reading.

A

There we go. Thank you.