Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes-Defined Monitoring [I] - Gianluca Borello, Sysdig

Description

Kubernetes-Defined Monitoring [I] - Gianluca Borello, Sysdig

Over the past few years we’ve all learned how Kubernetes can dramatically change the process of deploying an application, improve reliability, and accelerate operations. As Kubernetes matures, I believe it will have ripple effects on other elements of DevOps, including monitoring.

In this talk, we’ll explore the question, “What if Kubernetes also defined and automated monitoring?” We’ll explore some of the available tooling to answer questions like:

*What are the right ways to instrument Kubernetes minons & pods?
*How do you effectively get visibility into aggregate microservices vs just containers?
*How can Kubernetes automate the act of setting up monitoring dashboards and alerts?
*How can teams use Kubernetes to allow them to isolate monitoring data more effectively, so that the right data is exposed only to the right people?

This will be a demo-driven session, based on a real Kubernetes environment, using a variety of tools at our disposal. Attendees should have a basic understanding of Kubernetes deployments as well as monitoring requirements.

About Gianluca Borello
Gianluca is an engineering manager at Sysdig, where he wears many hats. He's a core developer of sysdig, an open source troubleshooting tool for Linux and containers, and spends his days dealing with backend development, performance analysis and cloud infrastructure management.

Prior to Sysdig, he was an engineer at Riverbed and CACE Technologies, the company behind Wireshark. He's passionate about Linux, open source technologies and distributed systems at scale. He holds a MS in Computer Engineering from Politecnico di Torino, Italy. Gianluca has presented sysdig to the DevOps audience at many events including KubeCon, Docker meetups, Sharkfest 2014 and the Facebook Engineering Tech Talk.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Good afternoon you folks, like monitoring, no listen, oh well, too bad you're, going to see a lot of it. Alright welcome to coup Burnett is define monitoring. The problem that we are going to tackle today is again monitoring, so let me share in five seconds what monitoring should be like? Monitoring should be like this.

A

You have your very fence infrastructure and you should essentially pipe out from your infrastructure a bunch of metrics from every layer of your infrastructure, so lower level infrastructure application processes, custom metrics, then you should to show these metrics to a knife that I engine that doesn't sort of aggregation, metadata collection, trending pattern stuff like that, and then as an output as an end user.

A

You should just get a very nice set of unicorn dashboard that remove all the noise that you might have in your infrastructure and really focus on the things that help you do your job better.

A

What happens in reality, though, is that you, most of the time you're unique on dashboard, gets butchered because in containerized environments, monitoring is much more challenging and we are essentially going to talk a lot about this. So we have to dig have a lot of customers who tackle these problems monitoring.

A

So we started to come up with a few best practices that we think are really really accurate and because people in the field adopted them a lot and so I'm going to shoot these best practices, then I'm going to share how these ideas can be essentially implemented in a modern monitoring tool for container and then I'm going to share some essentially more visionary proposals for nice.

A

Improvements to the current monitoring workflow so will be interesting for the most part, I'm going to use tools from the six digit code system, so I'm going to use the open sources and troubleshooting tools to dig and then cystic, monitor, which is our distributed infrastructure monitoring, troubleshooting tool. Why? Because they are very awesome but don don't take these. This talk as a pitch essentially for these tools, because it's absolutely everything that say will be absolute in a completely vendor-neutral way. So as long as your favorite monitoring tool respects these principles, you'll be fine.

A

So why your unicorn dashboard get butchered they get butchered, because things are fundamentally more complex in a containerized environment. So you take up simple infrastructure like this, with three basic building blocks or a database I web server in the cache in tier and when you deploy it in your in your infrastructure. You get something like this, where you have the different components of one one instance of your in front of your application.

A

Live in separate, essentially, are hosted in completely different nodes, and so they communicate with each other using all sort of virtual networking layers that, of course hinder to some degree the visibility and then you also going to deploy multiple copies of the same application, and so you end up with a big mess and you're monitoring tool needs to be aware of all this mess needs to be aware of how to properly make sense of this data so that you do so that you don't have to do it.

A

All right back I can. So how does this impact monitoring? From a practical point of view, we identified four things that are harder to do in a dynamic and flexible container izing infrastructure, first of all, get in the data. This is the very first step and this harder because of course, you have the containers in the way we are going to essentially analyze much deeper each of these points, the second one is making sense of the data.

A

So once you call once you are able to finally extract these data from your containers, how can you organize it in a way that allows you to see what is health of your application? Then we have essentially a troubleshooting, because again, monitoring is very much connected to visibility.

A

Visibility is, is the enablers for the monitoring and for the troubleshooting as well, so they are very connected and then there's a full point that we started seeing a lot of cystic, which is the people, so micro services enable teams, teams of people to stay very close to their infrastructure and follow the whole application for the whole lifecycle cause it's extremely easy to deploy an application on kubernetes, so, most of the time, the team of developers, the variety application then take care of everything, take care of the deployment and they want to also take care of the monitoring, and so there's this kind of affinity between teams of people and micro service and the application then needs to be monitored.

A

So for monitoring this will be taking notes into account. Let's start with the first item, getting the data. This is, of course, the very first step. Containers are great because they are simple: they are lightweight, they are isolated, but for the same exact reasons when it comes to visibility, these exact features become a big pain in the back, because now you have this further level of isolation between you and your containers.

A

Sorry between you and your applications, which are essentially the container and in practical terms, it means that extracting stuff it's much more difficult, and so how can we get these data? What are the best practices in the industry? We have seen essentially two models for extracting the data out of the container. This first model essentially leverages the pods in Coober native, so here we have two containers that are, of course, wrapped inside pods, and this container somehow needs to be instrumented, so some metrics need to be extracted from this container.

A

So what happens is typically you deploy a monitoring container sidekick container inside which pod so that this will share the network, namespace and volumes and the monitoring in space, maybe by pinging on localhost. It can extract a bunch of metrics from the container that you're looking for and then this one can push it somewhere else or you can just expose them to another poor so that you can use another monitoring tool that does the pooling this works very well. It's a very elegant model.

A

The main issue with is that you really need to add one more piece of infrastructure. Every time you need to monitor something so at times it can, it can get a little bit cumbersome, but it's a completely reasonable approach, another one which is the one that we tried, that this league is essentially following the principle of not having the user to modify the configuration of its infrastructure at all. And so what happens? Is that sig taps at a much lower level with a little kernel module?

A

We intercept all the system calls that all the containers, the processes the containers are doing and also the processes outside the containers are doing. Then we take this rich stream of data. We correlate it with the metadata that applies to each container. So if I see Apache doing a system call inside inside container to I, don't know that container to leaves is part of this pod. This deployment in this namespace and so I correlate all the metrics with all that metadata, and that turns out to be a very powerful approach which are the advantages.

A

The advantage is that, obviously it's very much plug and play you don't need to modify anything in your infrastructure to see so you get immediate value and then you can also use it for monitoring things outside your container, so it can be infrastructure wise. The disadvantage, of course, is that it's you know you can see how it can get a bit invasive so, especially if you're concerned about security and stuff, like that it raises some some other concerns, but for the most part, this is what we settle at sis digas.

A

We find it to be a very, very reasonable approach, considering the values that you get so after you get the data it comes to the other part which is making sense of this data. This is probably the most important slide of my talk. It's I think all these items are extremely are extremely relevant, and so these are the guidelines that you should adopt when you get the data to make sense of it. So let me explain them the first one is: you should not be involved in monitoring instrumentation. What does it mean?

A

It means that you should not come up with clever ways to extract metrics from your containers or instrumenting in one way or another. You're monitoring tool should provide your very user friendly and very easy to use and as automatic as possible framework to collect matrix at all degrees, from your infrastructure, from your application from custom metrics. So you shouldn't really get into the business of doing instrumentation because that's probably again not part of the value that has that as a company you provide.

A

The second item close real closely related to this is that you should not be involved in producing anything that is not a custom metric. So a while ago, we're talking to a customer. This customer were super happy said: wow I've wrote the script that I run is semi container, that's using a using essential, a jmx command-line tool. I'm scripting it and I'm extracting a Java heap usage, metrics and then I push them in my monitoring tool. No, that's horrible!

A

If you find yourself spending your time writing scripts to extract something as basic as a you know: Java hip usage, magic you're, doing it wrong because it even if it's inside containers, and even if it's easier, even if it's more difficult to extract jmx, metrics inside container I, understand it but see if you're monitoring tool should do that for you. So the only time where you should be involved with custom metrics with metrics is when you really need to produce them from within your code.

A

So you know you use taxi or use whatever other protocol for for producing custom metrics and you want to practice. I, don't know the queue size of your transaction, dispatcher pool or something like that stuff. That is really relevant with the code, and that's the only time where you need to do that.

A

The third point, which is probably the most important among the most important point, is that every metric should be tagged when you're dealing with containers, since they are so dynamic and things get rescheduled all the time just having access to the metric is not an unless you are able to put those metrics into the right context and in the right perspective they are fundamentally used useless. Let me show you what I mean with a practical example. So this is a cystic monitor. This is our essentially distributed infrastructure monitoring tool.

A

Here you can see a table where I see a bunch of containers for my infrastructure for each container, I, don't know, let's get one word press, for example like this one I can see the CPU very basic system metric, you know, and that's cool, but in itself did doesn't tell me anything because what does this container means? Where is this Cove? What what application this container is?

A

Part of so, if you're able to correlate this metric in the proper context by applying all the tags that apply to this container, which in the case of Coober net is, is all the hierarchy of Paul deployment and know that you'll get much more value. So if, instead of looking at this flat table, I start and I slice and dice my infrastructure by name space, so I see that I have a production in space that has a wordpress deployment that has to wordpress pods inside it and then I get disappear for each one.

A

Now this matter, it makes sense, because now I'm able to compare this appeal in its right context, I'm able to compare with the other members of the of the of the group and I'm also able to get aggregated magic at the micro service level. This is a metric. That is what must be automatically generated by your monitoring tool. Your monetary tools should automatically do that and then you can also dig much deeper. You know, for example, in these cases we are using the system call interface.

A

We were able to capture, for example, layer, 7 payload, so here you can see for my whole microservice, all the HTTP requests are are being done with the with the response time and know that, and this is all possible because we know the monitoring tool automatically correlates the HTTP request with the meta data that applies in the context of the cluster orchestrator.

A

So again, if there is what probably there is one lesson that you need to take away from this whole talk is use tags x sensibly and the other point always regarding tags that you should not be in the business of manually tagging.

A

Your metrics, if you think it's a good idea to push your custom message from your code and then with whatever SDK you're using for pushing metric manual applying it all the tags, for example, I'm pushing this custom metric and I'm going to I'm gonna, I'm going to put the pod I'm going to put the deployment of the tag. Know you're, monitoring tool should do that for you. Why?

A

Because one is extremely error prone to you know being in the too heavy having to carry the burden of tagging each metric yourself and to it extremely, is extremely difficult because, most of the time in the point in the code where you are pushing your metric, you have no idea in what namespace you're running, because the namespace is here and you're running an application inside the container, and you have no way to do easily a self reflection on all your infrastructure start to navigate all the metadata and pushing it down.

A

It would be extremely difficult to keep out of state, so you're monitoring tool needs to absolutely take care of that. One hundred percent very important. The last one obvious, is that you should collect everything and really try to not filter anything upfront. So I like to compare this with the word of vlogging like when you do logs, you typically log everything, and then, when you need to do a troubleshooting, you use grep, you use elastic, search or whatever you want to just fit. Look for the stuff that you need at that specific point in time.

A

Same thing should be four metric: I know that this means that, especially in today's environment, with a lot of applications and reach frameworks, and all that this means having your monitoring tool support a massive scale. That's fine! It's it's absolutely worth it. So this is how you make sense of the data again I like this like very much. We learn these essentially on the field by experience working with customers. So the other item, again as I said, is essentially troubleshooting, which is very, very related to monitoring invisibility.

A

This is just a random screenshot that I got from github on cue. Burnett is where you can see how, when you need to do troubleshooting of a containerized applications, you can always end up with some pain, because either you are running a very strict based image and you don't have the troubleshooting tool that you might want, or the troubleshooting tool that you might want is not see group aware or there's a weird proc file system layout, and so it won't work or you have to cross the network name space or something like that.

A

So everything that I said up to this moment completely applies also to the troubleshooting field, so you're troubleshooting tools should be completely aware of the fact that you're troubleshooting a dynamic and flexible infrastructure in a containerized environment. So what does this mean? You've probably seen this demo before so I'll. Keep it very short that is always very fun. So this is an infrastructure that is running on my laptop, as you can see a tuning spaces with a bunch of pot. So there's enough raw material to do some serious trouble, shooting so I'm going to use.

A

This is dig CeCe's. Dick is essentially a curse. This base UI for our CI comma line, troubleshooting tool. It has the same features, it's just very easy to look at, and so here when I open it. It shows me all the processes that are running in my system and that's fine, but since that we are doing container troubleshooting I'm going to switch a view to my system and then going to select containers by looking at our containers now I'm, seeing all the containers in my system, along with the resources that they are consuming good.

A

But very basic, let's filter for one container, so let's search reddy's. It shows that we have to Reggie's containers running in these in my in my machine. So let me select one, for example this one and if I press enter I drill down into the container- and this now will show me the sorry I entered in the one that keeps hold of the net or neck space. So in this one I see essentially the ready server process.

A

So I can see how many resources consuming, but while we're inside the container, let's take a look, maybe at the network connection that these registers is generating and again is just across the turning space, because we look at things from a system called point of view and while we are at it, wouldn't we like to see your content that has been transferred on this network connection.

A

So I can press f5 and I go into a commode, and this will show me the readies transactions that are going on on the wire, essentially on the network connection and stop for a second and think about how I did I didn't use any PCP dump or anything overly complicated I just opens his dick from outside, and these by fetching all the system calls and correlating them with the proper metadata. In this case, a simple container name was able to show me all the contents of da yo. This is very powerful.

A

I use it in production myself all the time. So it's very good, although there is still one thing that it's the kind of bothers me, which is I've, done a navigation essentially starting from started from containers. But what do I know about these two containers? I, don't know which one I was really interested in because again as a cooper, netted user I reason in terms of Cooper Nettie's extraction. So instead of starting my troubleshooting, why look at the containers? Why don't I started by looking at the namespaces?

A

Even if I'm, using a low level system, troubleshooting tool? I? Can start by my troubleshooting by looking at the at the cooper net is metadata, so here I am using the production, inspiration that can jump into it, and this will show me the part in this production in space, and I can see that now I have just one ready spot in this case called the other one within the other name space. So I can do this a top-down approach also to troubleshooting. Let's pick something out. So let's pick something interesting.

A

This job up, pod I can enter into this one and they can see that there is just one container I can enter into this container and I can see there is a tomcat and that there is an nginx.

A

That probably does something I don't know, probably ssl floating or something like that, and if I pick, for example, one the nginx worker process, I enter into the process and now I'm seeing the threads that are that that this that this this thread is essentially made by and while I'm inside here, I can look at the activity that ingenix is doing and notice.

A

I am looking the individual HTTP requests and responses from outside the container again pretty cool, but don't let yourself be fooled by the fact that I'm just we can network activity because, let's reach do something else. While I mean these nginx. Let's look at the files the 10 g NX is touching. We can see that nginx is doing a constant amount of outbound oil towards VAR log n, GN x axis of log. Let's see what is the content at five?

A

This is showing me the log entries as they are happening in the system and while I'm outside the container, so I didn't have to do a Dalek, SEC and NS enter, and this is really cool. So I like to think at this together, tcp down for your entire you of the system, it even works with pipe. If you do a pipe lining in shell, we don't have full pipe, something which is the you can see, what's been transferred on that pipe, really really cool.

A

And then I can also save all this information to a trace file, and then I can work on this straight file off line with the same work for it I would use now. So this is a workflow that we borrowed from the network board. You know, tcpdump rewards are awesome tools and we said: okay, let's apply, oh also, these two system troubleshooting. So this is what I can say about a troubleshooting last point and then we'll be done.

A

What about the people, as we said, as I said earlier, microservices really enable teams of people to stay close to the life cycle of their applications. So we think that monitoring should definitely be part of this equation. So monitor should enable teams they should enable people to stay close to their microservices. So acidic, we experimented with this concept of teams, so you have your monitoring tool that from the left it takes in just a massive amount of metrics from the right. It ingests all the service metadata.

A

So, in this case, from you know, from the kubernetes api server, it watches all changes and constantly correlates all these metrics with the cooper natives metadata. Then this is the trick. If we know that one team always a subset of the infrastructure, it could be a namespace, it could be a deployment. It could be something as something defined by just labels. We can choose to route.

A

The subset of metrics that apply to this team to a different entity from a monitoring perspective, which means different set of log, is essentially a sandbox for monitoring, so that people can use a big cluster that this can be shared across.

A

Multiple teams and every single team can have a unique view of the infrastructure as they like it, which- and this can be good to you- know- reduce noise, reduce clutter but also provide proper isolation, because maybe I don't want my developers that run the stage in container to see the sequel query that my production server is doing against the credit card.

A

With this you can, you can achieve it, so we teams again designed to reflect people integrated with with Cooper native support, isolation and customization, but what I'm going to show you now is these other item, which is self-service so silly? We said: okay, this concept of team is pretty cool, but you have to create teams manually right.

A

Every time you deploy application, you go in the UI, blah blah blah, and you do this or also the API, but wouldn't it be nice if we could bound if you could bind together really infrastructure definition and monitoring definition, and so we started with this experiment.

A

So if you remember my wordpress deployment, it looks something like this, so it's just a standard deployment and then I have just a very standard container I even have a password in clear text, path to the environment, variable horrible, but then I have in the annotation sections again generation is the essentially section that Cooper net is provides for pretty much any resource, which is arbitrary, key value, pairs for arbitrary consumption, and so we said cute the annotation to put a little input a little information about what we like in terms of team and some say: okay put this annotation that says: I want a team for this service and will be the members of the team for these team.

A

I want these dashboards, because I know that they're going to be useful for them, then I want the others, because again alerts is very important. I want them to go to these email address and then I want. The alert. I want a simple alert that will alert me if I get the slow response time, which means, if over a sustained period of 60. Second, the average of the response time goes above 500 millisecond across my whole service, then I have another one which may be the CPU.

A

It's also a very good thing to say, which is alert me whenever the CPU across any container. So this one is different. This one not do any aggregation. This one is taking a look at this individual container. That is part of the deployment goes above. Eighty percent, when I, when I submit this to compare net, is and when I deploy.

A

This hour is big in user space that is listening to the cabanas API server from watch changes will read these annotations and will automatically create that team definition for us that, then we can consume in the UI and will keep it in sync with the annotations. So if I change annotations, everything gets reflected to the UI. So if I go to where it was before, this is what we call the root account. So here, I can see everything, but if I switch over here, I can see it.

A

I have to two teams created automatically because I deployed to version of my wordpress deployment, one in the one in the Deaf namespace a one-day production, and if I switch to this team now, I'm just in these two containers that are part of the deployment, and this is completely dynamic.

A

If one container Goes Down and gets rescheduled the essentially our our monitoring system will recognize that a new container comes up and share the same set of deployment and stuff, so we'll arouse it to the correct team, and so I can do a basic exploration like this. I can again slice and dice it by the metadata, so I can see production wordpress and I can again do the same sort of troubleshooting, where I can see all the system metrics and everything that I care about and then I also have dashboards that are automatically created.

A

In this concept, I created a my sequel basic view where I'm able to take a look at all the sequel request, the this WordPress, this WordPress deployment is doing- and this is all created automatically based on a template. So I just provide the template of the dashboard that I want, and then this, like, a a stamp that gets replicated every time. I create a new team with those annotation same thing, with alerts with alerting I have all my alerts created automatically with the proper notification channels created.

A

So a team overall looks like look something like this. When I go to teams, I can choose hey. I have a big cluster of Coober netted, but for this team, whose members at this one's just show me all the containers that are part that satisfy this criteria, which in this case is namespace and deployment, I could choose anything. I want assuming I'm the administrator, and this really. This really provides a lot of value.

A

This really provides isolation, because then you know, if you have a user just belonging to the team, it won't be able to see the other container or the matrix on the host. So there are lot of use cases that these enables again divers of production, split team per micro service, or you know many internal. We know that other companies, q Burnett, is really as a you know. Some sort of platform as a service, for you know their various teams, so this is also useful and so very generally this.

A

Actually, the implementation is very, very straightforward, trivial, but the concept of not only having infrastructure as a code, but also monitoring as a code, is really appealing because, again it just is it just is I didn't allow you to say these are some very interesting pointers, especially if you want to try out. These latest features are just shown. This last link over there is a very recent blog post. That is also a tutorial, so you can go and try it out.

A

It's again, it's it's very I, mean I'm sure you can see the value of this and again, as I said, not a complete vendor agnostic. So if you have a monitoring tool is able to achieve these things cool, but again the principle should be should be considered, should be kept in mind. This is all I had. Thank you for your time.