Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Monitoring Kubernetes Clusters with Prometheus by Fabian Reinartz, CoreOS

Description

Keynote: Monitoring Kubernetes Clusters with Prometheus - Fabian Reinartz, Software Engineer, CoreOS

"Kubernetes is a powerful system to build, operate, and grow a Cloud Native architecture. But how can we stay on top of what’s happening across thousands of pods that are dynamically scheduled across hundreds of nodes?
It needs a system capable of monitoring all individual units across the entire stack while enabling users to drill down from a global view to individual instances. Prometheus is an open source monitoring system designed with exactly this goal in mind. As it turned out, Kubernetes and Prometheus is a match made in open source heaven. Fabian will explain common challenges when monitoring large scale infrastructure and how Prometheus provides high-level observability without giving up low-level insight."

About
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everyone today I want to talk about monitoring and we have seen quite a few really great products of the last few days that really help you setting up cabañas casters easily, but we don't have the same kind of stuff available for basic plumbing to actually keep your cluster running, for example, in monitoring. So um there are a few challenges when you want to monitor a community's cluster, because just an environment which is way more dynamic than anything with experience before so.

A

First of all, you're breaking up your analytic applications into micro services, and then you horizontally scale these micro services. So now you have instead of a few static instances of your vacation, thousands of pods distributed across nodes in a way that you can't even understand anymore.

A

So we just have a lot of stuff to monitor and it's constantly changing, because kubernetes also enables you now to basically do several rollouts of new software versions every day and constantly scale up the applications horizontally, and it can even now help you scale your entire cluster and based on the capacity currently needed.

A

So I don't have these thinking.

A

So if you want to monitor your horizontally scalable services- and you want the high level overview of what's happening and if you just can inspect a single part, this doesn't really help you because that's the dimension along which kuberdas allows you to be fairly tolerant. So if a certain part is not reacting anymore, that's fine! It's not impacting your user in general, so your monitoring system must be able to get these this high-level overview and to determine whether your service is actually giving a users appropriate experience.

A

But if there's a problem impacting your service users, you must actually be able to drill down into all sorts of dimensions, because it's quite a complex system. So maybe your application is buggy. Then you must be able to determine which requests are actually tricking. These erroneous responses, and but maybe just a certain cluster node- is failing, which will impact and ports across many different services, and you must be able to look at the monitoring data from all different angles to determine where you actually have to now fix the problem.

A

So, let's see if remissions can actually address all these points and prometheus is a poor base monitoring system initial develop at Sound Cloud, but nowadays it's developed by many different nice, and actually she has a common heritage with Google Google's internal boxes. Then because pork came with someone consistent Bachmann, which was largely and that's an inspiration for, vary quite a few concepts. We have infirmities nowadays and the most important one of these is the multi-dimensional data modeler.

A

So imagine you want to instrument your application with a counter for HTTP requests, total Maitri application and quite naturally, whenever you get a request, you just increment this counter by one and all as well. But now you want to have more insight into what's actually happening, so you want to know what kind of recourse was made. So you add another dimension in this case.

A

Here we have a passed label which now tells us which parts of application was actually requested, and so, when you want to add further dimensions, so, for example, the return status code or the done the use request method, and you can quite easy to see how this ends up with hundreds of individual counters for one single metric and recall this individual and points time series, and if you might blow this by the total number of metrics, you have forcing application and you multiply it again by all the pots running in this application, and you might refer to this again by all the different services.

A

You have you quickly see that you end up with millions and millions of time series even for a fairly reasonable sized infrastructure. So you need a monitoring system. That's really able to handle this huge amount of data, and promises was actually designed for that. So Prometheus scrapes, this video fine-grained data and stores it in this wrong manner. So we don't do any pre aggregation on right so that all these dimensions are still available to you when you want to go and investigate, and the querying is really important.

A

Of course right, and so let's say we want to figure out. What's 99th percentile latency is for my individual HTTP passes across our service and, as I said before, you don't want to notice.

A

For every part you want to notice across the entire service, so he has a metric which represents a histogram bucket for malukas latencies and now I can quite easily aggregate by preserving the past dimension and the dimension indicating which bucket I'm talking about and then I hand this down into the histogram quantile function, which then allows me to calculate any average percentile of this latency service right on the fly.

A

What I get is this really nice and meaningful overview of just a few points indicating what the current 9s percentile agencies and spread out by Rica's pass, and these queries directly translate into your loading as well.

A

So we can just take this curry and edit resh hold of one seconds, and then you have a load that will notify you as soon as any particular endpoint stops responding in less than one second, and in other examples here so let's say we have a metric which tells us how many free bites are on any particular hard drive on any particular node in your cluster, and you can look at the last hour of data and then make a linear prediction.

A

So if the current behavior continues, what will happen in four hours and it's allows now to check ok if the number of three bites in four hours will be less than zero.

A

The district is full and we can actually now wrap this into an alerting rule and Prometheus will now notify us as soon as this condition is true, which gives us then about four hours to react and fix either the course or, for example, accidentally hard website, and this is one single loading rule which actually allows you to cover every single hard drive in your entire infrastructure.

A

So apparently, for instance, is quite good at modeling and timer series and helping you to get this high-level overview and also allowing you to drill down if you need to, but now we have to solve the issue of having just so many targets. We have to monitor individually inside of our cluster, and we solve this problem by directly talking to the kubernetes api server and we can just get an initial set of things that can be monitored, but then there's this constant change going on and to deal with that.

A

We just place watches on the API server, which notify us immediately as soon as anything changes. So if you schedule deployment and the new pot gets created, promises will be immediately notified and can immediately start monitoring this pot as well. The same goes for EM scaling a deployment down. So if a pot goes away, promises will immediately know and don't alert.

A

You certain part that it seems to exist is no longer there, and this allows us to basically keep always in sync with your IP I server and that's basically the most high-level overview of your entire infrastructure. You can get, and this actually allows us now to go in further, so by being a poor base monitoring system and pulling our target information ourselves, we have access to all this meta information, so we don't just get an address from which we can get metrics by HTTP calls. We also now know more about this address.

A

We know to which party belongs and which labels a spot has and to which services pop along and in which names as it's running, and then we can take all this meta information actually and in which our metric data we get from duplication by further dimensions.

A

So the application provides us with the base metric and its internal dimensions, and we just add, for example, the namespace or the port name as a further label, and we can then use both external and internal dimensions later on in our queries to do a creation and filtering as we as we need so meteors seem to address all these problems kind of nicely. So now, let's try to get up a basic cluster monitoring, and this starts out with all these different components we see here. So we have a CD, which is the most important part.

A

If exid is unhappy, you aren't happy and because it holds the entire position state of your kubernetes cluster and then you have a quality. Of course.

A

Your notes and notes don't expose metrics in the Prometheus metric format, but the node Explorer small utility, which you can deploy as a daemon set and then to collect all sort of metrics for my operating system and make it available to promise yes and then, of course, you have the couplets and the cubelet is the local agent of kubernetes running on every cluster node, and this one exposes metrics about internal state. So, for example, what I can talk to the API server or not, and the same goes for actually any other Creators components.

A

So we have the API server, the controller manager and the scheduler and all these exposed metrics about their internals. And lastly, this cube state matrix and cube set fanfics is a fairly new project. It's just a small utility which allows you to expose business logic, metrics of kubernetes and in a metric kind of way.

A

So, for example, the deployment is not something real like memory, it's just like the abstract concept of your cluster, but you can actually expose metrics on it, so you can say and how many desire replicas are there in my particular deployment any cannot expose metrics on and what two clusters point of view is of how many of these replicas are available or unavailable, and you can then use these metrics data on to building and dashboarding to, for example, notify you when a deployment got stuck and lastly, we have C advisor and C advisor is pretty much like node exporter just at the container level, so it gives you the same kind of metrics just for every single container running in the entire infrastructure, and if you combine all these together, it's actually quite a good picture of metrics across our entire cluster nd.

A

If you get all these metrics, actually we are pretty complete if it comes to monitoring the cluster itself. So, luckily, for us, these boxes here all expose their metrics in the promises format already, simply because kubernetes as well as HCD, chose this format to be another way how they expose metrics. So we merely after now, point promises to every individual component and collect the metrics that are readily available yeah. But now we have to run from easiest and that's kind of a problem and not really. But police is not a micro service.

A

You can't just create a port and kill it whenever you want to it's a sample application. If you want to it has position storage, it connects to alert management and dashboarding and there's just kind of bit of stuff to take care of, and this is kind of an entry barrier if you want to get started with Prometheus, especially if you want to, if you have to learn promises itself and then have to see how you can make this all happen within of kubernetes, and that's why it's caresse.

A

We came up with the operator concept and which simply means we take operation, domain knowledge of certain application and express it as source code, and then we make this knowledge accessible to a user of kubernetes cluster via third-party resources, and this information shows as follows: we have a premises, object which looks like any other Creators objects. We have a yellow file where you specify the kind you can add, labels, etc, and that a specification just has the basic data about the premises setup.

A

We would like to run it in our cluster and the operator now we'll watch this resource and as soon as you, edit, which happens fire queue control as anything else. It will immediately start deploying the appropriate set up and manage this lifecycle for you, and this also means we can now, for example, change the version in our promises. Resource to 1.3. I have one point three point: one and the operator will automatically take care of doing a safe and disruption free update of your entire setup.

A

So if you combine the two, we actually have a default way to monitor clusters, because all these components we've seen available in every cluster, except for the node Explorer and keep site metrics, and these are really easy to deploy, and we now have a component actually allowing us to declare them define how we want to run from ECS and taking care of anything else. And if you combine that we can actually come up with a one command deployment or full end-to-end trust.

A

The monitoring which is which is basically really available after like three minutes and can actually look at how this looks like if it's deployed. So one of the things you of course get is a Prometheus setup in general. So here we see the Prometheus UI and you can go to the targets page and the targets page just shows us all the entities being currently monitored by this Prometheus, as we can see their monitoring of a signal density cluster.

A

Here we are monitoring the API server, the scheduler, the controller manager and everything a couplet, and we also deploy it as part of the whole setup process and now to export in every node and cube size matrix. And you know this really well-known set of metrics. We are getting and this because of that, we can also now deploy you and provide you with basic, alerting on all these different metrics. We know they exist, and so we can see this here.

A

They actually two are incorrectly configured ones up there, and but these are alerts covering Etsy itself, cubelets, your API server and all the other components, and they will notify you as soon any of these components is not behaving as you would expect in a healthy cluster, and the same goes for dashboards right and we know which meter metrics assist and therefore we can now provide you with your final instance, which is also deployed as part process and which comes with a bunch of default dashboard.

A

So here's the dashboard which gives you an aggregated view of all the nodes, belong to your cluster. So you can see how much your memory or this spacer is in total across your entire cluster, and we can also know, of course, drill down. That's an important aspect and we now have a similar dashboard, exposing the same metrics. But we can now in this drop down here up there in the left corner and select any node. We like and we'll get the same dashboard just showing the metrics for the respective node.

A

And similarly, you can also look at deployments. So here's the dashboard showing for every deployment which you can select by a namespace and name. It shows you, the aggregated, huge resource usage across all pots belonging to this deployment and also as convenient graph down here. We can observe ongoing warning updates or scalings in real time.

A

So all of this comes with just a single command, of course, for advanced clusters. We probably need to tweak on two values, but we work on that, but this is actually pretty much complete, end-to-end monitoring with little effort. But now your samanta mantra here on services and that's a bit more complicated because the per layer your own- and we don't know on which ports your services expose metrics or what they do it at all.

A

So we can't provide you with the default configuration, but in general, the best for people to actually set up monitoring for service and interpret the data. Other people wrote it and instrumented it because who could be more competent and to interpret metric data than the person who actually determined it would be there in the first place.

A

So, ideally, everyone, developing application and deploying it inside of a cluster would be able to self-service and monitoring for this as well, and ideally, monitoring would be part, would be a cluster feature which any anyone can use. So, just as people can do, the only Seconal also do their monitoring without ever having to know about how to operate from ecs or how to configure to find the appropriate monitoring targets.

A

So the operator kind of handles a part of deploying and operating from ETS continuously that's kind of nice, but we need a way to configure for our custom services how they can be monitored and for that the operator introduces another another third-party resource, which we call a service monitor and the service monitor our declares for a group of services how they can be monitored in a common way is actually pretty straightforward and because we can now use kubernetes idioms to express how we want to try to happen.

A

So, first of all, we select which services this service monitor should apply to, and we know, label selectors from career services and position volume types for example. So nothing new here if you're familiar with the basic various concepts, and then we simply specify that for all these services we can monitor them on the name port called web on the slash metrics endpoint, and we want to scrap these metrics every 30 seconds, and this is just a declarative definition of House of monitoring happen.

A

It doesn't do anything at all and which is a good thing actually, because now we want to connect that to a national monitoring system and to be used again labels. So we we label this service, monitor which the label ta equals front end, and in case you haven't noticed it's acting up on all services having the deprecated was fun and as well, and then we include the service monitor again by a label selection in our promises resource, and then we get a full picture.

A

Looking like this, so we now not only only have Prometheus, we also have service monitors and any Prometheus connects to a flexible set of potential service monitors based on labor queries and service monitors in return connect to a set of services also expressed by labels.

A

So all these relationships are inherently dynamic, so whenever you add a service, it will just automatically fit into the picture without any direct configuration and all the operator watches both resources and actually also all the relationships between all these components and then, of course, as before, deployed the premises over for you, but will also generate a configuration based on the current clusters. Data can see, and as soon as anything about these relationships changes. So you add a new service. You add a new service mentor. You modify your service mantra.

A

All these changes would be immediately processed by the operator and it will reconfigure the premisses server instantaneously.

A

So what you end up getting is, first of all, we have now a full end-to-end cluster monitoring, where we don't have to worry about operating procedure itself, and now we also have a definition which clearly separates House of monitoring happen with the actual implementation, and ultimately this can also allow you, in the long run, to just rip out promises if you want to and then just plug in your monitoring system into the existing service for infrastructure.

A

This is really important. I think like for the future, and but it's actually get this really nice full picture, and you can now configure a full Prometheus set up just using kubernetes idioms and never having to worry about promises, operation and configuration in any way, and all of this is of course, open source and it's all in pretty early stages, we're working hard to make a production where the as soon as possible. But we also need you to try it out and give us feedback so we're looking for that. Okay, that's! Alright! To show Thanks.

A