Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monitoring: Follow the Data from the Darkest Corners - Piotr Szczesniak & Frederic Branczyk

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Monitoring: Follow the Data from the Darkest Corners of Kubernetes - Piotr Szczesniak, Google & Frederic Branczyk, CoreOS (Any Skill Level)

Have you ever thought what are the sources of useful signals in Kubernetes? How to collect those data? The data that are crucial for understanding the system. How to ingest them to a human friendly tool and make them useful? In this talk Piotr and Frederic will cover Kubernetes native signal pipelines. The presenters will explain how to take advantage from them, extend and integrate with full blown monitoring solutions. The problem will be tackled holistically: from Cluster User, Cluster Administrator and Monitoring Vendor point of views.

"About Frederic
Frederic is an engineer at CoreOS contributing to Prometheus and Kubernetes to build state of the art modern infrastructure and monitoring tools. He discovered his interest in monitoring tools and distributed systems in his previous jobs, where he used machine learning to detect anomalies indicating intrusion attempts. He also worked on projects involving secrets management for distributed applications to build sane and stable infrastructure.

About Piotr
Piotr is Tech Lead/Manager working at Google since 2014. He works on GKE/Kubernetes for 3 years, joining the project in its early days. Piotr leads GKE Monitoring team and Special Interest Group “Instrumentation” in Kubernetes open source community. Piotr graduated from University of Warsaw, Poland. During his studies Piotr was an intern at Google, Facebook and Samsung."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, I think it's time to start welcome everybody. This is talk about monitoring. The title is: follow the data from the darkest corners of kubernetes. We would like to show you where the data comes from water and what happens with them once they are collected. My name is Fiat I'm working at Google on gke and kubernetes ter. In addition to this I'm leading a cig, instrumentation with me is Frederic. Frederic works for the that heart chorus for Mary Carlos and he's also leading sig instrumentation.

A

Okay. So, let's start with a quick sorry like this is the usual way to start such presentation who run production workloads here, who is responsible for them, who is you know, have something in common with this hands up: okay, I think that the majority of you, okay, the other question- is for those who run those production workloads who has monitoring of the production workloads here and up okay cool.

A

If you answered yes to both questions, that's great, but if not especially, if you answer that you have some production workloads that you are responsible for and you don't have monitoring yet it's really bad yeah. We'll explain you how you can fill this gap, but first I will recommend you this book. This is a really great book.

A

This is this book is written by my colleagues from Google. It shares all our like. Okay, a lot of our sree production handling experience. That is a Google we gathered for many years. One of the key success of running production workloads is to have a great monitoring. This is explained in this book. How to how to you know what are the? What principles do you do you want to have? Do you want to follow when handling production stuff?

A

The best thing for this book is that it's for free, so you can just go ahead and read it in the Internet. Ok. So, let's get back to kubernetes.

A

So few years ago there was big discussions in the community. Whether kubernetes is whether monitoring is a part of core kubernetes, and the answer is not now governor. This is not about monitoring, but it offers some monitoring capabilities, some basic monitoring capabilities, so there are components like horizontal pod, autoscaler, scheduler, cube, CTL table and note, of course, that requires some basic monitoring in order to perform housekeeping operations and the other operations that they are responsible for.

A

So this means that, in order to have a fully working kubernetes deployment, you need to have some monitoring yeah so introduced matrix API, which provides those basic resource usage metrics, so that horizontal code autoscaler can scale your workloads based on the metrics that are there. Scheduler can make better scheduling decision using the data about this usage there, and it was a cube, CTL top comment that uses exactly the same data to make sure that you better understand.

A

What's going on with your work with your containers, especially what is the research usage of them? Matrix? Api is implemented by matrix server. Matrix server is a system components of kubernetes, that is, that should be deployed to have the single cluster matrix server, periodically scrapes all nodes to get matrix from there and then expose them using this matrix API, he doesn't have any application level matrix. It doesn't offer any historical matrix. This is just latest resource usage of containers running in a cluster okay. So let's take a look into matrix, API.

A

What's this, so basically, it exposed are two types of matrix: one is about node resource usage. In addition, two times demand window in order to identify the time period from which those matrix comes from. There is just resource resource usage, and there are also pot matrix again, the same pattern, timestamp window and list of containers and and and and for every containers. We have resource usage of those containers. Okay, so let's take a look at going on on a node.

A

So on every single note there is cubelet instance running a part of this cubelet. Is she advisor that is compiled into this cubelet and there is summary api, an api exposed by cubelet that exposed matrix coming from this particular node. So this means matrix about all containers running on this machine plus matrix about system components running on this machine like cubelet docker, cube production.

A

So this was the core matrix pipeline. This is how it's called and coming back to our sentence. Kubernetes is not about monitoring, so in order to have a production ready solution, something that you want to run in a production you need more.

A

So, in addition to this car matrix pipeline, you need some third-party monitoring solution. So in monitoring Word, you can divide solutions into two categories. One is push based model. Another one is pull based model, so there is a lot of discussions which model is better.

A

So push model was in the way that your pod, your deployment, is responsible for delivering metrics to third-party monitoring solution. So the the easiest approach is that your pod periodically send metrics to some third-party monitoring solution. Pull model was in a different way where the third-party monitoring solution is responsible for scraping, matrix from well-known endpoints, so usually in a loop.

A

Third-Party monitoring solution, periodically send request format like get sent, get metrics request or all known monitoring targets. So the right approach is I.

A

Don't know there is no. There is no right approach. So, let's take a look into two examples of third-party monitoring solutions: one working in a pull model; another one working in a push model so we'll take a look closer disparity. Monitoring solution will start with primitives working in a pull model and I'm passing Mike to read.

B

Erica, thank you so maybe first as an introduction, why am I qualified to speak on this I'm? Actually part of the Prometheus upstream team and I actually see a couple of other people in the audience, so hopefully I, don't say something bad about their components, so Prometheus is obviously a pole model and because I work on the prometheus I believe that's the correct thing to do.

B

No, that's not true!

B

In Prometheus. We believe that, in order to be able to get started easily with monitoring, pull is just the easier model to do and in push by mechanisms. At often you often need to do clustering in the very beginning, because, just because of how the systems work and then Prometheus, we just wanted to keep it really simple for everyone to get started, and it's really reliable. That doesn't mean that the other systems are not reliable.

B

It just happens to be a bit more complicated to get started with that said, a lot of the push based mechanisms are SAS vendors. So it's actually really easy for you to get started. So it's really up to you to decide what you want to do and I'll be representing Prometheus here and Petra will later tell you about the Google product.

B

So, to give you an overview, what you can do with Prometheus I want to go through all of the criminality stack basically and tell you about all the metrics that you can gather in this stack and what useful things you can do with that. So naturally, what I want to start with is the node and on the node.

B

There is a component that we call the node exporter, and this is actually probably what you think about it, and it's just a an agent running on all your kubernetes nodes and or non communities nodes, and then you can collect all of the system level metrics, so CPU memory, file, system, crazy, systemd things really whatever runs on your node, probably has the node exporter. Has some metrics about this, and one of the really useful things that people like to do with this is, for example, something that causes outages in various ways.

B

Is a filesystem usage so something very popular that people do with the node exporter? Is they look at the file system, this space that's available and they want to be able to fix this problem before it actually hits, and so before this node becomes useless. I want to be able to fix it. So Prometheus has something very neat here, which is a linear prediction.

B

So what it does here is that you can look at the your individual node and see how much data it's been writing to disk in this particular example over the 6 past 6 hours and if in the next four hours, yes in the next four hours, no in the next 24 hours, this this space is going to run out, then I want to get an alert, for example.

B

This would be the expression that you want to do for this, and in that way you can make sure to fix those kinds of problems before they actually cause an outage, and that's really useful for a variety of things.

B

So next on the node that is not describing the entire node within your kubernetes cluster is likely pods and the solution today for this and we've already seen this in parts of piotros slides that see advisor is very central in monitoring today in in kubernetes, however, within SiC instrumentation, we are looking into ways how to replace the advisor for something.

B

That's a bit more pluggable, for example, there's the effort within criminales of device plugins, so that we can have GPU vendors supply drivers or any any sort of extension for kubernetes, but at the same time we want to make sure that they have the ability to expose metrics about these kinds of things and, if you're interested in in this topic, this is something that we're discussing right now do join us against rotation.

B

We have a deep dive later if you're interested in these topics- and this is the Buried link to the proposal that we have out right now and really pods and containers- is what we build all our beautiful dashboards with, and that way we can have really nice overview of our infrastructure and see which namespace in our cluster is maybe acting up or basically we can slice and dice all of this great information and just view it, but over our entire kubernetes cluster, and this is really useful.

B

So now that we've more or less or almost covered the node, let's continue with the kubernetes expo components, because all of kubernetes components actually expose Prometheus style metrics, which means we can collect them and alert on them or just graph them. So I said that, because we haven't quite left the node, because the couplet, as you all know, in a kubernetes cluster, sits on every every node and its purpose is to create the environment for pods and then tell the container run time to actually manage those pods.

B

So the couplet itself has metrics about all the pods it manages, but also, most importantly, what a lot of people have issues with or I shouldn't say that something that you would want to monitor is exactly this interaction from the coop that to your container runtime.

B

So if there are any problems, because containers are a complicated thing, you would want to maybe have a look at the errors that are happening within these and interactions or the latency, and these things can be really important when you're, maybe providing kubernetes as a service to the rest of the organization or even selling kubernetes as a service to others. And if you have a service level objective of some scheduling time or some time how quickly pods must be started, then these are the kinds of metrics that you may want to look at.

B

So now we have actually left node level and the kubernetes api has a ton of responsibilities within criminales. Basically, all of the traffic at some point goes through the kubernetes api in the kubernetes cluster.

B

So it really has to be really well monitored in order to make sure that your kubernetes cluster still stays healthy, and just one example that I put on here is the ability to see how many, in terms of percentage, how much percent of the API calls that my that are hitting my kubernetes api server, our airing out with either 4 or 500 errors.

B

So, whether that's that's a useful grouping and for your your case, you'll, have to evaluate but I'm, giving you the tools to build your learning and now another case of, for example.

B

If you have an an SLO that you need to schedule pods in a certain time, this happens very often in cases where you're providing kubernetes as a service to others, so that you can, even if your auto scaling your kubernetes cluster, that you can see the end-to-end scheduling latency so that, even if you are resizing your cluster, that you know exactly even if I'm adding another node, the enfin scheduling time has been four minutes or something whatever SLO actually makes sense for your organization is, is up for you to decide.

B

But this is a very useful latency, where you can see latency analysis where you can see the 99th percentile of your end-to-end scheduling, latency and.

B

Going going forward when we have our kubernetes cluster, so we have our No. We have our PI server web scheduler and now to actually make kubernetes useful. We have our controller manager, which manages our deployments. Stateful sets and all these great abstractions that kubernetes gives us, but that also means that it heavily interacts with the kubernetes api.

B

So we want to make sure that not only from the side of the API server we're handling these things, but this is actually a metric that every single component in kubernetes has because every every component shares the same restclient implementation. So we can see that all all requests that end with something: that's not a successful request over the past five minutes. If that has a high value whatever high for us means, maybe it's a maybe a percentage is better for you.

B

Then we can alert on something like that. So, finally, now we've actually covered all the kubernetes components themselves and one super useful super useful thing. At least I like to think that, because I maintain, it is coupe state metrics which doesn't report status of the component itself, so API server tells you all the requests that have been issued against itself, whereas coupe State metrics gives you metrics about the kubernetes objects that exist.

B

So something that is very common to be used here is, for example, a deployment has a number of replicas that you wanted to create for your pods, and then it has a status and tells you how many are actually available right now. So this is something that a lot of people do because and you've probably all done this.

B

Maybe you already use coop site metrics, but when people get started with kubernetes, they often start monitoring with coops, ETL or coop control, whatever your favorite pronunciation is, and they check coops ETL get deploy, and you see the number of available and the number of none available. Replicas and all of this information actually comes from the kubernetes api. Obviously, so we can create metrics on this and pretty much every object there is in criminales actually has metrics like this.

B

So it's super useful to create alerting rules for this and then finally, our very last component in kubernetes is etsy d, so etsy D is the the heart of all of our operations in kubernetes. So we really want to make sure that HCD is healthy. So does anybody know what the most common outage for HCD is or whatever you encountered anybody?

B

What was that ultra discovery that this full anything disk related is really what what causes etsy D to not perform in the way it should be, and at Korres we develop at C D. So we have a lot of experience running at C D, and this is actually the single most common outage of Etsy D clusters. That city simply cannot keep up with writing the right ahead, lock onto disk so yeah here you go, install alerting roll in your cluster and hopefully it'll prevent you from having a real outage before it's too late.

B

So now that we've seen all of this in theory, let's have a very quick lit. Look at some real data.

B

So I have a Prometheus server with a ton of targets, so I have automatically discovered all my urban eTI's components and monitoring components. With the Prometheus service discovery, we can see the API server, my blog controller manager, scheduler of state metrics, the couplets and all of the other components that we've looked at and.

B

Then we can use all of this wonderful data to create beautiful dashboards and we can see the utilization across our entire cluster or we can see it based on all of our namespaces and we can see which namespaces we can drill down on to each namespace or once we are in that namespace. That is acting up in some way. We can look at each individual, pod and so on.

B

So this is super useful, because at the end of the day, we can most of the time boil some problem down still to CPU or a memory problem. At least that's some side effect, so these dashboards are really useful.

B

Now I recommend everyone to either share them or build them yourself yourself, if they're not good enough for yourself and also the an effort that a lot of us in the prometheus community have been trying to work on is the ability to share alerting rules, because, since we're all running kubernetes clusters, we all run the same components. So we should all be experiencing the same problems. So it's really really. What we should be doing is as a community.

B

We should be collaborating on these things and share share this yeah, so that is it from the Prometheus side and now Piatra with dr. ever ok.

A

So this was an example of pull based more with pull based monitoring solution because parameter scripts, endpoints a system system components or from your applications, and there is other set of monitoring solutions. Third-Party monitoring solution that works in the other model. The push model stack driver is an example of of such solution.

A

So I will show how we integrated at Google stackdriver with GAE. This can be a reference implement integration for other providers as well. So we have a kubernetes cluster. We have some notes on top of the picture. There is various tag: driver API stack driver works as a sauce, so it's offered your API. You can go there and see your data and matrix, so matrix are collected into two ways.

A

Hipster is pushing resource usage matrix those system-level matrix to stack driver, so we use we used the the common solution for this purpose for stack drive for application level matrix. We have some integration with Pramod use so that remedial scraper scrapes matrix from your application exactly the same way as the pedometer stacked up and then matrix are eventually sent to stack driver monitoring.

A

The second piece of the integration is metadata: how do we collect those metadata so on every single node?

A

We have a stack driver metadata agent deployed, so this is a kubernetes daemon set which basically watch objects from kubernetes api, computes, interesting information for us and send them to stock driver metadata, API and third piece of this integration is logging, so we have stagger over log engagement, which is flew nd when is deployed on every kubernetes node within the same demon said that stagger our metadata agent is deployed to and is pushing all those logs to stack the river logging API.

A

In addition to this, you can imagine also that just your application can send that I thought to stack divert directly and there is also an effort in the community. I. Don't know whether this is this is. This is probably under CN CF, yes to standardize, matrix format for the whole cloud world, so there's multiple companies involved around the word. I, don't know the details about this. That is what what's the name Frederick do you know open.

B

A

There is open matrix standard that is being designed trying now it's on on github, so you can find this effort as well, and stackdriver also offer some dashboards based on the data that are delivered to to stock driver.

A

Google just announced a new stock driver monitoring for gkf or kubernetes product, and here the dashboard so coming back to our original motto. That kubernetes is not about monitoring.

A

A

We have core metrics pipeline der. We have third-party monitoring solution. The question is basically how to connect third-party monitoring solution with kubernetes. So the answer is custom metrics api. This is an api that is a part of kubernetes, but doesn't kubernetes doesn't offer an implementation of this api, so this was an innovative approach, because before every single kubernetes api had a default implementation, and now we just introduced an api and it ups to fat party monitoring vendor to provide this implementation. There is a few such implementation already and more are coming.

A

That's it. Thank you very much. Any questions.

A

No questions if you want to learn more, come to seek instrumentation deep dive today, well, dig into internals of the all the components that we explained here. Thank you very much.