Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Federated Prometheus Monitoring at Scale - Nandhakumar Venkatachalam & LungChih Tung

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Federated Prometheus Monitoring at Scale - Nandhakumar Venkatachalam & LungChih Tung, Oath Inc (Intermediate Skill Level)

In Media Build and Products under Oath, We run 12 production Kubernetes clusters running across our data centers with ~1200 machines with multi-tenant deployments. We monitor our cluster with Prometheus, each cluster runs a Prometheus instance and overall a single federated cluster with a persistent storage. Total time series is ~17mi (max 5mi /instance) with samples ingestion rate is 300K (max 80K /instance). We have built mind-blowing dashboards at a federated instance like Controller, Scheduler, API server, DNS, Kubelet, Etcd, Utilization overall and per-tenant namespace/ deployment/container gives high visibility. We leverage Alert manager which provides powerful alerting capabilities alerts on call on cluster status, nodes availability, scrape status, fd usage etc.We would like to share our experience of how we monitoring multi-kubernetes cluster with the multi-tenant environment

About Nandhakumar
Nandhakumar Venkatachalam is a Princ Production Engineer, Lead for Kubernetes Infrastructure/ Cluster management team at Oath Media Build and Products. He is a subject matter expert and solution architect specialized in high availability. Nandha has been under Oath for 11 years and has been dealing with operations at scale for more than a decade. He has delivered and led high profile product launches for Yahoo Fantasy, Yahoo Sports and contributed to flawless Fantasy season year after year. Prior to Yahoo, he had worked as a Linux System Administrator at IBM.

About LungChih
Lungchih Tung is a software engineer in core infrastructure team at Oath Media Build and Products. Lungchih has been working on building core infrastructure with Kubernetes, monitoring system and automating operations of cluster management.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Good afternoon, everyone, my name, is Nandu Kumar from what.

B

My name is dawn. She I'm Ronald, yeah.

A

Welcome to yet another permitted stock, we have seen a lot of talks around promises. It's pretty exciting, to see the community going on per mitosis, so before I go on, so we are from old. How many of you know what is what.

A

Wow yep I'll catch up with you later I'll, get you a swag yeah so, as he said, oldest a company created by Verizon when it acquired Yahoo and it also matched with AOL and Tumblr TechCrunch and all other products we are from both.

A

Specifically, we work from work on a core platform which provides the parameter kubernetes set up for all our media products, which supports Yahoo, Finance, Yahoo fantasy, and you know front page news and all these applications run on our kubernetes cluster.

A

I would like to take you to journey of how we achieved a good visibility on our monitoring system, very specifically around the kubernetes clusters. So before I go there, I want to just show you how we started with the kubernetes itself right, so we started with the simple cluster in a single data center and the first. The next question comes always hey. How is your monitoring looking? Yes, we started with the kubernetes dashboard.

A

It gives you a nice feeling about the CPU usage, the deployment usage, the memory percentage and, and also the node pressure details like how many parts can be allocated and all these details is this sufficient for us, probably not so we wanted to address this problem in a better way. So we looked at couple of operations we have so we traditionally we monitor yeah in the business for 20 years right. So we have been monitoring all our components using our one agents, so we have open TS to be pretty solid database out there for monitoring.

A

So we have an agent running. It runs the CPU collects of emeriti network details and all these details are sent to the central monitoring system. So we pretty much. We did the same for our kubernetes clusters too. We were happy because we were able to get good health about the the external system. Metrics.

A

Why we keep getting questions right. Are we monitoring really the kubernetes ecosystem? Rightly probably not yet so, then we looked at the search, hey. What's the best way to monitor kubernetes citizens back in 2016, I'm talking about so the first result you get in the Google is hipster right, so we looked at yes. The queue kubernetes dashboard is powered by hipster. We looked at the option to persist that data does kubernetes desk was real-time, there's no long-term storage right. We tried with influx DB.

A

We had definitely good success with that not going into much details about it. At the same time, Prometheus one point: X was there: we've used Prometheus one prometheus operator to get started immediately and just sorry to say that Prometheus is one point. I did had some performances, you could not meet our scale, so we were seeing a memories. Are falling down and the disk space gets filled up very quickly, so we were trying alternate approach. That promises also supports the remote write plug-in.

A

So we take that plug in and try to send the data to our in our system. So you know we were doing a lot of experiment with that, but definitely we are missing some insight into our kubernetes cluster, so I'm going to talk about it.

A

So, on the other hand, so the business is growing right that we were success with the kubernetes set up on one data center. So now we wanted to proceed on multiple data center so as as the cluster grows- and it is our footprint looked like, so we have twelve independent clusters, two key workers nodes- and you know 12k pause and 50k containers all the stacks right.

A

So we keep getting questions hey. How is your cluster visibility looks like how much CPU allocations is still available? Do you have a capacity to take more applications on board so who is the biggest consumer like we? We have a name space based isolation, so who is the biggest consumer in our cluster?

A

To answer this question, we sat back and we came up with a bunch of requirements within ourselves. We in order to be be successful, monitoring your kubernetes infrastructure. We have. We need to have an answer to all these questions, so we need to have an ax. We need to be monitoring or control, plane, components at CD, API server, and you know any atoms. You name it like. We wanted to grab all the dashboards our metrics and get a visualization out of it at the same time. So this the the applications we are getting more applications.

A

We wanted to make sure we have a good control over our our users like how much you know how much? How is the deployment doing? How is the pod doing even the namespace level utilization I just want to call out I, don't want to repeat some of the presentation from keynote fabian mentioned about the prometheus to find X, it's pretty great. So just I'm just put those numbers here. It's only so parameters. We read about the blocks Prometheus to point. X was pretty great and attack, ATX implement the storage and some computing improvement as well.

A

So we we wanted to take a good step on it and we really spend a good amount of time, understanding how Prometheus works and and how we can really go further and achieve over achieve our goal of monitoring of kubernetes ecosystem.

A

So next to focus is, of course, I think there was a talk from Bob cotton yesterday about all the monitoring equals all the promises, metrics available and kubernetes ecosystem, so pretty much I listed the same, so we have a net CD C advisor, Kuebler, API, server, cube state matrix from pretty these are the pretty important metrics. So we said and focused on grabbing all of them not to mention the promises has a really great service discovery built in. So we use that to scrape you know any services you have. You have any services.

A

If you want to scrape all the end points, we just need to set one on allocation, so we use that to scrape all the kubernetes metrics into the Prometheus's, not to mention about the file based discover, which I'm going to show you right now. So so we now we started with. We listed so many components there right, so we started with each and every component, one by one. We grabbed all the metrics started with the API server, so we kind of we used here the file based discovery, which is not bad.

A

So we have multiple environments. It's a bare metal server, so we have a luxury to you know hardcore the server name, so we put them in the server server list here we use the service, account token credential, so parameters, let's scrape it so the one another thing is: you can see here. So this is how we add the data center label. I is pretty cool, so we have multiple data center. If you want to label as well as part of the scraping itself, this is one way of doing it.

A

So this is going to be useful when we see in the in the in the Federation world. So after we collected the metrics of the API server. Now we move on to other control, plane, components like HDD and cube scheduler and the Control Manager. We have some challenge there. So HCD is a huge restriction. We have that the ports are only accessible only from master, as you can see in the diagram, so we wrote a simple proxies so far, reverse proxy. It listen on five, four four three and any call comes to that proxy.

A

It only receives slash metrics and it forwards to the local porter. So in this case, if h-series running on two three seven nine is going to forward, then what should write here yeah. So, as you can see here, it's it's a parameter, two, three, seven, nine to the metrics API, so the metrics proxy will just intercept the call and forward it to the localhost two, three, seven, nine, it's less metrics and points, or so we collect at City matrix. Very similarly, we collect the controller scheduler. So this is this way.

A

What we have done is B. Basically, we collected all these metrics into the into the into the committees, so so I'm going to candle what the lung she's want to talk about, what we have achieved with this matrix so how we came up with the Federation.

B

B

So, as we mentioned earlier, we have twelve clusters, we have production environment, Canaveral, environment. Each of them has six clusters total. We have more than 2000 notes, so these are big clusters. Even Prometheus is very performant. One single promising server will not be able to handle all the data in these clusters.

B

So how do we solve this problem? I think the the answer is simple: we need to use Federation right, so permit is offer this nice feature called Federation. The concept of order of Federation is simple: it's just a hierarchical structure, we're in each cluster. You can set up one primitive server and that promise server will be responsible for scraping data in that cluster, and you know teaching to that. You can pick one cluster and set up your federated.

B

The provision server and your favorite provinces will be responsible for scrubbing data from all other individual promises server. So this is how we make it scale, but in order to do that, there's still a couple of things that need need to be done. So the first thing we do is that we aggregate our data wherever get our test series. So why do we? Why do we need to aggregate it? Because by aggregation we can reduce the amount of storage of our data and in the federated, the promises server?

B

We only scrape those aggregated the data so and of course another benefit is that if you Eric your data wherever you're needed, you just need to call your location metric and then your data will be showing up. It's super fast, so, okay, the next thing is, since we reduce the amount of the storage space of our data and you know federated from its server. We can have longer retention period, I think for us in our federated formation server. We set the retention period to be one year.

B

So so, if you want to debug for something or just for visualization perverse, you can just pull data up to one year, and the next thing is that since we we have our federated for missus server, so this becomes the most important server we have right. So therefore, we need to secure our our data in the Federated four basis server. So we add a permanent storage e to that. So in case, there's anything wrong goes in your federated Prometheus server, the pal crash.

B

You might want to delete the part and after you bring back the part, your data won't won't be lost, so they will still be there and now, of course, with the federated server. You have unified it display for all your data. You don't need to go to one provision server to see your data. You want in one particular cluster and then go to another one to see data in another cluster.

B

So this is a another benefit of using Federation.

B

So this is our Federation configuration I. Think. Actually we in a federated provision server, we script, two kinds of data: one is the irrigation data that we just mentioned earlier. The the other kind of data is the important community components, control plan, components so I think in the red box. You can see that we script data for like Kobe is cool scheduler, Cooper, FBI server, EDC, the all those important component control and components.

B

So these are so important, so we don't want to lose any data for for them, and you can also see that we scrape the data with the name starts with either cluster or colo-colo means the colocation. It's just referred to our data center. So these are the irrigation rule that we created. So we also script than here so.

B

Let's look at some of the equation rules we have so to start with. We take some of the rigid rules from premises operator. If you are interested, you can just search it and you can grab a couple of nice rules there. So it's gonna be a good starting point. So, in addition to that, we also build our own aggregation rule, so mostly they are for CPU and the memory, because these are two most important resource. We have in our community clusters.

B

So this is this is an example of our aggregation rule. So this is for CPU utilization, although the rule looks complicated, but we can, let's first focus on the metrics we use on the top. We use container CPU usage seconds total. This is the CPU usage for our container and the label. Replace is just a function to create a another label.

B

In this case we create a label called controller from the controller is basically the sales deployment and we sang up all the CPU usage for all containers in one you can see in the end of the expression on the top. We group them by color.

B

So in the bottom we use matrix called containers back CPU shares. This is a metric that give us give us CPU allocated to a container. So similarly, we sang up all these for all the containers in one Colo and then with all these, and we came there if the CPU solution.

B

This is similar to the previous rule. The difference is just it gives us the different level of the specialization. This is the namespace level or you can see. We preserve two labels Colo and the namespace.

B

This is one more example. This is for controller level. We have three labels here: : M space and the controller. This gives us the pad level, and this is give us the container level. So with all these irrigation rules, no matter whatever you want to see, you want to see a civilization for deployment or for a pod for a container, for name space or for a cluster. You can use this area.

A

B

So now now we have all the data information servers right. So what's next, what should we do with all those data? We want to monitor our system right, so we want to get alert. If there's anything goes wrong in the cluster, we can get a notification and then we can fix the problem. So to that end we use a lotta manager. Then the manager is a separate library. You can deploy separately and it offers nice feature to help you to manage your alerts.

B

For example, you can you can use it to group your alerts, you can silence your alerts if you need some time to debug or you can use inhibition, so it has a nice UI like the pictures you're showing here. So it's pretty straightforward to use.

B

So that's that took a take a look of example of the other two rules that we have an example. This is for no not ready, so we used metric metric, called coop know the status condition this matrix tells us which node is going down. We also add some useful labels there, for example, severity in this case it's warning and a Colo environment with color and environment we can tell which alert it is coming from.

B

This is another example of whatever rule we have. In example. We use the irrigation rule we mentioned earlier. This is for CPU utilization. If the CPU decision for a namespace goes beyond 75%, then we will get this alert. You can see that the severity is critical in this case. So with the severity you can t always alert it differently. For example, if it's a warning, you may just receive email notification. If it's critical alert, you may want to page your uncle and fix the problem immediately.

B

So this is the alert template. Basically this. If you have used email notification, it's gonna be the same as you can see. Your email lists all the labels and it also description summary you can. Basically, you can some basic the park steps there and we will receive this notification pretty much. You have everything to deal with this alert.

B

So, of course, we couldn't miss the most important alert, which is to alert our primitive server. Obviously, this server goes down. None of this will work. So to do this, we set up alerting our federated parameters server and, if there's and if any individual committed server goes down, a lotta will be triggered in the Federated premier server and we will receive the notification.

B

As for the Federated appointment server, we can have a simple cron job that was periodically to check the status of our federated from Asus server. If there's anything wrong in our federated fermius server, you can receive notification for this.

B

So next I'm gonna head over to nada, to talk about the dashboards yeah.

A

Thanks long, she before I actually done want to the dashboards any guess. So we have we mentioned that we have 2,000 servers and he guess how many outlets we must be. We may be getting in a week all in there.

A

That is my manager. He knows everything, yes, how much 50 it's pretty close, not too close. Oh, so we get close to four alerts a week we have 2000 bare-metal servers like servers, keep going every day at least one or two servers keep doing, keep going down. So some of the concept in the let manage at lunch. It cover that you can so we set the severity to warning if one node goes down, so we don't get a lot paged for that of course.

A

So if there's critical alerts we get page so so we come up with a complex, alerting rule say you know: if there's one node goes down, don't alert. If there's a 10, node or 10 percent of the node goes in a cluster goes down then page me. So all these things possible with a lot managed. It's it's a similar explore expression. We use for dashboarding also yeah. Thank you, so I really want.

A

So now we have as many as as he mentioned, that one application of using the data is alert for alerting the other application is yes, we wanted to use these data for dashboarding. That was our initial goal, so not to bore it's going to be exciting, I'm, hoping that this is. We really got a good product out of it yeah, so the first dashboard, so you came up with the executive stats on our cluster- shows how many nodes, how many deployments you know how many namespaces.

A

So this is all a federated dashboard you can drop down in in a drop-down. You can choose a particular data center. You will be able to get in a single data center test stats. The next one is the cluster view. It's pretty cool right, so we within a single single spill single from single place. We are able to achieve what's happening on the entire cluster, so so this gives us this. This dashboard helps us with.

A

The capacity planning is basically shows how much Headroom we have for suborning new applications and also what is the current utilization of the cluster. This is pretty cool and I. Think some of the all these dashboards are derived from the alerting rules, so the aggregation rules we rewrote.

A

The next one is the namespace in a multi-tenant environment, so we have close to 8200 namespace and we wanted to be able to figure out who is our biggest user and how are they performing in terms of usage like? Are they consuming? Are they are they being a good citizen, getting a good, CP utilization? So all these things we were able to track it from a single place. So this is pretty cool, so this is one of my favorite dashboard.

A

It's a deployment so, as you can see here, there's a green and a yellow, so that shows that previous release and the current release. So from a single plane, we were able to observe the differences, the the memory difference and the CPU utilization difference over the week. You know: is there any previous releases?

A

Sorry current release, cost any code issue or anything like that, and and then application users able to enjoy watching the auto-scale happening and container restart councils pretty useful for our application team, not to mention we have a lot of kubernetes control, plane components here, so we, the controller scheduler, we pretty much. You know watch all these dashboards like in is for controller. We have a CPU sage, we have each and every controller. Amid some metrics Rozonda part autoscaler deployment controller. You know, demon set controller.

A

You can pretty much watch how these controllers are performing scheduler right. So the we have something similar for api server. So in api server is the core here right, like you: have all the clients talking to api server via other Beaman sets or a deployment or a controller cubelet, even your custom controllers. They all talk to api server.

A

We had ran into couple of issues where the api server was rattling from the go client, so we saw a flat line on this QPS for a particular client, so we were able to catch it pretty nicely from these dashboards.

A

So a cube lat. We have thousand two thousand nodes, so drop down and select a node and observe. How is the darker latency how's, the runtime latency. Everything thanks to I, think Tom also showed this dashboard, and so we just grabbed shamelessly from the core OS repo and and we were able to get a good view about Thea. Its CD metrics- this is one of the interesting dashboard, so we had a. We had a war room for alert alerting session.

A

So one of the challenge we had was we have so many systems is hard to track, which machine has which package version. So we came up with a tool we just read: Ron wrote a simple scrip which labels a node with the darker version or Debus version or RPC, find some critical demons. So from from a graph on with a pie-chart plugin, we were able to get a good. You know miss realization around it, not to mention so.

A

The scale at which we operate is that we have 25 million time series flowing on to the promises, and- and it's we be- we be very careful about this- is the ELA linus the promises, federación time-series. So even though put together, we have close to 25 million. Excuse me, Federation just have three million, so we get what gets into the Prometheus Federation. So that way, as he mentioned, we were able to provide a long, long retention yeah.

A

So that's it from us, so I can be reached on Twitter and the dashboards we have demoed here, it's all available on the Mike github repo. Yes, Thomas is working on a good approach of how to share the dashboards. Eventually, we will try to use the case on net over lattice mix in so right now. These are plain: J sounds out there and feel free to try out.

A

Thank you. We got a question there.

A

Yeah, that's a great question, so we have two data centers, so the one we shown in one data center, the diagram was simplified version. So we chose two. We have six data centers totally. We chose two data center for a Federation, so.

A

Yeah we were just discussing this morning: I should have got some picture of what our TV sees. So we have a. We have a 15 member team and we have a couple of TVs, so these dashboards just revolves there and of course we have our thing too. We have a slack integration, so we get a lot so, but when we come to an office we get a good view of our cluster status.

A

Right everything is independent graph on our dashboards right now, but you can just drop down and get to a different dashboard. I believe graph owner has a drill down version of getting into a dashboard I think we should be working on that.

A

A

Can you repeat.

A

Yeah so the question the do, you have any redundancy on the federated. We have two data centers scraping the federated metrics. So if one goes down, we have another one.

A

Yeah Prometheus is a single server one. So for federated. We just have two servers running in two data centers. So.

A

Right, the question is: why don't we use that yeah? So we did try with with primitives one point X, so that time we were not able to permit is one point was X was not able to scale over need. We were filling out disk, so we went on another route of trying with remote right and now, with the two pi X. We don't see any issue like a TX improvement, the performance so right now our X, the storage, is able to handle one year worth of retention. I was just talking to Gautham in the afternoon.

A

We would probably try Thanos or couple of contacts right, so the couple of plugins they are working on, so we would be trying with that for a long-term storage.

A

Hey-Ya, as a question is how do we secure so we have multiple data center. How do we secure the federated parameter scraping? So it's a big organization. Some of these problems are solved for us, so we have an ingress controller sitting in front of every kubernetes cluster, so we were able to jump onto that securely. So in in in Yahoo we have internal ports, it's not publicly or publicly open when it travels through a Colo to Colo a data center to data center.

A

We have a data center level encryption, so it's taken care for by the infrastructure team.

A

Thank you so much you so much.