Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automatically Making Dashboards Load 100X Faster - Shreyas Srivatsan, Chronosphere

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Automatically Making Dashboards Load 100X Faster - Shreyas Srivatsan, Chronosphere

High cardinality metrics often cause alerts and dashboards to time out when they try to fetch too much data. Prometheus provides recording rules to speed up queries by pre-generating the queries, however, they have to be configured manually and require reconfiguring alerts and dashboards to point to the recorded series. The performance degradation often happens as new metrics are introduced with more instances or deploys and a working query may break all of a sudden. In this talk, we will show you how slow queries can be preemptively detected and automatically sped up without any manual reconfiguration. We detail two approaches to achieving automated speed ups - one that is based on recording rules and the other is based on the M3 Aggregation tier. We will compare and contrast both approaches and show examples of how one can leverage either open source method to achieve the same results.

https://sched.co/ekCu

A

Hi good morning, good afternoon, good evening, depending on where you are today, we're going to talk about um making dashboards like automatically faster, uh using some tools we already know about like recording rules and also we're going to dive into kind of the m3 aggregation tier.

A

uh My name is shreyas.

A

uh I'm a software engineer at chronosphere uh we're working on building a hosted, metrics and monitoring platform targeting large-scale high-throughput use cases, um I'm interested in all things, observability, primarily kind of focusing on making use of metrics and traces to solve like interesting problems. uh Prior to this, I was on the observable team at uber working on kind of like building out and scaling the alerting platform there.

A

The agenda for today is kind of setting up the problem, uh looking at kind of like high cardinality metrics, and why they're a challenge, um then we're going to talk about kind of aggregating, metrics using recording rules and the m3 aggregation here and kind of like how that is uh yeah how we can kind of make dashboards faster using that, then we kind of talk about how we make things easy to use, uh and finally, we kind of round out with the demo and kind of a q a so to set up the problem.

A

For most of this talk we're going to take c advisor as an example which provides essentially resource usage and performance metrics of running containers. This dashboard taken is kind of the c advisor dashboard from grafana.

A

As you can see, it has some amount of aggregated information and some amount of like individual kind of like pod level information. So it has like it's tracking 5000 containers. You have some aggregate sets like cpu usage, network traffic, etc. You also have like per pod stats, like cpu usage and memory usage, which are useful, so the aggregate stats kind of give you an idea that something is wrong and then the more like per pod stats kind of help your root cause better.

A

um So this has like tons of dimensions uh because it wants to kind of slice and dice metrics in different forms. So what do these dimensions actually do? If you consider a simple metric kind of cpu usage, you end up having like 16 000 series and it takes 20 seconds to actually query this information like which is really slow.

A

This is because, like you can see the so many different tags on it like there's a part id there's an instance there's the image and so on so forth.

A

If you only consider the two labels that we're interested in- and we actually aggregate this using a recording rule, then this query becomes a way more manageable, like 230 series and just takes like half a second to actually like get back information uh and in most cases this is actually all the information we want.

A

Now. The reason for getting so many series here is like, even though it's kind of like containing it's kind of something, is tracking one instance.

A

As things get restarted like as the pod gets, we started reschedule like a new time series shows up, so there are more things actually getting stored in the time series database which don't have data currently. But when you actually like query for like 30 minutes or like a few hours of data, you end up pulling time series which should not exist, so it basically results in like slowing dashboards and dashboards, which kind of like constantly decreasing like degrade over time.

A

As mentioned. This is because kind of the cardinality of dimensions keep increasing. As you add new instances, roll out new images like things basically yeah, we keep adding dimensions and things keep slowing up. Okay, so this basically results in kind of slower dashboards, slower dashboards results and kind of browser, locking up and a bad user experience.

A

So when an engineer notices this, the dashboard needs to be optimized like how we actually kind of go up. Go about doing this. The first step is kind of figuring out which queries are the culprit. You inspect the requests. You could inspect the request from a dashboard to look for slow queries.

A

You could do this in kind of chrome inspector.

A

You could use the prometheus query log uh to do this and kind of associate back and yeah and try to associate back to the dashboard, but this is difficult because the prometheus query log has like all the queries running in the system and the dashboard itself, like you, may have many many dashboards, but given that we figure out kind of which are the culprit, you probably want to do some form of pre-aggregation of these metrics to make these queries faster and over the next few slides, we will kind of talk about a few of these.

A

Like pre-aggregation like a few ways, we can actually pre-aggregate things. uh The first thing I want to talk about is recording rules. This is something prometheus provides: support for and kind of, they're like widely used for this purpose.

A

They basically allow kind of pre-computing queries and storing back aggregate time series to the tsdb.

A

Once you do that, you essentially instead of making the query which kind of looks at all the series, the dashboard is just making a query which is like picking back one series.

A

um So it's like super fast and the caveat is that now the dashboard now has to go and you need to point it back to the pre-computer time series you need to go and like change the dashboard to actually access the real-time series. Recording rule looks like something like the below. You have a record which is basically the new time series you're storing stuff in um and then the expression which is like whatever prom q and expression.

A

You want to get executed, so you can put anything as complicated as you want here and, as mentioned, the record is essentially a new time series um so to basically create recording rules. You need to know what to pre-compute. So we figure out the background queries by doing some formatting analysis on the dashboard.

A

Then you basically go configure these recording rules and you can go and change the dashboard people kind of query the recording rule metrics instead of the underlying metrics. So it's a real long manual process. Let's say you've actually done this.

A

What happens when one of these metrics changes like the query for the recording rule has changed or a new panel becomes slow? You basically have to repeat this process all over again and kind of do the same manual processing again and again.

A

The second thing is: recording rules are very expensive. They basically execute and pre-compute the score at regular intervals.

A

So a query accessing many time series can get expensive very quickly, especially when kind of the dimensionality of the underlying metrics keep increasing, and given that these kind of run alongside other queries in the system like dashboards and alerts, they have the potential to kind of orient the query engine.

A

So those are kind of a couple of like really bad aspects of kind of using recording rules and which we need to kind of be careful about, but as mentioned before, sometimes for underlying metrics, we don't actually need all these dimensions, so it would be nice if we can actually get rid of these completely and not store them. What we care about is just the aggregate and we don't really care about kind of the underlying metrics themselves.

A

So would be great if we can do that and that's kind of where the m2 aggregation tier comes in. So m3 is a remote storage for prometheus, and the empty aggregation here allows us to kind of move the expensive recording rule computation to streaming aggregation.

A

So when prometheus remote write comes into m3, it sees that some metrics need aggregation and kind of it forwards it to the m3 aggregation here. The empty aggregation tier basically knows what rules to apply on what metrics it applies. Those aggregations and then sends it back to the coordinator to actually like persist it into long-term storage in m3db, so the aggregator basically allows down sampling, dropping or aggregating metrics prior to persisting them to the time series database.

A

uh The aggregator supports kind of two different types of rules. One of them is called roll-up rules which allow aggregating these metrics. uh Then second is mapping rules which basically are kind of dropping random metrics uh we're going to first we're going to talk about both of these, but first we're going to dive into roll-up rules.

A

Roll-Up rules are a way to aggregate metrics, so they basically have a series of transforms which are applied in order uh to kind of change and generate a new metric and the metrics kind of applied to depend on kind of what filter actually matches. It also has something called a storage policy which basically determines like where, like what like, where to store the generated time series in m3db we're talking about kind of one by one. uh The first step is kind of what we call like a transform, an increase or a delta transform.

A

uh The underlying metrics, which come from prometheus, are kind of monotonically increasing like counters, so we can't really aggregate them, as is so. We need to apply the equivalent of kind of the prometheus rate function first, so that's exactly what the increased transform is essentially applies. The prometheus rate function gets diffs between successive data points and kind of generates, a new time series for kind of use by the next level of the transform.

A

The next level of transform is called the rollup. It essentially sums the deltas by the unique dimensions specified in the group by, in this case we're only interested in container name and namespace, so it does a sum by container container namespace in like fromql language and it stores the generated thing as a different metric name.

A

Now, the metric name is interesting. uh If you're actually using a mapping rule to drop the original metrics, then we can basically store the rolled up metric as with the same metric name as like as the original metric.

A

If you're not planning to kind of drop the underlying metrics, then this metric needs to be different, just like it's in in the recording rule case, but the advantage of actually dropping the metrics and storing the metrics using the same name is that dashboards alerts or anything which is actually querying. These metrics now just query the aggregate rather than the high dimensional metric.

A

So that's something to keep in mind.

A

Now, finally, like after we've rolled things up and we've kind of reduced the dimensionality, we want to store this back into the tstb, so we need to go back from dealing with deltas to actual kind of accumulatively, like increasing, like monotonically, increasing time series, so we apply kind of a cumulative add operation uh for each of the metrics uh to kind of get the aggregated like monotonically increasing time series.

A

This is basically sent to m3db namespaces, identified kind of by the storage policies and things get stored there now, as mentioned before, like if you actually want to go and store the metric kind of with the same name as the original like original metric, then we need a way to kind of drop the like the raw like high dimensional metrics.

A

uh This is where kind of mapping rules come into play, so mapping rules also have a filter uh which basically says. Oh, these are the metrics that I'm actually interested in um match these metrics and just drop them do not store them, so the roll-up rules would still apply, but they won't actually get stored to the tsdb.

A

So the combination of roll-up rules and mapping rules uh we can get like aggregation like really easily like at ingestion time uh so to summarize kind of the aggregation here. uh It basically allows for kind of ingestion time streaming, aggregation uh the metrics can be aggregated or rolled up based on, like whatever rules you provide.

A

Different functions are possible and I think the key thing really is that there's a way to kind of like drop the raw metrics based on these matching filters, which gives us the like the great characteristic that dashboards and alerts and different queries could automatically sped up.

A

uh So we really have like two ways of doing things uh you just want to quickly like go over pros and cons of recording rules and roll-up rules, so recording rules are generally like their general purpose and they support full prom ql. So uh if you have this like super complicated like expensive query, which we want to speed up, then recording rules are probably the way to go.

A

The caveat is that they're expensive because they run against the regular query engine and they also kind of affect other queries, so you're actually crying like. If you have a recording rule going and like aggregating across 20, 000 time, series recording rule every minute or every interval goes and queries those 20, 000 time series and does aggregates and stores them back and the third kind of yeah. I guess bigger point- is that all the data needs to be stored. So there's a very high storage cost. You cannot um like with recording rule.

A

You will store the low dimensionality information and high dimension and aggregate information at the same like high dimensionality information and aggregate information. At the same time, which also means that kind of your dashboards and queries and alerts need to be modified to actually like hit the uh hit. The aggregated metric instead of the like the original metrics rules, because of the way they happen at ingestion. Time are much more efficient to run.

A

We have the option of only destroying the aggregates. We need and dropping other series, and if we go the route of only storing the aggregates, then you kind of get automatic query speed up, uh because aggregate ends up having the same metric name as the result matrix the biggest kind of caveat with roll-up rules. Is they don't support full prom ql, but rather like specific aggregates?

A

We are. We have been kind of adding more and more of these uh aggregates aggregate functions over time, but it's unlikely that recording like grow up rules would support everything. That recordings would support.

A

So now that we have these like two things like two ways of kind of aggregating things like how do we kind of make it easy uh to do these aggregations?

A

So we're gonna talk about kind of uh two we're gonna talk about kind of a couple of different things uh around like uh so we're going to talk about this tool, which we call the high cardinality analyzer, which makes it easy um yeah, which basically makes it easy to go and like do this yeah, which basically makes it easy to kind of go and do this analysis and kind of like create these recording rules and roll up rules.

A

uh So I'm gonna share screen again here, so this is actually available as a tool on github. The link to this is posted sometime like in the later part of the yeah in the latter part of the talk. So we have um it we're basically going to make use of the prometheus like query logs and we're going to analyze them and we're going to do like some operations to actually go and like speed things up.

A

The purpose of this example I emitted some metrics locally, just to kind of get like some basic information uh for the purposes of this demo. So it's not really high cardinality information with too many series. It's just a handful of queries.

A

um So what does the query log actually have? It has some query information here of what the exact query was when it started and then different stats like how long it took to eval how long it took to sort like that query, preparation time, any inner evaluation, and so on um so kind of just to run this.

A

uh We have like this high chronic analyzer. We point it to a sample query log.

A

We give it some target, saying I'm only interested in queries which have kind of a minimum query time of like 0.01 seconds, because it's a smaller example and then we have kind of oh.

A

I want to only like filter out queries which happen at least two times, so this actually shows you some like some options here uh then the query log, like the analyzer, has this way of like generating recording rules for these queries that we identified, uh and we actually see recording rules here, and it has another mode where you kind of can generate roll-up rules and generate mapping rules.

A

So what exactly is this like? Analyzer doing so, let's jump back to the slides kind of talk about that so analyzer, like primarily kind of uses, the prometheus query logs it logs. All of these queries, kind of like run by the engine and the query log has information about kind of where time was spent in the query.

A

So analyzer is kind of an offline process, a standalone tool to generate recording android roll-up rules, um as mentioned it uses the prometheus square log to find good candidates for aggregation, uh and it provides some recommendations for recording rules or kind of m3 aggregator roll up and mapping rules to create and speed up expensive queries.

A

uh So it just provides recommendations and then users kind of have the option of taking these recommendations and going and creating the actual rules themselves uh then kind of speeding up the dashboards. So a few steps like we would do is you basically have like days worth of prometheus query logs?

A

So we have enough information to know like which are the repeated queries which repeated expensive queries which are being run on the system. So we want to kind of find these most commonly hit expensive queries uh once we kind of get information about that, we want to kind of check. The cost of these queries are due to number of series, so the cost of a query could be due to multiple different reasons.

A

It could just be because it's very important, like large chunks of data, uh large chunks, of like a few series, essentially over, like a huge like sequence of time, or it could be like creating many many series of data uh like yeah many series of data and the cost like, even when it queries many series of data.

A

If it's actually returning all the data, then you can't really speed things up, but if it's actually acquiring many underlying series from the tstb and then it's kind of aggregating them together into like a fewer set of series like those are kind of the candidates like candidate queries, we're actually looking for.

A

So we want to basically look at kind of like the cardinality of the queries, which are actually being run and kind of use that and only optimize those queries.

A

So once it kind of identifies these, these queries, which actually need to be need to be need to be sped up. uh We provide proposals for kind of recording and roll-up rules to create, um and then users are kind of like free to go and like configure these rules as necessary.

A

If the user kind of goes and creates recording rules, then dashboard and other places where these rules are being applied need to be changed.

A

If we're talking about kind of roll-up rules, then- and the user is already gone and like the user also said that they want to kind of drop the underlying metrics, then the queries will kind of get sped up automatically.

A

As the query actually captures the aggregated metric um in case, the roll-up rule is not dropping the underlying series, then, like um then you'd like recording rules, you'll have to go and kind of like change, uh change, dashboards and other places where things have run. uh So we built this tool, um it's available open source. The link to this is in the next like in, like the last last slide, um so yeah. We encourage you to go and like try this out and let us know kind of how this works out for you.

A

uh So thank you so much um we're open to questions now.

A

If you want to kind of know more about kind of the m3 aggregation tier, like reach out, do reach out to us on the m3 slack channel and there's a link to kind of the high cardinality analyzer available here.

A

Yeah, there's a link to that on an analyzer available for you to kind of try out. Thank you again.