Cloud Native Computing Foundation PromCon Online EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: How to fill in historical data for new recording rules - Jessica Grebenschikov

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: How to fill in historical data for new recording rules - Jessica Grebenschikov, Prometheus

Have you ever created a new recording rule and wished there was historical data for that rule right away? If so, you're in luck because now there is a way to fill in historical data when new recording rules are created. Find out how in this talk.

A

Hi, my name is jessica and I am a software engineer and a contributor to prometheus, and this is my lightning talk on how to fill in missing data for recording rules.

A

So many moons ago, back in 2013, an issue was created that made this request to have the ability to persist, data for evaluating recording rules in the past and I'm happy to say that, eight years later, uh we finally have a pull request that got merged recently. That implements this feature, and this talk is about how to make use of that feature to fill in uh recording rules from the past and so just a tiny bit of background. What is a recording rule um in the prometheus official documentation?

A

It shows here um has some information on recording rules that they um are essentially way too. They are ways to pre-compute, um expensive queries and save the result so that you don't have to evaluate those expensive queries over and over, but rather the data is saved over time.

A

So here's an example of a recording rule that figures out what is the current network traffic um per second, and you can see the query expression example here that gets run, um so the problem with recording rules is that the data only exists from creation. Time on when you create a new recording rule, there is no past data for that rule, so um we're going to use this example through the demo of making a rule that tracks the network traffic per second.

A

So here is a config file for recording rule example, um and if you so, when I created this recording rule, uh then you can go look in prometheus. You can look on the dashboard, so, on the left hand side. This is the query that was executed, and so you could see for the past hour, there is actually data time series data that exists for that. However, if I just put in the name of the recording rule, there's no data here, essentially just because it's brand new, so it doesn't have any time series data.

A

So the goal of this is to fill in the past data right here, so that these two match, and um now we have the capability of being able to do that with prompt tools. So I added this feature into prom tool. uh This is the sub command. Tsdb create blocks from rules, here's all the options. So if you do run that like so, um you need to provide a start and an end time.

A

So in this example, there's an hour window that we just looked at on the previous chart that our window is what we're going to fill in and then the url to the prometheus api is passed in and then the output of this sub command are blocks blocks of time series data for this recording rule so providing a location of that output directory.

A

This is where all of the blocks will be so execute this command. You can see some logs. It says blah blah blah, we're running this rule, processing it and then afterwards you can ls into the output directory, and you can see that there's one block created to inspect that block even further there's another sub command. You can kind of see more data about it, so you can see it it's about an hour here and there's 70 samples which the evaluation interval is 60 seconds.

A

So this is about what you would expect um in order to make use of this data in the prometheus instance. These blocks once they're validated um they need to be manually, moved over to the data directory of the prometheus instance. So I did that um once I did that you can look in the logs and you can see prometheus now sees that there's some over uh lapping blocks.

A

So when the next compaction hits um it's going to merge those blocks together and once that happens, then the data is available back in the prometheus graph. So back over here, this is the first chart we looked at. You can see. The actual query is right here and we have all that data and then the recording rules over here now now we actually have all this past data for the recording rule, um and you can confirm that it looks pretty good and that's it so for future work.

A

For this um I would like to add a another sub command that will validate the blocks for you and also move the blocks over to a specified previous editor just to eliminate those manual steps and then um still, we need to add still markers. I'm gonna do that soon and anything else.

A

If, uh if anybody uses this feature and has any ideas how to make it better or there's other um use cases that are needed or bugs or whatever, um let me know, and then lastly, I just wrote a quick blog post that uh has these kind of the same information from this uh talk and with a little more detail and links to docs and stuff so check that out. If you want- and that is it thanks, a bunch.