Cloud Native Computing Foundation PrometheusDay EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Easy anomaly Detection with PromQL - David de Torres Huerta, Sysdig

Description

Lightning Talk: Easy anomaly Detection with PromQL - David de Torres Huerta, Sysdig

How to create an alert on a service whose load changes over the different hours of a day? How can I alert on a process that has different usage over different days of a week? Anomaly detection is one of the main challenges that Prometheus users face while setting up alerts. Systems are usually dynamic and the use of resources and behavior depends on external factors that vary over time. Setting up alerts with static thresholds in these environments generates a lot of noise, causing alert fatigue in the operators and ignoring important notifications camouflaged among false positives. In this talk, we will see the different kinds of anomaly detection, when to use them and how to implement them in promQL. Although PromQL does not have specific functions for anomaly detection, as it has for linear regression, it does provide the building blocks to create different kinds of anomaly detection. We will also discuss the possibility of creating new PromQL functions that would make it easier to create this kind of anomaly detection alert.

A

Thank you so much hi everyone. I am david torres from cystic and the first david from sydney that you will see in this lighting talks. By the way we are hiring if you are called david. Maybe you have a chance and okay we'll talk about uh easy anomaly detection, a little bit of an introduction actually early in the morning, you're made an amazing presentation about anomaly detection, and actually this can be seen as a continuation from that, because, first time that you see anomaly detection, it is like when you learn how to use a hammer.

A

Everything seems like a nail and it's not like that. You have to learn and know when to use them and for what. So, let's start, let's first uh see what is an anomaly to learn. If something is anomaly to have to compare that with something and to be able to model that we will use a little bit of statistics we'll make a simplification. This is like.

A

Okay, let's consider a spherical call: okay, let's consider our data to fit on a standard, normal distribution, but these have some advantages and they are that normal distribution have the property that we know that in the range of average minus plus a standard deviation, we have approximately 70 of the samples and if you go further to twice the standard deviation from the uh average, we have 95 percent of the data.

A

So even there is a seed score that it is. How far is data from the average measured in standard deviation? So this way we have a way to calculate if a value that we are evaluating is an anomaly or is this something normal?

A

So, let's get to the work first, we will see how to calculate something anomalous in a group. This is a easy example. Let's imagine we have a lot of temperature sensors and one of them is getting crazy, but how do we know it is crazy or not? Where is the point?

A

Well, uh proncus is giving us already the functions to do that and actually in bjorn's presentation, we saw already a snapshot of this. We are using average function and standard deviation, and we know that using this. Actually, you see the blue part of the expression is getting that part of the anomalies and the orange part of the expression is taking the another, the bottom part of the anomalies.

A

This is easy, but what? If we don't have other group of similar samples or sensors or time series to compare with? Well, maybe we can compare things with what happened before so maybe with this anomaly. If we know that our time series is smooth and suddenly there is a peak or a valley- and we can say: okay, let's compare with the last five minutes or the last hour or the last week.

A

Prom q already has functions for this. We have the average over whatever, or here we have the average over time, and we have the standard deviation over time and adjusting the window.

A

We can use them to say okay, if the value that I'm trying to figure out, if it is anomaly or not, is over the standard deviation of the last five minutes, plus the standard deviation x times or the contrary for the bottom part. Maybe this is an anomaly and maybe it is worth investigating, but this is okay. If I am looking for external temperature, for example, something I know it will always be smooth, but what happened in normal life, normal life? We have bad processes, we have things that happen every week every hour every day.

A

So maybe this peak is not anomaly. Maybe I can discover that this happens before. Let's see how we treat seasonality, the trick is that we have to take the spans of seasons and we have to take the previous one to the future, to the present time and the another one again back to the future and so on. We can do this with the offset function, but there is a problem. If we only use the offset function, we are still having the same time series. So we will have duplicated data in the same time series.

A

So here's the black magic we are using the label, replace to add another extra label- artificial one. So they are not the same time series and then suddenly I don't have one time series, but I have a lot time series and I can use the same trick done before. I have a group, a group anomaly detection like we saw before so I can say: okay, if I can make the average of the previous thing that we saw plus twice standard deviation whatever. Then I'm setting up this red limit that you are seeing there last.

A

There are some scenarios. They are not common, but sometimes you can do that where you can make or you have a function, correlation function that can predict a value according to another time. Series, for example, in case of the energy consumed by a building, in fact, is a function of the external temperature. So you know if outside you have 25 degrees, the building should consume x, kilowatt kilowatts.

A

If not, we have a problem, so you can model this directly in from ql, you can write the correlation function, plus the error or correction function. Minus the error to have the bottom limit and just check if it is outside or not this safe band and okay. That's all thank you. So much.