Cloud Native Computing Foundation Prometheus, 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Alerting in Cloud Native Environments [I] - Fabian Reinartz, CoreOS

Description

Alerting in Cloud Native Environments [I] - Fabian Reinartz, CoreOS

In a Cloud Native infrastructure, component failure is normal and expected. The loss of a single node or a dozen hard drives is automatically handled by the systems running a datacenter, removing the need to page someone at 4am.

This calls for an alerting system that understands service availability at a global scope, yet is still able to give detailed reports if and when there is a service-impacting incident. Prometheus achieves this by defining alerting conditions directly on time series data. The resulting alerts are grouped and aggregated into comprehensive and meaningful notifications.

Fabian will walk through the philosophy of time series based alerting, the Prometheus architecture behind it, and how practical anomaly detection can be implemented.

About Fabian Reinartz
| Fabian Reinartz is a software engineer at CoreOS and one of the core developers of Prometheus, a monitoring system and time series database. | Previously, he was a production engineer at SoundCloud and worked on information retrieval during his time at Saarland University.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So I'm Fabian I will record and I'm also Prometheus maintainer I've also been a barista Tina before at sound out and so I've been doing the stuff for quite a while. Now and I want to talk about alerting and cloud native environments, and the original was a loading based on time series, but this sounds virtual fun, I, guess, and but we actually want to talk about alerting with time series and and just to start out very theoretical and boring.

A

And what's the time series a time series is a stream of values and their couple of timestamp and an actual value and and the tightest term is always monitoring, monotonically, increasing and and to give meaning to the stream of samples. We have some sort of identifier and that's an our case here for Prometheus a metric named HTP requests total here, which we then split out into more granularity using labels. So not under this one counter. We have several counters for every instance. We have for every pass.

A

We have, or we said if we have for each unique combination of labels, and then we have to several times who is associated with a single metric.

A

And if we have that we can do interesting stuff right, we have now data that we came through in in this case here. We query all time series for this particular metric and this particular top level and because they're counters we now want to get the per second value so from just a counter counting up request, one by one by one. We now want to know what's actually window of the last five minutes, my beakers per second, that my applications are experiencing and applying just to sound.

A

The plan series gives you a pretty big result which doesn't really give you any good inside and what's what you're? Looking at and that's why you have a creations you can aggregate along with these flavor dimensions. So here we want to preserve the paths and said as I mentioned, and what we get back is unique topics of our past and set of labels and the currently associated recurs rate, and this actually optimistic is doing so.

A

Prometheus is a monitoring system and Tennessee's database, and it knows how to talk to your service care system like kubernetes and discovers all things that are interesting to monitor in your infrastructure and based on this information goes out and collects sample data from these, so called targets which can be routers, which is nodes, your micro services. Basically anything you want and having stopped that in our database now on our Prometheus node.

A

We can also query it and some of the query language you have seen before and thereby expose it and our HTTP API to things like asana, our own UI, etc, etc, but it's mostly geared towards basically graphing stuff, either for dashboard or on-demand. If you want to investigations to any knowledge, and now we want to do a low thing and and how we do this all right. Well, we have a ton of data.

A

We have millions of different time series and bridges data points associated with them and, of course, it's quite an repeating idea to just do all the French things, and we know from the terms like machine learning, anomaly, detection code rating stuff, making everything cell feeding, because we have big data right and big data generally means we can make everything magical.

A

So previously decided against this and very strongly and as a simple reason is that we're actually monitoring at scale and we really Auto users to instrument applications way more than they do. I usually means now and which means gets way more data. Then then it passed and when a machine learning etc might work quite well at a smaller scale. If you have little data and if you scale it up by a factor of 10, 100 1000 and something will always correlate right.

A

If you just look and look enough, some curves will always sort of add up and seem to make sense without having any sort of casual relation and and in the end, to eliminate these false positives, which will be the very bad for alerting right, because we get paid for nothing and you would have to put in huge efforts, and the problem is, if you will eliminate very aggressively, there's a good chance that you will somehow also cut out some others that should actually be firing, and because now you have false negatives and if you're alerting for the negatives are not really an option.

A

So if your datacenter is burning down and you don't get paged because you're machining so everything's are probably okay and doesn't really book, so um the question is having all this data can be still. However, the comprehensive approach that's easy to use and gives you full coverage and insight to add on things going wrong in infrastructure and, if you think about it, and our time series represents quite well the current state of everything about notes about micro services, other routers and everything, that's basically instrumented and for everything that's running.

A

We have an idea of how it would be running. That's commonly known as SLA or I said, oh, and if you don't have one, you should probably think about what it should be, but we generally can define based on the current state, whether the current state matches the desired State and if it does not Delta of those two, that's what you want to alert on. It's actually pretty simple.

A

So what does this mean practically and practically this means and everything doesn't alert, and we consider loads, usually things that page me at night right or during the day and everything that's paging. Should be original actionable right, it should be indirect problem for our users if the user is not impacted and there's not necessarily a problem and the user can be anything in right if you are writing a micro service and we just sort of serving data for other micro services.

A

These are your users, if you're placing customers directly, these are users, etc, etc, and you only really care about the boundary between your system and your user and they'll say exactly where I said is defined and they are for gold signals and which the sort of meant to capture things to a little and one of them is latency, and if a user makes a request to whatever service you provide, they expect a timely response might be a few milliseconds a few seconds, maybe even hours, but they certainly some sort of timeframe in which they expect the response.

A

So that's a good thing to learn on obviously Ennis traffic. If your vacation should receive approximately 1000 Rica's per second and is receiving zero and some things probably up, it's might be on the way between user and chat system, but it might also be the consistently subs just sort of not even accepting requests to the point where it can measure them. So it's also something and to what offer and it closely related.

A

We have errors if users make requests and gets not served the response I expected because of errors on the server side and you're violating it at a more certainly and should definitely load on that, and the fourth golden signal actually is not symptom based we're just kind of breaking our paradigm here and instead it's cause based and there's a reason for that, and but first of all what what what course based alerting and causes everything that potentially lead to problems in lethal symptoms of problems.

A

So if I can't connect to a database which is sort of my downstream dependency- and this might cause me not being able to serve a Rica to my user but does not necessarily imply it right, if I'm caching so long enough to for my dependency to recover- and it is just a potential cause, but not an imminent problem and in general, these things should not be something tool to notify someone about in a disruptive manner.

A

These are usually meant it about helpful context if I get paged for and high error rate of my service and I can now look at what currently cost based warnings- and we see- and the only exception is sort of our fourth column signal, which is saturation alleged capacity and that can be stuff like prices and running full memory running for basic set of stuff running for and and the reason why we make this sort of more of a cost.

A

Based thing is that if we aren't run out of faces in space out of memory, etc, etc. The effectors that are usually quite critical and don't resolve themselves quite quite easily and if I didn't run in full, might take hours of migration work, I might even corrupted data. So these are things that you really want to catch early and because, if they happen, they are not maybe cause of something they are most certainly and causing severe symptoms in the end.

A

So so far for the theory- and this applies to everything that is not bound to permeated in any way, and but we can look at how Prometheus sort of implement this idea of alerting- and we have our small DSL, where we can define an alert name and we can associate it with the condition and the convenient thing is that these conditions are expressed as Prometheus query language strings, and these are the same queries that you use for dashboarding and investigation during an outage.

A

So everything you can graph, you can alert on and that's extremely good, because, first of all, we hopefully know the Creveling is already and also if you get an alert, we can directly click on the condition that triggered it and we'll see a graph that relates to exactly this um and then at some other seven they're, like the four duration, which is sort of a great window before we actually send out an alert which is supposed to prevent flapping alerts right if a problem resolves itself within a reasonable time, there's not really reason to sort of propagate this alert to any human and their lives their patients, and we will see later how these are used and- and if you have seen from clear before it's um or in the example before we don't get just like a single result right, we get back sort of a vector of label, Perce associated with the value, and this exactly what we get in our condition here and every element in our result is one sitting in a dress.

A

So at an example, we have beer monitoring, STD and STD expose the material itself, which is called SIDS leader and it's zero.

A

If X D does not, it is notice, not see a leader in the sed quorum and one if it does and obviously not seeing a leader is very bad and, and one is okay and here and B obviously are in trouble and in other words, detecting exactly this condition would be as simple as saying para: SE, d, novena and then the condition being Exedy has that equals zero, which filters every time series that does have the value zero I put it out every time she was having the value unequal, zero and which means that our result basically is exactly two instances of seen before, which don't see a reader at this point in time, and each of these now form one in dirt and what we do before sending out derivatives.

A

We attach a label of alert name and we attach the Civic's label, which is part of our Energon definition, and this label will use later on to make routing decision on how and who should be notified. Who should be notified and how? So? These are just arbitrary labels. We can attach to any alerts being sent out and to make it a bit more complex and see how we can actually design them bit more complex and real world kind of alert.

A

And we have some time series here and or we have two metrics one counting requests and one counting the subsets of errors of these requests, and these are spread out by certain labels like the instance, so every sort of processable microservice and by the recurse pairs being used and by the whicker's method, and we want to alert on error rates being the errors being too high, and we can start as simple as saying alert higher rate. And then, let's take just all these sensors.

A

We have for the error rate for the errors we counted and take the per second value, because that's what we're interested in. So we take the per second value of a 5-minute window and what this will still give us for every single instance. Pass and method combination put one potential alert and that's of course, too noisy right.

A

We don't wanna get at 20,000 words if sort of the whole service is buggy and to mitigate that we use an aggregation, so we apply a sum which connected this whole thing nicely in a single value, and we can see here that, apparently our service is experiencing 534 errors per second yeah. Okay, that's bad, and and because we don't want this to happen, we set a threshold and make this arrow trigger if greater than 500, and so the result we get is sort of expected.

A

We alert us if the condition hits and the problem here is this F. So it settles because it's quite fuzzy. What does it actually mean like? This is a five note service, a five thousand note service.

A

How many because is getting and right and whenever it's totally depend on the total traffic you are receiving so over the day we might have a sort of curve which changes as if users, in busy reasons rubber, has a system or something, and we might just get more traffic over months or weeks of time or we were to send a traffic spike because you were mentioned in news. So there are many many different ways that can sort of change.

A

The number of requests we are getting and the error rate we are seeing depends directly on these requests and the semantic meaning of the load changes basically based on based on values that aren't represented in the condition, and so what we ideally want is we want this sort of um threshold to be adaptive to the total traffic, and we can do that easily enough and by just dividing the error rate, which we calculated by the total number of requests. Right and now we have a fraction which is certainly way closer to what we want.

A

And so, if more than one percent of our requests are errors, then we want to take an alert. And now the picture looks like this and it's certainly way closer to what you want.

A

And it's not right anyway, um because we are losing out of detail right and we have this label that curry language of all these time series having a lot of care and IOT and what we are we're, condensing them into a single value, and- and this unfortunately, can cause signal cancellation.

A

So let's say you have one weakest pass, we'll just hit very, very rarely like in a contact form right, and it's maybe hit like once an hour compared to your index pages that we hit like two one times a second, and even if you, our contact, form, I wrote every single time you use I want to see it. It will not show up in the result, because totally overshadowed by the high traffic your index page receives and the special is somewhere down there.

A

So we are certain anything properly. It's not detected at all.

A

Luckily, we can also solve this and, as you've seen before, we can preserve certain dimensions in notifications right and so for this time series- and we say yes aggregate but keep certain things around so I can have more detail and it want to preserve the internet, the path label. We achieve exactly that. So what we get back here. It's a nice per instance pass combination of the total error rate, and in this case the previous example will be reliably sort of detected and it's still wrong.

A

I'm intentionally, of course, to just show you what can be wrong because we preserve the instance label and in the case of a microservice, for example, that actually did I mention, along which we tolerate fader right, because we are building within a system that can handle certain errors and in the horizontal scale of the system, the single instance. Misbehaving is certainly one of them, so um we'll see load as it is. If one are of one South instances misbehaves, we will get paged and we have nothing we can do.

A

We can probably shutdown the instance and bring up a new one, but it's not necessary because there was no problem for users to begin with and great sort of going to burn out, because we get all these nonsensical errors a lot.

A

So the reason. The answer, of course, is just pick different dimensions and previous comes to the video nice query. Language feature which is sort of inversion of some by which is some result. So we specify the dimensions we want to get rid of, which is way safer right because we know are not necessarily aware of all the dimensions that our metric has and but we're usually aware of the dimensions.

A

We are fault tolerant along and this case here at the instance- and we say: ok, I, agree at everything away, but the instance and what we get Vegas itself gets the most insights we can, because we preserve the past as well as a method which was not considered before and sitting sort of alert. We would like to end up with and it's it is realign alert which will cover one ten, five thousand ten thousands, microsoft instances.

A

It doesn't matter because acting based on a pool of time, series data and you don't have to sort of tweak anything if your infrastructure roughly stays the same or the applies things infrastructure is the same. It will just scale automatically sort of with that's actually going to install any new checks for every new thing.

A

You deploy it's sort of adaptive yeah and that's actually typical alerts for symptom based pages and once the latency look pretty similar, and it will also serve that if you're sort of having it typically Microsoft architecture that almost all alerts will look like this and that's actually a good thing and because it makes things simple right, there's no black magic involved.

A

Here you have a few distinct patterns to stick to and to get reliable, alerting covering you end to end and don't have to consider and any of these potential tens of thousands of causes leading up to the program and out to an interesting part and the capacity situation. Adults. There are kind of a different class, but also following a very similar pattern. Every time, and so a typical thing to detect is my digit Oracle and what we typically do nowadays is we probe every sort of note or file system and check?

A

Ok, how cool is it and if it's a 70 percent everything's good and if the eighty percent everything is good and if it's at any one percent, then we will page someone, because obviously everything changed between 1881 and- and this has several problems.

A

First of all, you are not using 20% of your disk just surface like 20% buffer to react, and if your disk, that's one full and but it also triggered immediately, if you just add a 2 kilobyte file, putting it over the specials, even though afterwards never nothing changes ever and and also doesn't detect if your disk is almost free, but then sort of extremely quickly filled up by some sort of slightly software, and it will only let you if it's almost too late.

A

So the solution here is for freedom Prometheus and we have history. We have time series we not just don't just know what's happening now and we don't just know that occurrence for is over disk. Is 80%? We also know what it was 10 minutes, 30 minutes and 1 hour ago, and based on that, you can actually look at the development of the data over the last hour in this case here and and interpret the metric telling us how many free bytes there on any particular file system and based on this one hour window.

A

From now to the past. We can do a linear regression and just predict into the future.

A

We are think to end up if it continues as it does, and we do so for 4 hours now, okay, so yeah, and if this prediction has that setting four hours will have zero or less free bytes on a disk, then the current development on this disk is indicating that we have to do something- and this holds true, no matter whether we add 80 percent, 90 percent or 10 percent of this usage right and if you as 95 percent and you add one megabyte file, but don't any nothing afterwards, this alert rocks.

A

We can break you up because there is nothing to be done and this sort of applies universally for all sorts of Federation capacity alerts right. If you want to load on Hydra script and running it, file, descriptors running out, inodes running out memory running out, and this sort of always applies and.

A

Now we're looking at annotations annotations are this other thing that we have seen before and they allow us to just check additional non-identifying information to alert. So if you want to sort of provides a more insightful human, readable message with all alert, we can just add some text in you and you can begin as property. We can use the labels of our results, so our results has labels for the device in question. The amount point and the note is running on.

A

We can all add this into a message, and this is also again their full alert which will cover every single file system on every single half-life or every single node of your entire infrastructure.

A

So and that sort of okay- that's how you write, alert, how we, how we define what is problematic and what should be done in the first place, and and now we come to a lot manager which is sort of just thing component in your premises, ecosystem and no matter what system you have. You should have some alert management layer and how it works. Is that committees evaluates the rules which is specified?

A

It's protected data started and immediately evaluate the data in alerts, which is really nice, because everything is set contained right and not crossing fair domains and and if it consider some other firing, it will push it out to a res manager, and there are managers, then responsible to do the sort of more fine-grained stuff like determining and which uncoil person should receive, which alert, which is has no real verb, is being directly in your monitoring system and it has some more important functions. Actually.

A

So, if you consider these rules, we have defined to just be roots somewhere. We don't really care about which parameters are where they're coming from and but they're sort of. They are these things that are just emitting streams of alert going your way- and you probably see below here- that's probably a pattern that you have encountered when being on call right. You get woken up at 4 a.m. and you get a page telling you.

A

The latency is high for service ex Enzo and us for this weakest person, this weakest method, and he said ok and ok, and then you get the next page telling you the same thing just like some slight modification, and this just continues for 10 minutes, you're getting pages and pages and pages and pages and they're all sort of tying down tying back to the one single incident.

A

So the question is: why are you getting 20 pages if there's one problem and- and we actually want it to be this way after all, because we want this inserting very levity right, because you don't want to overshadow problems by being too generic in our language as you've seen before and and how we can still solve. This is with an alert management layer and what your load manager does is.

A

It gets a stream of alerts and looks at it evaluates how it should be actually digested and brought to the user and by grouping it together along certain label dimensions and by major disables making routing decisions so which team should be paged about which service and and also deduplicating. In case, you have seven permeated servers that are sort of emitting the same alert in a deployment and what we then get is we get one single page telling you?

A

You have fifteen others for sex in zone a us, and these are the rough problems here: high latency higher rate and Casella being slow, and that should be almost everything you need to sort of know what is up and then getting through your laptop and actually start investigating and handing the incidence and all the individual URLs. You see here, it's actually not too important, and so there the big picture is what we are interested in and they also some other stuff. So we have, on the one hand, silencing right.

A

So if you take a service into maintenance, you can sign it orders that are having a certain label, for example, service X.

A

It's pretty straightforward and another more advanced feature is inhibition and actually pretty cool, even though rarely you ready to get because that and necessary that often and but it basically allows you to have certain innards mute, other alerts, and so, if you have sort of a higher priority alert like your data center burning down, that's probably going to lead to all sort of other things firing, but personally sort of encore for the servers running in some rec in the data center.

A

Probably won't really care or can't do anything about this incident, and so there's no reason to sort of wake. Someone up if there's nothing to be done, no matter how critical it is.

A

Coming back to anomaly detection, so of course everybody to want it and we have seen the Prometheus has apparently some statistical features like linear regression and everything. So maybe you can do something after all for fanciness, but just first of all, this is just all bonus. These are things that might be nice and informative, but never make these things pay to whatsoever and don't ever rely on these in critical conditions and that's it and, let's suppose we have a request rate for our service, which we sort of and move into.

A

This new metric here called job request, 285 minute and- and we want to sort of get a smoothing view of this right, because it's sort of spiky, but we want to get the overall trend over the course of the day. And here we have powered windows functions in parameters which allow you to set the parameters and what you get back essentially is a smooth red curve here, giving you a good representation of the overall behavior okay. So actually, if we have to smooth in view, we can actually build anomaly detection in a way.

A

So let's say we want to sort of detect if the recurse rate right now is above or below a 20% sort of window around this mutant view from one week ago and there's a collectable fee sort of reflecting this curve in here. So we take the current precursor eight and subtract.

A

The smoothing grade from one week ago same day same time and and check if it's out of the 20% window up or down, and then we can notify you in some hopefully non-invasive way about a traffic being honest nominal and that's good stuff to experiment with, and but it will trigger on a question to you, but virtually gone Superbowl all these times. Right, don't make these things page and another example, and that's prob'ly example why it's really really bad to do stuff after all, and so no need to understand this in detail.

A

What we're going to do here is we have the average latency over five minutes for a bunch of micro service instances and we want to detect if one instance is out of the two times standard deviation of the average of all these instances. So one instance is being particularly slow and and two standard deviations is a pretty good measure. I guess, and so that's exactly what for you're sort of representing here.

A

We take our friend latency and check for every instance and check whether it's greater than the average of all the agencies of all instances, plus they are two times standard deviation and that's working fantastic until you realized that if all a piece of the rough is the same, your cent deviation is going to be effectively zero and so now even the slightest sort of deviation will trigger the alert and pretty bad. So let's just change this condition with another condition and get the condition and sell it.

A

Okay do all of this before, but only if we are in total twenty percent above the overall average and good one sort of each case, elided great, and until you realize that, if you put up an instance, Excel has a code cache and this is being slowed overall and that does matter because it's evolved up here it and but you will still get paged. So we have to sort of not consider any instance that has not received sufficient traffic so far.

A

Sorry, another condition our fee above and the only fire of this thing and if this instance has received more than one week at the second- and here we have built a completely functioning educate, considering anomaly detection, and the point is that we had a very well scoped problem. We knew what we wanted to solve.

A

The data was very narrow down from millions of series and billions of samples to just a few, and even then, with all our domain owns we have and knowing the cases we want to catch, it is really hard, and this probably is still not sufficient right and so just rule of thumb. Don't do this, stick stick to symptom based pages, stick to detecting violations of ETA and- and you are generally on the safe side.

A

So some depth, symptom based pages is what you want. You can add, cost based warnings.

A

It's really helpful right if I see that my service is being slow and I can detected it, because the database cannot be reached properly- that's great, but only page on symptoms and design, your loads to be adaptive, to change, right and sort of be aware of of dependencies between certain metrics, like error rate and quickest rate, and and preserve as many dimensions that you can and explicitly remove dimensions that you are fault-tolerant along, not page you if there's no actual problem happening and for anything capacity planning related linear prediction is probably what you want to do yeah and you can do most of that.

A

Just stick with the basics and the rollers we get from this whole thing. They are not really meant for human consumption rights. You need is a load management layer which digests everything and gives you this nice precise overview of, what's happening and gives you one page for all incidents, and your incident is not measured by how many pages are triggered and I.

A

Think perfect, four minutes any questions.

A

As the question is, if we have no way to intelligently deduplicate alert, okay, and so the dumb way is same labels. If they exactly the same right, we can did you forget them, of course, if they're roughly the same, the point is: if EF is grouping, they would generally go into the same group. So in this case we don't care about the individual words anyway. They will just go into the same group and be one is diffic ane way, so just based on this grouping behavior, it would probably not cause you to get page twice.

A

Oh I have any visible effect from these duplications if they cannot be sort of deduplicated properly.

A

Actually, yeah and Exedy has, in the github repository for v2 and v3, set up alerts that are working.