Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Alertmanager on Its Way to High Availability [A] - Frederic Branczyk, CoreOS

Description

Alertmanager on Its Way to High Availability [A] - Frederic Branczyk, CoreOS

The latest releases of the Alertmanager introduced a high availability mode. Frederic will guide you through the contract of Prometheus and Alertmanager and why developing the high availability mode is unique to the problem the Alertmanager solves by giving an introduction into the principles used to implement it.

About Frederic Branczyk
Frederic Branczyk is an engineer at CoreOS, where he contributes to Prometheus and Kubernetes to build state-of-the-art modern infrastructure and monitoring tools. Frederic discovered his interest in monitoring tools and distributed systems in his previous jobs, where he used machine learning to detect anomalies indicating intrusion attempts. He also worked on projects involving secrets management for distributed applications to build sane and stable infrastructure.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, hey my name is Fredrik I'm, a software engineer at kora and I'm here to talk about Newark Manager on its way to high availability, I work on upstream prometheus and alert manager and kubernetes, and pretty much everything connecting the two worlds and I'm branch on github, so a lot manager on its way to high availability.

A

But what does that actually mean what am I going to talk about so when they talked about and as from what happens from an alert firing from prometheus to an actual notification being sent out, then the actual high availability contract between Prometheus and alert manager, so how we actually achieve high availability between the two systems, the actual implementation? And what is that? What kind of implication does that have on an operator operating alert manager in a highly available manner? Just as a before? It's not talking about everything else.

A

How does cores fit in to working on these projects? Well, this all fit into our product tectonic, which is health driving kubernetes, and as part of that, we want to have self-driving monitoring systems. So that's exactly why we want to be in this community. Be active, drive innovation so just to give a general overview for those who may have not worked with an earth manager itself. So far, alert manager is the component that received alerts that are fired from a Prometheus instance, but technically it doesn't actually have to be Prometheus.

A

It just has to speak the same protocol and behave in the same way as Prometheus does Tom. What is then does, is it groups all of those alerts into one notification and it potentially deduplicate them so that when alerts are hiring again and again that we don't actually send out notifications more often than we want to, and then once it's done all of those things it actually sends them out with your favorite provider, so paid your duty, black female, whatever you want pretty much and there's also a web Punk integration.

A

If anything you want to use, is not fair default included, then you can build it yourself, and one of the features that are important about the talk today is financing. So that means, if you, for example, has a plan maintenance window where you know your database is going to be offline. For, let's say 8 hours you can already like you can put that into alert manager, and you won't be notified about that, because it's an alert or a notification that you're aware it's going to happen.

A

So this is the general architecture overview of Prometheus and alert manager, and specifically today we want to be talking about how alerts are generated from Prometheus, which is partly through prom, ql and alerting rule and those then potentially result in an alert actually being hired against alert manager, and we wanted to see what that contract between Prometheus and alert manager looks like so just as a as an example of how alert manager could work is prometheus, is constantly evaluating these alerting rules and sends out alerts for fires, alert against alert manager, every rule interval.

A

So that could be every 10 seconds. It actually evaluates these alerting rules and actually tires alert against alert manager every every 10 seconds. So that's, basically the contract that Prometheus and alert manager have along themselves and then, when alert manager receives all of these alerts. As we can see here, then it actually grouped takes all of these and we've seen before there were multiple alerts on highlighting 3.

A

It says: ok, there are 3 high latency alerts, there are 10 high error rate alerts and to cache server, slow alerts and, depending on your infrastructure, you can group them to whatever you're needed and potentially, depending on how you structure your notifications, you can actually include all the individual or it's in your notification so shortly or compact alert manager is the component that reliably sends you notifications and if we're talking about rely reliability, we want to talk about high availability and that's essentially, what we'll we're here to talk about.

A

So just as an example, I want to talk through why this is important to do so. You start a project and you write micro-service number one like service number, two Micra service number three and, as you get start getting users, they have some kind of expectation that your service is running. You have maybe even have an SLA once you have paying customers, so you want to start monitoring your application with Prometheus.

A

And at this point, you're only scraping the metrics from your clients or your service with Prometheus, and that allows you to write, alerting rules so essentially what you then do is you hook up Prometheus to alert manager and you actually get notified if anything goes wrong?

A

Well, but if your product is successful right, you start scaling out and you want to be highly available with your service because you have some kind of an SLA and then you start scaling up scaling up, but you realize you're scaling up all all of your users are facing services and have them highly available, but not your monitoring system.

A

So you add an additional Prometheus instance to monitor all of those targets again so you're highly available with Prometheus, and then you also send all of those alerts that you developed before with your initial Prometheus instance against alert manager. But here in this diagram we can already see again we have a single point of failure, which is alert manager in this case, so alert manager goes down. We will not be receiving any notifications and that's essentially our most crucial point at this point right. We have alerts that are firing.

A

We know that something is wrong, it's just not reaching up. So we add another alert manager instance, but because of the features that we that we saw earlier, we still have a problem at this point, because alert manager in itself is deduplicating these alerts. But now, since we have two instances, it actually sends out these alerts. The notifications twice, that's exactly what we don't want. So what we've come up with four alert manager is that we implemented a gossip protocol between other planet or instances.

A

And I want to talk through what just? How does gossip article wants to does work and what kind of implications that has for you running it, but first of all, I want to say: why did we actually be couple alert manager from prometheus? Well, it's exactly because of the complexity that we just saw that we can keep all of this state sharing gossiping integrations with notification providers that we cannot keep all of that separated in a separate component, and even that component could potentially be used by other monitoring systems.

A

They just need to speak the same protocols that Prometheus sends against alert manager and yeah. That keeps everything simple in Prometheus from alerting side. So we only evaluate these rules and everything else complex. The distributed system can be done in the alert nature.

A

So let's see what an example alerting will would look like in Prometheus, so here I'm taking one of the metrics that are actually exposed by a CD one of the chorus product or project and in sed a problem could arise if there is no leader in an ad cluster. So we say our alert name is no leader and this alert triggers. If some time series is does not have a leader at this moment and if that stays for 10 minutes, then we fire this alert and we can add some description. What does that?

A

What that actually means? We can template some things in there, but this is a very simple alerting rule how we would write it for prometheus and let's be clear that this is actually something that is evaluated in Prometheus, not an alert manager, but this does result in prometheus, constantly or in repeat and rule interrelation interval does actually send every 15 seconds or, however, you have configured Prometheus to evaluate the rules. It actually sends these alerts every time that triggers and that's very important for the high availability model between the two.

A

So because we constantly send these alerts, how we can tell that an alert has been resolved is that if the resolve timeout has been reached, for example, if after five minutes, our others are not firing anymore, we know that the problem has been fixed and then this is an example configuration for the alert manager which groups by the job so that we can logically group all the alerts for let's see our HCD cluster, and then we can say that we always want to collect for 10 seconds or the alerts that are coming in, and then we want to send them out in a batch, and then this example just because it's very simple I use the web hook, but it could be any other integration that we have so from a high level perspective.

A

This is what another it goes through once it's reached their newark manager. So the very first step is the financing. Where we decide do we actually want to continue going through this notification pipeline or have we silenced that alert? We don't want to even consider sending it out.

A

The next point is very crucial for the high availability mode and, if you're not running it as high availability, this is basically skipped, but, depending on your position on an alert manager instance position in the cluster, it waits for a certain period of time for the gossip data to come around from up from other instances and potentially deduplicate, because the very next step is to deduplicate whether the notification has already been sent out.

A

Then it actually sends out the notification and then the important part where actually sends the notification that a notification has been sent out to the other alert manager instances, and we will see why that is very important. I'll have a visual demo of that layer, so what data is actually actually gossiped? The point that we just looked at it's the send notification that data is profit, but also the silences. So whenever we create the silence, an alert manager, those are also communicated via the gossip protocol.

A

What is not gossip is the alerts that have been received by the firing system, because that is just that's just the sheer amount of that would be too large to be able to gossip in larger, set up where notifications are a tiny fraction of the received alerts, and how did we? How did we achieve this?

A

Well, we use CRD T's, so conflict-free replicated data type so that that means that we basically can merge all these states without conflict, and we always end up with the same data basically and that's actually wrong with not well-suited for CP systems but as well suited for AP systems in cap theorem, because this is exactly what we want to achieve with high availability.

A

We want to have an available and partition tolerant system and how, if we implementation wise do this, we use a library called mesh by week, work and because we're using CR DTS here or because mesh requires us to do to have a an eventual e consistent model. That's what we have. So we have a last writer winds model where time stamp based, basically, whatever the lightest time stamp is wins.

A

Arguably, we could replace that with vector clocks, for example, but so far we haven't had any problems, so we're sticking with that for now, but it could definitely be exchanged but yeah. Basically, we can just merge all these things based on a unique ID and it's quite important to know how these unique IDs are generated to be on to be able to understand the availability model, but just looking back at. Why are we actually implementing a complex distributed system ourself and not use something like sed?

A

Well, ICD is actually not the kind of model that we want to achieve.

A

Icd is CP, not ap, and the advantage would also be that the alert manager is actually single binary. So it's very easy to operate and it's just less moving parts, because if we had a distributed data store that would be backing for our system, then that would be a lot harder to operate or even troubleshoot if there are any albums with it. So, let's look at silences.

A

We have to alert manager instances and we want to perform a create action on those and want to see what that means in our gossip protocol. So the first thing that actually happens is some user actually creates or wants to create a silence through the HTTP API, for example, and what alert manager 0 because alert manager 0 received as a crest request, actually does.

A

Is it inserts it into its own in memory database and once it's done that it basically takes the Delta that was just inserted into its own database and gossips that to all other other manager, instances with the unique identifier which for silence is just some randomly created, unique, ID and then, since alert manager, 1 just gets that Delta. It merges it into it. Data set and eventually they all end up with the same data.

A

So how would that work with a with updating a silence, because, let's say someone got sick and you are moving the maintenance window to next week. So you want to change the start and end date or just the start date of X Island.

A

So you just need to know the unique ID which is in all of the databases of the road manager, instances and say what you want to modify and what then happens is that the alert manager 0 merges that into its own in memory database has the Delta that has actually changed profit that to the other alert manager instances and they merge that into their state, and we are actually consistent.

A

So for the finances that's an easier model than for notifications, because for silences we can always just create a new, unique ID, but for notifications we need to be a to recreate the ID in a consistent manner. So let's just see how we want the alert manager to behave in certain cases. So in the first days, I want to show a case where there's actually no failure, but so, like the happy case, with a non silent alert so that it actually gets sent out.

A

So Prometheus evaluates some alerting rule and decides that it's going to be firing, so it fires them again. Both alert manager, instances and because alert manager. Zero is position zero in the processor. It waits for zero seconds and alert manager, one for five seconds, and so, while alert manager, one waits for the five seconds. The manager zero actually continues through the notification pipeline so at first once to deduplicate the alert, but it sees that there's actually not been in the notification for this alert so far, so it actually sends them out with paid paid.

A

Your duty, for example, and then gossip the state that this notification has already been sent out and once alert manager one is has waited the five seconds. Then it will or wallet it's waited. It received the gossip data merged it into its state and therefore, once it goes on to deduplicate, it does not actually send this notification out so, but what does then happen when there's a network partition from one alert manager to the other so that it was not able to gossip the the notification that has been sent out?

A

Well because we're we were aiming to to build an available system. We always want to say we at least once send out our notifications. So the terrible thing would be for the alert manager that it doesn't send out a notification right. So in a failure case, we always make sure that it least one notification gets sent out, and so, while alert Major General was trying to gossip its data. It failed because of the network partition and then alert manager.

A

One was trying to deep duplicate, but actually send it, because it has not received gossip data, and so we have a notion of at least one sending a notification. So how does that work? Let's say that we're actually able to generate this unique identifier for each of the notifications ad to find where Prometheus or system that behaves like Prometheus fires, an alert against the alert manager zero in this case it wants it sent out at the notification to its database. So things like is it resolved?

A

When was the last notification sent out and it works very similar to the silences log? We're just take the Delta again and gossip that so. But how does that? How do we actually get the unique identifier then? So what we do is when we look at the actual route. A alert goes through in the alert manager. We always say things like group by group by job, for example, so at runtime we take the values of these labels and.

A

Take take all those from all the alerts that are going to be in one notification: we hash them and then XOR them with the entire route that it's gone through, because the route can be a tree. So we need to go the entire route up and then, at the end we actually concatenate that with the receiver, because a route can send it out to multiple receivers. So at that point we then know that we've send it out for that specific receiver for that specific set of alerts in a notification.

A

So now that I've shown how this is supposed to be working in theory, let's see that this actually works so I've prepared a little demo that large enough so I have a simple alert manager configuration here very similar to the one that I've shown in my presentation.

A

I picked up that I group by alert name here and whether that's a good idea or not that's another discussion, but just for the sake of this presentation, we have a group weight of 10 seconds, so we'll wait for 10 seconds to actually send out a notification, and it wasn't that time.

A

We group notification and we would send it out every one hour if these alerts keep firing and we send it out with a wet work and I've prepared a proc file here, how I'll run three alert manager instances and an example webhook, which will listen on the address that I had in my configuration and the web. Folk is really just a very simple.

A

Very simple: go application with just listens on HTTP on port 5001 and then just writes everything that it received to send it up.

A

So, let's better string and everything's running so let's go to the first alert manager, web UI, which would be 1993, and we see that there are no silences created yet so the first thing I want to show is that gossiping of silences actually works. So we go through all of these alert manager, instances and see.

A

No, there has indeed not been a silence created so far, so that's we can take a random alert manager instance and just create something so alert name equals high latency, for example, and I need to insert a an email, and the comment could be something like hi from pucon and we create that and we see down here that the alert has been the silence has been created, and now we can check at all of our other instances that this silence has indeed been gossiped to the other instances.

A

So now we've seen that finances creating work. So let's look at an example where there would be actually something firing against the alert nature. So we're simulating what prometheus or comity is like system would do in this case. So I've, a small script here called send alerts which just sends a couple of alerts to my learn manager. Instances and we'll have to be quick here, because within ten seconds we should now see that a alert manager actually sends it out and the webhook server printed it out to standard out.

A

And what we want to see is that, even though these alerts have been sent to all of the alert manager instances, only one of them has sent out the notification and since they're the web server prints out everything it receives on standard out.

A

We have only gotten one JSON blob, and that is exactly what we wanted to achieve, and that is basically at the end of my demo, and that makes the end of my presentation just at the very last night, we're also hiring in Berlin or all over the world and at all our foreign offices. And if you have any questions, I'm happy to answer them.

A

um Okay, so the question was: why didn't we choose something like in in a form of a leadership? So what we wanted to achieve is that we don't have any master. We want to have a map in this system because we want any any note to be able to go down at any point in time, and this is seem to be still available because typically systems that have a leader election are typically CP system, so particles or rock, for example, their CP systems right. But we wanted to have a an IP system in this case.

A

Okay, there are no other crushing.

A

So the it's an in-memory database, but it gets snapshot not shot at every snapshot. Interval that you want, but there is a there's a time time where, if the last snapshot was done and within that time and like t0 and t1 and then inter crashes, it doesn't actually process that to disk. So that can happen, but we are. We are on track with that. We do want to implement some kind of a right head lock or on something in that manner. To be able to do things like that. Yes,.

A

So 1,000 notifications is like there's it's essentially free, so we haven't actually extensively benchmarked storage, but like memory usage is very low because we don't actually persist anything of the alerts that are coming in and that's the high traffic right. We only gossip and store the things that notifications that we've sent out and we also garbage collect those after some retention time. So even if you do scale up to, let's say that you need 16 gigs of ram.

A

You can still tune the retention period to be so that you only use memory within that controllable range.

B

A

A

Okay, so the question was: if I now scale-up alert manager will it has? No data does send out all the alert. No actually, when an alert manager joins the network, it gets all the data from the other alert managers and it also periodically periodically share all of their state and merge it into each other. So that way, we make sure that it means periodically. We always have the same state.

C

A

We don't currently have a plan for anything like that, but you can using the web hook. You can basically implement whatever you want right. I mean you still need some kind of a notification mechanism right like if you want to like emails already implemented and other than that you'll probably have to write a web hook. If you have some proprietary system, let's say I.

B

Wanted to use a question so general literature. We don't want to duplicate each of like actual management of people, that on call schedules, expiration policy that we need more for systems like pages opportunity, just like kind of a separate 25:4. Let managers really just about the routine health alert notification.

A

Okay, I think that's it, though. Thank you.