Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Evolution of Metric Monitoring and Alerting: Upgrade Your Prometheus Today - Bartlomiej Płotka

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Evolution of Metric Monitoring and Alerting: Upgrade Your Prometheus Today - Bartlomiej Płotka, Red Hat, Björn Rabenstein & Richard Hartmann, Grafana Labs, & Julius Volz, PromLabs

Infrastructure metric monitoring is constantly evolving. Similarly, Prometheus, the most adopted observability tool in the CNCF ecosystem, is under active development and constantly growing. Every release brings something new or improves existing features to help increase the reliability and debuggability of modern clouds. But Prometheus is not about a single project anymore. There’s an ever-growing Prometheus Ecosystem, including Alertmanager, tons of exporters, standards, client libraries, and long term storage projects, enabling the best and truly cloud-native observability story for you. In the first part, Prometheus Maintainers will introduce you to the universe of reliable monitoring and alerting with metrics via Prometheus with specific and actionable examples. After that, we will make sure more experienced users can learn as well, by explaining the advanced usage patterns of the Prometheus and new, useful features available in the newest versions.

https://sched.co/ekHn

A

Hello, everyone welcome to kubecon and I hope, you're all having a great time, welcome to this roundtable discussion on the evolution of metric monitoring and editing via prometheus. My name is kunal and I have been contributing to open source ever since I was in my freshman year. Currently I'm a cncf intern working on thanos and I'm really excited for this particular talk. So, let's just start by introducing the speakers that we have for today's panel, why don't we start with julius.

B

Yeah hi there, I'm julius, I co-founded the prometheus monitoring system back in 2012 at soundcloud uh the last years I spent freelancing around it. Just helping companies build stuff on top of it use it get training cetera, and this year I created my own prometheus related company called prom labs, so basically continuing what I've been doing, but also building commercial software on top like prominence so yeah. If you need that or any help with prometheus, please reach out.

C

Yeah hello, my name is bjorn. I have been with prometheus almost from the beginning and that's a fun story, because I kind of watched julius and matt proud his co-founder, creating it. I thought it's like a very weird idea and nobody will ever use this, but they still lured me over. Like I was sitting, I had this privilege of sitting in berlin as one of the kind of only google engineers who could work from from berlin back then and and yeah julius was there at soundcloud.

C

They invited me for beers and told me the story, and at some point I don't know if it was just prometheus or just because they're nice people to work with. I came over and then yeah that was 2013 and that's long ago now and we were starting to work on this and I have never stopped working on prometheus ever since, and now I'm here.

D

Hi I'm richard I um joined prometheus in 2015. um was looking at or for a new monitoring solution for for a network for for an isp, but the data center also monitored. The data center with prometheus basically fell in love with with prometheus immediately because it's just more powerful and you can do actual math on your monitoring data and I've been around ever since.

E

Yeah, my name is bartek and I'm a software engineer at red hat. Currently we are doing you know everything observability uh mainly on openshift. um However, my journey with prometheus started four years ago when I was in my previous job as sre, and we were doing at the very beginning, rotations between teams and I was supposed to do game engine and I end up actually exposing metrics and doing observability of promotions, and I fell in love with that.

E

So this is how my journey started and then I made I become from juice maintainer and we started thanos as well on the way.

A

Thanks a lot for the introduction everyone and uh before we move forward with like uh some more like, we will be talking about the what prometheus is and like how it started. So it's uh even if you're a beginner in prometheus or someone who is much more experienced, there's the best of both worlds that you can get out of it. But before moving on to all of that, I have just the first question that I have is: what is actually what actually is prometheus and how did it start.

B

So uh prometheus is a metrics based monitoring system that includes a time series database that allows you to monitor software devices, anything that you can really get numeric metrics out of and then either make nice dashboards based on the collected data, or also integrate that data into your alerts, to wake yourself up at night, hopefully not at night to fix a problem and detect these problems and yeah how it started.

B

So I found myself in my dream: job at google in 2012 being a site, reliability, engineer in zurich, working to keep one of google services online and working and there at google. We had a great monitoring tool called pokemon. Everyone hated it, but everyone also said it's the worst tool, except for everything else, and when I then left google to go to soundcloud to berlin.

B

Ironically, actually, I really liked the job at google, but I really wanted to just go back to my hometown of berlin. After so many years, I was at soundcloud and another ex-googler also joined madproud proud, and we were both really missing the kind of monitoring tool pokemon that google had and we looked at everything in the open source world back then, and we're really not happy with the data model. The ui, the query, languages or complete lack thereof.

B

The efficiency in storage um being able to deal with dynamic environments, so soundcloud back then already had built a complete own cluster scheduler before docker existed before kubernetes existed many years before that, and there were hundreds of microservices running on that thing with thousands of instances, and it was basically impossible to find out when there was a problem like a latency spike. Was it all of the instances or just one like where what's really happening in detail on that cluster?

B

So we were really convinced that we should at least try to build something that is inspired by borgmon to help like both our own job, but also create something for the open source world that we could use in our next jobs and that others could use and um yeah. We basically started in our free time and eventually after it became useful enough, we started introducing it at soundcloud and it was still like a long path uh to actually make it work and make like convince people.

B

I remember talking to bjorn initially I think like in an elevator, and he was like what you're recreating pokemon. This is totally crazy um like.

B

Why would you do that, and I think yeah now now he's happy um and yeah, but that really helped in the end, get a totally different level of insight into what was happening with the microservices in that cluster, um get way more precise, alerting um and helped teams work in a totally different way than before, like even testing their own software, when it got released and so on um and yeah, then we really finally published it properly with a blog post and everything from soundcloud and others uh in 2015, and we didn't really expect it to you know we didn't really expect anyone to really get what we're doing, because prometheus does a lot of things differently than other monitoring systems before it.

B

With this query, language pull model instead of push for the actual data collection and, like many little other things, but I think it really hit the nerve at that time with people starting to adopt kubernetes needing something to monitor stuff running on top of it and prometheus, really integrating well with dynamic environments by service discovery which we might get into later.

B

The rest of the story. Basically, you know like it got really large adoption from there. uh We were really happy and a year later, roughly uh we we joined the cncf and yeah. By now we're we are happy with cncf project, and it's for some reason became the basically de facto standard in open source monitoring, which is amazing, but was totally not what we set out to do initially.

A

That is, that is, in fact, really amazing and, like julius mentioned, something about the time series so like uh can someone uh give a bit more insights about the time series for our viewers and like how is it? How does it sort of compare from something traditional like nagios,.

C

Yeah I mean I guess I can take that um so nagios was kind of the gold standard in in whatever 2012 2013. When when prometheus started- and I mean it's- many people are still probably using it and it's doing a good job in certain traditional scenarios.

C

But um there is this one thing that niagara's essentially is your run checks um and if they fail, you get an alert. It's kind of this binary thing uh it it fails or not. The only kind of thing you can do is that you count checks and, if, like three out of five fail, then you alert or something like this there's a bit of a time dimension.

C

But there is no real um time series in there right- and um this was I mean that existed a bit with like statsd and graphite, which was fairly new at this time as well. I mean it paved the road already uh and kudos for that. Definitely, uh but then people always thought about dashboards right. You you write lines on onto a dashboard, uh but uh like yeah, this this vision of doing this all in one and have alerting com combined with that that that was a really really important paradigm shift.

C

uh We usually tell people that the cool thing is that you can do like it's called trending. I think, or you can say, okay, I don't have to alert if, if I cross a threshold, if the disk is full, I can actually alert if it's if it's um getting full right. So this will allow me to alert way earlier. If there's a steep increase in this usage- and I can totally uh write through like 90 disk usage, if this is just constant, it might be fine right.

C

um So this is the one thing there's also more to that to make alerts more meaningful like if you alert on on, like a a curing system being behind, you can actually stop alerting if the cure is getting shorter, because it's still fairly long and that's bad. But you know the system is already recovering, so you don't don't need to alert.

C

So that was the one big thing you could also um by time. Series, you could uh have this whole notion of a counter where, like um I mean, let's say: apache like ancient apache, already had like this kind of matrix module, where it would tell you how many requests it was serving per second and how many it has served all together, and this is essentially redundant information.

C

If you, if you just record in a time your database, how this counter evolves, then you can take a rate like differentiate the the counter and you you get the rate and you can actually define if you want the rate over the last minute or last 10 minutes, it's so much more powerful. But it requires you to record this as a time series and that's what prometheus did and it unlocked an enormous amount of things you could do, and it was not just for drawing lines in the dashboard.

C

It was also for alerting, and the other thing is that we labeled those time series, and that was uh where we went from like those dotted hierarchical strings from graph rights. That's the to a label model which is like non-hierarchical, and you can slice and dice along all these dimensions. That's really important what julia said to really go down to to certain failure, cases or root causes- and this was such a nice coincidence that kubernetes also labels everything right.

C

So kubernetes came out when we were already done with the prometheus prototype essentially and the kubernetes people. They could have noticed what we're doing with prometheus, but they they didn't, know right. It was an open repository, but nobody knew it and then it came out it both was like greek words, kubernetes and prometheus. They are both 10 letter words, so you could say k8s and b8s.

C

The logos are like exactly complementary, colors, orange and blue, so for anybody outside it looked like this was designed on purpose to work together and and to fit together. But it's not true. It's like really converging evolution, essentially and also like you can do a lot of prometheus monitoring for non-kubernetes systems like at soundcloud. We did it for our own homegrown orchestration platform, but you can do it, use it for everything. It's not.

C

It's really separate developments, but it played perfectly together and that's also, I guess the reason why the first two projects in this cnc are kubernetes and prometheus.

A

Absolutely and yeah: that's really yeah go ahead.

D

So I I just want to to make one shout out to savix, because even back, then it was doing time series it was pull based at least mainly. You could also do push it allowed you to do basic math um still prometheus blew it out of the water, but already back. Then you had something which was. It has all the main things, but not as as nicely integrated as prometheus did.

A

Yeah, absolutely that was a really good explanation for like why promises started what it is, but I'm currently like. uh Let's say I want to use my let's say I have a java application. So if I want to use prometheus, then how do I actually like send a metrics to prometheus? So how does that usually work.

E

And that's why you know prompt users is very unique in this field, because you don't actually send metrics to promoters from qs, collects that metric value from your application- and this is the you know- sometimes controversial discussion between pull and push model, so prompt use is primarily a pull model here. um It allows you to configure certain scrape targets and- and certain interval with uh you know how often you collect certain metrics and with a simple http endpoint. You know page that you expose in each of your applications.

E

um You just point permit used to those endpoints and computers will collect this. The data from those endpoints um periodically, and um this is essentially how produce collects in regular intervals um the values of your metrics that are exposed within your application, and that was pretty novel. You know back then, when prometheus started, because everyone used to uh something called like black box or close box monitoring where, for example, in nagios, you are creating scripts around um around your system that checks from outside what is happening within your application right now.

E

It's um there is this direction of um exposing more observability kind of signals within your application. So maybe you can count, you know your q sizes um http.

E

You know uh request and server latency and stuff like that, which is extremely important, and you do that by not pushing this data to um to some monitoring system, but actually just um allowing some other system to collect it and primitives leverage that um this um well. This might mean that you know the code um has to be instrumented like you need to use some client library in, and there are plenty of um libraries um for for the major you know languages available um and and supported by either community or promote use maintainers.

E

But at the end, with this, it's much much easier and more efficient than pushing metrics, because you don't need to worry from the client perspective on the application into. You know how to handle failover scenarios um rate limits retries. How do you buffer this data? Essentially, you can perform pretty stateless applications and have reliable monitoring. On top of that, this also allows you to simplify discovery and configuration so make it kind of top-down discovery, um and I think julius. You can tell more about that.

B

Yeah so once you have those applications that have instrumentation like an http endpoint, where prometheus can pull metrics from the next question is how does prometheus know where it should pull from? um Obviously, the simplest way would be to just statically configure some endpoints in your prometeus configuration, but that worked maybe 20 years ago, when you have these static database servers and web server and they never change.

B

Nowadays, you have cloud instances popping up going down. You have kubernetes on top, you have on top of that kubernetes like many changing micro service instances, and the key here is service discovery, so prometheus can integrate with different types of service discovery in your infrastructure. The most prominent one would be the kubernetes service discovery where it continuously talks to the kubernetes api server to get a constantly updated view of what should exist in the world right.

B

So for a monitoring system, it's really important to know what should be there and what is it and so prometheus um no uses service discovery, information, let's go with the kubernetes example, but there's others to know what should be there. How do I pull from it and then also enrich the pulled time series data with information it got from service discovery, so it will know what kind of part it's pulling from in kubernetes, for example, um which kind of environment is in?

B

What's the part, name, etc, etc and um yeah that just really helps prometheus deal with these dynamic environments and allows you to define alerts when prometheus is trying to pull from something that currently should be there, but isn't because prometheus will also not notice that automatically and yeah. So there are service discoveries integrated in prometheus for uh instances of the different cloud providers for kubernetes, mesos, dns, zookeeper and some other ones, and there's also a an interface to plug in your custom one and um potentially, if it's.

B

uh If, if you are missing a built-in service discovery in prometheus, that is for something really popular. We might be able to include it in prometheus itself.

D

And this also has a political angle, of course, while we have that system which which allows you to nicely transport your metric data, this is inherently tied to the name of prometheus and we got hundreds back then thousands these days of integrations and exporters from people running their own stuff or instrumenting their own stuff writing their own exporters. Of course they needed it for their own things.

D

Yet there was this political angle of competing projects or competing vendors, not wanting to report, something which, which carried the name out committees, which is why openmetrics was started, which is basically an effort to to standardize the permissive exposition format with some slight changes, um and it's been that concept has been in use for like five years now, we're actually and I'm really meaning this. We are close to uh to publishing the actual draft within the itf. Just yesterday night we finished the to-do list, so we have to find punishing.

D

um So what is this? It's basically just taking permitted exposition from it, taking all that goodness putting it into its own thing. So you don't have that political angle of supporting something from a competing product, while also allowing you to to have an official standard within iitf, where you can just say: okay, rfc one. Two three please support us, which is especially important when you come to networking hardware or more traditional vendors. Of course, they usually work on its standards, at least in the networking space.

D

So this is why why why I started this because it's just important to to not only have prometheus, as this super nice efficient, powerful data engine and framework for doing your observability, but also permeating this concept of labor-based metrics throughout the whole ecosystem throughout this industry and other industries, and that's basically the intention behind this, and I think it's kind of working which is super nice and again within weeks you actually publish the standard.

D

Not really yay.

A

Yeah, that was uh that was that's really helpful, but uh the like bartok mentioned about you know the pull versus push mechanism. So what if, like the question that I have even from like the viewer's point of view, so how do I actually like fetch the data and actually like visualize it to visualize the data that I have collected? So how does that usually go.

B

Yeah, so there is a query: language called prom ql in prometheus, which really forms the heart of building alerts, building dashboards, but also doing ad hoc debugging digging around in your data, and this query, language was initially inspired by what we were used to in pokemon at google and now, of course, it's of course not exactly the same, but it uses similar principles and it is not a sequel like language like you would find in some other time, series databases, but a more functional language, which just allows you to select data and then wrap more and more and more transformations around the selected data, and I think some of the core features that really make it powerful is that you can do math between whole sets of time.

B

Series, for example, dividing a whole set of error rates by a whole set of total rates, for example, automatically joining them on either completely identical label sets or related label sets. So there's modifiers you can use. So it allows a level of insight and math between time series that wasn't really seen before that.

B

It is, admittedly, it has some sharp edges um and is different from what people are used to in query languages, but I think it really pays off and it allows you to do. You know very precise alerts as well.

A

Yeah thanks julius, and what about visualizing? The data that I have collected.

D

So there's there are several approaches to this. Obviously you have the ui built into permitted itself, which is super nice for exploring your data and which I still use quite often because it's just very snappy and nice, but there's also something which is called prom dash, which was the dashboarding tool for prometheus, and it is an absolute pain to use and to create dashboards with promdash.

D

I never liked it and just by happenstance, also around the same time 2014. I think it was um torkle founded uh or started the graffana project initially to visualize uh graphite data, but there was a plug-in for for prometheus as well within within a few months.

D

I think, and basically, as I was mentioning, I was trying to find new monitoring for for my for my internet or for my backbone um and was looking at both grafana and prometheus at the same time and just kept using the two with each other, while mainly ignoring promnesh, because it was just so painful and at some point talking with carl. I just realized hey. Why don't we try and make grafana the actual default recommendation for prometheus, which we just passed within?

D

I don't think prometheus team existed back then, but like within within the group which was around prometheus. We discussed this and and basically decided that. Yes, this is not the focus of prometheus anyway and having something which is taking care of. Actually, the visualization part for for fixed dashboards is absolutely nice and something which helps the project in as much as we don't have to focus on this.

D

Of course, these days uh grafana has the explorer ui, which basically mirrors and even enhances what you have within prometheus proper, which doesn't mean that that doing it within permeated is still not also um fully fully okay and and sometimes the quickest way to get something done, but for dashboards.

D

Basically, we switch the official recommendation towards grafana in either 2015 or 16. I think I think 15.

A

Yeah thanks a lot richie. I do have a follow-up question on that um like uh when we're using like monitoring and alerting tools. So one of the one of the reasons and one of the most like advantages that we can take off is uh figuring out and detecting the incidents that might cause. You know some some damages. So do I need to glare at these dashboards to detect those incidents, or is there like another like a better way to do that.

E

No, that's exactly what what was the kind of one of the primary goals of promoters um from use was not only about dashboarding graphing. Those data is it's really about alerting and kind of reactive work and and focusing on incidents only when uh and on focusing on running workloads only when when there is some something wrong going on, so um this is kind of natural evolution. When someone introduced a monitoring uh first, there is no monitoring, then, for example, promote use is introduced and they try to.

E

um You know visualizate some data, some some health of the system, and then you set up alerts, so you can actually stop. You know looking on those health systems to monitor yourself. You can actually let promote use to monitor for you, and um this is really really kind of novel and amazing and kind of the way going forward, because you can, as an you know, operator, devops sre, you can focus on future work.

E

um You know expanding your business um instead of operating and manually watching you know health of your system, um and you can, you know, configure let's say, rules that will trigger an alert that will be sent to alert manager, which is part of the promoter's ecosystem and alert manager, allows you to route to the um notification system um of your choice.

E

It can root the pages due to your other kind of systems that allows you to send message to your phone slack, whatever email, jira and what is also revolutionary here is that you can actually um trigger an alert on the symptom of potential seedlings incident. That will happen soon. Potentially, for example, you know disk space is running low or you predict that the cpu saturation will happen soon, because there is like growing increase and nothing is is going down uh or maybe memory utilization is is constantly growing. So you predict that.

E

Will you run out of the memory soon, so you can actually predict that ahead of time um and and essentially react faster, but the true kind of value in in true truly automated kind of system. Infrastructure monitoring is um kind of slo based alerts, and this is kind of there's huge ecosystem and lots of talks around and tools around around that. I really recommend um the tool that wrote matthias my coworker.

E

It's called prom tools and it lets you generate. You know error budget based slos, which um you define. You know the service level objective that you care for, because you don't care if one user maybe request, fails you care. If you know a significant amount of users um cannot reach your service, for example, for some amount of time. That is, maybe um you know described in the contract, so you can alert you can. You know, adjust the alerting to match those um contract behavior and also only be notified.

E

For you know human response um only very very needed um to avoid noise.

A

Yeah uh that answers my question thanks a lot for sharing, and I just have one more question for julius, like what was the was the naming uh inspiration behind prometheus like we know for thanos, it's uh marvel avengers and stuff like that, then there's loki and uh what was the inspiration behind like yeah.

B

Well, I mean to be honest: we initially we just needed a name, so we did what everyone does. We went to wikipedia lists of greek gods and goddesses and all titans, and all that, and eventually we did stumble over prometheus and a it wasn't taken.

B

Yet in that relevant space we could even get the github org, of course, that existed already at the time, but it was not used so github actually gave it to us, but also we just noticed, like the symbolism fit nicely um a we kind of you know prometheus stole fire from the gods and brought it to the humans the outside world.

B

In a way I wouldn't say, okay, I would say we we were inspired by google's pokemon and built something like that for the open world um and the second meaning is the fire right, like a monitoring system, gives you insight, as does fire. So it's illuminating things and the torch is, is a nice symbol for that so yeah. It worked out really well and the fact that then, all these other cloud native tools also ended up with greek names like kubernetes and and istio, and so on. uh That was a complete coincidence.

A

All righty yeah thanks a lot everyone for joining and everyone who is watching this and uh will be also available to answer any questions live so to make sure that if you have any questions, any follow-up questions you can ask those and uh have a great kubecon.

B

D