Cloud Native Computing Foundation PrometheusDay NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: How Not to Use Prometheus? - Shivangi Motwani, Zeta Suite

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: How Not to Use Prometheus? - Shivangi Motwani, Zeta Suite

Today Prometheus is one of the standard tools used for monitoring and alerting. Like any other tool/library we use, there tend to be common mistakes we make as beginners. This presentation aims to highlight the flaws made and overlooked while using Prometheus. Few of them being, Are all your metrics coming up in Prometheus on time? Are you using the rate/increase function correctly? How do you get metric value as instant/time series when using PromQL? Why alerts are not getting triggered? Why is the latency of your query high? Is the choice of your labels correct? Let's make your journey of using Prometheus as your Monitoring buddy easier. Along with that presentation seeks to provide a crisp and handy checklist you can use while making your dashboards and creating alerts. This checklist would also help review the Prometheus queries for your colleagues, making the review process faster and your dashboards and alerts bug-free, functionally correct, and uniformly interpreted (at least from the Prometheus point of view).

A

Hello, everyone so yeah uh how not to use Prometheus uh I'm, shivangi and I'm. A software developer at Gita suits uh I have been using Prometheus for around a year now and in this talk, I'm just going to share my experience. So it's more of like how we should avoid beginner mistakes and save us a save us some time in the long run. So yeah uh in this talk we'll see like at each stage of using Prometheus.

A

What are the kind of mistakes we can make and how we should be avoiding it to save our energy, so maybe like setting our expectations right or while we are instrumenting our code to querying or setting up the alerts so setting our expectations right. What I have seen in my experiences that one of the biggest mistake that we can make while using Prometheus, is to expect it to work as a logging or tracing solution or at times expecting it to give us 100 percent accurate results.

A

Prometheus is good enough for us to make operational decisions, but expecting it to uh give us like expecting it uh to be used for the use cases where we care about each and every increment would be a bit more. So we should try avoid that, and when we have these expectations set right, we can avoid like lot of mistakes in long run. Along with that, we should uh be careful about thinking. uh Prometheus are Standalone solution for a long-term retention.

A

Maybe we can look at uh look to integrate Prometheus with Solutions like Thanos or cortex or M3 right so yeah like once. We have our expectations right most likely. Our next step would be to integrate Prometheus uh entering Prometheus along with our code right. So uh one of the mistakes that we made uh while starting off was not using our metric names right. It's it's very simple mistake that we can think of, but it can cost a lot.

A

We are registering our Matrix once, but we will be using it way more often, and it's not just you who would be using it. The entire team would be using it so having a non-clear matrix name can create a lot of issues there. Apart from that, most of us face the cardinality issues lot of the time. I think the best way to deal with it initially is to have a rough idea of what cardinality would be introducing with your Matrix and just have it like.

A

Maybe if you are creating a PR or in your comments of just mention that this is what card terminality, you are expecting. That saves a lot of time by this time, like most likely, you would be having a Prometheus setup, uh and you can see your Matrix in in some end point, uh but you have to configure the target later on, so that uh Prometheus is able to scrape your Matrix.

A

You can play around with the scrape interval and retention time as you like, but uh just keep uh just try to uh keep the scrape interval and retention time within the limits. Otherwise, you can see like broken graphs, which would eventually not be very useful, and you have to go back refactor and, like just spend a lot of unnecessary time over there. Along with that, uh relabeling has been something which comes in very handy. uh You don't need to deploy your code again and again.

A

If it takes time, you just can use the relabeling in your configurations, and you are good to go for some time till your deploy your code changes.

A

So uh this is uh like once you configure it it's the best part, because now you are ready to see like uh what your application is doing.

A

You can query your data right, but when we query uh Prometheus uh there there are multiple chances that we can overload Prometheus by the way we are querying um our Matrix, so just try to like uh have uh just try to like avoid querying uh High cardinality Matrix, with lot of variables that you could put lot of load on your Prometheus and your queries can time out making your entire monitoring thing and effort go useless.

A

So just just be careful about it, along with that, try to split your dashboards as much as possible, based on your use cases, instead of having everything in a single dashboard. It's it's better for both the ways like when you try to visualize your dashboards. It would be easier to understand what's going on there along with that, uh it will be like way it will. It would load up pretty quickly, so yeah try to do that.

A

uh So, while we were setting up the alerts we we had way too many noisy alerts, and one thing that we learned was every now and then we would figure out. Okay, the timing that we set between two of the alerts was way too less or either we were not setting a proper dependencies. So, for example, if I'm getting uh alert uh on that, my success rate of the application is pretty low, but I'm also getting the alert that my application is not working so like having the dependency between two can help.

A

You not get overwhelmed with the alerts um and as important as it is to get uh less noise in alerts. It's equally important to uh not miss out the alerts. So um if you know like how you can avoid uh having the missing Matrix uh try to do that, but at least have some idea beforehand that you have missing Matrix for uh whatever use case and try not to have alerts on top of it or expect alerts to be triggered on top of it.

A

So yeah I think oh, like these are some of the use cases which we can take care of at the initial stage, and if we avoid it, we can save a lot of time going back refactoring our code.

A

um These are a few of the resources that I have found pretty useful in my journey of learning Prometheus and if you would like uh to uh dive deeper into these topics, they are like pretty interesting to go through um yeah thanks and like free feel free to like um use the slides if you like, So yeah. Thank you.