Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Evolving Prometheus for More Use Cases - Bartek Plotka, Red Hat & Chris Marchbanks, Grafana Labs

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Evolving Prometheus for More Use Cases - Bartek Plotka, Red Hat & Chris Marchbanks, Grafana Labs

Prometheus is a well-known CNCF Graduated project that allows cheap and effective monitoring capabilities using metrics. With a rich ecosystem of exporters and integrations, Prometheus allows you to collect metrics from your applications running in Kubernetes and beyond in no time. There were many opportunities to learn about Prometheus, its data model and capabilities at every past KubeCon around the globe. This talk will be different. As the cloud-native ecosystem evolves and more organizations adopt cloud-native approaches, all projects are innovating and seeking improvements. We mean things like Kubernetes clusters “as cattle”, multi-cluster applications, eBPF, edge services and more. To keep up with demands for different deployments and architectures, Prometheus evolves too. In this talk, you will learn how you can use and extend Prometheus for those different use cases. In addition, we will highlight advanced, experimental, or upcoming functionality to help with your use cases.

A

Hello, everyone. We hope you have an amazing day at kubecon and you have some spare energy to learn something new and exciting about prometheus project. Before we start, though, I would like to you know to you to know too that maintainers team and community created so many materials about promotions. Already, you can read right.

A

You can find those on our website promoters.io. You can find that on blog posts, youtube other cubecons, fast cubecons, and you know in to be honest, like it's kind of hard to prepare a talk, a deep deck from cues that doesn't repeat some knowledge but fear not. We found some cool area that might be a little bit different for you uh this year.

A

So what we want to do we'll try to give you an overview of all newer features and deployment possibilities that we have created to accommodate many many various requirements from you know cloud native community that adopts uh you know the technology, the cloud, the communities and and all the good is there um with more companies. There are much more, um you know, use cases and different uh integrations that we have to evolve, and this is what we want to show, but first some short introduction. I have chris with me.

B

Hello, I am chris marchbanks and I've been a member of the prometheus team since 2019. I'm currently working at grafana labs on engineering on some machine learning capabilities in my free time. I try and spend as much time as I can skiing and climbing, with my wife in the mountains outside our home in colorado.

A

Well, I miss king as well hello. My name is bartek plotka and I'm a principal software engineer at red hat. I am from used maintainer for a bit and I also co-founded thanos project, which kind of is a product for scaling. Permit use. I'm also cncf attack, observability tech lead and I also write a book with aurelie, which is called efficient, go so before we jump into those advanced cases.

A

Let's introduce prometheus to maybe those new here who just want to learn about what it is and what it does, and let's make it very briefly very brief. So technically speaking, prometheus is just a single stateful go binary that we deploy in one of uh or two replicas next to the workloads. We want to monitor so the things that that produce some some data, some metrics, that we want to observe and grab.

A

So when you start this prometus server um with the typical configuration it will first reach the discovery that you configure discovery service that you configured, for example, it might be a dns service, it might be a kubernetes api.

A

It might be ecs that scans through the virtual machines that you have deployed and using the labeling mechanisms or various different strategies, it obtained the targets that we want to scrape that we want to essentially monitor so with those targets we essentially reach them and pull metrics from those targets which typically are just applications, and we do that with every configured interval, which generally means something between one second and couple of minutes, and we call this process a scrape right and then scrape is essentially a very fast hdb call to the end point that those applications exposes and it usually exposes those uh using the open, metrics or prometheus text exposition format, and so this exposes essentially the current values of the gouges and counters that are in the system.

A

Right now. It will also you know, watch the service discovery for any changes that might appear dynamically to make sure we always have the correct place we take the metrics from and they will be. The available data that we pulled can be then accessed through different means. Yeah. We have various permit to use apis rest apis, including query api with flexible prom, ql language. um You can then connect rafana for some dashboarding experience.

A

You can also configure your own recording rules so to make some fast aggregations or alerting rules that will trigger an alert when, for example, from you see that there are too many errors on your application.

A

Others then are sent to the other manager for routing purposes and then sent to your email, client, phone or or slack to notify you or or some further machine processes, so that was general. You know overview of, promote use very basics. In this talk, however, we want to look on various, maybe more advanced use cases, but actually I want to highlight that those are getting more and more typical, because you know with uh advancements of of technology.

B

A

Cncf projects we can actually you know abstract so much complexity in a in a faster way. So you know you might hit those cases pretty soon if you are even new to the to the cloud native ecosystem, so we will talk about uh scraping features, backfilling signal, correlation, multi-cluster, agent mode and multi-tenancy.

A

Let's go for it.

B

Yeah so, first up the core prometheus, the scraping is still adding more features to allow more and more use cases. I want to highlight a couple of those in the service discovery component.

B

First off. In the last six months, six new service discovery implementations have been added to prometheus. I especially want to thank all of the contributors who have implemented them. These new integrations allow for more and more users to easily configure and get started with prometheus.

B

I'm especially.

A

B

About the generic http service based service discovery to allow users the ability to use service discovery for anything they can think of. For example, in the past, I've had to use sidecars and kubernetes operator to fairly distribute load among a fleet of prometheus instances.

B

Today that would be possible using only the http service discovery.

B

With all of the new service discoveries and just the continual growth of highly dynamic systems, prometheus has also improved handling of potentially untrusted or misconfigured targets.

B

First, it is now possible to limit the size and the count of labels sort of especially useful. One time write a service writing arbitrary error messages into a label value, sometimes as large as 100 kilobytes in a single label value quickly brought my prometheus instance to its knees.

B

Second, you can also now limit the body size of a scrape which can catch some high carnality use cases that may also cause issues in your prometheus instance and finally, starting in the most recent version of prometheus we're going to allow the configuration of scrape intervals and timeouts via relabeling in your in your service discovery.

B

This means things like a generic kubernetes job will be able to customize a scrape timeout, a scrape interval on each service or each pod, without having to create more and more complex configurations for one-off cases. Combined. All of these new service discovery integrations and just the additional control you have over scraping your targets, allow more use cases and more safety than ever before now back to bartek to learn about the new backfilling capabilities in prometheus.

A

Thank you um so, in this kind of um feature you know what is very very important is that we need to understand that, for some people like for most of the of the users of of any observability solution, data is precious because once collected you usually cannot. You know, collect it again for the situation that happened before so sometimes you want to persist that data even across different systems.

A

This is why there are more and more cases when users want to import bigger data. Sets of you know serious metrics to promote use setups, for example, it might be due to migrating from two promoters from other systems or even between prompt uses in a different locations, or maybe bring back up backed up data when you lose your, for example, persistent disk and data backfilling was always possible, not maybe always, but for for a couple of years already, but yet only recently with community, we made it much more approachable.

A

So let me explain how you can achieve that and import large number of metric series today. So first comment that I would like to introduce or like the solution is: um maybe you are familiar with the prompt tool. A prompt tool is like another binary that builds with the prometeuse project and um it allows you to perform certain operations on your local machine or on the server. It's like a cli tool, so we recently added a command called tsdb create blocks from, and then it allows you to define, create a tsdb blocks from different inputs.

A

A tsdb block is essentially a data in a format that prometheus understand and and uses to store metrics uh in an efficient way right. So if we can create in an easy way, those maybe complex, binary format blocks from very easy uh formats like openmetrics or oriamo, then we can, you know, just create those and put it in output and then literally just copy or move those blocks into your prometeurs.

A

Local storage and promptus will immediately load those and even start compacting or recompacting vertically for for efficiency purposes right. So this is why we create the the dcb create blocks from openmetrics, where you provide open metric format, files for different the same time periods, and you can really literally use something something like fromtio's exposition formats, open, metrics and and pack multiple series for multiple. You know timestamps and efficiently, create this db flow from that on your local setup or somewhere else, and then move the blocks. So that's kind of the general backfilling solution.

A

But there is one thing more that we did.

A

So it's a common thing that when you do recording rules or even alerting mostly for recording rules, you are recording those roles so evaluating the uh certain expression for longer time, and then it might be a you know, it might be some destruction at some point of time, like let's say for three days, uh your clusters were down for some reason or like your your machine with promotions were down and you have a gap of three days now what what you can do right now, you can backfill rule backfill this data, which means that you can perform the same recording rule but for the past data and catch up and and build this gap of data and import directly into the promoters again, which is amazing, because you can just fix that the previous gaps, if you still have a raw data for this moment right- and this is especially important when you have like distributed um rules system, maybe with the cortex system and thanos, but it might be useful for permit us cases as well.

A

um So for this you use ddb, create blocks from rules.

B

Thanks spartak next up, there's another new feature in prometheus, which is exemplars exemplars, provide a way to associate context or other information to your metrics and are commonly used for adding tracing information to counters or histograms.

B

So what does an exemplar look like here's, an example of exemplars being exported using the open, metrics format. We can see two exemplars one with a trace id of fast and one the trace id of slow, but you can add any data that you would like to these. That is helpful for you.

B

In addition, you're provided the value of the observation of the exemplar, as well as the time stamp for when that value was observed. These fields allow the prometheus, ui and grafana to plot exemplars on time, series charts.

B

For example, here's how exemplars are represented in the prometheus user interface graphing, the 99th percentile of request latencies for cortex. Then you can click on any of the exemplars present in the graph to find a trace id that you can look for in your favorite tracing tool.

B

If you filter the series to only look at certain pods or error status codes, you will also only get exemplars relevant to those series, always matching the context that you're looking at having the ability to quickly jump from an alert or a dashboard to an example. Trace is very powerful and streamlines. Both incident and debugging workflows.

B

Then client library, support for exemplars is still growing currently both java and go fully support. Exemplars and python has an open pull request to add basic support that will be merged soon in order to enable collecting exemplar data in prometheus, as well as sending it with remote write to other systems that might use that exemplar data.

B

You can use the exemplar storage feature flag. Also note that exemplars are only collected when using the open, metrics exposition format.

B

Openmetrics is negotiated by default when prometheus scrapes a target, but some libraries, such as client goaling, require feature flag to enable open metrics.

B

I hope you find exemplars as useful as I do, and now back to bartek, to discuss multi-cluster use cases for prometheus.

A

Thank you, chris yeah, I mean exemplars. This is something I I really would love to use more and more these days it helps so much to correlate with uh with traces or or really to get the workflow done much much easier. But now, let's talk about very common thing that that are starting to well emerge uh for many of us. So you know we have we used to have just one big kubernetes clusters, but nowadays, with um you, know, companies and startups building solutions.

A

You know that can spin up communities clusters in like minutes or like even seconds like it's, a very common that we have many of them like in red hat. We have dozens of thousands of them running around the world. So how do you even you know, cope with that? And and what are the monitoring challenges or deployment models that we can use with? Promote use?

A

So you know so we have one cluster. uh Let's imagine it's like you know in europe and it's kind of production we might have different environments, you have applications and then, as we discussed, there is a prominent use running there to monitor them and what you do if you have more of them now something that I want to show as a kind of anti-pattern.

A

Never never do that essentially never try to put prominent use outside of those clusters in let's say another region or another um another cluster. Some are very remotely in a different network and and try to use scraping mechanism. So you know very low latency 15 seconds interval or even faster scrape between those zones right, and the reason is that you need to have like a very reliable network available, reliable situation to actually have reliable monitoring from that.

A

So you want to have the promotes very close to your application or really the ingestion part, the pulling part very close to the application.

A

This is why never do that never put primitives in a different failure domain, but what you can do instead right so generally, what always was possible again, not always but probably like three four years was character, federation and essentially what it does is that you federate, so you can say scrapes from one parameters on top of another right, and usually you do that only for a set of metrics with maybe even different interval.

A

So you can actually group and aggregate data on on the second layer and on some other cluster, for example, and be able to query that data from a single place and have all the data there. If, for example, your primitives will die on or your cluster totally die right, you still have data and um that's kind of a great approach, but it has its own trade-offs.

A

First of all, it's a double scrape again, it's again scraped across the domain. So it's not. You know very reliable for all cases and um and overall you usually need to have you cannot. You know fetch all the.

B

A

You have to fetch, like very small percentage of the data, to to to make it even possible, so there are limits to this to the solution.

A

So what kind of you know we build or like community build um is, for example, remote with integration, the promotions, so this allows systems like thanos to just attach to the to the prometeurs so build.

A

We can kind of have a sidecar that kind of transform that um data to some jrpc api, which is kind of easier to use internals, and then you can place some kind of querying component aggregating aggregation component on global level to have to be able to query the data from multiple locations, multiple clusters and I'm talking about hundreds, thousands and and every query- will try to find a proper place and find out to the relevant places for the data and aggregate and perform promptql evaluation.

A

On top of that- and this is thanks of the remote read protocol, which allows fetching series from promoters directly again, the limitation at trade-off here is that you are.

A

You are accessing the data which is still on those cluster kind of components on those clusters. So when the cluster is down- or maybe network is unreliable between you and the cluster, you don't have visibility of those metrics. The advantage is that you don't move any data right. Well, you don't move persistently and you don't persist anywhere.

A

You only move it when you need them right and generally to be to be honest, like we, we love scraping a lot of collecting a lot of data, but we don't use most of them, so this might be great solution for you, but it's not only that. I think what what was what is getting more and more popular and what what people use is remote right, which means that you switch um kind of to the opposite uh model.

A

Instead of you pulling of the data, you are pushing it somewhere else and actually persisting there for, for uh you know forever. Sometimes, if you have like a cheap storage, so prometheus then uses remote right feature to stream the data samples, every sample or the samples you want to stream.

A

So maybe portion of the data that you have inside cluster into them some other cluster and then there are different, many solutions that allow you to receive this data, including thanos cortex, and recently we created even some system that allows you to to kind of configure that in a kind of sas product observatorium. But there are so many other solutions that you can have uh have them listed here and- um and you know that might be the same thing that that will work for you.

B

Beyond just those big clustered systems like cortex like thanos, like things like that that you're sending to, if you don't have a ton of data- or you only want to send a small subset of data similar to what you do with a traditional federation- is that native prometheus is the prometheus binary can now be enabled to receive remote write requests.

B

This means all of your clusters can just send a small amount of maybe the result of recording rules to a central prometheus where you can evaluate global rules and global alerts.

B

An advantage to this is you get all of the nice remote right semantics with retrying requests, or things like that. If there is some sort of network partition that occurs between your central prometheus and your many clusters, this also feeds into edge clients which we'll talk to a little bit a little bit next.

B

So what happens? If you just can't store your metrics next to your next year apps. This happens in lightweight clusters running just like raspberry, pi's, iot machines or maybe dynamic clusters that don't exist very long and don't have long, don't have storage for prometheus or you just don't have persistent disks wherever you're trying to collect metrics from that can make a traditional prometheus architecture very challenging and many times it's very hard to scrape data from an edge client or things like that. The upcoming prometheus agent seeks to solve these use cases.

B

So prometheus agent is a mode in prometheus, just configured with a flag that you can run prometheus and all it does is collect data and forward that data via remote right to another prometheus instance, or any of the remote write receivers that we just saw on a previous slide.

B

This new mode is based on the grafana agent and is contributed by grafana labs running in agent mode. Will minimize the disk and memory consumed by prometheus, while still having a write-ahead log for reliable sample delivery, feel free to follow pull request 8785 for updates. As this work is completed, then beyond the just agent mode and the remote write receiving capabilities, we're also working on improvements to the core server to better support resource constraint, environments, for example.

B

There are some settings recently added to prometheus to just lower the amount of disk space that prometheus needs to provision for every block of data, and we are welcome to any more any more contributions that will help enable these low resource use cases.

A

Thanks, that's pretty exciting.

A

So last but not least, let's talk about use case that pops up um here and there again, which is multi-tenancy right, which is essentially you know what, if we can, you know, reduce the cost of running um from used servers by essentially running one of them for multiple teams that don't necessarily like each other or like, don't want to see each other later um how to enable that- and this is something that we try to evolve as well, and I want to explain how you can achieve that right now, especially on the kubernetes system.

A

So you can you know the problem here, as you can imagine. Let's, let's be specific is, is that in the cluster you might have you know the instances that you want to make sure they are only you know, single or like multiple replicas but kind of single element there, and it you want to make this reusable for multiple teams that and you want to kind of isolate their experience.

A

So, but you know the problem is: there are lots of shared resources here in grafana you specif, as you specify a set of dashboards, you specify a certain configure configuration. You don't want multiple teams to access the same set of dashboards because they can impact each other. Maybe you know create a dashboard, the same name and suddenly everything will will uh not work for another team, the same with administrator, and this is actually very critical.

A

You, you kind of there is normally a a single place of configuring, routing and providers, which tells you know from what alerts should be rooted to watch uh notifier, like maybe page page or duty, or this kind of slack or or whatever. You know. If you allow configuring this by multiple people that don't know each other and don't know each other kind of goals, then well, we might have a clash here and similarly to prometheus itself, where we specify set of scrape targets. So what should be monitored?

A

And we for what are alerting rules all of this. If it's configured from a single place, then you know we might have a serious incidence uh when people are impacting each other from the configuration standpoint. So this is what uh you know: prometheus operator is aiming to solve and in the community's ecosystem.

A

um So you have promised use operator and graffana operator, which kind of recently emerged those servers. They essentially define a certain apis, allowing each team to have their own set of you know: rules, recording rules, alerting rules, service monitors which specifies what things should be scraped. What targets should be should be specified for, promote use um and other manager configurations, so routings and and and those providers and dashboards. You can have your own set of dashboards and it's only your dashboards.

A

You don't need to see you know other teams resources so by providing that through communities apis, we have, you know by default all the airbag and kind of access controls. So we make sure that you know this team can only access, namespaces, so resources, so those dashboards and and configurations that only relate to them, and then we can use graphene operator and prompt use operator that will watch kubernetes apis and essentially merge those things together. So merge another configuration into one.

B

A

Monitors into one script, configuration of parameters and and inject that to the single resources like prometus prometheus, another manager and grafana, and this is what we use in production right now in red hat and and and we really, I mean, um recommend doing that if you have like multiple multiple teams- and you don't want to impact each other, and there is also one important point which is at the end. uh You know configuration is one thing, but second thing is how you access this data while providing some data isolation. You can do that with grafana.

A

Thanks to the you know, access control mechanism, many of those but promote use, doesn't have that. But it doesn't mean that you cannot do it because there are already community driven and very well supported uh proxies, for example, prom label proxy. That allows you to essentially inject a certain label into the promql making sure you know your team only sees your own data, nothing else, and we use that heavily on cortex system tunnel system, so all those systems that derive from promoters as well.

A

So it's like a very, very flexible solution here, and you know all those things that we improved on the scraping part that that chris mentioned in the very beginning uh so making sure resource constraints are, are uh you know, respected and, and also uh you know, there are scrape limits and all of those um isolation mechanisms. This helps as well, because you also limit the the possibility of um of you know one team providing you know so many metrics or so broken metric uh metric endpoints.

A

That will, you know, be stopped and be rate limited before it. It kills the whole probability of server, for example. So those approaches are exactly for this purpose for the multiple multi-tenancy cases, so you have some links for those operators. If you want to check them out- and you will you can you can totally um check those out and help building and improving those as well, but there are certain things that we always you know evolve into, and we have two that we are super exciting to excited to announce.

A

And the third first thing is this: in idea of ingestion scalability automation, what it means is that, thanks to the agents mode, that we kind of start to uh you, know implement and provide, it means that in a sense ingestion, so those the scrape mechanism and service discovery is stateless. So you know the data is pulled and immediately forwarded to some uh remote server. This allows us to scale them dynamically and, using you know, horizontal, auto, scalers and communities or any other solution, because suddenly we can dynamically assign scrape targets uh into multiple agents.

A

So you, if you're, probably if your cluster scales from one node to thousand, we can dynamically. You know uh also scale those ingestion pipelines. It's not like a ones promote you server that suddenly you need to have more of those which which are stateful it's kind of harder to achieve. So I'm super excited to see those uh mechanism and we we we. I hope we can also work with open telemetry to make that to make to improve that part as well.

B

Yeah, we also have high density histograms that has work going on right now to support the new histogram format will provide both higher resolution and higher accuracy of histogram quantile calculations and a significant cost savings in terms of how expensive histograms are to store and just how much data is in your prometheus tsdb.

B

I highly recommend checking out just a few days ago, at the day, zero prom con event. My colleagues ganesha dieter gave a talk about the work they've done in this area.

B

um Please check it out and finally thank you for listening to our talk, provided some links to reach out to the overall prometheus community as well of each as well as each of us individually.

B

Now we have a few minutes to answer any questions that you have. Thank you.

A