Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Stream vs. Batch: Leveraging M3 and Thanos for Real-Time Aggregation - Gibbs Cullen, Chronosphere

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Stream vs. Batch: Leveraging M3 and Thanos for Real-Time Aggregation - Gibbs Cullen, Chronosphere

The debate over stream vs. batch processing has been ongoing for years. While batch processing is optimized for large volumes of data, stream processing allows for real-time analysis. With monitoring workflows aimed at minimizing time to detect incidents, having real-time insights is critical for maintaining reliable cloud-native applications. Monitoring business-critical applications can become difficult at scale. How do you continue processing large volumes of real time data while maintaining valuable insights? There are OSS metrics solutions designed to ingest high volumes of data, but they also need to efficiently aggregate metrics for viewing and analyzing these volumes in real time. This talk will explore how two popular OSS projects, M3 and Thanos, have approached the problem of real time aggregation. The audience will learn how stream and batch processing methodologies have been leveraged by the community to aggregate data in real time, and the tradeoffs of each approach.

A

Okay, well uh welcome everyone. Thank you for joining. uh You know so I'll be talking today about stream versus batch leveraging, m3 and thanos for real-time aggregation.

A

I guess I just had a quick intro, but take the mask off all right sure. That's good yeah! No! We want to be able to hear okay um yeah, so just quick intro about myself, so yeah developer advocate at chronosphere, where I help out with the m3 open source community as a contributor, and then I'm also a member of the cncf of durability, tag and prior to chronosphere. I was a product manager over at aws.

A

Okay, so just running through the agenda for the talk today, we're going to start out going through the problem statement and an overview of streaming versus batch aggregation, then we'll go through stream, aggregation with m3, followed by batch aggregation with thanos and then quick overview at the end kind of comparing the two.

A

So why does aggregation matter for real time?

A

Looking here at this example of a c advisor dashboard, you can see that so see advisor is a way to get your resource usage and performance metrics of all of your running pods or containers so kind of cpu memory, kind of your infrastructure level metrics.

A

It runs as a damon set inside of a cubelet and then this particular dashboard we're looking at all of the um pods within our gateway application, but essentially you're just able to get a quick uh 10, 000 foot view of all of your of all your applications uh with this kind of with this dashboard so zooming in for this example we're going to look at the this particular panel. Here, that's highlighted looking at cpu usage across all of your gateway, pods or containers.

A

So you know you can see here that it's it's kind of showing an overview of all your pods, but if we look at behind the scenes at what's what it takes to really produce these results, you can see that it can. It can lead to quite a bit of time for your queries to fully render these results. So the way to advisor is is aggregating or pulling this metric.

A

This container cpu usage metric is it's pulling labels across all of your pods in your in your pod group, so as a result, you're getting roughly 51, 000 time series and that's going to be taking over 20 seconds for your results to render so and that's just to query this this metric. So if you did any sort of functions on top like summer max, you can imagine it would take much longer, but in most cases you typically don't want to.

A

You don't need to look at your your metrics at the per pod level, and you really just need that aggregated view to see what's happening across all of your your pods or containers.

A

So you can see here. In this example, we've taken the same metric except we've, aggregated it against two labels, so container name and namespace, and by doing that, we're kind of we're also taking the sum of this metric at a rate of one minute um and by doing this you're kind of aggregating prior to query and that's going to reduce the load on on query time by quite a bit. So you're going to be seeing.

A

You know seeing your results much faster, so less than a second and that's because you've aggregated down to roughly 230 time series.

A

So yeah, so that's just an overview of kind of how aggregation can help with performance, but a couple different ways of doing aggregation is streaming in batch, so just to kind of give an overview for those who might not be familiar between the two two ways of aggregation. So with stream aggregation, you um have data being collected continuously or you know streaming continuously.

A

The aggregation is going to be done and performed in memory on the ingest path before being written over to your time series database, so you can kind of see in the little diagram at the bottom there, and this is typically very useful for um for for information. That's needed immediately. So for like dashboards, for example. um As you know, your data is going to be aggregated in real time and then available immediately for query.

A

Once it's been written over to your time series database and then on the batch side a little different, because you have you're going to have your data collected over time and the way it works is um you know, you're going to have your aggregation performed by reading your raw metrics from your time series database before then. Writing back the aggregated result, so you can kind of see the the two arrows pointing from the batch job there on the diagram, but this is typically meant for large large quantities of information.

A

That's not necessarily as time sensitive um and then the data is going to be aggregated in batches over time.

A

um So for the purpose of this talk, we're going to be looking um primarily kind of at how prometheus or prometheus compatible solutions perform um or do aggregation, whether it be batch or streaming, but just to start out giving an overview of how prometheus does aggregation um is they do so prometheus uses what they call recording rules to do aggregation and what recording rules are essentially um are that they um allow for pre-computing of frequently needed or computationally expensive queries before then storing back the aggregated results to your time series database.

A

So any the execution of these and the pre computation of these are done kind of in memory as a single process. Your regular intervals that you set so every minute, 30 seconds, for example, so using that cron job type process- and this makes it really useful for dashboards.

A

So by using by doing the pre-computation you're able to kind of have much faster results than if you were to kind of have to re-um, re-evaluate your expression every time it's needed and then, of course, as it's supported by prometheus you're gonna have full access to from ql.

A

But typically, if you outgrow a single prometheus instance, you may want to use a remote storage solution, so some of the most popular ones being thanos, m3 and cortex, which are all prometheus remote storage in prom ql compatible and they use a combination of batch and stream aggregation.

A

um You know to do their to do the aggregation, so we're going to now focus on uh m3 for a streaming streaming aggregation and then we'll get into thanos for batch aggregation.

A

Okay, streaming aggregation with m3, so just quick overview of what is m3. So it's an open source metrics engine comprised of four main components. So there's the distributed custom built time series database called m3db, then there's the m3 coordinator, which is our down ingest and down sampling tier, followed by the aggregator, which is optional to run depending on your use case. But it's the streaming aggregation or distributed streaming, aggregation tier and then.

A

Finally, we have an optimized distributed query engine as well called m3 query, and then m3 was built back at uber and open source from 20 in 2016 to help with their metrics monitoring, use cases internally and is now used by many other companies, including chronosphere, and it was designed to be prometheus. Remote storage and prom ql compatible.

A

So, just going to show a high level overview of what the architecture looks like, so you can see on the right side. You have instances of prometheus it's going to send in metrics 2m3 via the coordinator using prometheus, remote right and then you can have you know your coordinator and then optionally, your aggregator um to do any sort of down sampling, aggregation or um before kind of sending over your metrics to m3db and then on the read side.

A

Similar thing, you're, gonna kind of send any query: requests to m3 via the query tier um using prometheus, remote, read.

A

Okay, so uh streaming aggregation with m3 so with m3. The way that it does aggregation is basically it moves. The kind of prometheus recording rule computation to streaming aggregation, and it does this by uh through what they call roll-up rules.

A

So roll-up rules are essentially m3's approach to aggregation of high cardinality metrics and it takes the similar it kind of solves the same problem that recording rules does just with a slightly different approach and how it works. Is you can kind of see the diagram here? It kind of aggregates across multiple time series um and then and then what the aggregator and the coordinator do um is that it'll reconstitute um this new rolled up our aggregated metric as a new histogram gauge um uh or histogram histogram gauge or whatever, or counter metric?

A

Yes uh before, then writing it back to rather than writing it to m3db, and so then, once it's written over 103db, it's going to be immediately available for for query and just what this is just kind of showing that description visually.

A

So, essentially, there's going to be three main steps here, so the first step is going to be sending in your metrics to m3 via prometheus remote right so from there you're going to have the coordinator and optionally, the aggregator for, depending on your use case, to do any sort of in-memory aggregation um on the ingest path and then from there.

A

The coordinator will send over this new, aggregated or reconstituted metric 2 m3db for storage.

A

Okay, so some pros and cons of streaming aggregation with m3, so, first being on the pro side that you know it is very you're going to get really quick query results with with this approach, because you're doing all of your aggregation prior to prior to query so you're going to have your results already ready to go, and that also makes it so that you're going to have very few uh requirements on the query or read side of things, so you're able to then kind of use, kind of you're able to kind of um deploy more more, like other things, instead like uh scaling up a higher number of alerts or recording rules, because now, because your your load is a lot less on on um on your time, series database from the from the query or read side of things and then on the con side, uh it can be complex to to operate and deploy uh and require some additional overhead and then, additionally, it doesn't support arbitrary prom ql.

A

Instead, it just like I mentioned the coordinate is going to reconstitute these metrics as new aggregated counter timer histogram metrics before then. Writing them over to your time, series database.

A

All right so now I'm going to get into batch aggregation with thanos, so a little bit about thanos for those of you who might not be familiar, but it's a cncf incubating project. It was originally built at improbable back in 2017 in open source, and then it has several main components that we're going to talk about today. So there's the store, which is essentially the gateway to object store.

A

Then we have the query component, which is the horizontally scalable and stateless query, aggregation and deduplication tier and then the sidecar, which is one of the ways to deploy thanos and it kind of acts as a proxy for prometheus via kind of remote and read, read and write apis and then and then there's the compactor, which is responsible for down sampling and block compaction and kind of applying any sort of retention policies.

A

And then finally, the ruler or the rule which basically uses thanos rule command to evaluate any prometheus, recording or lording rules and then finally, like m3 thanos, was also designed to be prometheus. Remote storage and prom ql compatible.

A

All right, so this is just a high level uh architecture, diagram of thanos. So, as you can see here, we have a few instances of prometheus with thanos running as the sidecar. We have the query component fanning out requests to each of the various instances, which will then kind of pull, metrics um aggregate them and kind of duplicate them inside the query tier, which will then be informed by the ruler for any sort of recording rules that that you may be wanting to run.

A

And then another thing to note is that thanos does or the sidecar does write over metrics and blocks of two hours by default to the object storage. So you are also able to then query for longer term metrics.

A

Okay, but how does uh batch or how does thanos do aggregation um so basically you're going to have the way it works? Is your raw metrics data is going to be collected by your prometheus instances prior to query aggregation so from there? Your query component can then perform any sort of metrics aggregation or prompt ul queries on top of your metrics that get pulled from your prometheus instances and then and then you have the ruler, which will then implement any recording rules you may have before. Then.

A

Writing your new aggregated time, series data or metric to to object, store and showing this visually kind of four primary steps. Here again, it's a very paired down view of the architecture, so we're not showing each of the components. But um so you have you know first step here: metrics are collected by the thanos side, car store um and then from there, you're gonna have your ruler, which is gonna kind of issue.

A

Your your query, your recording rules and then the metrics will be pooled to kind of um for to mute that query by using kind of reverse index querying and reading from storage, um and then the third step is going to be having the query component kind of evaluating that result on the the query result on your pooled metrics and then from there. The new aggregated uh metric will be sent over to your object store, and you can see here as well like. There are the query.

A

It has two different ways of kind of querying metrics, so you can query directly from from your prometheus instances via the store api for more, like real-time queries, and then you can also kind of access more longer term data through your object, store.

A

Okay, but what are some some pros and cons of this batch aggregation with thanos?

A

So on the pro side, it does fully support and is fully compatible with prom ql, so you're able to kind of run those arbitrary prom, ql's queries and it's also, especially as it compares to m3 it's more simple, to operate and manage, um especially if you're wanting to scale up and down your resources as you're not having to constantly like redirect live flowing traffic or metrics, and then on the con side.

A

You're, basically adding. I guess, comparing to stream aggregation, you're, adding an additional step by having to re-query and then read and write back your results to storage, so kind of by doing all that over the network. It can be expensive or kind of lead to large resource consumption, especially for like larger queries, and in addition to that, you can have slow queries, especially you know when looking at recording rules or cron job type queries.

A

If your, if your query components are going and querying a lot like a large amount of metrics from your various prometheus instances that can take that can take. um You know.

A

Quite a while for for those metrics to fully be queried, and at that point you may, depending on your intervals, that you're running your your queries at you may miss those intervals if it takes too long for your query component to fully query and aggregate your metrics, and um so not only that, but it also, you know, could lead to your query component being overwhelmed completely. So uh just one thing to to note on that side, but um but yeah.

A

So that's uh now we're just going to jump into an overview basically of everything we just discussed so um kind of.

A

How do you choose like you know you want to do streaming with m3 or batch with thanos, um so you know kind of recapping on the m3 side, I think you know the main pros is that it really does alleviate query requirements on your time series database by kind of doing a lot of that pre-computation prior to prior to query so that you can now use a lot of that those resources now for additional purposes like more recording rules or alerts.

A

um However, it is compared to things a little bit more complex to operate and deploy, and it doesn't fully support arbitrary prom ql. Instead, um it reconstitutes these aggregate metrics as uh counters gauges and histograms, and then on the batch side with thanos, um it's more simple to operate than m3, especially when wanting to kind of scale up and down your resources, and this also does fully support problem ql.

A

However, by having kind of reevaluating and rewriting and re reading your metrics over the network, it can lead to large resource consumption and kind of slow query results as well.

A

So I mean these are obviously two examples of streaming in batch that are specific to m3 and thanos, but I think they both do demonstrate some of the the you know the the benefits and trade-offs of more like high-level like um streaming and batch in general. So um you know you can kind of hopefully use this to apply similar learnings to your particular use case that you may have.

A

But finally, you know if you, if you did kind of want to see if there was a way to do both together. I actually gave a talk with rob skillington up here: who's, cto and co-founder of chronosphere. We gave a talk at prom cod on monday about how you can use the m3 coordinator to provide streaming aggregation along with a remote storage solution like thanos or cortex, so definitely check that check that out. If that's something of interest but yeah, that's basically all I have so. I think we'll just open it to questions.

A