Cloud Native Computing Foundation PromCon Online EU 2021, 21 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Sponsored Keynote - Fun and Games: Using Prometheus to aggregate disparate metric dat... Matvey Arye

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Sponsored Keynote - Fun and Games: Using Prometheus to aggregate disparate metric data formats at a major gaming company - Matvey Arye, Timescale

In this session, you’ll learn how and why a major gaming company used Prometheus to simplify querying, dashboarding, and reporting on their load testing data, by aggregating disparate metric formats. You’ll learn about the challenge of centralizing monitoring data from disparate sources, why it's becoming increasingly common, and an example of how to overcome it by using Prometheus in an innovative way. You’ll also walk away with a nuanced understanding of the importance of flexibility in observability systems, as well as frameworks to advocate for more Prometheus adoption inside your organization.
We’ll focus on one story, but the problem of aggregating disparate metric sources is an all too common one.
In this case, a major gaming company uses Prometheus to conduct load testing before releasing its games into production, so that every game runs reliably and performantly. Each online game is composed of many services, run by independent engineering teams. Given the mammoth scale of the company, it was not possible to standardize on one tool to monitor metrics. Each team used its favorite metrics monitoring tool, which includes all the usual suspects, like Prometheus, DataDog, Graphite, Zabbix, New Relic, StatsD, etc.

This gave rise to an interesting technical challenge: how to efficiently centralize load testing metrics and monitoring data from disparate metrics sources into a consistent format and data store?
The gaming company solved this problem by using Prometheus and Promscale -a versatile long-term store for observability - to centralize, store and analyze the disparate load testing metrics. The result is a simple, unified system that allows the team to run queries, deliver useful dashboards, and generate reports to analyze load test results in an on-demand and self-serve fashion.
We’ll also discuss some other dimensions of storage flexibility we believe are beneficial to support complex, hybrid environments, often found in the real world.

A

Hi today I'm going to be talking about using prometheus to centralize desperate metric data formats. My name is matt and I've been working at a company called time scale. Since I finished my phd actually, the cto of time scale was my phd css advisor, and so I didn't fly far from the nest so to speak. I was one of the core architects of time scale db and for the past year and a half have been leading prom skill, which is a project to allow users to easily ingest prometheus data into time scale.

A

In my old time I like going skiing, going to see the shows, traveling and taking photos, but during the past year covet has made most of those things impossible. So I really love to geek out on those topics with you over twitter, slack or email. Today, I'm gonna tell you about an interesting use case. We had from one of our users and later on in the silica I'll. Tell you why. I think this use case really illustrates the power of prometheus and prompt scale.

A

This use case involves a gaming company which produces many games, and the way the company is structured is that each game has a team which controls its infrastructure and observability system, but the team we were actually working with the load testing team. This team is responsible for making sure that each of these games can scale to handle a large number of users that play these games over pcs consoles, etc. They do this by load testing each of these games regularly via a ci pipeline and generating a report based on the results.

A

The problem, however, that this team has is that each of these games has a different observability system and it's a challenge to bring all of the data together in the one place in order to combine with the test results and generate the report. The architecture looks something like this: the load testing team has a low generator which generates load onto game services, running in the staging environment of a game's infrastructure, and there are many such games remember. The load testing infrastructure is monitored by prometheus, pushing data into prompt scale and storing it in time.

A

Scale db and the query engine is a homegrown dashboarding system developed by the load testing team. Now the results of the load test are apparent in the results of the observability system inside the games and, as we mentioned before, each game might use a different observability system. The load testing team wrote data collected to collect the key observability metrics from each team's observability stack and push those metrics into prompt scale.

A

This allows all of the load testing data to be centralized in the one place, including both machine metrics and game metrics. The benefits of this to the load testing team is that they're able to save metrics into their own system to analyze it over time, which makes it easy to run reports and to build visualizations using sql and promptql queries.

A

I asked this user. Why did they use prometheus?

A

They mentioned that prometheus is an industry standard with an easy to understand data layout and it's easy to use from ql, as well as a gaining wide adoption inside the company, and the team wants to be on that kind of bleeding edge and the team runs kubernetes and prometheus is very cloud friendly, and so it seemed a natural fit now. Everybody in this conference knows all of these reasons for using prometheus, but I think it's good to be reminded of them so that we could advocate for prometheus usage inside our organizations more effectively.

A

I also asked the user why they chose prom scale. One of the main reasons was backfill turns out that this team had legacy data. They wanted to be in the same, centralized data repository as the new data they wanted, the flexible querying they wanted both push and pull. As I showed in the architecture slide, they found it easy to use and deploy. They mentioned the time scale to be slack channel as giving good community support and they needed long term storage for prometheus anyway. So this checked multiple boxes for them.

A

Now, I'd like to stop and listen to why. I think this is a good use case to think about, and I think the main reason is that it shows that prometheus and his data format and promptql form a good basis for these hybrid systems which combine different metrics together for analysis and reporting, and this ties in very well with promiscuous version, which is to be the flexible database for prometheus and other observability data. What do I mean by flexibility?

A

I mean to both for both push and pull for different data formats? Obviously promises remote rate, the prometheus exposition text, format and json, so booth for promql for dashboarding queries and sql for longer term analytics support for backfill different data types, including not only promising flows but also strings and symbols, as well as different, observability modalities.

A

Right now we support metrics thanks to prometheus, but we want to add support for logs and traces as well. The flexibility allows you to do is to put a lot of your observability data into a single database, but why would you want to do that? I think there are two main reasons. One is operational and one is analytical on the operational side running any kind of database or data store is hard because it is a state for service, so you have to deal with pvcs or hard disks.

A

You have to think about backups high availability fail over security scaling and tuning. These are complex systems, and so why would you want to run multiple types of these systems if you could decide on just one? The other reason is that having more data inside the single database allows you to use joints to do analysis across these data types. This allows for correlations more complex analytics as well as predictions and ml. If you want to learn more about this, you can go to the following website to get slides and the resources.

A

From this talk, I encourage you to check out tops, which is the observability stack for kubernetes, which is a cli tool that allows you to deploy a observability suite inside of kubernetes in under five minutes. That sweet includes prometheus grafana, prom scale and other components and, of course, check us out on github.

A

It's worth mentioning that we are hosting a bit of a further discussion later on in this conference, about dealing with this kind of messy world, where you have multiple metric systems or legacy and cloud deployments, and this plethora of complexity, and I hope to see you at that discussion. Thank you and you can find me on twitter, at 7ny or by email at matt. Timescare.Com thanks.