Cloud Native Computing Foundation PrometheusDay EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Storing Continuous Benchmarking Data in Prometheus - Matvey Arye, TimescaleDB/Promscale

Description

Storing Continuous Benchmarking Data in Prometheus - Matvey Arye, TimescaleDB/Promscale

Prometheus is most commonly used for observing live production systems. In this talk, we’ll cover another great use case: benchmarking. Usually, distributed systems are benchmarked by using a benchmark driver to apply load and measure performance. This is typically the only data recorded for the benchmark. The problem with this approach is that it gives you visibility into performance output but no ability to diagnose why performance issues occurred. By using Prometheus in your performance benchmarks you can measure resource usage metrics and internal application metrics across all your components giving you the insights you need to understand the reason for performance issues so you can fix them. This will not require a lot of additional effort because you can reuse the observability infrastructure that you should already be implementing in your application as well as the dashboards already built into Grafana. It also allows retrospective analysis of benchmark runs since the data is stored in Prometheus. In this talk we’ll explain how we set up such an environment as well as share lessons learned about tracking benchmarking runs and keeping the result data organized.

A

Hi today, I'm gonna tell you about storing uh prometheus, storing continuous benchmarking data and prometheus.

A

So when they ask you, how do you benchmark your systems today? What are some answers you would give to that.

A

Well, when we ask people, they usually talk about either what they benchmark, what kind of data they use for the benchmarking, what kind of load they generate where they benchmark, whether it be on a local laptop or on aws or somewhere else, and how so? What kind of automation do they use? Do they use a ci cd system? Is it in kubernetes?

A

Do they use popular chef? But today I'm gonna talk to you about the question of why.

A

Why are the benchmarking results that you see uh the benchmarking results that you get right and we actually have some non-benchmarking tools to determine this question in the developer world we have profiles whether they be heap profiles or cpu profiles in terms of operational systems. We have tools like evpf or observability data right, but the problem with all of these tools.

A

The first problem is that they work on one component at a time and that doesn't work well in the cloud native environment, where you have a lot of micro services working together and open performance problems actually arise in the interaction between components, and you will never see this. If you look at one component at a time, the second problem is that these tools are not integrated into your benchmarking flow, and so often what happens is developers get into the cycle where they make a change, they run a benchmark.

A

They see that something is slow and then what's the next thing they do. They try to reproduce this locally on their local machine. In order to use these other tools to get an idea of why the performance is the way it is, then they analyze this data, they make code changes and then the cycle starts again right, and so the question is: can we reduce this friction instead of having to run benchmarks, then download everything locally and then run benchmarks again? Can we do something smaller? Well, I wouldn't be giving the circle.

A

The answer was no. So the answer is yes and the way you do this is you integrate prometheus into your benchmarking setup?

A

So why am I talking to you about this? Well, I work on a system called prom scale is a remote right system for prometheus. So it's where you can store your prometheus data and we benchmark this system using the techniques. I will tell you about now.

A

So how does the architecture I am proposing? Look like? Well, you have your uh benchmarking setup, where you're benchmarking, a single system. uh You have a load generator that generates the load onto the system, and then you send the data back to prometheus, which can then visualize it with grafana.

A

If you have multiple systems, then the users have either the same or different. Load generators apply load onto your two systems and they both send data to prometheus.

A

To get a little bit more complicated if you are running these systems on two nodes, that's also fine and you could get additional node metrics uh using a node exporter, also sending data to prometheus.

A

So now that we can see that in your benchmarking setup, in addition to the traditional benchmarking results, you would get from your load generator or from a measurement tool. You also get application metrics for the application under load application metrics for other applications in your setup as well as know the metrics.

A

uh So now you could start asking much more complex queries. You could ask into end uh questions about how does the system perform um kind of uh the overall throughput or latency of the system right or you could ask infrastructure queries like how much cpu or ram is the system using while performing it as a benchmark?

A

This allows you to in turn answer queries about, given a certain amount of load. How many resources do I need right?

A

um The second advantage of this setup is that you can store your results.

A

And you do this by splitting the setup into two components: one environment is restarted for every test that contains your systems under load and the other environment. You just keep running as a long-lived thing right and over time you will just get the results for new runs into the same prometheus uh server, and so this allows you to track all of these things. I previously talked about over time as well as insert uh information about each benchmarking run.

A

You could get information about benchmarking parameters, so the parameters you provided to your load generator as well as application parameters. Now, if you've ever done benchmarking for a long enough time, you'll know that, coming back to a benchmark after a month, you already forgot exactly what cli parameters or other kinds of parameters you provided to it. You also forgot which exact version of the system. It was that you were benchmarking with this setup.

A

If you expose this information through simple prometheus metrics, you now store it side by side, and this data cannot get out of sync.

A

In addition, you can perform easy historical analysis, not only about how fast your benchmark ran and how much load it could handle, but also about things like resource usage or other types of metrics. For example, you could track the cash hit of your system over time, as your system evolves.

A

uh You can also do this using grafana panels, so then you will have nice graphs over your system. As you continue developing it.

A

The another advantage of this kind of setup is that you reuse your existing observability setup.

A

So, if you think about it, the um question of why is fundamentally a question of figuring out, what's happening inside a black box, you have a system that is running and you have to figure out why it is it's running as fast or as slow as it is, and the inside here is that this question of a why you need to answer both when you're benchmarking, your system and when you're operating the system and the same kinds of information that will give you this answer in an operational setting will also be useful in a benchmarking setting and this.

A

So uh if you've already built uh an observation system uh for operating your system, you can reuse it in making inferences about your benchmark, results and you're, going to reuse kind of a few components here. One is the metrics that you instrument your system with, but also the grafana panels and the message of analysis that you have already developed, and so you get into this uh kind of dichotomy.

A

If you are, if you have good uh observability signals already, then those signals will give you a lot of insights about what's going on in the benchmarking run. However, you, if you don't have that this method will force you via benchmarking, to add additional observability uh instrumentation into your application, because you were needed to figure out what's going on, and so this creates kind of this nice cycle where improving your performance also improves your observability.

A

So this is kind of a positive feedback loop. If you will, where metrics improve performance, improve.

A

Metrics and so in summary, um this approach has uh three main advantages. One advantage is that you get kind of historical analysis and storage for free. You don't need to deal, as you often would with excel spreadsheets, where you record your benchmarking results, you get a view of multiple components at the same time, and this will also force you to improve your observability signals, or these observability signals will give you insight about.

A

What's going on so in the end, what you get is you get a nice end-to-end view of the system as well as an answer about why right.

A

And so um how do you get started uh using this kind of a technique? Well, it turns out that prometheus uses the same technique itself, so what I would recommend is starting to look at a tool called prom bench which will spin up an entire prometheus cluster with some load for prometheus it loads. The data monitors the cluster.

A

This tool is already integrated with github ci cd, and it will show you how to do this in your own setups.

A

But I believe that this technique is really just the beginning, because observability is more than just metrics right. It involves logs that are involved, there should be retracing, and so, if you take this idea- and you add uh other observability signals in it- will improve your understanding of your benchmarking results even more, and this is actually the vision of problem scale where you can put all these signals into the same system.

A

Right now it supports prometheus metrics and open telemetry traces, but more signals will come in the future.

A

And that's about it for the talk and we have a booth outside and we're hiring. So, if you're interested in working on these kind of systems and developing them, let me know.

B

Awesome. Thank you very, very much.

B

And by now you all know the drill and any questions raise your hand or even stand up.

B

C

So, with um with benchmarking, is there do you have a preferred tool for generating almost like reports or like artifacts, that communicate the inputs and outputs of this load testing that you've been doing.

A

um Where the report is, you would just all of the inputs and outputs just become prometheus metrics. So all you have to do for input say.

A

Is you um instrument your load generator to generate prometheus metrics those prometheus metrics get stored in prometheus and you just create a grafana or whatever kind of panel to take a look at it for outputs, and this is, I think, the kind of thing that's not always entirely clear if you think of old-style benchmarking, there's one result but in actuality, if you look at these complex micro services based systems, there's no one output, they kind of the whole behavior of the system is the output, so yeah.

A

You could measure something like request rate or latency, but they could be much richer. Sets of outputs, including you know how much ram cpu was used, etc, etc.

B

A

Right. Thank you.

B

Very much any any other questions anymore. Bjorn. Can you check slack for me nothing on slack any more questions here in the room, then we were very quick, and that means we get an unplanned coffee break for 15 minutes. We reconvene at four at five to four. I would ask metal matzah to get his microphone.