Cloud Native Computing Foundation PromCon Online EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: PromQL Compatibility Testing - Verifying Vendor Claims for End User Benefit - Julius Volz, PromLabs

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

PromQL Compatibility Testing - Verifying Vendor Claims for End User Benefit - Julius Volz, PromLabs

Prometheus' query language PromQL is one of the most important interfaces in the Prometheus ecosystem. Organizations rely on it to build their dashboards, critical alerting, and automation. With Prometheus becoming more popular, we are seeing more and more PromQL implementations outside of Prometheus itself, both in open-source projects and in hosted monitoring providers. While this is great, not all implementations that claim to be PromQL-compatible actually implement PromQL correctly. To avoid user confusion, ecosystem fragmentation, and vendor lock-in, we want to encourage fully compatible implementations and at the same time increase transparency around the actual compatibility of different solutions.

In this talk, Prometheus co-founder Julius Volz will present an open-source PromQL compliance testing suite that can shine a light on the compatibility situation. He will also outline test results of actual vendor PromQL implementations, dive into some of the encountered behavioral differences, and give recommendations for anyone wanting to implement PromQL in their own product or service.

A

All right, hi everyone, I'm julius, I'm one of the co-founders of prometheus and I'm also the founder of prom labs, my company with prom labs. I help other companies use prometheus, get the most out of their prometheus, build stuff around it. So I do consulting training products like a training website or live trainings, but also a prom ql, query builder called promlance, so if you're interested in any of that check out promlabs.com, but today I'm going to talk about the prom ql compatibility efforts that I've been working on since last year.

A

So this is also mostly going to be an actual prometheus open source talk, so prometheus has become really successful. It's become the de facto standard for metrics based open source monitoring.

A

This is really amazing and has already also brought a lot of other players onto the playing field. Who want to have a piece of this who want to interoperate? Who you know want to build a business around it or just you know, have a different alternative that is compatible in this diagram. I'm showing a number of different open source projects and hosted services and other projects that promises can send.

A

Data to this is not an exclusive list, just a number of examples, and in general this is great because you know the the standard prometheus server might not be the best for everyone, for example, for someone who wants a truly horizontally scalable alternative cortex might be better or for someone who doesn't want to run their own storage systems at all. A cloud-based service might be very attractive and so on and so on. So alternatives really offer different trade-offs for the different use cases of different people in general.

A

The interoperability then also gives people a larger ecosystem of puzzle pieces to put together in ways that they see fit, and as long as this competition stays healthy and open, we will get better monitoring for everyone, but at the same time we have to be vigilant and be clear about compatibility expectations and actually delivery of those expectations.

A

So if someone uh does not implement one of our interfaces correctly but says they do so, you know they might be misleading users or surprising users when they least expect it in the worst case, an alert that you don't get at night right. That would be bad or you might just be, then locked into their specific behavior in feature set, which is no longer compatible with the rest of the ecosystem, so you can't easily switch anymore and ultimately this leads to ecosystem fragmentation, making the ecosystem weaker.

A

So we want to really encourage people to be both clear around their communication when it comes to compatibility and to also implement solutions to be compatible as compatible as possible.

A

So originally, when we released prometheus, we focused a lot on the implementation of the different components internally like how does the prometheus server work internally? What does it do? um What does it target internally? How does it track metrics? How does alert manager, dispatch alerts and so on? But of course, all these different boxes, the components also talk to each other, and especially now that people came in and said. Okay, we are going to create prometheus, compatible solutions or that interoperate with prometheus in one way or another.

A

The lines between these boxes, the arrows, have become more important. The exposition format is a pretty prominent one for transferring metrics from a target to prometheus. So you know that is currently being standardized or has been standardized as open metrics, which is the standard for sending metrics to prometheus. That has its own test suite and spec.

A

There is the remote write protocol, which is being used to send data from prometheus to some remote storage system, which could be hosted, or you know, another open source thing that you run yourself. um There's the alerting protocol between prometheus and alert manager. We haven't really seen that being re-implemented much because it's pretty specifically tied to those two and a bit low level, uh but a fourth one, of course that we have been seeing uh support crop up from different people in hosted services and open source projects has been our query language promql.

A

So that is also what I want to focus on today: people re-implementing promql or offering it as part of their solutions and the compatibility statements. They make around that versus what they actually deliver.

A

Prom ql is really an important piece of the prometheus story. So, typically always when you collect the data and you store it, you then want to first do prom ql on it to do anything useful, whether that's dashboarding, alerting, ad hoc debugging, you know automating the icd pipelines or even exporting it for later use.

A

Typically, you want to run prom ql on it, and prom ql is quite a powerful language. It's a big interface for sure uh it has a lot of subtle behaviors and you know that you can get wrong or right. So it's it's quite important for users to not be surprised and so on to to get this implemented correctly.

A

So there's a number of projects, vendors, hosted services, etc. That now have prom qr support or claim partial or full prom qr support.

A

You have open source project again in here you have hosted services and so on, and we're going to go into some of these a bit deeper a bit later so, but the question is: how do you even test compatibility?

A

The issue here is that there is no formal spec for promql.

A

You can read the documentation on prometheus.io and it will tell you the relevant or most relevant bits and pieces from the user's point of view, but there are still a million small edge cases and subtle behaviors, and so on that, if you just read that you will probably not get that right, if you just completely re-implement it from scratch.

A

So how do you test that? So what I ended up doing is basically taking the prometheus server itself as the reference, implementation and testing against that like comparing any vendor implementation against the prometheus server's own behavior, I wrote a tool for that called the prom ql compliance tester.

A

It takes a test configuration the reference api, which would be typically your prometheus server as the reference and then the vendor or open source project promql api that you want to compare and test, and then it takes a number of test cases in in the example that I used, I had like over 500 different prom qr queries, and they expect that both of the reference and the test system have been pre-populated with the same data, so that you would expect in the best case, exactly the same answer coming back in a query.

A

So the tester tool then runs the same query against the same. Data set in both implementations, compares the results and writes a test output and the test output will include how many tests failed or passed, and a percentage score. Basically, you know just based on.

A

Did they really return exactly the same results modulo some very slight allowances in floating point values here and there or you know it pre-sorts the result before comparing because the sorting is undefined uh completely in prom ql. Unless you use the sort functions um and for the cases where there's a failure, it actually gives you a detailed error message or an actual diff of the difference of the sample values returned, so you can debug and figure out what is actually going wrong.

A

So test summary, might look something like this.

A

This is just a very small excerpt of one. That's currently linked of one of the blog posts.

A

That's in this presentation on the prom labs home page in the resources section, you can basically see each query that was executed and for the ones that failed either get an error message or get this kind of div output. Where you know in this case, you can see that there's slight differences in the actual floating point values for the quantile operator.

A

So this project started as a prominent project in the promlabs github org, but just now april 2021, I donated it to the new prometheus compliance repo, where we, as in the prometheus team, all want to collaborate on. You know bringing more focus on compliance and compatibility testing to the various prometheus interfaces, remote right, also and open, metrics and so on. uh So it really made sense to donate this prom ql tester there as well, so now it lives there.

A

I did two initial test runs of different vendor implementations and wrote blog posts about that last year. So you can find the full details in under these two links today, I'm just going to talk a bit about the latest results from december 2020.

A

One note I didn't run tests in the recent months after that, so it could totally be that some vendors have different results now, hopefully better ones so keep in mind. This is the state from december 2020.

A

But of course, like we hope to have updates again in the future as the prometheus.

A

Team, so here are the vendors that are included in the last test run.

A

These are not all of the ones that I had initially listed because you know, for example, I had log z listed, but I think I didn't even know that they had a hosted prometheus service yet or didn't exist yet, and so they were not included and wavefront has some quite limited, pronql compatible or not well capabilities, I would say, because they're not quite compatible, uh but the http api that they had was so different that it wasn't even possible to run the tester tool against it.

A

Yet I have been communicating with them, so you know, potentially it will become possible in the future, but I can't say anything about that yet so, for now I I tested, you know all the ones you can see here.

A

I want to give one word of caution about interpreting the numeric test scores because either don't just look at the score alone, because obviously, like some differences in behavior, they cause one test failure. But you know, depending on the difference in behavior, it might actually be a really impactful breakage or a tiny difference in floating point value.

A

So really look at the detailed test results to to see like if this really matters for you and to see how bad it really is, and also some differences are more general than others. So they might like affect multiple query types where this one specific little bug over in a popular function potentially, but that only test once so. It depends a lot of course on my test, query set and so on.

A

So look at the detailed results if you really want to see what is going on and want to make an informed choice when choosing a prometheus compatible system or prom compatible system. So this is a quick result. Overview table here of the different systems I tested.

A

Most of them are a hundred percent or close to a hundred percent. I have two outliers here new relic and victoria metrics, which are quite far from that um we're gonna see why that is later, then there's the feature of cross-cutting issues that can be isolated in the tests. So there are sometimes in an implementation, general query bugs or issues that cause potentially even all queries to be different.

A

If you don't factor out this issue, for example, an older version of cortex had a slight query: input timestamp, parsing bug, so every time stamp in the output would also be off by like one millisecond and then all the results would not be the same, and basically you know to work around that and still make the rest of the results comparable.

A

I added certain query tweaks that you can turn on for test targets, saying, um send a query in such a way that this bug does not occur, so you can still run the rest of the uh of all the comparisons.

A

So the query tweaks that are necessary to enable that are still listed here and if you click in the blog on those details, you will also see uh what exactly those query tweaks were that were necessary.

A

Okay, let's take a look at some of the results just briefly so the first one was chronosphere. I don't have to say too much about that because they got a hundred percent everything passed. They are based on the open source, m3 db system coming out of uber.

A

The good thing here is: they reused the native prom ql engine code. So you know that's already very helpful for getting very close or 100 cortex as well is reusing. The prom ql engine code is an open source project got 100 uh one little note here um in an older version when I was still using the legacy chunk, storage mode with it. It couldn't execute queries that didn't contain a metric name. So for that some of the queries were failing.

A

um I only got 99.62, but you know if you run cortex now with the new block storage, you should get 100.

A

Grafina cloud is cortex based and they are using block storage, so they also got 100 modulo one little cross cutting issue. They align the incoming query timestamps to the resolution step to enable query caching, uh so this may or may not be a problem for you depends, uh but it's something that should be pointed out: it's not technically correct, prom, ql evaluation, but factoring out that issue they get a hundred percent.

A

uh M3 as the pure open source project, I also just tried running that myself. Storing data in there querying it back out and also got 100 great metric fire is another cortex-based hosted service. um They are still running a slightly outdated version of cortex. At least there were in december, and so they didn't get the full hundred percent, um but you know I, I hope, maybe already they're getting 100. If I would test them now, they are aware of the issue and basically a cortex update, hopefully will bring them to a hundred percent.

A

uh New relic is a hosted monitoring or application performance monitoring uh player that's been around for a long time, um full disclosure. I have been consulting with them a bit on this trying to improve the prom ql support. They were building um the big challenge in their implementation is that they they wanted to reuse what they already had. So they have an existing query. Language called nrql and an underlying database in our db, and they were conceptually not really compatible with prom ql data models and language concepts.

A

So it was not really possible to to were feasible to to transpile everything faithfully into nrql. So you know as some examples. Obviously you see the score they got in in my case was only around 31 with some cross-cutting issues, so they don't. You know, support certain features like binary operator, modifiers quite important, I would say not all functions.

A

We have in prom ql, staleness handling, special float values and a bunch more of a little different behaviors that you can look up in the blog post, so you know pointing new relic at a grafana dashboard or pointing a graffana dashboard with a prometheus data source at neuralic might give you quite similar looking results sometimes, but it's it's going to have quite some differences and and not supported, queries as well.

A

Then there is another project which is like so prompt scale is an adapter for the time scale database by the time scale. Folks, it's open source and it's also reusing the native prom ql engine and again great job getting 100 thanos same case, open source, reusing prom, ql engine from upstream getting 100.

A

uh As a last case here in the alphabet, I have victoria metrics. They are also an interesting case in that they really position themselves marketing wise as a drop in replacement to prometheus, but they have their own language dialect called matrix ql.

A

But you know in the first sentence, typically describing it, it says: metrics ql is prom ql backwards compatible. um But then, if you keep on reading, there is a bunch of exceptions listed how they are not compatible and they have extra functions and they do quite a bunch of things quite differently um and yeah. In this case they got around 60, not quite with some cross-cutting issues as well. um You know they have different behaviors around when they drop metric names. They select one more sample in range vector selectors than prometheus.

A

Does they don't support the staleness markers that prometheus? Does they remove nan float values from outputs? They also don't store full float values, so you can't get the full float values back out of the database, um so that also sometimes caused the thresholds of the comparison of the comparator to say: hey, you know this. This query actually failed uh and there's a couple more differences uh in backwards compatibility that are not quite met here.

A

uh So this is just you know something you might want to look at if you were to choose a solution so that that's it for just diving, a tiny bit into individual vendors and projects. um There are still some open questions around this effort in general, so like. How do we deal in test reports with slight differences versus larger differences?

A

Do we report them just as one test failure, or do we want to like characterize them in different ways and how would that even be possible in an automated fashion? So maybe it's just not, and we just in general, want to encourage people to get to 100 anyway.

A

um So you know, maybe that will not be as important. um There could also be some behaviors in the native promptql engine that we actually want to treat as undefined.

A

For example, the subquery alignment step, the subquery step alignment, or you know sorting- is already undefined, or I already ignore sorting in the tester, but there could be other things that we don't actually want to compare in behavior and then there's also the question of how to version the compatibility and the test results over time saying like with which prom ql version are you actually compatible and how much all right there's some related future work going on as well.

A

As I mentioned in this compliance repo, we are linking tests and specs for open metrics and we have tests in there for remote right. Someone just proposed creating remote read tests as well. Obviously, there's the prom ql work in there as well now and potentially future interfaces as well.

A

Ultimately, we want to get to a point where people building systems that are prometheus compatible to get certified marks potentially even be able to self-certify themselves, but we we can't say that for sure yet uh so, initially for sure we, you know these tests that give you some certification will require some manual effort likely input from the prometheus team as well to say that yes, it's actually compatible, but ultimately we should figure this out as a community.

A

So please watch this compliance repo and contribute to it and potentially like discuss on the prometheus, open source channels, the typical ones, the mailing list, the chats and so on. All right so stay compatible. Thank you and I wish you a happy prom con.