Cloud Native Computing Foundation PromCon Online EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Future is Bright, the Future is Prometheus Remote Write - Tom Wilkie, Grafana Labs

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Future is Bright, the Future is Prometheus Remote Write - Tom Wilkie, Grafana Labs

"Prometheus remote write was introduced almost 5 years ago as a way for Prometheus to send samples to external storage. Since then the protocol has been implemented by ~30 different metrics systems - and in version 2.25, Prometheus itself accepts remote write!

In this talk we'll discuss the use cases for pushing data to Prometheus via remote write - and more importantly, what it's not meant for. We'll present a more thorough specification of the protocol, and some of the future plans and what a remote write v2 might look like - streaming? metadata? exemplars?"

A

Hi, everyone welcome to the future is bright. The future is remote right. My name is uh tom, I'm one of the prometheus team um and actually primarily work on the remote write code. uh In my day, job I'm the vp product at grafana labs, I'm still trying to figure out what that means.

A

After three years, um I also started the cortex project alongside julius and more recently started the loki project, the kind of prometheus inspired log aggregation system when I'm not when I'm not coding uh when I'm not working, I like to make 3d printers and occasionally uh brew brew, my own beer.

A

So without further ado um today, I'd like to talk about four things right, I'd like to talk about remote right. How did it start? What is it? What is it for um I'd like to talk about standardizing remote right and a lot of the recent effort efforts over the last two months to define what it means and to test that I'd like to talk about what's next or what's in the immediate future for for the remote right protocol?

A

So I'd like to talk about metadata, I'd like to talk about exemplars and finally, I'd like to talk about uh you know further off in the future for remote right, um some of the ideas we've got. I noticed the slide says exam plus I'd love to know what they are. So what is remote right? um Actually, funnily enough, the story of remote right kind of matches, my story with prometheus, the first pr I did to the project was to to switch the remote right system um from grpc over to protobufs and http.

A

um This was because it was quite hard at the time. You know this was almost five years ago. It was quite hard at the time to get uh to get grpc to go through um an elastic load balancer, and I wanted to use prometheus remote right to send data to to cortex and really that is what prometheus remote right is for.

A

It's for sending data to other systems prometheus, sits there scrapes the samples scrapes, your job, scrapes metrics from your instrumented applications and exporters, collects them stores them, but can also forward them on to other systems.

A

So we did this. um We did this about uh five years ago um and and over the past five years, we've seen many vendors kind of take notice of prometheus and and really add, support for prometheus remote right to their to their products. um You know they've they've made it so that you can send data from prometheus to pretty much any any of the metrics vendors in in the world.

A

In fact, if you look at the prometheus docs you'll see there are over 30 different projects um that accepts from the accept prometheus, remote right or send prometheus remote right and really kind of it's amazing that that only over five years has been so popular um and we're not resting there. You know we. We really want to kind of push the interoperability story uh in prometheus as far as it will go and more recently uh in the not the most recent, but the one.

A

Before that release, we added the ability to actually have prometheus receive remote write requests as well, so you can now configure a prometheus to send data from one place to another right. This really solves the kind of global federation problem in a very different way. You know we noticed with with federation. You know. Federation is the way you can send.

A

uh You can set up a prometheus to scrape the metrics from other prometheus servers, so this allows you to have a prometheus server, let's say in each region and then a global one that scrapes all the data from the from the regional ones and get that kind of central view where you can run your kind of central aggregations.

A

However, the challenge here is that this requires you know the the global prometheus to to be able to scrape all the edge ones right. So you have to open up firewall ports and figure out ways of uh securing. You know three or more different prometheus servers with the the push-based approach coming from uh prometheus remote right. You only have to open up that one central location and on and kind of lock down and authenticate one central location and you can have the edge locations push to that central location.

A

um This might not sound like a big deal, but imagine if that uh these edge locations um were on ip addresses that were changing or were maybe on flaky networks. You know the remote right protocol might be a better a better fit for that. So this is in the uh in the second to latest uh prometheus release. It's experimental, we'd love you to give it a go see if see what works see, what doesn't work and see if it suits this use case.

A

So that's remote right, that's how it started. What it's used for.

A

What we have noticed with all this adoption by by many many people is there's been some differences in implementation, and we want to make sure that this ecosystem is interoperable and and really that all the users of all the different components can have the best possible experience.

A

In particular, one of the things we've noticed is: there's a bunch of projects popping up that that fill this kind of scrape prometheus metrics and send them else elsewhere. Using remote right, you know obviously prometheus does. That uh is the original one um about a year or two or a year ago we launched the grafana agent, which is a kind of stripped-down version of prometheus, using all the same code, but kind of stripped down, lighter weight version that doesn't have any local storage.

A

You can't do queries, but it will just scrape your jobs and send them using remote right, there's. Also, the victoria metrics agent uh influx's telegraph will scrape prometheus scrape jobs instrumented with prometheus metrics and send them elsewhere using remote right and then more recently, you've got open telemetry, which will also scrape jobs using prometheus metrics and send them over remote right.

A

So these are the five systems that I'm going to look at, and you know there are other systems I have to kind of slip in at least one meme into each of my talks.

A

So we started this effort to standardize it right. We we wanted a document that described what prometheus remote right was what it meant to say you were compatible with that. We also wanted to explain a lot of the reasoning behind the decisions. Why we've done it this way? It wasn't all just arbitrary and finally, we wanted to give some thought to. How would we future proof this protocol? How would we upgrade this protocol? How would we make it so that, when we introduced a v2, we could be kind of backwards compatible?

A

I'm hoping that this will allow us to remove the experimental flag from prometheus remote right. One of the key things here is that we're not changing anything with this standardization, uh we're just documenting the current behavior, how it currently works, because there's so many people that use this- that, I don't think. Realistically, we can change it uh at least not in a uh incompatible way.

A

You know this will also allow us to offer some kind of compatibility guarantee, and I just want to say a big thank you to everyone in the community. Who's commented on the doc who's proposed changes. It's really been a big team effort, so now that we have this standard, it's time to test those. Those agents that I mentioned against this standard and to that end uh over the last week or two I've built a test harness that runs an instance of each of the agents.

A

Exports some metrics for them to scrape and then configures them to send that data back to the test harness via remote write. We then, in the test harness in you, know, export various different, metrics and, and you know, test examine the the response we get over a remote right and check it matches what we expect. You know, and you know this is relatively straightforward. You know we export a counter. We check, we get a counter back, we export a histogram, we check, we get a histogram back, but there's a lot of nuance to this protocol.

A

There's a lot of different areas which some people do or don't implement, and so we wanted to kind of figure out what level of coverage we had. These are early results. um You know, I'm filming this talk uh almost a month before you'll be watching it so hopefully uh over the course of the next month. Some of these vendors and some of these other projects will improve their compatibility with remote right and we're actively working with all of them to do that.

A

So, to start with, obviously we have prometheus right prometheus implements all of remote right. The specification is just a document that describes what prometheus does so we'd expect it to pass all these tests, the grafana agent, using all the same code that prometheus uses um just implements the same thing right and again, we kind of expect it to pass all the tests. Interestingly, uh we found a bug in the grafana agent using this test suite uh it turns out.

A

We were not removing uh duplicate labels, um so we fixed that in the point one release it was, it was one line of code. The victoria metrics agent does pretty well. Actually um it does everything it has. Some slight inconsistencies around how it does the up metric.

A

It doesn't actually, um as far as we can tell, send the up metric when there's a failed scrape, which is kind of important for the test harness to set to to tell the difference between the agent not doing anything, and the agent successfully kind of failing um and also the victoria metrics agent doesn't implement the staleness markers, which are really important way. We can tell when metrics go away telegraph, not quite as not quite as complete as a victoria metric agent in general.

A

It doesn't do up or staleness at all, but it you know it doesn't quite have the right job labels there. You can program it to add job labels, um but it's kind of missing some of the features of service discovery that require that would add those job labels.

A

Then. Finally, the open, telemetry collectors, the newest kid on the block, so not not not as far along in its support it doesn't propagate histograms correctly, doesn't do a lot of the job labels correctly. No up metric, uh no, no stillness markers, we're actively working with the open, telemetry team to try and get better uh better support, and hopefully in a month or so, um we can we'll see if they've got better support.

A

I'll be I'll, be hanging around for a kind of live q a at the end and I'll make sure I I bring uh updated results for that, so next uh metadata and uh exemplars, not exam plus um so first metadata, not a lot of people know actually that that prometheus exports, um prometheus uh client libraries allow you to add kind of help, text and and and type metadata uh to every metric in your application and uh prometheus scrapes.

A

This and stores this in memory stores the latest values in memory and then has an api, so the the client library, their client applications to prometheus, like grafana or or the prometheus ui can can query this and use it to help build these kind of uis. um So grafana builds this particular ui. Just allowing you to see what each each metric is.

A

We want to enable um systems that implement remote right to have the same information and basically the same experience uh as prometheus and so back in uh february last year. So over a year ago, josh added support for um uh metadata to the remote right protocol. So it's an extra field and it's kind of a bit best effort right, we're we're taking some of the metadata and sending it on a period.

A

Alongside your samples, we want to make a series of improvements to this. We want to write the metadata to the right head log. We want to send it alongside the same metrics that it's supposed to be sent with instead of just kind of arbitrarily sharding it um and and rob from uh from chronosphere, actually has a pr that achieves some of this.

A

That we want to work with him to get in over the next few months, so the next example um we want to give is exemplars right, spelled correctly this time in exemplars, allow you in grafana to to overlay kind of dots on a graph, and then, when you click on one of those dots one of those exemplars, you can jump straight to the trace that that dot kind of represents right. So so bjorn added exemplar support to client golang quite a while ago and callum added exemplar support to um to prometheus.

A

In the last release and together you can implement this really cool experience that really speeds up kind of incident, response and debugging, workflows and and really kind of makes the whole system feel a lot more integrated prometheus doesn't actually care where these traces are stored. You can store them in jager. You can store them in zipkin or you can store them in grafana tempo itself.

A

um We want to make this available to uh to remote right. Endpoints to to people who are implementing remote right should also be able to receive exemplars and and offer the same apis in the same experience and to that end callum from grafana labs, he is adding uh a remote right, uh for example, so this is gonna, be writing the exemplars to the right ahead log and then tailing that right, head log is, is as part of the remote right code and and sending them out in batches.

A

This will also enable kind of long-term storage of exemplars and and some really cool use cases around that callum assures me. This is going to be merged by the time you uh you're listening to this talk, and- and we really hope that this is going to be in the next major release of prometheus.

A

So finally, what's next, what are the kind of more long-term things we want to do with uh with remote right? First thing I want to talk about is atomicity so prometheus. Has this really cool guarantee that um sorry before I go into the guarantee?

A

A lot of you will know that prometheus metrics are normally you know, sometimes composite and actually made up of multiple time series. You know the example here being a histogram which is made up of uh time series per bucket alongside a kind of a count and a sum time series um this is this is kind of how we build up these histograms and how we can tell latencies in your application, and so it's really important that when you run a query, you only see a complete scrape.

A

You see a consistent snapshot of a scrape, you, don't you don't see, kind of partial data or half updates from one scrape and half updates from another scrape um prometheus offers this right. It's it's quite a cool feature of tsdb and it's uh it's. It's really uh really useful.

A

Remote right, unfortunately, doesn't do this actually prevents systems from from offering this guarantee, because of the way the remote right, uh the remote right, client inside prometheus, um splits up uh batches of requests to send them in parallel right. So this means that you, you know remote right systems might actually get um metrics or samples for different series within a histogram in in a different order. Right and we'll actually see these partial states.

A

This means they don't have the opportunity to implement this, um which we, we think kind of sucks right. So we want to fix this. We we haven't decided yet how we're going to fix this. You know one of the thoughts is that we're going to make sure that this scrape the entire batch of samples that are gathered in a single scrape are written to the right head log in a single batch, which is kind of already the case.

A

But then the the the remote right system reads that entire batch in a single um in a single kind of scrape in a single read of the writehead log and sends it out to the remote system in a single batch as well and so kind of aligning this throughout. The entire pipeline will will at least give the kind of uh the systems at the other end of the remote right will give them the opportunity to uh to offer atomicity.

A

um Another thing we're working on solving is kind of, or improving at least, is that the handling of four to nine so four to nine is the is the status code that remote systems will send when prometheus is sending samples to them too quickly? It's 429 is kind of rate limit is a back off right, so prometheus is is designed to back off and retry on 500s now 500s indicate there was something wrong with the system. You know it's the server's fault we couldn't handle the request.

A

You know please try again, um but but prometheus doesn't retry 400's right and it that's on purpose, because 400s indicate there's something wrong with the request right and that request will not succeed. If you try it again, in particular, like 400 might be an invalid request like it's just gibberish like please, don't send me gibberish, you know it might be that the you know you've hit some limits on the total number of series. There's no point in trying again because you've hit these limits and it's weird.

A

I think that 429 is a is a is a, is a rate limit right because you know, actually, if you back off and retry the 429, uh the request that received a 429 response might actually succeed. Either way.

A

What happens is you know we talked about how prometheus is uh collecting data from your applications and writing them to a write head log and then reading that right, head log and sending it to a remote system, and, if that, if that, if there's a network outage or a heaven forbid, an outage on the remote system, um then that will start to buffer up on disk right, and these samples will buff on disk and prometheus will basically wait.

A

You know periodically retrying, but wait for um wait for the system to come back up and then, after that outage prometheus will try and replay that right, head log to the remote system to fill in any gaps.

A

You know prometheus will replay as quickly as it can and it can replay pretty quickly, so it replays many many many samples and that upstream system sent a 429 saying you're replaying too quickly. At that point, prometheus will basically just uh drop the data.

A

So after after doing all this really hard work to to batch up the you know to buffer up the data on disk and and make sure it's there. For you know, during the period of the outage, when the system comes back, we replay 429 and drop the data and it's a real shame.

A

We do this, so we think we can improve this by backing off on four two nines and and knowing when to kind of balance, um you know catching up against kind of falling behind, because you know you never want a system to kind of get behind and stay behind.

A

So finally, and another area we're actively looking in uh and investigating and looking at improvements for the remote write system is its bandwidth usage. You know I I perhaps touched on a bit earlier, how we designed the system to be pretty pretty simple, pretty stateless right- and this is a key principle that I think has really made the protocol easy to implement right. There are no interdependencies between messages in the protocol and this simply just drastically simplifies downstream implementations. It makes things like cortex easy to write.

A

It makes the um the adapters between prometheus and graphite and prometheus and influx. It makes those adapters simple to write, and I think this has been pretty successful.

A

um You know we've seen, as I said, 30 different uh implementations of remote right and I think one of those one of the reasons this has been successful is because it's a relatively simple process protocol. However, this comes with downsides right. It's expensive right. The the fact that the batches are stateless means that they have to repeat labels. Many many times you know the same label was sent in most batches and this really eats bandwidth.

A

You know we used between 10 and 20 bytes per sample to send via remote, right and prometheus only uses one or two bytes per sample on local disks. So you can see, there's a big big room for for improvement here and our internal monitoring at grafana labs um just to propagate samples between our internal prometheus nodes and and our internal cortex cluster, we're doing over over 90 megabytes a second of remote right. So there's there's probably a 10x gain here and there's various ideas.

A

We think a lot of the work we're going to do on atomicity and and batching, and you know, building these consistent batches will allow us to have a kind of symbol table in the in the remote right um requests that that will probably reduce reduce bandwidth usage. Quite a lot, and that's it really. You know we've covered kind of what is remote right. What's the history, why does it exist? How did it start?

A

um We've covered our efforts over the last couple of months to standardize remote right to really document how it works, why it works that way and then test implementations to make sure that they work this?

A

The way they should we talked about- what's coming kind of, you know pretty soon, hopefully, in the next release or two of prometheus, we should uh improve the way we send metadata via remote, right and and start sending exemplars right and finally, we've talked about some of the ideas we have for the more long-term future, how we want to make it uh remote right, atomic how we want to uh deal with back off and retry on four two nines and and how we want to reduce the bandwidth consumption of remote right.

A

So I'm gonna, I'm gonna, hang around now um take some live questions. If any of this was interesting, please do please do ask and thank you very much for listening. Bye.