Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ecosystem Tools to Select the Best Remote Storage Solution - Matvey Arye, Timescale

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Ecosystem Tools to Select the Best Remote Storage Solution - Matvey Arye, Timescale

In the past year, the Prometheus ecosystem has developed multiple open-source tools to help users evaluate different remote storage systems to decide which one will better fit their needs. This talk will give users a high-level overview of a number of those tools and discuss how they can help users decide which remote storage system to use. The tools fall into three categories: compliance, benchmarking, and data migration. The first class of tools checks remote storage systems for compliance to the Prometheus standards. In particular, there is now a PromQL compliance suite that makes sure that query results are correct as well as a remote_write compliance suite. A remote_read compliance suite is in development. The second type of tool is a remote write benchmark which allows users to take their existing Prometheus TSDB blocks and “replay” them into any remote storage system that uses the remote_write protocol. During replay, users can choose to test various scenarios such as increased scrape speed or cardinality. The last type of tool allows users to migrate data from one remote-storage system to another. This prevents vendor lock-in and promotes flexibility for users as their storage needs change. We will go over the basics of how prom-migrator works and show a quick demo.

A

I hope to see you here next year. Hopefully the pandemic will be more managed by them. uh So, first of all, a quick recap of what the remote storage systems and prometheus are prometheus has the ability to send its data to an external system for long-term storage and analysis.

A

This is usually done to analyze performance of your systems over longer periods of time or for auditing, with similar reasons.

A

Right now on the website, there are 27 different remote storage systems listed and looking at this list, you might say ah how do I choose, which one is right for me right, so this toolkit, for you is all about the tools that exist in the ecosystem to help you make this decision.

A

uh So a little bit about me, I'm the developer of one of those 27 systems problem scale. uh The system is able to store prometheus data for the long term, and today we announced that we are also able to still open telemetry traces within the system, but here problem scale. We also believe in contributing back to the prometheus ecosystem.

A

So some not all, but some of the tools I talked about were developed in the house by parameter scale, but are applicable across the remote storage systems.

A

So today I'll talk about three different criteria that you should use for: picking a remote storage correctness performance and the ability to migrate data both in and out. There are a lot of other topics you should consider. This is not a decision you should take lightly. You should consider cost. You should consider the availability of different query languages and operational concerns like high availability, security, etc.

A

But unfortunately time is limited and we don't have tools to verify those things anyway. So I'll just talk about these three and I'll start with correctness, and I start with it for a reason. I think often people go straight to performance and forget to ask about how correct a certain system is and having a system. That's very fast, uh but not correct is like having a fast car with a really bad steering. You'll be fine until the first turn, so the first tool I discussed was developed by julius woltz.

A

It's now prom labs he's one of the founders of prometheus. He wrote this great tour that goes across these remote storage systems and tests. How accurate the query results are for different systems and the gold standard here is prometheuses up. So the question is given the query over these other systems. Are the results the same as what prometheus would return? If you have stood the system there and this you can see a whole bunch of systems, chronosphere, um m3, uh thanos, cortex prom scale will have a hundred percent uh compatibility scores, the others.

A

Well, uh they all have varying degrees of of problems, and what I would say is you know uh a promiscuous really believe in in the standard, but you might not necessarily believe in it or you want to trade off some performance to get some sort of non-compliance and that's fine, but you should really think about the trade-off you're making deeply before you make it.

A

The other tool here is a tool written by tom, wilkie garvana. This is a bit different. This doesn't test the remote storage systems, but the test system is at the right to the most storage systems. So prometheus is not the only writer. If you will, there is the grafana agent. uh There are others, and this says how compliant those systems are when writing data and when you're designing your system holistically.

A

You know you should think about this as well.

A

There are other uh correctness considerations you should think about uh whether the compression used by the system is lossy or not uh how crash safe the system is and the latency of seeing your data in the system, but there are no tools per se to check this, but these are other things you should think about.

A

The second thing you should consider is performance, and the first tool to measure performance is a tool used by prometheus itself is called prom bench. It deploys an entire simulated environment for you and generates loads on the environment and then measures how well prometheus um performs within the environment right- and this is a great if you don't have your own data, this could simulate data for you and you can easily configure, in this case prometheus to do a remote write to whatever system you want right.

A

However, if you do have prometheus data, prompt scale wrote the tool code, the remote array benchmarker, which can take your prometheus data, read it out and play back into any remote storage system that accepts remote, write so critically, due to the exact same logic as prometheus itself would so uh you can be pretty sure that the tuning that uh you perform in the trying out the remote array benchmark will apply when doing when using prometheus itself.

A

um When playing back this data into the remote storage system, you could do several transformations. You could change the timestamp to be the current timestamp. Instead of whatever was previously recorded in your data, you could kind of multiply your series, this kind of simulates, the flowing more instances of the same service. You could speed up or slow down playback and this simulates changing the scrape loop interval.

A

You could also play around with the configuration of the remote rate parameters in order to tune the system, so these systems are kind of different prom bench works off of simulated data, but it can test both in just a queries. uh The remote, the right benchmarker is great. If you have your own data already, but I can only right now test ingest and by the way I should stop and say at the end of the talk, I will have urls to all of these tools.

A

So, okay, last part is migration, because once you pick a system sometimes you will need to change your mind. Things happen.

A

um The first tool to do migration is it comes with prometheus itself is called prom tool and it applies to any remote storage system that writes data in the exact same format as prometheus assault does, and those are cortex using block storage and thanos and suppose backfill. So, given current data, you can migrate data from a previous time period, which is important when you are starting up a new system right, because you need to start up your system and, at the same time migrate all data into it.

A

The other system is the pro migrator. This is another system within prom scale. It can read data from any uh remote storage system that supports remote, read and write data into any remote storage that supports remote rate and it kind of does the obvious thing by just piping the remote wreath into the remote array, but it has some nice properties. It limits the memory usage. So even when you know migrating terabytes of data, you can tell it to use 500, gigs or whatever it will show you progress as you are doing the migration.

A

If the process crashes or you want to pull it, it is able to resume with left off and it is completely stateless all right, so you don't need to store your state there um and again it's universal. So any system that is supposed to remote read to any system that's supposed to promote red. Unfortunately, narrower systems support both remote, read and the remote rate and the. Furthermore, there are some things that you have to uh watch out for, because writing to a fresh database is different than back fitting data uh to a database.

A

That already has data right, and so, when choosing a system you might want to think about hey. Can I get data out? Can I get data in uh so uh these are all the urls to the various tools that I just discussed and um yeah we're hiring as well. So come talk to me if you have any questions about any of the tools I just mentioned. I know it was kind of a worldwide tour, so yeah.