Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TSDB Data Compactions on Scale Without Headache: Now & Future - Ben Ye, ByteDance

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

TSDB Data Compactions on Scale Without Headache: Now & Future - Ben Ye, ByteDance

Data compaction is common in modern databases. The same happens in most Prometheus-based systems like Thanos, Cortex and Prometheus itself. Compaction plays an important role in those systems, improving query performance and storage usage significantly. In this talk, Ben will explain how compaction works in TSDB and how TSDB-based compactor works with object storage. Audiences will learn about the challenges of operating compactions on scale and how to mitigate those issues. Next, Ben will introduce the new improvements that were contributed to this space and new toolings that rewrite TSDB blocks. Last but not least, Ben will outline the future work that is planned to improve compactions in the Prometheus ecosystem further.

A

So hello, everyone, my name, is ben, and today my talk is about tsdb compaction. So a quick introduction about myself, I'm an sre at bad dance, working on kubernetes and and I'm a tunnels maintainer and also a promises contributor.

A

So, first of all, I want to give big thanks to ganesh for his uh great tsd block series. So if you are interested in kcb and want to learn more about compaction, please definitely check out his blog post.

A

So in his blog post he defines the compaction as a process of creating one, a new block from one or more source blocks, and this is actually a high level overview of tsb compaction and in this talk, let's dive deep into it. So because the definition mentions a tcp block, so I will first introduce what is the tcp block.

A

So a tsv block is created from an in-memory head block to disk every two hours, and this way there's no time overlapping by default. So different blocks will cover different time ranges. So you can think this as a way to organize your time series data partition by time. So, for example, when there's a query comes in, it will only touch the blocks within the required time range. So if you query the most recent data, it will only hit the most recent block, so each tcp block is actually a minute database itself.

A

It contains four parts: a json file to store some metadata index file for helping you search series and a trunks directory to store your data samples and finally, a tombstone file.

A

So, let's start with the metadata file first, so it includes some block level metadata like the time range and some tsdb stats like number of samples series and trunks and some additional information about compaction and in other systems like tunnels. We extend the metadata file to include some information like the external labels or down sampling resolution of that block.

A

So next, let's look at what is index and trunks, so we all know that in tsdb the smallest unit is actually a sample which just a pair of timestamps and value.

A

So, let's imagine if we map all these data samples to a two-dimensional graph, for example, the x-axis is time and y-axis is serious, since the graph would look like this and you can find that each row just represents a series and a series includes two parts on the hand side we have some label sets which are just some combination of label pairs uh used to uniquely identify one series and for simplicity.

A

You can think uh all these uh series labels on the right hand, side can be stored on disk uh in some special format, and this is called the index file and it's used for looking up your series faster during the query time and on the left hand, side. There are some data samples and each row of samples actually belongs to each series and to store these samples on disk. They are encoded in some special format to save a disk space, and this is called trunks.

A

So when we run a query, it will first try to match the series from the index file and for each series it will try to fetch the corresponding trunks file on the right hand, side and finally, return. The results to the user so yeah, we covered the index and trunks, and I want to skip the tomstone file for now and let's talk about compaction first.

A

So in this diagram we prepared two uh compacts two blocks to compact together: block a and b are created from the head block, so they don't have any overlapping data by default and, as you can see, they have some common series like a job equals http server and also in the meantime, they will have some different series like this series from kubernetes environment, because we all know that kubernetes environment is very dynamic, so the parts might be created and deleted at different times.

A

So it's very possible that some series will exist in block a but not in block b. So after this setup, let's start the compaction. So I think the first thing in the compaction phase is to merge the several index files together into a larger index file. So you can see we have two index files from a and b and we merged merged them into a larger one.

A

So during this process the common series labels, such as the labels from http server and node exporters, they are merged and duplicated because we don't want to keep two copy of them and for the different series like the kubernetes metrics, they are just sorted and combined together.

A

So after the index merging is done next, we want to merge the trunks together. So let's switch to the two dimensional graph again. So what we want to do is to first iterate over all the series from the newly created index file on the right hand, side and for each series on the right hand, side we will try to fetch the trunks from different blocks and merge the trunks together.

A

So uh this process would look like this so because the two blocks don't have any overlapping data. We just need to combine these trunks together, horizontally and yeah. You can see for these different series that only exists in uh block a because b. Don't have this kind of data, so we don't need to merge anything and yeah. Finally, the results would look like this, so we got a larger newly compacted block and the index and trunks are merged together.

A

So this is how the compaction works, and but this is the most common case, because there's no data overlapping and another case is called a vertical compaction and uh yeah. You can see block a has and block b have some overlapping data, and this case happens a lot when uh prometheus supports backfilling.

A

However, the compaction phase is actually still the same. What we need to do is still emerging the index file together and then we going over all these series on the right hand, side and try to merge the trunks together.

A

So the only thing different is that we might got some overlapping in this case right and finally, we will get a newly compacted block like this and the only like overlap trunks is outlined here and for these overlapped data we need to rebuild the overlapped trunks and maybe do some deduplication, but this process is actually very naive in prometheus right now, because the duplication is only one-on-one, which means only exactly the same trunks can be duplicate so yeah.

A

I think that covers all the process of compaction and next, let's take a look at why we do we need it so yeah, let's think about the query scenario so uh before compaction.

A

uh If we do a long-term query, it will try to match all these small blocks and hit them so that in this way we need to go through all these small index files and try to merge the results together back to the user.

A

But if we can do some kind of compaction, then finally, we will only have one larger block in this case, so the same query will only hit one larger block and we don't have to do anything related to like merging results or some online deterification.

A

In this case, so query performance will be significantly improved and additionally, because each index files from the small blocks might contain some common series, as I mentioned earlier. So when we do the compaction and merging process, we can save some disk space by duplicating the index data, okay, so yeah. Finally, let's take a look at the tomstown file, so why do we need it?

A

So, let's imagine the case: if we want to delete some serious data from a block, then we can send a deletion request and specify like the series matches and the data time range we want to delete and for performance reason. These data to delete are temporarily saved in a file called tomstone and it's these data are deleted during the compaction time.

A

So, while we iterate over all the series- and we find that if the series and trunks actually match the data from the tombstone and if it matches, then we drop this and so that the newly compacted block will not have this data so and additionally, if the tomstown file contains enough amount of serious data, the compaction itself will be triggered to clean up the data in order to save more disk space.

A

So now I think you have a high level idea about what is a tomstow file, but in the perspective of compaction, it's still just some kind of modification to your index and trunks data. So yeah to me. I think the essential part of compaction is merging the index and trunks so yeah. I think we cover the compaction, but actually before the compaction phase, there's another phase called planning and it's used to choose which blocks we want to compact together.

A

So in this diagram there are five blocks in the tsdb directory now and the planner choses to compact. The three blocks together into a new new block f. So to choose these source blocks. It follows some uh planning algorithm and right now. The planning algorithm in promises is a little bit simple and it only checks the time range and number of tombstones of each block, and this information can be easily found from the metadata file.

A

So I will not uh dive deep into the planning algorithms here, but ideally we can extend it it to be more intelligent and in tunnels we have supported one planner called index size limit planner to limit the maximum like index size of the compacted block.

A

So yeah, I think that's all for the compaction in promiscuous tsdb itself. Next, let's take a look at some like more large data scale scenarios in some system like tunnels or cortex, and actually I will not cover cortex compactor here, because it's built on top of the tunnels compactor with some like additional features.

A

So a common like tunnels determinant looks like this, so we might have two clusters and each cluster has one promises with one tunnel sidecar and in prometheus we will configure the cluster name as the external labels and the sidecar will try to inject the external labels to the tsdb blocks and upload them to the object, storage every two hours and besides, we also have one central tunnels, compactor running against the object, storage and the compactor works in four steps and first because compaction needs to do some planning.

A

First and planning step only requires the block metadata, so the compactor will have a separate job to fetch the block metadata from the object, storage and download them to a local disk, and after this is done, then it can start planning using this local metadata and the third step. If there's an available compaction plan, it will try to download all these required blocks and compact them locally.

A

And finally, it will try to upload the new block to the bucket and delete all these source blocks. So, as you can see, you can find that stop. Step 2 and step 3 is exactly the same as the prometheus compactor and the tunnels. Compactor just adds step one and four to get it working with the object, storage, uh environment and actually uh there's a little bit difference in the planning phase. So planning is actually done separately in different groups, and each group is identified by the external labels added to that block.

A

So you can see here we have blocks from cluster europe and u.s, so these blocks should be planned and compact together, because we don't want to have a super larger blocks, for maybe all the cluster go away and yeah. So, finally, we got two blocks for europe and u.s and uh yes, that's how thanos compactor works. Basically and next, let's see some extensions we made to the tunnels compactor to deal with our like large-scale data use cases and yeah.

A

The first scenario is actually super common where we deployed two promises: instances at each cluster for high availability purposes, and in this case a new external label called replica, is added to identify the source premises of these data blocks.

A

And then the problem of this setup is actually the data duplication problem because of the grouping mechanism and the external labels for different replicas are actually different. So we will have two groups for two replicas, but actually they are still in the same cluster right. So, finally, we will have two compacted blocks like in this cluster us with totally overlapped time range for the two blocks, and obviously you can see. This is not good, because, firstly, we double the space usage by having two blocks with almost the same data and also during the query time.

A

We have to do some online. The two duplication, because also the query, touch the two blocks. The query performance is not as good as like, with only one block so to solve this actually like. As I mentioned, we can use the built-in vertical compaction mechanism to merge the overlapped blocks. Right and tunnels provide a flag called the duplication, replica label, and if we specify it, the replica label of the replica external label can be ignored during the grouping and planning phase and all these six blocks will be planned and compacted together.

A

So, yes, this is good because we utilize the default vertical compaction mechanism and use it to improve the space usage and the query performance. But the question is: is it really good enough and what about the trunks data?

A

So? Actually the trunks data is still problematic, because the trunks data are from high availability promises pairs and actually they cannot be perfectly duplicated by the default vertical compaction mechanism, because we have two promises instances and they cannot actually perfectly collect samples at exactly the same time. So the timestamp of these samples from two promises cannot be the same. However, the default vertical compaction only works for one-on-one vertication, but in the case of different timestamps, it doesn't work. So.

A

Finally, we still have double space usage for like this replica data so and it would be better to have some like smarter, vertical compaction algorithm to do this duplication for this use case and actually what we want is to have it to have something like this. We want to maybe only keep one sample from only one replica and that's already good enough to us, so this can be done by extending the like duplication, algorithm used in the vertical compaction and in tunnels.

A

We supported the same penalty, duration, algorithms used by the tunnel squarer, and we applied for the location at the compactor side. So this saves the space usage a lot and another challenge we made of internals is to how is about how to deal with some data manu manipulation, requests from the object storage, for example. We want to delete some high cardinality series, or we want to relabel this series or rename this metrics, and the problem is that these blocks are from object, storage and they are immutable by default.

A

So can we modify the existing index and trunks easily so yeah, because today's topic is about compaction and actually tunnels uses this compaction mechanism to achieve this request.

A

So let me use uh relabeling for an example, so we have this kind of reliable in configuration and what it does is that the first reliable request just renames this node cpu metrics to another name and another reliable request- just drops this label with the name code.

A

So in this case we are still trying to do a compaction for this. For this block we want to modify, and while we like iterate over all these series on the right hand, side, we just do this relabeling to this to each series label and if this series series label changes, then we can do some actions accordingly, for example, in the case of label, replace it's super easy because we just needed to rename the labels on the right hand side. So only index parts need to be changed.

A

We don't need to modify anything related to trunks, but the label drop case is a little bit trickier because the code label, once the code label is deleted. Then these three series will be merged into one series right, so in this case we need to also merge the corresponding trunks files and combine them and do some duplication in this case. But anyway. Finally, we re-labeled this pcb block and we got a new block after this compaction.

A

So this is cool, so why not? We have this support in premises as well. So recently I opened the pr to support this kind of use case in the prom tool, and actually the required change is super easy.

A

So first I added one interface called modifier and it has only one modify method and this is just used to apply some additional modification logic to the index files and to the trunks files during the compaction phase and for the promises built-in write method. It's extended with a list of modifiers to apply this logic and to view it more easily. So this is the graph again with the modifier, so you can see the modifier just works as a middle layer uh in the compaction process and during the modif during the compaction phase.

A

We still need to merge the index files and trunks together, but we apply the modification to them before writing a new block.

A

So what can we achieve using this modifier? So let's do some brainstorming about down sampling, so you can simply think about don't sampling as a process of increasing the script interval in premises and in the diagram. You can see we done sample data from 15 seconds to one minutes resolution and after down sampling, we have fewer samples in this case and you can imagine in the case of down sampling. We just need to change the trunks file and we don't need to modify anything related to the index.

A

So actually we can implement a down sampling modifier and with the configuration on the left and high left left hand side when we do the compaction, we find all the matched series with this mattress label and just rebuild these trunks with this configured resolution.

A

And yes, the last brainstorming case is about dynamic retention. We can also implement a retention modifier and during the compaction time we can check whether these series match the required matches and for these matches series we move them to a new block and keep them for a longer time. So yeah. I think, that's all for today about my talk so yeah. Thank you for everyone for listening and I'm ready for the current part. Thank you.

B

We're gonna have a slightly shorter q, a for this session to catch up a little bit on time, because we've run a little bit over. Does anybody have a question they'd like me to repeat for our virtual audience.

B

All right I have one, then um you mentioned the two dimensions of vertical and horizontal that you compact by. um I think I'm correct in saying. Horizontal is your time there and vertical is um overlapping data. That sort of thing yeah.

A

But actually uh I think vertical and horizontal is I mean the in z graph. You can see when we try to merge the trunks together. It's just still like horizontally merging right, but the if there's no time overlapping. We don't need to do anything. We just change these trunks together, but if there's any overlapping, we need to rebuild the trunk so yeah. I was.

B

Going to ask: do you see any other dimensions that, besides those two that you would.

A

Like them yeah, I don't think so yeah, I think only only two dimensions.

B

Well, thank you very much.

A

Okay, thank you.