Cloud Native Computing Foundation PromCon Online EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How we converted our Cortex data to TSDB blocks - Peter Štibraný, Grafana Labs

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How we converted our Cortex data to TSDB blocks - Peter Štibraný, Grafana Labs

Cortex has recently gained support for storing data in Prometheus TSDB format (using so-called "blocks engine", also reusing some parts of Thanos). In Grafana Labs, we had hundreds of terabytes of data in previous format, and we wrote scalable tooling to convert it all to TSDB blocks. In the talk, I would like to present the tooling, show some details from Prometheus TSDB internals, and show how simple it is to generate TSDB blocks using Go code and what pitfalls to avoid.

A

Hello and welcome to my talk about how we converted our cortex data to tsdb blocks. I am peter, and I work as software engineer at grafana labs, where I work on the cortex project, for those of you who are not familiar with cortex here is a short introduction.

A

Cortex is a horizontally scalable, highly available multi-tenant long-term storage for prometheus. That provides global view across all of your permit. Use data, cortex features, blazing fast queries and is hundred percent compatible with prom ql cortex is currently an incubating project at cncf.

A

In short, cortex acts as remote right target for prometheus servers and allow squaring the data back year. 2020 was the year when we have introduced a new storage engine into cortex called the blocks engine. The storage engine uses tsdb blocks exactly the same as prometheus itself does. In fact, we reuse the tsdb library from prometheus to write the data in addition to that, cortex also reuses some components from thanos and we have optimized some of those components for the benefit of both cortex and thanos.

A

Why did we spend all this effort? Tsdb blocks are simply much more efficient at storing the same data compared to the storage used by cortex. Previously, in addition to that, we also get extra features that cortex did not have before like support for queries without the metric name, pertinent data retention or much simpler data cleanup from the long term storage.

A

Introducing the new data storage format poses the question about what to do with the old data stored in a legacy format, and that is what this talk is going to be about.

A

Original storage in cortex is called chunk storage. It has two parts index and store for chunks. On this slide, we see example of three chunks. Each chunk contains samples for one time series for a specific time range. Typically, a few hours.

A

In addition to those samples, cortex chunk, has submitted data about itself like what series it belongs to stored as a set of labels or minimum and maximum time.

A

Each chunk is stored separately, for example, as individual object in the cloud storage or individual cell in bigtable. As you can imagine, this generates a huge number of objects or cells due to the replication individual sample. May end up in multiple chunks, although cortex chunk storage does have some features to reduce this duplicity for cortex to be able to locate the correct chunks during the queries it uses index index is stored in no sql database like bigtable or dynamodb index needs to have multiple features.

A

First of all, index needs to allow for efficient lookups based on time, because promql queries are always restricted to some specific time range to make them run fast. We need to restrict search to this time. Range only index must support, search by specific label names and values.

A

Next index must also support multi-tenancy, even though the index entries for all users are stored in the same database.

A

How exactly index looks like in cortex has evolved in time and cortex uses so-called schemas to describe each version of the index on the picture. You can see simplified view of so-called version 9 of the index with entry types like label label value or series here is the fun challenge: how to efficiently return chunks for cortex uptime metric with job names, starting with queue.

A

Let's take a look at tsdb blocks. Now block consists of multiple files on disk. Each block has a unique identifier that encodes timestamp when this identifier was generated and random part for uniqueness inside chunks subdirectory. There are so called segment files which are numbered in this image. There are two block also has an index stored in a single file.

A

Metadata are stored in a small file in json format and finally, there are tombstones which cortex currently doesn't use. This is what meta json file looks like. It contains some information about the block. Most importantly, the time range that it covers, but also some stats and information about compaction.

A

Now, let's take a look at segment files. Those numbered files under chunks directory segment files contain individual chunks. These are similar to cortex chunks, but much simpler. They only contain samples. Chunks are stored, one next to another in the segment file until they take roughly half of gigabyte of space and then another segment file is started.

A

Each chunk has so-called reference number, which is basically the position or the offset of the chunk in the segment file combined with the number of segment file. 001 in this example encoding. These two values together gives us a reference number index in tsdb block is a single file and it is quite complicated internally for maximum efficiency.

A

But the basic idea is simple: it contains information about all time series in the block that means list of labels, list of chunks or the reference numbers together with minimum and maximum time for each chunk index also contains all label value pairs for fast lookup and these label value pairs are then mapped to series ids, and that's basically it now. You know: what's inside your tsdb blocks, it's interesting to see that prometheus does not encode series type like gauge or counter in the index.

A

What is important about the esdb blocks is that they are standalone. Each block is independent from any other block. Blocks can be merged together into larger blocks through the process called compaction, and this helps to save the disk space, because big part of the index is typically the same. That is, if you have the same label, names and values for longer than couple of hours to learn more about esdb blocks. I can highly recommend series of blog posts by my colleague, ganesh vernicar, who maintains the sdb library in prometheus.

A

Slides, contain the link if you are interested, so how are tsdb blocks used by cortex cortex components, put incoming data to blocks and upload them to object, storage like gcs rs3. We define our own structure for storing blocks. Basically, each user has its own directory full of blocks, and we also put little extra metadata into meta json file, but otherwise we just use plenty sdb blocks in cortex. We generate two hours blocks at first, but later we compacted them into one day blocks that is, each block covers 24 hours of data.

A

We have seen how cortex chunk storage looks like and what tsdb blocks look like now, how do we convert chunks or cortex chunks to tsdb blocks to generate block? We need to find series and chunks that belong to this block, since we want one block per day per user. That means finding series for specific user that has samples in that specific day.

A

Unfortunately, cortex index was not designed with these requirements in mind. Maybe thinking what isn't that? Isn't that? What index is for yeah well index is designed to do lookups for label and value pairs, but not to find all series for a user.

A

In fact, it's even difficult to quickly find all users in addition to that, cortex uses different versions of index. Fortunately, in our production databases, we have only used version 9 and later, which are compatible. So that is what we have focused on.

A

On the other hand, we can read the entire index and generate lists of series and chunks that should be put into each block, and that is actually pretty easy and efficient to do. It just requires single full scan of the index, and this observation draw the design of conversion tooling.

A

Our conversion tooling, has three main components: index scanner for doing full index scan and generating plan files, which is basically a list of series and chunks that belong to a single block, block builder, which uses single plan file, downloads, all chunks and constructs tsdb blocks and finally, scheduler which monitors available blend files and distributes them to block builders.

A

I will shortly talk about these components now, but if you want even more details, there is a link to the design document.

A

Scanner scans, the index tables in cortex. Our index is divided into individual tables. Each table covers one week of data when scanner scans the table. It is processing all index entries in some order. Due to how cortex stores index entries. This order is basically random, because cortex prefixes each entry with the hash for better key distribution.

A

At least that is the case when cortex stores index in bigtable, which is what we have used in our production.

A

While reading all index entries scanner only selects entries that describe the mapping between series, ids and chunks and stores these mappings into a file. We know minimum and maximum time of chunk because it's part of the chunk id so we know into which days the chunk belongs.

A

So, in the end, the scanner produces one file per user and day, and this file contains complete list of series ids and their chunks, and this is called plan file.

A

If you have thousands of users in the index, scanning single table will produce seven times that number of users of pain files one per day per user plan files are then uploaded to the object. Storage bucket and scanner can continue with the next table until it processes all of them.

A

If new table appears, which can happen because entire conversion take takes many days and customers are pushing new data in. In this time, scanner can still process the new tables.

A

Originally scanner has supported bigtable only, but now it also supports dynamodb, and there was some community work on adding cassandra. Support scanner is not horizontally scalable by design. It's typically pretty fast. Already block builder is the main component of the tooling, because it generates the sdb blocks.

A

Block builder is told which plan to work on, and then it downloads the plan and fetches all the chunks from chunk store, quote builder then builds tsdb index and segment files and uploads generated tsdb block back into bucket, where cortex can find it block builder takes care of many details that need to be right.

A

It deduplicates samples from multiple chunks in the same series chunks such chunks may appear because of replication used in cortex builder can also fix some bugs introduced in cortex chunks over time, for example, duplicate labels builder, of course, takes care of sorting series before building index. Otherwise index would not be correct, and that's so without getting unkilled builder obviously knows how to produce meta json file with cortex metadata important point about block builder. Is that once it has the plan, it doesn't need to interact with cortex index.

A

It only needs to fetch chunks. Cortex chunks already contain full information about the series like label names and values, and that is all that block builder needs.

A

This also shows one reason why tsd is more space efficient than cortex chunk storage, namely because the information about the same series is not repeated in every chunk when it is stored in tsdb.

A

Instead, the series information is only stored once in the tsdb index. Block builder is horizontally scalable. You can run many of them to get blocks built faster.

A

We have said before that index scanner writes plans to the bucket. Scheduler is the component that finds those plans and sends them to the blog builders.

A

Blog builders communicate their build progress using another small file in the bucket, which is regularly updated. As long as block builder is running and working on the plan, if scheduler finds that the block builder hasn't reported its progress recently, scheduler will consider such builder as crashed and will abort the build when blockbuilder is done with the plan it uploads finished file. Next to the plan file. This tells scheduler that, given plan is finished, if build has failed, blockbuilder will upload failed file. Instead.

A

This is a very simple orchestration, using object. Storage note that the scanner doesn't need to read the files it only issues list commands or information is encoded in file names by doing single list, scheduler knows which plans are available which are in progress or finished.

A

This allows scheduler to update its in-memory state every few minutes so that when the blog builder asks for the next plan, scheduler can return one. How did it work? Well, it worked. We have successfully converted all stored chunks data into tsdb blocks and we could downscale our bigtable instances.

A

What have we learned that simple distributed system is good communication using buckets by listing of files works just fine, and it's easy to manipulate the state of jobs by manipulating files in the bucket, for example, if you delete the failure file, scheduler will see that the plan needs to be built again and will give it to the block builders.

A

Were there any problems? Of course, there were first issue we have run into was related to meta json files in early versions of both builder. We did not correctly set all json file details, especially compaction section cortex compactor was then confused by such box and deleted newly built blocks soon after they were uploaded to the bucket.

A

We did not catch this in testing because we didn't run compactor in cluster, where the test was done, but once fixed one once fixed, we were quickly able to just redo those blocks again.

A

Another issue we have hit was related to memory.

A

When writing tsdb index, we were trying to sort all series in memory, but some of our customers had so many series that we could not fit all the labels in the memory during the block build after retrying the job several times with more and more memory, we eventually gave up at 30 gigabytes and stored series data on the disk instead with some sorting afterwards that fixed the issue and last point I have- is horizontal scaling for the win by using plan files.

A

We split the whole conversion task into many small sub tasks, allowing block builder to be horizontally scalable, that is, we could just run more of them to proceed faster. One of our clusters contains thousands of tiny tenants and number of plans on this. Cluster was in millions, but we got it converted in just couple of days by running many.

A

Many block builders, if you have your own chunks data that you would like to cover to blocks all this tooling, is part of open source cortex codebase in the last part of the talk, I want to show you how you can write the sdb blocks by some simple go code.

A

Of course, if you want to convert some data to tsdb blocks, you can also use the new prometheus backfilling tool, but sometimes it may be more efficient to write the sdb blocks directly without converting data to open metrics format. First- and it's also easy and fun to generate the sdb block we need to just we need to generate sorry. We need to do three steps, write chunks into segment files, write index and red method, json file, and we will use tsd library from pro methods to do that.

A

You can find full example at provided link.

A

This code example shows how to prepare chunks for a single time series from given list of samples. Each sample has a timestamp and a float64 value. We need to iterate through all samples.

A

We convert the timestamp to the unix timestamp in milliseconds and because that is what prometheus expects to find in tsd block. We check we have appender for trunk. If not, we create a new sort chunk which is used by prometheus and we get a pander for this chunk. Appender is used to write individual samples to the chunk.

A

We are not only building chunks but also chunk meta information with minimum and maximum time. Now that we have appender, we can write samples to chunk.

A

We simply append sample to the appender and update max time in metadata. In this step, we assume that timestamps for samples are increasing. That is how prometheus expects to find samples in the chunk, after writing, 120 samples into a chunk. We finish the chunk, append metadata to our list of metadatas and set appender to nil, which will cause creation of new chunk in the next iteration, and that's it. This creates chunks.

A

Chunks are written to the disk using chunk writer. One can get the chunk writer from tsdb chunks package, and it currently has these two methods. Right chunks and close red chunks can be called many times. We give it a slice of chunks to write, and, after writing them all, we can close it.

A

There are few important details. Chunks must be sorted in the same order, as is the order of series in the index prometheus. Does it this way and prometheus may use this fact for future optimizations within single series. Chunks must be sorted by increasing time.

A

Calling great chunks will update reference field of each chunk. Metastructure. This reference field is important in the next step writing index index. Writer has two important methods. First, you must call add a symbol with all symbols, that is, label names and values, and these symbols must be sorted and each symbol must be added exactly once.

A

After writing, all symbols to the index. We can finally start adding series to the index to add a series. We need to pass in ref number set of labels that the series uses and chunks and the slice of chunks meta structures.

A

These chunks, meta structures must have a reference field set and, as we have said on previous slide, it is set after chunks are written to the chunk writer.

A

The reference number passed as first argument is somewhat strange: it's really not needed, and the implementation of of index writer in tsdb library only uses it to check that it's increased between calls. After writing, chunks and index. We are almost finished with the block. Last piece that is missing is meta json file and this file is represented by block meta type from the tsdb library.

A

We need to fill all the fields, and most of them are pretty self-explanatory. Version is currently one. Unfortunately, this constant is not public in tsd library, compaction information must be set as well, and we simply set block source to itself the failure to do this may confuse compactor.

A

After writing. Meta.Json file to disk. We have a complete block that we can use with prometheus for thanos and cortex. We need extra bit of metadata in the meta json file, but nothing too complex.

A

You can find the fully functional example code at this address. It contains a tiny program that generates the usdb block with the single series.

A

If you put this generated block into prometheus, you can query it and see a sine wave.

A

If you want to use this approach to generate the sdb blocks, check out prometheus storage, documentation and section about backfilling, there are some small things to keep in mind when converting very recent data and overlapping blocks. On the last slide, you can find some links to learn more about the blocks engine in cortex and tsdb blocks.

A

Thank you very much for your attention.