Apache Cassandra Apache Cassandra World Party | July 2022, 20 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Using SSTable immutability for incremental backup | Apache Cassandra World Party 2022

Description

Presentation from the second Cassandra World Party on July 20, 2022. Learn more about Apache Cassandra at https://cassandra.apache.org/

Manish Khandelwal of Ericsson presents “Using SSTable immutability for incremental backup”
This lightning talk will cover the idea where we use the immutability feature of SSTables to perform an incremental backup. The main idea is to copy only those sstables which are new after the first backup was taken.

About the Presenter
Manish Khandelwal is a Cassandra Consultant at Erisson and a Java developer mostly working on the backend side of the system.

Apache Cassandra is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Linear scalability and proven fault tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Get started: https://cassandra.apache.org/quickstart/

The Apache Cassandra World Party 2022 was a virtual, one-day celebration on Wednesday, July 20, 2022 held in anticipation of the launch of 4.1

Join the Cassandra community: https://cassandra.apache.org/community/

#ApacheCassandra #database #NoSQL

A

Hello, everyone I will be speaking upon uh how we, how did we use accessible immutability for incremental backup uh disclaimer here? Is that uh uh I'm not going to talk about the backup uh incremental backup provided by Cassandra is just a kind of a quick in the full backup we did for getting the similar results. So before going to how we should say why we are doing that. So we have a use case that uh which requires us to keep two backups at a time.

A

um So we should have enough space to keep two backups in all backup disk space. So uh generally we Dimension the product like that only but over the uses of over the period of time the data disk fills up, and uh so this use case we are not able to fulfill for one of the customer. uh So uh so now at runtime we have to solve the problem. So what we thought that we asked the customer to uh either uh increase that this space or find a solution to fit uh 2x data in 1.5.

A

This is space, so what we did that uh we started looking out for the solution, because the customer declined to increase the data disk size. So we saw the data as a stables and we found that old access tables were of very large size and they were quite all old, and so uh we identified that most of the data is again copied in the second backup. So we now knew that we don't need to copy the complete data. So before going to the solution, uh we will see what the access tables access tables are.

A

The files uh which Cassandra stores datas on the table uh as a stable is created, result of flashing of mem table and compaction process. They are immutable so once they turn they will not be updated till the compaction is over and they have a format, format, version generation, ID format, component and text generation. Id is basically used for uh differentiating the access tables. So this is the part which you should keep in mind, that generation ID and how the Cassandra backup process we take it.

A

The full backup we just trigger the node tool, snapshot command and it creates a snapshot on all the nodes. uh What node should snapshot is basically, it creates a hard link of all all the access tables in your a parallel directory to the uh your within within your tables directory. So it will be. The hard links are just basically a link to the existing file inode, uh so they do not take much space and they are created pretty fast.

A

So we use this both concept, the immutability of access tables and hard links for solution solving our problem. uh So what we did. Basically, we created a backup, First, the full backup and before taking the second backup, we created another directory in the backup directory called back up to, for example, and we just hard linked all the access tables from backup one to backup, 2 and then before. Taking the backup 2 means uh before triggering the Backup Tool from the real that data directory.

A

What we did uh now we have backup one and backup two, which are quite similar, uh so when we triggered a backup 2 from the data directory, what we make sure that we did not copy the files already in the backup too, and we removed the files which were not in the data directory. So now, backup 2 was the actual full backup but kind of an Delta backup.

A

So we saved a space using this process and this uh speeded up of backup process, because now we could do it, we had we have to copy only five percent or ten percent of data for each incremental backup and we have a multiple backup. Copies can be capped and All. Copy can act as a full backup in itself. So Cassandra 4.1 has introduced a feature which can be used to enhance this feature, which is a global, unique identifiers. So we can have uh like earlier.

A

The access tables were incremented in their uh uh uh Wave by the Cassandra so, but uh that was, if you terminate the truncate the table, the numbers will be generation. Id will be again uh repopulated, so that may create a problem. So now you can use a enable, unique identifiers in Access table, so that can help in basically uh uh overcoming this problem that you will always have a unique access table. Identifiers for your Cassandra access tables.

B

A

Feature can also be used.

B

Unfortunately, it is a lightning talk, so we do have to cut you off there, but no thanks so much for covering that it was really.

A

B

To hear about um the uh the incremental backup process there, so thanks so much for presenting uh and and covering that topic.