Apache Cassandra Apache Cassandra World Party | July 2022, 20 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cleaning Large Scale Data in Cassandra | Apache Cassandra World Party 2022

Description

Presentation from the second Cassandra World Party on July 20, 2022. Learn more about Apache Cassandra at https://cassandra.apache.org/

Rahul Xavier Singh of Anant presents “Cleaning Large Scale Data in Cassandra”
So you have tons of data in Apache Cassandra. Then you realize years later that you didn't have any TTLs on your data and now there's terabytes or petabytes of data that needs to be cleaned out. We'll cover how we used Apache Airflow, Apache Spark, and some carefully crafted DAGs to democratize data cleanup and other common data operations without giving users access to the Spark Cluster. This pattern can also be used to build other common data operations such as import / export of data to and from Apache Cassandra using a variety of tools including DSBulk, or Scylla Migrator.

About the Presenter
Rahul leads a boutique consultancy that helps clients design, build, and operate global scale platforms that serve and impact large groups of people. He's been working with Cassandra for about 7 years and has been contributing to the community by publishing cassandra.link and running a weekly "Cassandra Lunch" for almost a year. He's recently gotten involved in the Cassandra Kubernetes SIG and helps out by updating the blog with updates.

Apache Cassandra is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Linear scalability and proven fault tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Get started: https://cassandra.apache.org/quickstart/

The Apache Cassandra World Party 2022 was a virtual, one-day celebration on Wednesday, July 20, 2022 held in anticipation of the launch of 4.1

Join the Cassandra community: https://cassandra.apache.org/community/

#ApacheCassandra #database #NoSQL

A

Yeah so today I'm going to talk about uh cleaning uh data, um and you know in the big data world, there's lots of different options.

A

There's coldish, like you know, file, storage, there's warm data like data, warehouses, data lake houses and we have hot data, and when I put hot star it's like the stuff, that's on the cloud right and you know the cool thing is they all use cql, which is uh one of my main things uh that I love about cassandra is that you know you can start using cassandra, but there's so many different cassanders out there right, um and that just means that you know your skill can be applied in different scenarios.

A

um Now cleaning big data, you know: how is it different uh than cleaning um well little data or medium data? And it's basically the same. So I would say it's the same different day, but the problem is that you know you can clean data in the data engineering, meaning as the data is coming in. But what, if you don't do that uh when you get the data in right? What happens after the fact?

A

Well, you have petabytes of information to go through it in cassandra unless you've, obviously you know put it in a partition key or whatever it's going to take more effort in sql, relational databases. It's very simple! You you figure out what you want to do by you know finding what you want to delete and you delete it and you don't necessarily have to just do deletions. You could do select blah blah blah blah blah insert into this other place right or select as and do all these functions.

A

You can say insert into this other place and it would just work- and you know when I got into big data, especially cassandra, because it has cql which looks like sql. I was expecting something like that and there really isn't, but now with spark sql, you can kind of do that right. So there are certain databases or data stores that are connected to spark sql, where you can kind of do the same thing right and that's amazing.

A

If your database supports it- and you know the new york spark connectors, you can do some cool stuff, but for the most part, you're all not going to always get uh what you want with spark sql, because again it's a subset.

A

It's a you know it's hot, it's hive, so it's a subset of sql, um but if you're lucky, you can do the same things now the surefire way of cleaning data uh in big data, depending on which version of cassandra you have, which version of spark you have uh is to just have a function that you make. You know and here's an example where uh you know it's.

A

You know deleting a category of information in a big database of documents and you can even parametrize further along, but the benefit is that you have much more control over your filter criteria. You have much more control over what you would like to do once you find your information. So in this case we select using spark and we delete using cql using a cql driver session.

A

Deduping is a little bit harder. um It's it's not as intuitive. I don't take credit for writing the the middle code, which is do something with deduping. um It's just something I found on stack overflow and I got it to work and I've been using it over and over uh and it works.

A

You know- and this is the cool thing about spark, is once you get something to work, you can wrap it in a function either in a python function, scala or java, and then you can share it with people and that's great until you have to do things like migration of tons and tons of information, uh cleaning of tons and tons of information for hundreds of tables or hundreds of key spaces, and that's where airflow comes in uh airflow in this case is showing a this.

A

This diagram is showing a directly basic graph of all the things that we want to do to delete uh or move uh a big application. That has lots of information, and you know we can't just say take all these tables shove them from one place to the other. You know what, if we have other things that we'd like to do what? If we want to move a subset of that information um and what we did was we looked at this thing called spark migrator cell migrator.

A

You know somebody else has written that software and we adapted it to clean data. So we took a part of the engine ripped it apart and said. We like the way it selects information, but we're gonna also apply uh some deletion uh in here and using the same directive asycyclic graph that we made for migration of data. We started to use used to clean data, so we could say this particular set of tables in this particular set of key spaces. Let's move that information now here are some considerations right.

A

If you're going to use spark with airflow um figure out, you know if you really want to manage it or run it like for the most part when you're moving or deleting data, you don't really need to run the spark cluster forever right so like see, if you, if elastic map, produce, works or see if google data proc works for you right or data breaks works. For you same thing with airflow, there are aws airflow as a service.

A

Google composer is kind of like an airflow as a service, so you don't have to necessarily manage these things, um and you know other like key things about spark. Is that if you compile it, it can't be modified and that's really useful, uh so key takeaways right, um don't reinvent the wheel right use, apache spark like when you're dealing with cassandra and you're dealing with moving or cleaning information.

A

You can't really, you know, go wrong with using spark all right. It's like not getting fired for hiring ibm um and then, in terms of a scheduler. You have lots of different options. Apache airflow with python is the lowest common denominator, but there are other things like luigi prefact. You could even use jenkins to schedule the jobs for cleaning and migrating data, but there's so many different options available with airflow. You can't go wrong. Try it out thanks! So much guys appreciate it.

B

Thank you role. It's it's! It's good to see you um and and a fantastic talk there. I'm gonna have to put uh airflow on the top of my.

C

Practice to to take a look.

B

At um it's, it's it's pretty fantastic.

C

B

C

Great job and data.

B

C

Is definitely a kind of an overlooked art that kind of deserves a lot of attention in these circumstances, because um that that takes when I was starting with this sort of thing, it's like that took a lot of the ramp up effort is making sure that, like wait, a second, why is this? In my data? Oh, oh and you're gonna have to have that step back moment. um So thanks thanks. So much and good to see you guys sort of absolutely.