Apache Cassandra Apache Cassandra World Party | April 2021, 28 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Moving from Elastic to Cassandra | Apache Cassandra World Party 2021

Description

Presentation from the first Cassandra World Party on April 28, 2021. Learn more about Apache Cassandra at https://cassandra.apache.org/.

Charles Herring of WitFoo presents “Moving from Elastic to Cassandra” on how earlier versions of their cybersecurity operations platform were built on Elastic and in 2019, WitFoo moved the platform to Cassandra 3.0. Charles will cover why they made the move, the lessons learned and the benefits found in Cassandra. Additionally, he will share the testing they performed on 4.0.

About the Presenter
Charles' career in cybersecurity and data research spans 20 years. He started his career while on active duty in the US Navy as the Network Security Officer for the US Naval Postgraduate School. Charles served as a product reviewer for the InfoWorld Test Center, before performing 7 years of public sector consulting work. From 2012 to 2016, Charles worked as a Consulting Security Architect at Lancope and Cisco before founding WitFoo in 2016 to research approaches to mature the craft of cybersecurity operations (SecOps.)

Apache Cassandra is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Get started: https://cassandra.apache.org/quickstart/

The Apache Cassandra World Party 4.0 was a virtual, one-day celebration on Wednesday, April 28, 2021 in preparation for the launch of 4.0, the most stable database release ever shipped.

Join the Cassandra community: https://cassandra.apache.org/community/

#ApacheCassandra #database #NoSQL

A

The next uh presenter that we have uh is charles herring, uh charles charles's career, in cyber security and data research spans 20 years. uh He started his career while on active duty uh in the u.s navy as a network security officer, um charles served as a product reviewer for info world test center.

A

um So I guess, has a lot of experience, breaking new things um before performing seven years of public sector consulting work uh and then from 2012 uh worked in a consulting security as a consulting security architect at lancope and cisco, before founding whitfu, in 2016 to research approaches to mature the craft of cyber security operations.

A

um So today, child's talk will be about moving from elastic to cassandra. I'm super excited to see this because you know sometimes the venn diagram of those use cases doesn't necessarily overlap, um but cassandra does have some really awesome. Awesome things that you can actually use uh to power some of those elastic search workloads. So I'm really excited about seeing what uh what you learnt on this journey uh and with that charles over to you, appreciate.

B

It ben well yeah a lot of the research and uh doing cyber security. Work is figuring out how to deal with all the data, and so we've been through six full platforms and through that painful journey realized that cassandra is definitely, as raul pointed out. The last the last stop on the train track.

B

um The product that we work on is called with through precinct. We deploy it in on premise with our customers. We deploy it in public cloud, private cloud, hybrid of those things. We have a number of technological hurdles to get through, including those deployment options, the different disk configurations that they may have some solid states and some slow stuff. I think some of our customers run on old school uh five and a quarter arrays somehow, so um I'm gonna wait for a new slide by having there we go boom.

B

So basically what we do we take in a lot of data we process it. We provide some intelligence around uh cyber security operations, the business, what you can do to protect and respond to cyber security issues, which requires us taking in uh roughly about a petabyte per deployment per day. In most of our deployments, the analytics of our platform combined approaches from law enforcement, so how we can textualize and relationalize data the way a detective would do it. We use approaches from natural language processing to understand the data.

B

We also have some business constraints on it needs to work when we turn it on. So we can't turn a whole lot of levers depending on the different deployment pieces, um and we need to give sas level monitoring, even though we don't have the software running inside of our data center. We can't control that. So it's a lot of things for us to solve, for, as or as I like to say, a lot of things on the table that we draw a big circle around and say: oh, we need to.

B

We need to address those things so in.

A

The earlier versions.

B

Of of our product precinct, we built on elasticsearch, but we ran into some deal breaking technological problems. It was non-linear in scale, meaning added a note on we added on n number of nodes. We didn't get a number of scale back. The memory became increasingly difficult for us to manage, given the different configurations.

B

Iops disk iops is, uh uh as we've heard a couple of times today was also a killer and then uh index hot spots or what we call uh wide partitions.

B

When we looked at cassandra, some of the out of the box stuff, we got that we didn't even realize was we got data center awareness that was pretty cool memory, containment and the way that cassandra works, load, balancing from the data ingestion and throughout our pipelines, and you know what ultimately came a problem in elastic was a bad license right with elastic, changing it so being an apache foundation.

B

Project gives us a lot of confidence that we're not going to run into legal issues supporting our customers down the road. We did decide to lean into cassandra's way, not try to do cassandra elastic, but how would we do something cassandra's way? How do we make it happy and so first start with replication factor three to give us some flexibility in a different consistency, form queries and keep the schema simple.

B

Keep it down to a json blob, a partition key with a primary key for allowing us to do table scans, partition, fetches or row fetches when we needed them and be nice to the merkle trees.

B

A lot of the work that we wanted to do was I get into data ingestion, making our decisions as we're receiving the messages. How many rows do we put in each partition? How many we want.

A

B

Index every row and put that into a different json object and then do all of our analysis before persistence so that we're writing and avoiding wide partition or um hot spots. In that, as we started. Moving into.

A

Cassandra ford testing.

B

We found uh we also did java 8 to java 11 testing, which showed us g1 or garba's. First collection was really good. It was much better than working with old cms as far as memory memory and cpu, and it was pretty painless. The only issues we really had was um dealing with uh key certificates with getting to dips compliant encryption and replication across them.

B

um There is a blog post and appreciate putting that up, patrick the we did some testing on beta four, so we will do rc one here shortly, but read writes were about the same. Using our approaches a little bit faster with cassandra iv repairs were much faster, garbage 56, faster garbage collection was 34 faster and compaction was 52 faster. So a lot of great improvements in how we're maintaining these petabytes of data coming in each day. So I'm done there, you go. Let's win something back to you. Ben.

A

Thanks so much charles uh and I'm gonna pass it very quickly over to you melissa, but um hey can, I just say, love seeing those real world 4.0 stats like I, I absolutely love it um and it's great to see those real world um experiences backing up a lot of the benchmarking and a lot of the experiences we've also had uh internally for for my company. So um super super good news. Super excited to see that stuff come up on the screen.