Apache Cassandra Cassandra Community Webinar Series, 19 Mar 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Avoiding anti-patterns: How to stay in love with Cassandra

Description

You wouldn't try and use a saw to hammer a nail so why do people try to use databases as queues? Or perhaps more subtly, rely on atomic updates to mutable data distributed across the world (when you say it out loud is sounds a bit ridiculous right?).

Misusing technology isn't isolated to the distributed database world. I used to work on a small piece of queueing software called WebSphereMQ and what do you know? A good portion of our customers used it as a database!
In this talk we're going to go through the most common misuses of Apache Cassandra and why they’re not a good idea.
Topics we'll cover:
• Joins
• Mutable data / read then write / breaking the laws of physics
• Really wide rows
• Batches
• Multi partition queries
• Secondary indexes under the covers
• Tripping over tombstones
• Crazy retry policies

A

So today we're going to be talking about the various ways in which you can misuse. Cassandra are not get the scalable performance that you're after and I've probably done most of these and then had to redo things. In my experience with Cassandra, especially from moving from a a relational background and a quick introduction, though Christian did it well yeah, so I work as a Technical Evangelist, the things I'm going to speak about today, I've inadvertently done at Sky I spend my time doing. Talks and building example.

A

Apps I also maintain a testing framework for Cassandra, which is how I keep my coding tools coding, skills sharp, so its current days will be we'll be getting to questions.

A

But if we don't get to your question, then I will write up a blog at the end and try not to them all or, if you think, of questions down the line then feel free to ping me on Twitter for more than happy to answer questions later on, but here's the list- okay, not going to read them out, but one thing to notice is most of the things that we're going to speak about.

A

Today are actually Cassandra features so batches, and you know the driver, retry policies on tombstones and secondary indices, so I'm not going to say that the use of them in tight is anti funds they're, just the features of Cassandra that most people end up making mistakes with I'm using them slightly incorrectly, not getting the kind of scalable performance that they move to Cassandra for, but we're going to start off with joins because I when I first moved and started to use Cassandra as a distributed database.

A

The first thing, I noticed, was hey: wetware joins where're, guez group buying aggregates and there's a very good reason why Cassandra doesn't support these features. So when I, when I was trying to build systems with Cassandra I'm after two things and a scalability and performance, and what, when you combine those two things, what it essentially means is that if I start off with a small application, say three nodes of Cassandra and a few instances of my application and I'm getting ten millisecond response times.

A

I want to build the system which, as I scale it up to the tens of hundreds. If not, thousands of gnomes I still want to be able to get those 10 second release 10 millisecond response times. That's what I mean when I say scalable performance, so there are certain say, lack of features. You could say in Cassandra that one second sorry so there are certain features missing in Cassandra because you can never implement them scale early and one of those is joins and others aggregates.

A

That's because if we ever tried to make us Andra do anything which scan the cluster, then that might work for a small cluster, but never for a large cluster. However, what people do when they move to Cassandra? Is they say? Okay, Cassandra won't. Do the join for me so I'll just do it myself inside my application. I've definitely done this one.

A

So I tend to be able to explain things better and learn things more easily when I do it via example, rather than just theory, so I threw out all of the empty patterns that we're going to speak about today. We're going to use this this common example and it's it's one that a very similar system to what I build the sky. So essentially, what we're going to do is we want to store everything our customers do?

A

We want to store if they log in to our site, or they log in to one of our mobile applications or perhaps have actually come in to a physical store. We want to store what staff members are associated with that event. You know you can imagine how this can be used. They can be used if a call center that if a customer rings up and complained or it could be used for analytics to see what their, what a customer's done.

A

Historically, okay- and our assumption is that you know as we build and get more and more customers, and they do more and more things we're going to have to scale this Vista we're going to have to add more nodes to our Cassandra cluster. Okay, and if we try to model up, you know, quite traditionally, we probably end up with a couple of tables. Okay, you can see two distinct entities there.

A

We can see events, but we can also see the store you know and by the store I mean, did they come by our mobile app or our website or our physical store, um and you might think oh I want to model it like this I'm used to modeling like this.

A

But what we're going to do is try and work out what will happen in Cassandra if we do it this way and I quite like to explain things with animation, so I'm going to assume that you know a bit about Cassandra, you know about replication and consistency. Okay. So when you come to read this data, let's pretend we have an eight note book under cluster, so quite a modest cluster, and we have a replication factor of three. What we're going to do when we read this, we want to get an event out here.

A

So if suit assuming we're using one of the deficit of data stack drivers, it's going to speak to a coordinator, you're, going to specify a consistency. When you execute this query, we'll say it's quorum, I'm, assuming that this the coordinator, wasn't a replica for this data. We'd end up having to speak to an extra couple of nodes. Okay to meet that consistency, so we need to speak to two out of those okay and then, obviously the coordinator will respond back to the driver when that's complete.

A

But of course, because we've we tried to normalize our data. We need to go for Cassandra again. So what's going to happen here is we're going to have to issue a second query now chances are, the driver is going to speak to a different cause and they took it a bit low balances around them and if we're executing this a quorum again, then we're going to have to speak to yet another two nodes and what you'll see here is.

A

If you start, if you actually try to do client side joins, then you end up needing a vast number of your nodes to be up and responsive to. Actually, you know achieve one user transaction. So, apart from the fact that you're running latency, because you're going to the cluster price, you might think this is okay, because Cassandra is probably quicker than some of the databases that you've come from.

A

But still, if we're, if we're, this is six out of eight nodes- and this is actually probably the best case for client side joins, because it's a wife, a FISA warrant or more nothing. We had one customer event referring to one store. So that means that we only needs to do two queries. But what normally happens? If you continue to model your data like this and do crime side joins if you fall into um sorry, a more general anti pattern which I gratifies multi partition queries.

A

So you end up doing what some people call multi gets. So imagine that if one customer arm event you know master multiple staff, members or multiple stores, and that's that's where we're going to end up doing multi partition queries which are really going to hurt performance and they're really going to hurt availability in our applications.

A

So let's say that we had a new requirement come in for our customer event and we want to associate multiple staff members so here obtains the table so that actually it's a steps of staff members are associated with this event, and we wanted about some of the staff members say their job title. You know jokingly their favorite color. um What type of query is: is this going to lead to it's going to lead to queries that look like this?

A

So after we get back a customer event, we're going to end up taking doing a multi partition query to get the information about all the staff members is now sorry. Did someone say something nope, okay, so what we're end up going to do is this? Is this multi partition query and these are really bad for a few reasons, one is because we have to speak to so many nodes again so very similar to the client side join. But the other thing is, if more part of this query failed.

A

Let's say that we had stopped member 1. Actually, the node was down always slow or overloaded, but we actually got the information back for staff member 2, 3 & 4 we're going to have to retry that entire query.

A

Ok- and that means that we're going to you know, while things are going badly whence perhaps crews are slow, we're actually going to overload our cluster even more by retrying, large queries and what we're actually going to do is circumvent the the token aware load, balancing policy in the modern drivers, that's where it tries to communicate with a node which is a replica for the partition which you're trying to either read or write.

A

Soon, as we do these type of multi partition queries, the driver has no chunks, it might be able to go to the partition for, say, stuff. Member 1. That's going to be a different replica to say, staff, member 2, 3, & 4.

A

So, rather than look at animation for this we're going to start looking at some trace and a tool which I find absolutely invaluable for doing, data modeling locally is the Cassandra cluster manager and what it is. If you haven't used, it is it's a way to spin up small clusters on your single development shrink.

A

So, rather than like you know, having a single node, Cassandra cluster, it enables you to say spit up a three node or a six mode and it clips down the memory usage and the number of threads, so it won't be as resource intensive. If you haven't come across CCM before then, you can go to the website. It's on. It's on get up. I can send out the link after after this, but there's also a there's a course in the day stocks Academy about installing and configuring Cassandra, and it has a whole video about CCM.

A

So you can go and learn about it there as well. So what I've done for the example is I've spun up the six node cluster locally. Okay, I wouldn't normally run this larger cluster I'd normally stick to a CCM, the Vino cluster for most of my development purposes, and what we're going to do is we're going to insert some data we're going to look at arm we're going to insert some staff members.

A

So we have three Technical Evangelist here you know: we've got CH, Beatty, Luke and John, and we're going to have a look, a quick look at the trace, but if we did the query select star where name is in so this is a multi partition, query and we're going to go to three partitions. So this is what would happen if we associated these three staff members with a a customer event.

A

Now I'm not expecting you to read the trench, don't worry what I mean. What a highlight here is the column with the IPS of the note, so the one on the middle column and what you'll notice here that I did this query our consistency level, one I originally did it at quorum because I wanted to keep it as close to the animations as possible. But they have you. Do this query a quorum? Then there is no chance of me ever fitting this um trace on a slide.

A

But what I'll note what you can note here, we're down to a six node cluster here, but to do this one query: we needed four nodes to be up and responsive, so you can see how likely it is that this query can fail, and we've already talked about how we trying that query is expensive, whereas if we did this is you know, even if we did want to query like this, we should be doing a small individual queries, so we can retry them.

A

However, the best solution is cost to denormalize, and if you haven't gotten to Cassandra 2.1 year, then you're in for a treat because user-defined types in cassandra, 2.1 I. Think for me you make me happy about denormalizing. You know before user-defined types I might have had some concern about adding column after column after column, tomorrow um tables and then duplicating, but now I pretty much can see. You know when I do my data modeling is when ordinarily I would have a different entity, say, store or stuff um rather than create a separate table.

A

What I would like to what I like to do is create user-defined type, and here we can use them directly inside the customer events table, and this is really powerful, because now, when we query our customer events, all of the information about store and stuff are all going to be stored. Sequentially on disk and we're going to be able to read it very quickly again we're not going to have to go to multiple nodes in the cluster or we don't have to go to multiple nodes to recheck and fiscal tee.

A

So before we move on from clients like joins and a multi partition, queries, there's I'm going to show one slightly more subtle example: okay, so if you've ever done any time series data modeling with Cassandra you've no doubt come across the concept of bucketing. um If you think back to our data model that we've been looking at our customer events table, you know, you'd, probably notice that the customer ID is the partition key and thus all of the events for a particular customer end up in the same partition and end up on the same note.

A

Now, eventually, this could get too big. So if our customers are very busy and they managed to do hundreds of millions of events and stay with us as customers for a long time, eventually we'll want to spread their data across multiple partitions. So what we typically do, there is add a time bucket and if you look at the the primary key is the important bit here. What we've done is we've added something to the partition key. So it's not just the customer ID.

A

Now it's also the time bucket and we could make this the day of year or the month or the year, and this basically means that we would move a lot and we'd start using different partitions, but the problem that you can actually fall into here is: if you make your partition a your time bucket too granular. Let's say you really wants to future-proof your model. Let's say you made it.

A

Second, what you can end up doing is forcing yourself when, following a good pattern in Cassandra to actually follow an aunty fun, because if you make it seconds and I solid a little bit overly naive, you know I, think more! You make it. You know minutes or we scored or year, but if you did make it too small, it's very easy to fall into this kind of multi-part query, even when you are following, like the best principles, the best way to model your data in Cassandra.

A

So here we will be falling into the same problem as our staff member example, and just to say that, even if you end up doing this and you keep this down to a good number say two or three partitions, you really want to probably want to consider and benchmark executing these. As you know, parallel asynchronous queries, rather than you know doing it all in one big query. Just for the reasons we've already mentioned, you know so that a retrying is less of an issue and obviously you get the benefit of using multiple coordinators.

A

So that's that's it for client side joins and multi partition. Queries so the kind of takeaways are, you know we need to be normalized. We need to put things into user defined types rather than separate tables and what I find is like, and this is going to be, a trend throughout this entire webinar is I'm going to show trace because pretty much if you're, using a multi, node cluster and you're, constantly looking at race you'll be able to identify.

A

When you know your queries are using too many nodes and that's where you can predict that these queries won't scale.

A

Okay, so that's the next thing we're going to talk about is batches. Batches are probably one of the most misused feature of Cassandra I misunderstood, because when I came to Cassandra I was used to using batches in relational databases, and essentially they were pretty much always a performance game. You know, I can group together unfoldings in the same transaction and think that you get much far higher throughput. However, that's certainly not always the case with Cassandra they.

A

They sometimes have a a use case for performance, but most of the time it's different, so it was splitting it into two sections: we're going to talk about unload batches. First, then we're going to talk about log batches and the whole idea of this is is not to say, don't use them is to make you understand how they work under the covers. So you can make make an educated. You know decision about whether they're going to fit your use case. Okay, so what are unlock batches for so very similar to a relational database?

A

It enables us to send multiple statements to the Cassandra server server in one go now. That probably sounds like a great idea. You can amortized the cost of you, know the overhead of talking to Cassandra and send everything in one go and we're going to see how that's going to look like you know, for our customer events, type of example, and we're going to look at some examples using the data structure of a driver.

A

Don't worry if you, if you don't come from a Java backgrounds, the code is very similar and in most of the other Cassandra drivers, especially the the ones produced by date stacks.

A

So we just start off with some code. That looks a lot like this. So assuming we've prepared the statement in green at the top of the slide here then- and we've saved that prepared statement as insert statement the the text in purple, then each time we get a customer event, we combined all the parameters. Now. My assumption here is are all stored in this Java object. Customer event are we, then we can set the consistency level and then we can execute that statement. But of course you know with performance in mind.

A

You might think yeah I'm going to put these in a bunch and increase my performance, so the code doesn't change much. What we've done to the method signature is: we've actually started. Storing we've actually started taking in a variable number of customer events. So if you don't know Java, those three periods means that we can pass in many of them to send to see an array of customer events and what we do is we create four back statements. You know type and logs we loop proms, we add all the events to a box and then.

A

Okay, so, under the covers it's going to send all of the inserts.

A

Which shows us some feedback? I'll start this one again. So what under the covers? This is going to send all of the insert.

B

Cassandra, hey grace, wick we're getting an awful lot of feedback. You dialed in by your phone or via.

A

The computer I'm dialed in by the computer, okay.

B

We're getting an awful lot of feedback, so I don't know if it's your microphone does.

A

That just started illiterate and I just started hearing it. Yeah.

B

It's just started see.

A

B

Have cleared players now, so I won't move.

A

Okay, so what we're going to try and do is understand whether this is a good idea and I'm going to do it with the example of inserting eight records into an eight node cluster and we're going to assume that they're evenly distributed around the cluster. We're not going to worry about replication, because the diet, the animation, would get a bit too busy, so we'll just assume a replication factor of one.

A

So what the drive is going to do for this unload batch, it's going to select a coordinator, it's going to send them all in one go and then it's a coordinators, responsibility to break that batch up and then start sending it out to all of the nodes in the cluster. You know, depending on whether the replicas are now. There are some reasons why this is not a good thing.

A

You know this might be a good thing when you're running a single node database, okay, but what we've essentially done is circumvented some of the cleverness inside the driver. So the first thing is look at how many nodes in this example need to be up and responding for this entire query. To succeed, you know, if you do big enough matches or you'll essentially be doing, is making sure that every node in your Cassandra cluster have to be up for you to get for this to be successful.

A

Okay, so that's the first disadvantage, because I mean essentially means that if any, if only a few nodes in our cluster are overloaded, you know queries will start failing. The second problem is the fact look at how much work the coordinators got to do if we start sending these big batches. What's going to happen is that the coordinator has to keep track of all of these messages it sent out and when they acknowledge that that rights been accepted, so we've got quite an unbalanced customer of node here, which is the coordinator.

A

If you use batches wrongly or often see nodes in a cluster having much higher, say, CPU usage, okay, the other problem, is it completely ruins the performance of the token aware driver, so the modern data sucks drivers, don't send it to a random coordinator anymore, or they don't do say a round-robin policy. The default is to send it to no witch actually owns the data.

A

So we look see what this looks like if we were to not do a nun one batch, but we were to just a synchronously: do all of these eight insert separately what it would look like this?

A

The client would be sending it directly to the crack nerves and we go round and then would be done, and this is this is so much better because we're taking advantage of the token aware policy, our cluster is evenly balanced, we're not overloading particular nodes because they're coordinators for large batches and if a single one of these inserts fails, we get to retry an eighth of the work. Okay, whereas previously we'd have had to retry the entire batch you're, probably thinking. Why would you ever use them?

A

But there is a use case and you may get a performance gain if you do, and the use case is this, so if you need to insert a lot of rooms for the stained partition, so if you see what have we changed here, this probe looks very similar, but it's actually that there's a little difference which makes a huge difference, and this is the cut the fat that we're only going to store I'm already going to batch things together, but have the same customer ID.

A

Now, if you remember the customer ID with the partition key, so here what we've done here for building a batch, the things for the same customer and then with you know, apart from that, it's virtually the same code, we're looping around them, adding them to the batch and then we're executing them I'm, just to finish a section off with all animation. It's going to be as simple as this. You know with the token aware policy it's going to go directly to a coordinator. That is a replica for that partition.

A

It's going to store it and then it's going to respond. So you can suddenly see how suddenly you do get performance gain from this.

A

Ok, so that was unlocked batches, but logged batches, which were introduced in Cassandra 2.1 have virtually the same empty pattern, but it put quite a different use case.

A

So if you haven't used these before some people refer to them as atomic batches I prefer log batches okay, because in a distributed system it it's very hard to achieve any form of acid transactions or otama city, because you're trying to get nodes that have a or computers to agree on something which might be as a result of oh but Cassandra does have a feature which can help you up and that's what Baptist?

A

That's basically saying that you need to keep some tables in sync or you've done or malaises, then you need to make sure they don't get out of think somehow, then you can Cassandra can actually put extra effort into making sure that everything in the buff succeeds. um This is achieved and we'll show this for an animation again by first distributing the batch to a set of nodes, called the backlog and before actually writing them. But of course it adds a lot of overhead.

A

These are, there is somewhat slower if you're constantly benchmarking and you move from unlogged to log batches you're going to see a performance set. The code is virtually the same. So you know we apart from the the type of batch, so here we're just we remove that.

B

A

And we've just gone to you know any any number of customer events and we're going to say some type log and then the rest of the trade is the same. Let's see what's actually happening under the covers to see whether we actually want to be doing this so because we're going to go with eight say tripes again evenly distributed same as the first example I know our drivers going to select a coordinator, it's going to send all eight there now before the coordinator begins to write the first thing it does.

A

Is it collects a couple of nodes in the same data center, but ideally on different racks and it, and it sends the batch to them these aren't readable. Yet these aren't this doesn't start it's going through the normal right process. Okay, so once it's got it onto the batch log, then it's going to actually attempt to do the right. So then, we'll start sending the right out just as before now I haven't got this in the animation, but imagine that the coordinator would have fail right now. The big advantages of log batches are.

A

If the coordinator fails, one of the other backlog replicas will take over and finish it off. So that's the use case, but if you try to use these for a performance game, then you're going to see we're actually doing vastly more work than unmod batches. Okay, so we're going to even see worse performance and, of course the use case. Is this it's not for inserting lots and lots of rows in the same in the same batch. It's all really want to keep two tables and sync so, with our with the table.

A

The customer events table that we've been working with we're only able to query this by customer ID and then filter it by time. But if we have the requirement to want to be able to create by say stuff ID as well, then we store it twice.

A

We'd be normalize, and if you had a requirement arm that you needed to keep these in sync, then what you could do is create two preferred statements, one that inserted by customer ID another that inserts it by stuff ID and put this these two things inside a logue batch and then what Cassandra do is as really attempt to get both of these written, even if there's failures in the coordinators etc.

A

But you've always got to consider whether it is worth it because is it better to be slightly out of sync or have to have some recovery mechanism? If and the odd query fails, but to take this by 30% performance increase, that's up the decision up to you and you should think you should benchmark both.

A

So that that's it for batches, hopefully your think very carefully before you decide to use a batch again. Okay, not they've, got their use cases and they're really good features, but you know they're often misused, because people assume they're going to get performance increases out of them. So the next thing we're going to talk about is mutable data and distribution systems.

A

So apart from denormalization and schema migration, things like that questions are um you know the consistency of mutable data are probably what I get asked the most okay and it's a pretty hard problem and we're going to go through a pretty naive example to show you just how bad it can be if you do this incorrectly.

A

So if you're running a multi by if you running a single node database, so a relational database or perhaps you're running a muster slave system or even a master slave sharding system, now writes all go through a single point in these systems, which gives you you know, good consistency. It also gives you a single point of failure and a single point of a single point of say: your bottleneck. um Cassandra doesn't work that way.

A

You know you can issue, writes that any node in the cluster, but if, if you then decide to model your data, so you have mutable state, then you're going to run into some problems and most of the examples you'll find online where people say hey Cassandra loses data, it's normally an example. That's nearly as naive as this.

A

So what's going to happen here, if we were going to store account information, so people's balances, what we have to do it looks like luke, some Luke's overdrawn chris, has got loads of money and Patrick's not doing so, but okay. But if we were going to model our data like this inside Cassandra we'd have to read this and then write it back and say: add 50, pound or you know, I take money away, and that's that's really not a good idea.

A

Doing any read before or right in Cassandra normally means you're falling in to ante up. Okay, because Cassandra uses last right wins if you've read the dynamo paper or you've used like riot. You probably come across some vector clocks and in key value stores. Vector clocks are very important because you're very likely to have a conflict, but in a wide column, database like Cassandra in what you generally want to do, is just not what you not model it. So so you have mutable. There are mistakes and let's see how we do this now.

A

The reason I can't continue to use the customer event example because it was already modeled immutably. There was no mutable state there. We constantly append in new event, and we can do exactly the same thing inside this accounts example. So what we can do here is rather than store a mutable variable. You know, which can be just as much of a problem on a single computer. What we're going to do is store the event pennis we're going to log the deltas.

A

So here we have a partition key of customer and we have a clustering column of time and we've used the brilliant type, which is a time UID. So here as we are deltas to our accounts, we're never actually going to update a column and Cassandra is not his lust right wins on the column. So it's not on the row. It's not even on the partition, it's every single column and with this later model we're never actually abating a column. Now.

A

The complexity that we have to deal with here is the fact that we, it becomes more complex to read this data, because what we're going to have to do is calculate the balance when we read it, however, we're going to be storing all of these deltas on order on disk and doing a sequential read the read from Cassandra still going to be quick, but eventually, as the number of events gets very large, then of course it.

A

You know it's going to slow down to read still more things from events coughing, where perhaps at the end of each day or week or month, depending on the frequency of your data, you would snapshot this and roll it up. So you'd only need to use this pattern of arm teh log, structured data storage on the like the very recent mutable data when you're getting new events, so quite a small section, but quite quite a different mindset to deal with when you, when you store data and distributed databases without say a master or a single-point.

A

Okay, so you're always you're, always looking to avoid updating mutable data. Now cassandra does have other features, which means that you can get away with mutable data, one of which is lightweight transactions or, as I, prefer to call them compare-and-swap.

A

That's where you can actually read a value, stay abundant balance or a number of tickets sold, and then you can update it, but only update it if the volume hasn't changed. Okay, but the thing is, if you, if you think how that's going to be implemented under the covers Cassandra suddenly has to then, rather than just you know, append it on to it some into the commit log and go just go to a mem table.

A

It's going to have to coordinate amongst all the replicas okay and it's going to be much slower in the order of it has to do around for round trips between all of the replicas for your data and that's in Jung contended case. So in general, lightweight transactions are really good. If it me really, if you're going to hit that part of your application frequently. But if you want that scalable performance, you probably want to move over to a more immutable data model.

A

Okay, so next time you pattern so this time we're going to talk about crazy retry policies or what they normally boil down to is understanding failures properly.

A

So failures in Cassandra are quite different to failures in the acid relational databases and by fallen foul of this a number of times I, actually combined a couple of the empty buttons. I am Multi partition, query, which meant that my queries failed more often and then I didn't quite understand how failures worked when I started with Cassandra, so I then had a what I've plotted by now is a crazy retry policy. So I ended up doing the double whammy of ante buttons, but let's try and avoid doing this in the future.

A

So we're only going to talk about rights just because they're they need taking ones they're the dangerous ones. What we're going to try and do is really understand what I write failure really means in Cassandra, so we're going to assume that we're going to write some data, we're going to we're going, what we're going to write this a quorum. So the first misconception we often get with Kassandra is that your consistency affects how Cassandra writes data, which is not the case.

A

Regardless of what consistency you write the data, it's going to send it to all three of the replicas okay and some of those replicas might respond in time. So because we've got a replication factor of three and we're going to write this a quorum, then we want at least two of these replicas to come back to us and say yep I've written the data. If this doesn't happen, if say, replica threes come back to us, but replicas 1, & 2 would have timed out. Then what the coordinator is going to do.

A

Is it's going to send a right timeout back to the application? Okay. So the really important thing to note here is Cassandra. Does not roll back? Okay, like you, might expect if a transaction fails in a relational database, so the situation we're in here is that replica 3 has the data. Okay, it is store, replicas, 1, & 2, and we don't know because they didn't respond to us.

A

The coordinator, if you've got hits, enabled, will have kept the hints and will you know if they do go down and come back up, we'll get the hint when it comes up, but what I accidentally say did when I first started working with Cassandra when just a blind retry policy in ion or a never move to say a downgrading retry policy, which is in some of the drivers when you think about it.

A

If we written this at quorum and we've actually got it onto one of the notes, there's not much point we're trying that right at one, because it's already written to one and if you're, using a data structure driver the drop, is actually going to tell you this. So when you get a write, failure back all right failures are not equal. Some of them that they're all going to prove this information. It's going to include how many acknowledgments it got, how many it required based on your consistency level.

A

So you actually know you could already know you have a clever, a retry policy to say actually, if I did get on to one or two notes, I don't need to a try. I only actually need to retry the rights. If I got it, I got knowledge once it becomes slightly more complicated when we start working with lightweight transactions or compare and set and baptists.

A

And hopefully we really understand how batches working out, because we've spent quite some time on batches, but essentially what Cassandra and what the the data structure drivers will tell you is which part of the batch file. So, if you remember back to the animation, if we're doing a log batch and we failed to get it out to the you know, we failed to get confirmation that, in fact the Box log replica has worked. Okay will bit will get a write failure type of backlog.

A

So we know that we should probably retry, because we don't know that the bug replicas got it and are going to be retrying it. In the background, however, if we got past that phase, we put it on to the backlog, replicas and we're going to get a right type. Failure of batch, which essentially means that it's very unlikely that you're really going to need to a try. You'll have three nodes in the Cassandra cluster constantly trying to apply this. For you, the only time I could possibly think that you might want to retry.

A

This is if you, if that no desperately needs to read its own rights, and you don't mind taking a performance hit for unlock batches you're very much in the same same cases and just normal wrappers.

A

One last thing to talk about before we move on is idempotency, so there are certain types of rights in Cassandra that aren't idempotent and they are in the minority. So virtually every writing. Cassandra is idempotent. It's why the drivers can retry etcetera.

A

However, there are two cases when they're not and need to be very careful about doing retries or even using these. These particular features of Cassandra they're, very clean of critical, where it has to be exactly right, so one is counters, serve you. If you retry in increments, it's obviously going to increment it twice. The other is list.

A

So, if you're trying to append something to a list, if you accidently retry that when you shouldn't do you're going to end up in the list twice, so you shouldn't be using um counters for exact, like 20 of the accuracy, is of own prime importance, and you should always prefer to you steps advisor and be aware that they accept in Kassandra are ordered. So you don't need to worry about that, so run for the next one.

A

So Cassandra is a coup which I always I, find quite a fun funny, because people try and do it a lot. I've tried to do it and when I used to work on queueing software, people try to use it as a database all the time, and but it's a more general answer person that you don't have to be thinking. I'm using Cassandra Z cube to fall into this. What you could be thinking is at.

B

A

Let's use delete a lot in a partition and then do slight queries across that partition. You can fall into this this auntie button. This is a. It is really quite well documented one time I'm not going to spend much time on it and there's a couple of blocks here which he just go and read, but for this auntie pattern, I didn't create my own content. I actually took it from the top one. The data stacks text block I figured I work for the attack times. I was allowed to do that and there wasn't much point.

A

You know, spending time generating my own content trip. One of the reasons I'm still talking about those notes quite well documented, is even with some experience.

A

I still tried to do it at one point: didn't get anywhere in their production, but at one point, I had an application and it was running in two datacenters and then I had another datacenter which didn't have the application, but I wanted to send messages from between the data, centers and obviously I was thinking, should I just do something synchronous over HTTP or should I bring in some queuing software I thought I already have this technology, which is which is brilliant cross data center replication.

A

Perhaps I should just expand that Cassandra cluster out to the new data center, there's a reason why this is a particularly bad idea, and it's because of the way that deletes work in Cassandra. So how do you know? Cassandra doesn't read them right? Okay, so if you're going to do a delete, we don't go into the mem table or the SS tables on disk and trying and try and delete it. What we actually do is for market, as they, this column is deleted.

A

So when you come to read it, it sees that the what's called the tombstone with a later time stamp than the last value. So it knows it's deleted. The bad part of tombstones is, if you have a build up of them because they are kept around for some time, then you can spend more time scanning over the tombstones. Then you can actually reading your reading your actual data- and this becomes very apparent if you have a data model which looks like queuing.

A

So let's have a look at. Let's have a look at how that with them, I love to have a lot of being feature of. So imagine we have a cute accused table. You know we have a name of a queue and that's the partition key and we also have a in CUDA, which is our time UID, so they're going to be ordered on disk in a FIFO way, but first-in first-out and, of course, we're always going to read from the back, because we want the oldest message.

A

Okay, once we've read the message we're going to delete it. So that's a very few like data structure.

A

Now, even if you ignore the race conditions of reading- and then you know deleting and that type of thing you have to realize that what you're going to do is be building a large number of tombstones at the start of this partition, and if we're doing this query this, you know this one at the bottom of the slide here, where we select from you know, name equals Q, but we limit its and we just get one of our messages back we're going to be reading over every tombstone that ever existed for that Q.

A

Now, if you know about DC great period, the default for that is 10 days, so essentially we're going to be reading through 10 days worth of tombstones for every single read. This is one of these empty buttons where it will work for awhile, but I see your application. You know, is it has been in production for a while you'll be building up these tombstones and your performance will be getting worse.

A

We're going to look at race again to see you know see this in action because, as I've said, most of these anti patents are really obvious and trace. So this time it's a it's a three node cluster running locally and it looks like we've, been swig inserted and delete nearly twenty thousand messages in our tube okay, Anna Vissi, to read one records to read one column, you know which meant one message inside our cues we've had to scan over twenty, nearly twenty thousand tombstone and not surprising, that's taken some time.

A

So that's the reason why cassandra isn't too suitable for this type of data modeling. Now there are times, of course, I'm not trying to say never use deletes what we're trying to avoid specifically are lots of deletes, and then these, like slice or range queries over them, and there are a number of ways to avoid this. Okay, so let's see do need a data model which is is similar to this. So the first thing you can do is make sure you don't remain in the same partition so rather than you know. In our previous example.

A

Staying in this zones, a partition for the queue name, what we'd actually do is we've put some kind of time bucket in that, so, if every hour or year or month, whichever, depending on the frequency of your messaging you'd, be moving partition. So we have like you, know, Q Chris arm for the last ten minutes or the last day, etc. The advantage of this is, we don't go anywhere near the old tombstones.

A

The other thing we can do is give Cassandra a big hint that if we, if we know we're reading data and time order, then we know that the next message we get is probably going to be older, so newer than the last one, and we can actually tell Cassandra about this by adding a they are, what we call it a greater than inside our query. So in the queuing example, it's the last bullet point here we can actually say in CUDA is greater than say five minutes ago.

A

So this is a time UID and that way that that gives Cassandra a hint and then it can very efficiently avoid scanning over all of the departed. All of the tombstones.

A

Okay, so we're nearing we're nearing the end, we're onto our runs on last them anti-pattern spaces, some people's favors.

A

So if you've been around Cassandra for some time and you've been on Stack Overflow or the mailing list, no doubts at some point. Someone has told you, so whatever you do use the secondary index. You probably think this up why it's not like a really cool feature.

A

So what we're going to try and do today is explain exactly how secondary index is welcome to the covers, so you can really understand why, most of the time they're going to be they're not going to be a good idea and they're, probably never ever going to give you the performance that you can get from denormalizing your data, so we'll go back to our favorite table the customer events table and we'll go back to the added requirements so before we're allowed to with this data model, we're allowed to query by customer ID, but what we really want to be able to do is also queried by stuff IDs.

A

If we try and query by stuff ID, what's going to happen, is we're going to get this error message back. You've probably seen this in a CTL SH. It says no index columns present in by columns cause with equal operator. So surely the logical thing to do because it says there's no indexed columns is to create index. That's what we do. So we actually, you know, do create index on stuff ID, but let's try and work out how Cassandra implement this under the covers and why we're not going to get particularly great.

A

So here's some data, I've cut out most of the other columns. The only ones I've left in are the partition key the clustering column, which was time on the column that were interested in indexing by okay. So we've got the stuff ID. We have a couple of partitions: we've got CH, Beatty and rusty, and a couple of staff members, so we've got Trevor and Bill and what Cassandra is going to do into the covers? It's probably not too surprising. It reversed this. For us, young economic can work out which partitions are very staff.

A

Member the Avery. Instead of going them customer ID, the index table goes from staff ID to customer ID. So we, if you query by Trevor, you can go into this secondary index table and see you know all the CH baby on rusty have records for um Trevor, so I'd better go and look at those partitions.

A

But alarm bells should already be ringing here because, as soon as you use secondary indexes, then the chances are that you're going to end up under the covers doing multi partition queries, because unless you've got a one-to-one mapping, you know you're in SP, only hards the same number of staff members as customer IDs. Now there happens to be one to one, then, when we look up by Bill we're going to end up having to follow-up that query internally, that is to look up rusty, nth baiting.

A

So we we've already kind of discussed the disadvantage of this. You know if suddenly means that query is going to be reliant on many notes and it's going to be much slower and it's going to be much bigger if we try to retry it, but that's not the only downside to secondary indexes. The other reason that they're really quite bad is the fact that they're distributed differently they're, not like a normal table.

A

So in a normal table we, you know we have the partition team we find out which node it goes, and you know assuming a reasonably even distribution. We have roughly our tables split equally across all of the nodes. Now, that's not how secondary index tables work we'll talk about.

A

Why in just a moment, but how they're really stores under the covers is on each node there's only the entries for the secondary index table that um that have that that belong to the partition on that node now I've never been very good at explaining or understanding that in words, it's always so much easier, I think with an example and with some pictures. So that's what we're going to do so we're going to assume a really simple to know.

A

Custer, a and B same data as we've been mucking up, so the same index table um we're going to assume a replication factor of one just so we can concentrate on how secondary indexes work. So it turns out that C 850 maps to node a and rusty master node B. So then we can work out which parts of the index table belong on each node, so for the partitions for the staff members that map to CH basic those bits live on no date for the part of the secondary index that are much too rusty.

A

They live on no be okay and the problem with this is if we start to issue queries against this cluster, which don't include the partition key. What cassandra has to do is it has to go. Look rump all of the segments of this secondary index table taking imagine it has to go to virtually every node in the cluster, so it's going to be extremely slow.

A

The reason it's on one of the reasons that a theory of why it's implemented like this. It's simply because, let's say they were distributed like a normal table. Imagine that we only have two staff members, but tens of millions of customers. What if we distributed that table normally, then we'd only end up with two partitions. You know say we only ever had Phil and Trevor the millions of customer events.

A

We'd only ended up having two partitions inside this index table and thus it wouldn't scale the best way to understand this is by far to see trace. Okay. So if you're really keen I got some homework for you and we might be able to send you out prize depending on where you are in the world. So what I'd like you to do is install CCM so either go to the website or go to Dex tax academy on you know, workout, to install it spit up a six node cluster.

A

Okay, um I think you can obviously take the schema from the slides, I've put them on SlideShare or tweet that up so you've got the customer events table and I want you to create the secondary index on a hidden slide which you'll you'll get. If you download the slides, um there will be some data that we've been playing with. In the example so go and insert up the next thing to what I want you to do is through these two queries, so we're going to use the index both time.

A

If you see a and B okay, so we've got stuff. I'd equals Trevor as one of the clearest and stuff I'd equals Trevor and custom ID for CH Beatty for another one, and what I'm going to do is the spike on Twitter. Just send me an explanation of like roughly how many nodes in the cluster would be used and an explanation of why? Okay, because these two queries are really different, one is going to be a few lines of trace.

A

One is going to be pages of trace, okay, and if we can the first person who sends me an answer will hopefully send out some swipe, but no guarantees. Of course, the way you would really solve this problem is the normalization and the massive reason why this is always going to perform vastly better it's as soon as we do normalize our data, we store it by stuff, I deep. You know all of that information is suddenly sequentially stored on disk. You know we're not even having to go to multiple nodes.

A

We're not even have to do multiple disk seeks to really. This is a much better way of doing it, but of course people get sick of denormalizing.

A

So there's a lot of buds around a certain zero in Cassandra a moment called global indexes, and you know Cassandra is an open source project, so you can go and contribute I, don't necessarily mean to the code, but I mean to the discussion, that's happening here and maybe put your use case down, because you know they're going to have to solve that problem of the fact that the index can't end up with two bigger partition.

A

So hopefully I've convinced you that you should really think carefully before using batches or secondary indexes, so we're coming to an end there. So we've covered quite a lot. Okay, I'm actually spoken for a bit longer than I thought for the intro is slightly longer, but we've gone through batches and joins. um Hopefully, if you are, you really understand how secondary indexes work I'm going to follow this up with a blog explaining secondary indexes equations, so hopefully you're not going to fall into some of these problems now.

A

So thanks for listening any questions, we don't get time to now. I'll stick up a blog, so we will answer the wall it just it just might take a little bit longer, most of the anti-patterns that I've been when I've been building up. This slide deck I'm a much more detailed blog post about.

A

So you can go, read in more detail about what I've been talking about again if you've got any more questions down the line, just just finger on Twitter and I would really advise you to taxi CMO and go to this actually cut of me, you're learning or about right.

A

B

A

In the very British way will keep calm and use any quest any time for questions so.

B

Chris they're right there are absolutely loading loads of questions in here. I I, don't know whether we should start and get through a couple. So you you. Let me know if you're game for a couple well.

A

Let's go for a couple and I: don't promise to be able to answer them, but if I I'll find someone who can and get back to people.

B

Okay, how about this one this is. This is one that maybe we can get through. How would I go about solving limit of set queries without having to skip rows? Programmatically, for example, taking a simple page to customer table.

A

So did you say how to do limit queries on a set.

B

Yeah, how would I go about solving limit of sets of set queries without having to skip rows? Programmatically.

A

So at the moment you can't see you can only arms query things in Cassandra which are stored in order um for the for the person who asked that question of I'll ping you another JIRA, which is user, defined functions and at that point, um you're going to be able to actually execute some code on the Cassandra side, which is going to be able to like filter parts of partition.

A

So you won't need to store the data over so stay tuned for Cassandra 3, which is sometime in the next month or two, not surely, state shut up.

B

Okay, next one and I've got time for maybe three more. There are times when data simply cannot be written simultaneously and therefore must be joined as later time. What you recommend for joining needs an external tool such as spark sequel or something.

A

Else so spark is a fantastic tool for doing that type of thing, but bear in mind it's more of an analytic, slow type of query when you're doing those type of joins, then you're not going to get these single-digit milliseconds. So if it's a one-to-one joining and then yes they're inside your application, you know rather than building a separate tool. If it's something that you've realized later down the line, then what spark can be great for is rebuilding a new table based off old tables.

A

So, rather than doing the join in your operational systems, do the join and write it back to Cassandra so that your operational application can read it exactly as it is.

B

A

Great I'll answer everyone.

B

Who's asking whether the slides will be available so within about 48 hours of the webinar we post the video.

A

B

This so you can replay it and the slides, along with it, two planets Cassandra one last question Chris and thank you so much for taking the time today in order to emulate aq without falling on this anti path and I use the new date/time compaction strategy and TTL.

A

The thing is that, if you're doing that, that certainly helps because of the date/time compaction muscley reduces the overhead of deleting old data. You know so it just drops away entire SS table trying to build like a high throughput queuing system with Cassandra. You know you're still going to be reading over a huge number of tombstones, so it's still going to it's still going to be slower. The only you can really do is like move partition really quite quickly, so you're not reading over the tombstones, but there's very good queueing software out there phone.

B

Okay, great, thank you. Thank you very much. Chris really appreciate your taking part today and we'll look forward to your blog on the questions that that are absolutely loads of them. You've probably got 50 questions to get through here, so.

A

B

Everyone we look forward to seeing you next time and keep your cassandra learning going either by joining us at a Cassandra Day. You can check out the cities that will be in on datacom. We have free online instructor, led training. Again, you can find that on dates, tax, calm and then, if you want to take some self-paced learning courses, you can visit intersex Academy for those and then loads of great content on Planet designer, and so, if you're interested in taking your learning forward, definitely have you have it thanks.

A