Apache Cassandra Cassandra Summit 2014, 10 Oct 2014

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Netflix: Astyanax — To Be or Not To Be

Description

Speaker: Puneet Oberai, Senior Software Engineer at Netflix

In this session, we'll cover a quick introduction to the Astyanax Java client driver, powerful features, comparison to Java Driver and what to do with CQL3.

A

uh Hello. Welcome to my talk. uh Let's get started, uh ostianics yeah.

A

uh Astionics uh to be or not to be, that is the question uh so, to give you a little idea, um um astianix, uh if you guys are new to cassandra, you may not know this. It's a pretty popular uh java client for accessing cassandra.

A

So especially if you're new to cassandra, this talk will be very helpful uh for you as a good starting point um also for the veterans. Who've seen plenty of action uh with cassandra and production, it's pretty common for them to be on thrift, uh and we're also here to talk about that, because asthenics is largely a client that uh used to work with thrift, at least in the past, as in only with rift. uh So there have been some changes, so this will also be very useful for you um all right.

A

Let's get started, okay, so a little bit about me, um I'm punit! I work at netflix at the cloud data engineering group. uh This is the team that runs cassandra for netflix in the aws cloud.

A

I hope you got a chance to attend talks of two of my other colleagues earlier today regarding building the big foundation at netflix and also like uh the crazy active active thing that we do for resiliency uh at global scale.

A

um So yeah cnx is pretty core to that, and uh I am not the author of asthenics that credit goes to iran landau at netflix. He's awesome, uh I'm currently the custodian of osteonyx and I've made some pretty uh big contributions to asianx in the recent past.

A

That will really help understand the story between thrift and cql java driver etc, which is why I'm here to talk to you about that? um I know yeah.

A

Prior to this, I worked at amazon web services couple of distributed systems there funny enough sankalp kohli, the apple guy, and I were actually teammates uh at apple uh sorry at amazon before and uh then we diverged and he went and launched uh 75 000 nodes uh yeah. Well, we didn't do that um and you know it's okay, because I'm sure your netflix bill would be higher if we did that, so we're we're working pretty hard by the way to keep it eight dollars um all right.

A

uh Some uh trojan history uh so show off hands uh who uh who knows who cassandra was all right? Besides my team, all right who was cassandra.

A

Okay, uh so hector uh was uh the brother of cassandra and he, uh or rather the one of the big first java clients that was introduced uh uh was hector and st nx was largely uh inspired from hector. um So that's why we called it. Asian snx was hector's son um okay. uh By the way I want to get a feel of the crowd. How many of you are new to cassandra just playing with it and not the 75 000 guy, okay, cool, so how many of you are actively playing with cassandra in production?

A

Very nice. You guys will hopefully find my talk helpful. um How many of you are using java driver? uh Okay, how many of you using astianiks? Oh wow, okay, uh hector anybody awesome, okay, cool! So I think this talk will be pretty helpful. Okay. So uh let's get a quick introduction on asgenics. What does it bring to the table? uh Just um top level features, uh so it's very good. uh um It's been uh you had used in netflix, for I think two and a half plus years, so it's pretty mature.

A

Now um it does a very good job of load, balancing all the round robin strategies. uh Token aware, all that good stuff uh connection pooling yeah, this is actually a pretty big deal in stx. It's quite mature. It's gone through a lot of uh patches and fixes. It's really uh received some good maturity. Now uh it's been battle tested uh and especially like we run this at netflix scale in the aws cloud, so we do see a lot of network anomalies.

A

A lot of glitches instances get uh taken away all the time, and sometimes we like to do that to ourselves with all the chaos monkeys and all uh so ostianx is pretty good that way it's pretty resilient.

A

It actually does help out with all this chaos um plugable host discovery mechanism yeah. This is pretty easy. All I mean here is that you can supply your own mechanism. You know give it your seed nodes and it'll, take it from there it'll in it it's connection pool, but the other cool thing is um at netflix. We use our own discovery mechanism, and since this is totally pluggable, you can write your own. You can actually programmatically send signals to asti annex to add or remove your host from its connection pool.

A

We use this a lot at netflix, especially while doing operation operations on nodes like any kind of maintenance, or even if you want to isolate a node for just like debugging. Something first thing we make sure is that the application traffic is routed away from that node now granted that a coordinator can still send traffic to that node if it's not disabled in thrift, but at least, if you're using token aware algorithms with cl1 see a local one. This is very useful. You don't have to actually turn off uh thrift or anything on the node.

A

You can actually uh just surgically route traffic away from it uh via sdnx uh metrics. Okay inside ascentx has a lot of them. um It's not just the throughput numbers of read versus right paths and latencies and stuff like that. It's also you have a lot of insight into its connection, pool uh which is pretty cool. um It's uh it actually can help you even debug how the connection pool is doing.

A

You know as an example, if you're, if you're, uh if you have a fixed uh size connection pool and if you have persistent connections and you're reusing them for queries and if your query starts slowing down for some reason, uh you'll actually see your connection, pool, saturate and then you'll start getting timeouts. It's pretty cool like you'll, actually see the metrics will directly indicate that problem. um It's something pretty non-trivial to debug.

A

You know for a application running at large scale in the cloud, and these metrics can actually help you uh and yeah highly configurable and pluggable. So, oh, I forgot to mention this. Ascendx was one of the big contributions for the netflix open source software stack, so it was designed with the intent that others outside netflix will adopt it and will plug in place so all its components, as I mentioned, load balancing, metrics connection, pooling, if you don't like it, you can just add in your own. It's pretty uh configurable and plugable.

A

It's pretty nice, very modular. Okay, just uh giving you a little idea, uh it uses a very nice fluent query, syntax, a pretty big reason why a lot of folks use ostianics.

A

um I hope you can actually read that slide apologize for the small font, but I just literally took this image from my ide, so the top line we're just starting a mutation batch. This is a write.

A

The second line you can see, I'm preparing a row mutation, the column family, is cf time series. uh The row key is uh just cpu utilization, uh something I made up and then, if you can see the for loop, I'm just inserting 30 columns for that row. That's all I'm doing, and the last line is mutation. Bachelor, execute it's pretty simple flows pretty well, especially if you're exploring uh the client in the beginning, while you know new and figuring things out is pretty useful.

A

uh I noticed that I just did date minus one minute, so I'm just storing the last 30 minutes, I'm just making something up, and this is the corresponding read query. uh I especially wanted to show this to you. I really hope you can read that font. um I can read it with you uh it. This really shows how fluent uh the syntax is. It's very strongly typed, it's all java based and it still brings you that fluent query syntax, which is pretty cool.

A

uh So first line you have ks, which is just a key space object, and you are preparing a query for that column: family cf time series, you're asking for a certain row, cpu utilization and then you start doing a column slice. uh I'm assuming you guys know what column slice is you're, just asking for a range of columns.

A

So there's a set start set end, I'm just asking for now: minus 15 minutes, that's it uh and then execute and then below I'm printing the columns this code actually works. I even had the output uh below, but I removed it because the font looked totally weird.

A

You wouldn't be able to read this slide otherwise, uh but that's it that my whole uh intent of showing this slide is just to prove the point of the fluent query: syntax, where you can do something pretty non-trivial uh and it flows pretty well, and it's a pretty compact uh query recipes. ah This is the big one, so um asti annex also has a lot of recipes and I'll. I can explain what recipes in case. You don't know it, because I didn't know it. When I jumped onto the team.

A

A recipe is just a high level abstraction on the existing api. It just it's some snippet of code that does some non-trivial work for you that that's all it is, uh and we basically looked at certain use cases. Certain patterns in the company were like wow people do this again and again, let's show them how to do it right when we can just wrap it and contribute it back to asti. Annex that's exactly what a recipe is. um These are not the only recipes. Asianex has more of them.

A

um I don't think I have time to go through all of them, so I just listed the the popular ones like I'll go through the first one all rose reader.

A

um This is a recipe where it's kind of like a scattergather approach, where you can actually ask astinx to read your entire cassandra database and you know, send all the rows back to your uh function. Call uh it uses multiple threads uh and it'll. Do a pretty good job of like looking at your token range and you know, send one thread to each node to grab all the data and you know in parallel asynchronously, though they'll bring data back to you now why you'd want to read your entire database into memory is a separate conversation.

A

It happens a lot. I've seen use cases, uh but it's it's a pretty powerful recipe um distributed rollock. uh uh I think somebody else uh in today's summit uh mentioned that they're using distributed rollock. uh So this is another pretty important uh recipe, that's being used uh at netflix, where um you can actually lock a row uh to get exclusive uh right access to it in cassandra. Now, please don't confuse this with uh lightweight transactions. It is nowhere close.

A

This is just an optimistic uh lock, that's it, um but it works pretty well and we use it for use cases where like. If you want to send an email once and only once to a customer, you want to make sure uh that you, don't uh you know, send duplicate emails. You know it's.

A

Basically, the person has a distributed worker fleet and they're all working on these task items, and it is possible, with some distributed race, condition that you could get the same task on two nodes and now they're both going to send the same email to the customer and they'll look bad.

A

So we use this recipe to actually uh dedupe that, so it's pretty nice chunked object store.

A

I think most of us know that storing large blobs in cassandra is not a great idea, but at netflix we do have certain large blobs, like your silverlight player, that you can download from your browser uh when watching a netflix movie. That is one example of a large binary file. uh Folks do want to store that in cassandra because of our global replication footprint and all that stuff. They want to leverage all those features of cassandra, but storing a large blob is not a good idea.

A

So this is another recipe where it will seamlessly chunk up that bite stream for you and sprinkle it all over the database. It uses unique row keys for each chunk, so you don't actually have any hotspot on any one node. So it's pretty good and it maintains a nice abstraction uh for the client. Where you don't know, the chunks like you don't need to know, as tnx will split it for you and it'll also also stitch it back when you do a get.

A

So it's pretty nice and you can control things like chunk size and all that's good stuff, but this is just another example. So just wanted to give you an idea of recipes. There are more they're available at our github site, which is somewhere in this presentation, the link, so you can check it all out with a lot of examples.

A

Okay, so how does netflix use sdnix just want to give you an overall picture? um If you guys can read this picture with me like go bottom up, so look at the last row: there are a bunch of boxes there, blue boxes. Those are just the internal components. You know they map to some of the features I mentioned earlier, like connection pooling load, balancing, metrics config, yada yada.

A

If you go up a little bit, that's the big st nx api right there. So this is what takes all your queries uh and coordinates these components below and like executes them, for you in a reliable manner in the cloud.

A

um If you look at the two green boxes, that's just uh that's also part of the astx ecosystem, they're, just uh a high level abstractions one's a nice entity, mapping layer uh where you can annotate your object uh and persist it to cassandra uh via the astnx api.

A

It's totally built on top of the api and the other one is the recipes I just spoke about, and the little yellow box is uh all the other netflix dials uh and data access layers that we've written for all those objects that we want to store in cassandra- and I think maybe christos may have talked about this in the past, but we store literally everything in cassandra at netflix um almost almost so. This is pretty key by the way.

A

My main, uh the main point I want to drive home here in this slide is that blue box, in the middle, where everything, ultimately everything that's public facing goes through that uh front uh asthenics api and that's pretty key to remember for this presentation. Yeah you'll see how we use it later, there's no backdrop and as being used anywhere java driver. uh This is the uh this is pretty key, because it's uh now a pretty popular uh driver uh written over the cql and native protocol, uh and it's in java so seeing very heavy adoption.

A

uh So when netflix saw this in 2013, uh this is pretty attractive. It has a lot of good features that we want to use, namely uh to start the first one, the async interface. uh It does uh use neri underneath so it's pretty cool. You can actually uh use that async feature in java driver, um and you know we have other projects at netflix. I don't know if you have heard of rx java or whatever, like netflix, is investing a lot now in reactive extensions.

A

So we are looking at that. In fact, our entire client server mod model we want to uh see if that can go truly async throughout and netflix by. The way has a very, very big client soa stack. I think we have from allowed to say I think, 500 plus micro services, uh that bring that netflix experience to you um so uh that all you know should start going async, and this will be pretty key client and server communication.

A

Cursor support this. I view as the little brother of the aldors reader. You can actually again do select start from table. I don't know why you would want to do that, but you can do it, it's pretty cool feature uh or not, depending on how you look at it uh collections uh collection. This is uh also quite powerful. It enables a lot of cool data models.

A

uh A lot of people are very familiar with composite columns and also there is some sort of parallel here with collections, especially as you're trying to map your mind to the cql protocol, um flexible retry policies, again, another feature where, depending on your perspective, you may like it or hate it. You can actually change your uh consistency level on a retry. You can upgrade it or downgrade it. um It does seem a little non-dude as to. Why would you want to do that?

A

But there are a few cases where you actually can leverage that um and whatever the other stuff in sql. So we looked at this and we were like wow, okay, let's, uh let's start uh trying to leverage all of this, um you know and can we integrate asthenics uh with java driver? Can we replace asthenics with java, like what should we do so we started looking at our options um so by the way, just throwing this out there, like uh you know, for especially the veterans who've been using osteonyx in production.

A

This is a pretty big deal like if you're going to go over java driver, because there's so much in thrift that people can use uh to such an extent because it's so powerful, especially with the flexible schema-less design advantage of the cassandra data model. This is quite concerning to a lot of folks, so I just want to say this that uh don't worry about it. Astionic still works over thrift. In fact, in production today, sdnx is using thrift.

A

So as long as the community supports it, uh we will have it but anyways um um we are uh uh pushing to integrate with java driver, which is what I'm here to talk to. You about how we can bring the best of both worlds together, uh so I started the little exercise you know, and I uh pitched to the company. uh Hey guys, you know you wanna uh try using the cql protocol, like all nice new nifty features uh new things coming along.

A

um This is what they viewed me as, and uh uh the succinct technical feedback that I got from the developers was more like this. Where they're like? uh What do you mean uh new features? Only in cql, uh I'm using thrift, uh you know what am I going to do um and then I explained all the advantages of cql um and I actually I I can admit this. I understated the uh pervasive impact of sdnx at netflix. uh I did a little survey recently. um There are usually use some build system to find out.

A

There are 350 plus direct dependencies on the stanx library at netflix. This is just the direct dependency. This is not even the dependencies that pull it in via the transitive closure. This is the direct dependencies.

A

So, roughly, what I was actually really saying was this: like hey thousand people, uh can you guys, like all, go, refactor your code, so I can bring these new features to you. um Then the technical feedback was slightly different.

A

It's a true story. It's actually not funny, uh okay, anyway, sorry, uh but we we took this feedback pretty seriously and we went back to the drawing board and we did understand uh what they were saying. um You know when I largely speak uh to developers at netflix. I find them in two categories. uh One is the kind of developer that will say. Okay, I know what I'm doing. I know the thrift. I know all those mutation batch queries. I also know everything about cql or I'll. Go read it up.

A

I got it like give me full control. Let me go crazy right um and that's cool um and they can actually just directly start using java driver if they wanted cql. There are also uh provided they understand all the data models.

A

There is also this very large bucket of developers who are already using in production and are worried about. What's going to happen, if we just flip the switch, um and some of them also give very good feedback where they're like they don't care about the low level petty details about which uh protocol has what uh little idiosyncrasies and how do they work with them?

A

Some of them are asking for higher level abstractions where they don't actually care about the the low level details, which is, I think, my key point in this talk and I'll talk to you about I'll talk about it. More so went back and I said no wait. We do have a nice new adapter at astianix.

A

So remember that blue box, I showed you a while back where everything goes through that front api. So what we did was we re-implemented the entire asthenics interface to go over java driver underneath so java drive is the query engine um in a new version of sdnx, uh which is a cnx 2.0, and uh we have an adapter that can take your existing query and translate it to a java driver. Query underneath and it's totally aware of your schema.

A

uh It knows what your schema is like and hence what your sql query should be like, which, by the way, was really a bear, but we did it um so the cool thing is, uh your queries can now uh be async since they go over neti but they're still the same queries your api from the asthenics perspective is the same. So all the recipes, the entitymapper all got ported for free, which is good and asthenic still works.

A

By the way, keep in mind, thrift is not uh being abandoned. We actually do have even the thrift adapter you could use any one. You can just switch your api back and forth. Well then, there probably will be some caveats, because not every data model can actually work seamlessly between thrift and cql.

A

Sometimes it just is not possible, uh but we have a nice big blog at our website uh at our github site, which is in this talk and you can actually check out if your schema will work, but we basically looked at netflix's schemas and there are a lot of them and I think we literally nailed all of almost all so anyway took this uh design, pitched it back to the devs at netflix and the feedback was much better, so the babies, I'm sorry the developers at netflix are happier no.

A

No, uh it was a good lesson learned for us, um so, okay, very quickly doing an architectural uh review of what I just said. uh Remember that that's the blue box right there um in the middle asti nx api and you have two uh drivers underneath actually forgot to show a little box. There's a little blue box above java driver, which is the adapter um or you can imagine it inside the blue box, the cnx api.

A

But that's actually what translates all your queries uh back and forth to the java driver and you could uh just with the flip of a switch, uh go back to thrift. If you wanted to okay yeah the big question, what about performance, uh so I think the community has gone back and forth between java driver uh and thrift uh and st nx, and what are we doing, which is faster and there's improvements on the way, and you know we're talking about different technologies.

A

um I'll tell you largely what I noticed, um and this is also available on our blog post, which is on our site of what we found. I don't have those graphs in here, because that detail would be unnecessary at this point, but we roughly found uh java driver to be on par, uh or there was a it was a little slower, but the difference was insignificant enough. Like we didn't uh for us, it was still a green light for production.

A

um One thing you have to uh remember: uh please use prepared statements if you're using java driver. That is just uh just assume that it's a requirement if you want that production performance at par just a little digression into prepared statements, um there's actually no magic. Here, you prepare a query. uh Basically, you digest. The query create some byte buffers send that to the server there's some state there, and then you reuse it. So when your new queries come your values are just dynamic values and they just bind to that existing prepared statement.

A

That's how it works and that's why it's faster than not using a prepared statement.

A

um But if you look at uh some of the use cases that we've seen uh built on top of sdnx, where people basically will go build their own data access layer on top which can do all sorts of flexible queries uh depending on the you know, input criteria to the method. Oh, the query is going to keep changing. Imagine them trying to maintain their own prepared statement cache, so they could reuse it for some people that may be trivial.

A

Some for some people, not people, start using fancy techniques like they'll, maybe take the string query or something and they'll hash it to a value and look up their prepared statement in a concurrent hash map or something there are other ideas to be more robust, so you're not sensitive to like weird changes in the query and avoid collisions where you could tokenize the query and then sort those tokens and then rehash them, uh but whatever it is slightly non-trivial and you soon get into prepared statement management.

A

um So there was one cool, astinx value add here that we discovered sort of uh almost after the fact. So if you look at this query in the middle, um I showed a similar example before it's key space start whatever with row or with column slice and then execute.

A

If you, if you notice it's quite structured, which is kind of nice, you can view all of this as uh just little java objects underneath in the sdnx api. So the cool thing about this is all these structured queries have an innate query, signature underneath. So the thing about this query signature is stnx. Can now look at that structured query and say, like ah I know what the prepared statement should be, because I can predictably tell what the java driver query should be and that's right below select star from table.

A

Where key is this and column one is in this range? Oh yes, I said column one uh that is true I'll. Tell you more about it in another slide um so anyway, the the point I'm trying to drive across is that there is an innate query signature in this query. uh You know, instead of actually parsing your query, to understand what its hash code should be. You've naturally done it up front in sdnx by being forced to use a structure interface.

A

So this is one cool thing in sdn x y, if you're, using java driver underneath sdnx can just uh understand your queries and manage the prepared statements underneath for you seamlessly. So you don't have to worry about that, and that's also how at netflix we're going to keep that performance bar up uh when we switch to java driver, which is kind of a cool thing, um anyways a little bit more. So I think we're getting a little bit more towards java drive and sql now so something else I want to show you.

A

um We started looking at uh the sql protocol uh a little closely and something didn't seem right. um I have been asked these questions uh all the time that our rows really rows are columns rows our rows. Columns like what- and surprisingly I know, but surprisingly uh I'll, be honest with you technically. The answer to all those questions is yes, depending on your perspective, I'm not joking guys seriously and I'll show you I'll show you what I mean.

A

So something was not right and we had to like fix this uh for netflix, because things were getting pretty confusing for the devs. um So I'll give you a very simple schema, just as an example. This is a very simple example. Okay, your key type is an integer. Your column name type, which is your uh column comparator, is also an integer, and your values are just say some ascii or some strings right. The simplest schema possible.

A

Take these two rows as examples by the way this is slightly in the old world when you're using thrift. This is the true uh power of cassandra right. The flexible schema s design where you have two rows row, you know with two row keys one and two and the first one has five columns. The other one has three columns great, and these can tomorrow become wide rows or whatever you could do, column, slices and all cool.

A

You know you guys get it so create this using the cli and then log out of the cli log into sql shell and do select star from table.

A

I hope these will be the only two rows by the way. This is what you will see, I'm hoping that this is legible, for you guys at the back um note how you're seeing eight rows here I'll flip back to the previous slide. There are eight columns here and two rows I'll flip back again. There are eight rows here. This is what happened uh for dynamic columns in cql notice, there's something called column one up there.

A

So you see, I don't know if I can call it uh transpose, probably not the right technical term, uh but that's sort of what happened uh to your columns. uh They sort of became rows. So this is kind of what I was saying. You know when I see this. I think of the grizzly bear that I showed earlier, because it's not just that hey you're going to sql. You can just rewrite your query with this new syntax. No, no!

A

No, your model is changing depending on your schema and you have to adapt to this model earlier. You used to get two rows back or one row. If you ask for rookie one now, you will get five rows back in your results and you got to figure out how to uh whatever appropriately deserialize it into your object. uh So this is another.

A

How do you say this is a breaking change at netflix right, um so the cool thing is um just to keep everybody's sanity uh at netflix. We decided to maintain the original semantics. So since we have an adapter that intercepts all your queries, it also intercepts the result sets. So it will.

A

You know, bring your result set back to the old world, uh which is kind of nice, because you know that's how I don't have my butt shoot out by like 900 devs at netflix. Right like this is what we had to do to like not uh break everybody, so this is one other cool thing, uh so rows in our world are still rows and columns in our world are still columns and uh by the way this was no easy task like in assionics.

A

We had to understand what's a row key and then in the new way. What is a partition key? What is a clustering key? What is your primary key so as cnx does that by a way of like describing your schema under the covers, uh so it exactly knows what your schema is like and hence how to transpose your your columns back.

A

It knows that cool okay, so the takeaway for the big takeaway for the talk is that you know we've seen uh uh that there is a lot of value in providing higher levels of abstractions uh over uh other drivers, underneath be them thrift or cql.

A

um I'm just listing a few examples here, like time series or something we've been asked, a million times that if somebody wants to go implement this and I've seen various different implementations, some of them not uh following the best practices of cassandra uh sliding window. um You know like netflix: if you play a movie and you you know uh log out, then you log back in you want to play the movie. It asks you do. You want to resume playing.

A

Have you guys seen that the way that works is every few seconds the player is streaming uh heartbeats back to netflix, to show where you know where you are in your movie experience um and that's stored in cassandra and the person wants to implement a sliding window over it right because he doesn't want to keep everything out there for all the time and we have to actually we need data model experts to actually consult with the person to show them like how to best model this in cassandra, how to not create hotspots.

A

How to not have white rows. um Actually white row is a big problem like we recently. I won't take any names, but we recently had a problem where an innocent developer just ended up. Reading a wide row in cassandra, um you know is several gb and you know what that does to the jvm uh dutifully took the note down uh and then did what I actually asked them to do. Retry and go to the next replica, take that replica down and we did see a problem in production.

A

They were sort of following best practices- oh no, but but but the last takeaway is that uh we're sort of uh leaving this window of vulnerability open for them like they don't know like uh or you know they don't want to focus on that. They want to focus on their own uh uh high throughput service and we're supposed to solve uh and make the database super resilient for them. So we have an opportunity in sdnx to model these best practices through higher levels of abstractions like.

A

I actually wish at that point that I actually had a special column family, which was uh called the large row. The white row column family. So I could actually control uh how uh you can paginate over columns, uh and you know literally introspect the bytes coming back and you know keep it at certain thresholds. That's very valuable for a lot of people. You don't want everybody to solve this problem again and again and again um and oh yeah objects with attribute based indexing. We've heard this all the time. Okay, I have this object.

A

I don't know what columns are. I don't know what rows are? I need to store this object in, oh, and you know I'll pull out only a few attributes at a time we're like okay, good those are going to be the columns like we get all of that and then you'll say. Oh and I have these few attributes.

A

I want to search, like maybe like say, employee or something and I'll search by last name, and I want to get all the objects back that match this last name uh and we're like okay he's asking for indexes, and we don't want to give that to him, uh because that's another thing that we say right, don't use indexes in cassandra and his feedback is like okay, you chose this data store for me.

A

If I don't, how do I get indexing, whatever inverted indexes, whatever give me, give me a virtual table where this is just possible like solve this problem for me, because I don't want to focus uh my core dev cycles on this, because they want to focus on other stuff and that's totally legit and that's the overall feedback we've taken uh and that's kind of where we're trying to move sdnx in that direction.

A

um So, okay, so the big question. I know there is time for q and I know you guys are going to ask this so I'll. Save that what should I use um so I'll, be very honest. um Asthenics is all about uh more typed interfaces, higher levels of abstractions.

A

um If you don't uh want that and if you can write sql queries on your own, don't use! Ask genius, use java driver, it's totally fine it'll work pretty well, it's got all the connection, pooling support all the load, balancing support, it's doing pretty well actually with cql, so you don't actually need sdnix for that. But if you do see value in higher levels of abstractions or actually, if you want to even contribute those back to.

A

As cnx go for it, that's actually what that's actually how we want astana to live on and that's it question.