Apache Cassandra Apache Cassandra World Party | July 2022, 20 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cassandra-backed Streaming Graph with Quine | Apache Cassandra World Party 2022

Description

Presentation from the second Cassandra World Party on July 20, 2022. Learn more about Apache Cassandra at https://cassandra.apache.org/

Ryan Wright of thatDot presents “Cassandra-backed Streaming Graph with Quine”
Quine is a "streaming graph" built on Apache Cassandra. This powerful combination is revolutionizing what you can do with event stream processing and how easy it is to make happen. In this lightning talk, you'll see the world's fastest data pipeline built from start to finish: it consumes streaming data and builds it into a graph, stores that graph in Apache Cassandra, and triggers immediate results from graph patterns in real-time. One slide; live code; real results… all in 5 minutes.

About the Presenter
Ryan Wright is the creator of Quine, and has been leading software teams focused on data infrastructure and data science for two decades. He has served as principal engineer, director of engineering, principal investigator on DARPA-funded research programs, and is currently the founder and CEO of thatDot—the company supporting Quine. Ryan particularly enjoys making the philosophical ends of computer science more practical and useful. This usually involves problems related to language, meaning, and data.

Apache Cassandra is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Linear scalability and proven fault tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Get started: https://cassandra.apache.org/quickstart/

The Apache Cassandra World Party 2022 was a virtual, one-day celebration on Wednesday, July 20, 2022 held in anticipation of the launch of 4.1

Join the Cassandra community: https://cassandra.apache.org/community/

#ApacheCassandra #database #NoSQL

A

My hope today is really to introduce you to an open source project that we just recently announced just recently was released, it's the result of about eight years worth of work um and the reason it works. The reason it can actually kind of change. What was previously possible is because cassandra under the hood really makes a big deal. um The capabilities of cassandra has an have, enabled new kinds of streaming graph processing and that's what coin's about so.

A

It's free, open source software, quantum dot io and, if you're ready to hit the next slide, we can start talking about it.

A

The big question to begin with is what is a streaming graph, because we kind of made up this term. um It is something that lives squarely in between these two worlds: infinite data streams, so kafka, kinesis pulsar. These kind of things are delivering just an endless supply of records, probably at a very high volume and, on the other side combine that with a really expressive graph model. That gives you the ability to talk about data, to connect data in in ways similar to relational tables, but more expressive than that.

A

So all data fits in that shape and when you do that, you get the ability to run graph queries over your stream, combining together new data that just arrived with old data from minutes ago or months ago. Regard like either way, and so the the streaming graph is really. The combination of these two features stream data in and use graph queries to express complex patterns uh for what we're looking for. Over top of that.

A

um So that's that's. The concept and quine is new in this space for uh for really bringing that to bear in high volume situations and there's the next slide. So why would you want to do something like this? Why a streaming graph?

A

Well, some of these technical capabilities have really enabled new kinds of capabilities um and things that were hard or even impossible in data streams like multi-join patterns, which are just too slow to actually be practical in a high volume situation, those get solved with quine and enable new kinds of use cases like detecting money laundering or if you've got data sources from lots of different sources and different types, and your job is to join them together and handle out-of-order data delivery.

A

Then you can enable new use cases like cyber security, xdr platforms to actually monitor that and find those things with real-time analysis capabilities. You can watch for authentication fraud as it's happening and you can shut those uh fraudsters down before they actually get in logging into the system, so those were previously impossible and now they become new capabilities for stream processing.

A

Because of how cassandra works together with quine quine saves the node data in an event source fashion.

A

It means the deltas, the small little updates, those updates get written into cassandra in very small units, not the whole node information, just changes to its history, so that gets written very quickly and efficiently off to disk in cassandra and then cached in memory to do graph operations pulled back in only when needed by doing efficient range scans over known keys, combined with other cassandra capabilities like time, windowed, compaction and ttls to expire out data that we don't care about any longer.

A

Enterprise users have been able to run this in production and achieve just crazy results like a million graph events per second, as just even kind of an arbitrary stopping point. So they've seen it scale way past anything that anybody's been able to do before, and so, if you want to try using streaming graph together with your cassandra deployment, it's really a matter of set up. Your cassandra system start quine and configure it to use cassandra and then an api call will start quine consuming from your kafka and pulsar streams.

A

Data comes in. It gets formed into a graph using what we call an ingest query. So take each record out of kafka form it into a graph shape and and as that graph gets built and changing dynamically, you can issue what we call a standing query over top of it to say, monitor the graph for these complex patterns. I care about those patterns get monitored very efficiently when they get completed because of new data coming in combined with old data pulled out of cassandra at just the right moment.

A

You get real-time results that can trigger updates to the graph or can stream data out to the next system and so plug it into kafka or pulsar and deliver that data to wherever it needs to go plug it into slack notify some humans notify some users about what's going on and let them take action.

A

So, if you'd like to try this, it's really just the three-step process of download quine. It's free and open source software, startup cassandra by default, run it on your local system and then just run the run. The jar file run the executable with the cassandra storage, configuration option and you're off and running at quine dot. Io recipes you'll see examples of particular applications that are all run with just one little command line.

A

You can plug it into your cassandra instance for local testing or change a few more configuration options and run it in production on a real live cassandra cluster.

A

So those those kind of step-by-step instructions will take you from start to finish and, as we said, there's a bunch of a bunch of examples of of recipes that live live uh on the website that you can go poke at and and play with, and experience on your own um one caveat if you do use those recipes along with cassandra configuration um there's that little force configuration flag, we're missing a slide from this.

A

But if you saw the last slide, it would point you to quine dot io at the website, where you can see uh demos of it running, live fine quality, io, slash demo and there's a slack community there. If you want to ask questions, get help and and just talk with others who are using quine for similar purposes.

A

So all of this is new capabilities previously impossible that were now enabled by cassandra on the back end. For quine on the front end for streaming graph, thanks guys.

B

That was great. Are you also going to send us some blog posts or something interesting.

A

Oh yeah, we're writing all sorts of things. So it's uh uh lots to talk about and lots to say it's it's the product of eight years worth of work, so we're we're happy to share with everybody and talk about it all day. Long.

B

Yeah we'll be happy to hear even more five minutes is five minutes, but you said it's a lot of work many years so we'll be happy to hear more. Definitely thank you.

A

Very much, thank you all.