Apache Cassandra Cassandra Day Silicon Valley 2014, 1 May 2014

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Using Distributed Session Clusters to Support Millions of Concurrent Sessions (Cassandra and Shiro)

Description

Check out the full post for more information: http://www.hakkalabs.co/articles/millions-concurrent-sessions-distributed-session-clusters-cassandra-shiro

In this session Les Hazlewood (Co-Founder and CTO of Stormpath), describes Shiro's enterprise session management capabilities and how to use Cassandra as Shiro's session store, enabling a distributed session cluster supporting hundreds of thousands or even millions of concurrent sessions. As a working example, Les will show how to set up a session cluster in under 10 minutes using Cassandra. Watch this if you need to scale user session load.

This talk was given at Cassandra Day Silicon Valley.

ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.

FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520

A

So we're here to talk about clustering sessions, application sessions on top of cassandra, my name is les hazlet, I'm the cto and co-founder of storm path. We were disrupting the identity management space.

A

This in turn has a session id generator so as when new sessions are created, um clearly a new identifier needs to be associated with them. um This could be anything it could be a string. It could be a long for like relational databases, given that this is a cassandra group and we'll be using cassandra as a back end. You can create, for example, a time uuid uh generator that will generate timely uids every time a new session is created and then that, of course, could be the roki that you store.

A

That will store a session underneath or store a session in cassandra with, but more on that later, um there's also a session cache, so we don't always want to be pinging or kind of hammering the data store. Every time we do crud operations, so there is a cache that is shared by all application instances, so we have second level caching kind of plugins built in so people use, hazelcast or clustered get our ehcash with terracotta or gigaspaces or coherence.

A

You know basically redis it doesn't matter you could plug in any any number of second level caches and then finally, there's the data store under the hood behind you know, behind the session cache and, of course, in in this case, we're going to be using cassandra.

A

um So the purple kind of the purple uh objects on this diagram are really all related to persistence or crud for given session objects. In addition, there is a session validation scheduler that the session manager uses to make sure that sessions when they expire.

A

They are deleted from the data store or when they're known to have been expired or when they expire immediately kind of remove them from the data store. But this is really important. Every single session management mechanism out there tomcats jetties, doesn't matter. They all have one of these things and it's necessary necessary to make sure the session orphans.

A

Don't fill up the data store over time so like if you're, using a web application- and you know that application has created a session for you, you just close the browser and you don't you don't click log out or anything like that, your session's still alive. It still exists somewhere in a data store and if something like a scheduler didn't exist, all those orphan sessions that no longer have users using them will just fill up the data store over time. So you need something to remove sessions from the data store and the session validation.

A

Scheduler is the thing that shiro uses to do that and then, of course, their session listener. So you can have register many different session listeners to determine you know life cycle or be notified of life cycle events, so session started a session stopped. It's expired. It's deleted.

A

You can kind of react to these events in your application and perform operations.

A

So this is the the architecture of sessions or excuse me, shiro's session management, slice of the whole framework. Any questions about this before we start talking about cassandra's specifics, everybody on the same page, how this works?

A

Okay, all right, so we're just going to focus now on just the purple parts. So just the uh the. How do we? How do we configure shiro to represent these things in its configuration such that we can point to cassandra as the data store and we'll use a web application?

A

A configuration in this in this kind of example? um Again, as I mentioned, you can use shiro in any kind of different jvm context. We've got again people using it in command line apps. We have people using it in vertex, which is an asynchronous framework, akka play so there's all sorts of kind of environments that you would configure shirow in and in those environments you might have different environment, specific configuration techniques, but we'll just focus on the the standard servlet web application model. Here.

A

With shiro's web config, it's just a couple things that you drop in web xml and shiro will then protect all urls or any any kind of request that flows through the application will be intercepted by shiro on all of them and make security decisions as to whether or not the request is allowed to continue or not. So maybe a certain part of the application is supposed to only be accessed if you're authenticated or if there's a known user identity.

A

Shiro will make these decisions before they ever hit your application code and either send a response back or redirect to a login screen. What have you there's all sorts of behaviors that you can support in your application, because shiro is filtering all requests, there's some really cool, innovative filtering url, filter definitions, that shiro has configuring servlet filters in web.xml is not very nice, especially if you have a decent number of them.

A

Cheryl makes this actually much much easier via a custom dsl and, of course, as you might expect, there are jsp tag, tag libraries, support, jsf tag, library support, so you can show or not show things within the jsp or jsf page based on security state of the current user and, as I mentioned, transparent http session support.

A

So this is how you enable shiro in a web app every shiro deployment needs an environment, and so this listener will listen for the servlet context of startup, create a new environment instance, a shiro environment and then bind it to the servlet context, so that environment is tied to that particular deployed application, and you can deploy many different environments.

A

It's almost always a one-to-one correlation, so you have one environment per deployed web application and then, once the environment listener is deployed or excuse me defined um you, you then specify the shiro filter that will then use this environment in order to filter all requests and get access to all the security things that it needs to do to get this job done in addition to the filter as part of the filter. Definition rather, you've got to specify a mapping that says I want to.

A

I want to filter every single request that goes through the app that's the slash star. I want the shiro filter to handle it, and all of these request types should uh should be processed by sheryl again. This is just kind of setting up the overhead of of telling the web container to to let everything flow through shiro, so it can make security decisions and, of course, session session operations.

A

Once that's done. Instead of embedding all these security rules and configuration into web.xml, which is very nice, um you can configure shiro in a number of ways.

A

Out of the box, shiro comes with a an ini-based configuration mechanism that um that supports what we like to call as poor man's dependency injection you'll, see very soon that you can create object, graphs just by simple dot notation, and so this is the out of the box default lowest common denominator format, if you're not using something more advanced like spring or jee, annotations or juice. Or what have you so?

A

We recommend that you use those other things because they're typically more powerful, but if you just want to get off the ground very quickly, there's this nice ionized support all of the object. Graph configuration beam, config is in here and then there's these three other sections. You can define static users, static roles and static, url, filter chains.

A

Of course these are just users and their their passwords.

A

These are any of the roles that are assigned to those particular users, as well as permissions assigned or attributed to a role and again, I'm not going to get into all this stuff too much, because we're really here to talk about sessions not so much access, control and security, but one of the cool things is that this url section allows us to define filter chains as a set of name value pairs that I'll show in a little bit, but it's way easier to understand the flow of a request through a filter chain when you can see them list when you can see them listed here than having to figure out via all these different definitions in web.xml.

A

So we'll see an example of how that looks in just a second okay, so we've got this configuration mechanism. How do we build the components we need to talk to cassandra and then configure it within an application?

A

So there's two approaches. As I mentioned before, there's you can implement the session dao interface yourself directly to perform crud operations to your data store or you can use something that comes out of the box with shiro called an enterprise session or excuse me, enterprise cash session dao, and then you write a cash manager and so, as part of its architectural, you know, constructs shiro has a cache manager that can be shared across almost all the components in shiro.

A

So if you write one cache manager, you can just plug it into the session infrastructure and then all of your sessions just live permanently in the cache.

A

um So, for example, if this was redis or hazelcast, all of the sessions could live in the in that environment. But those aren't really persistent environments and you don't really want to lose sessions. If nodes go down, so cassandra can be an ideal approach. So cassandra is not really a cache distributed, cache architecture because it's persistent it's durable, so we don't really need to to go through this approach. We're going to write our own session dao.

A

It's very easy to do, though. So what does that? Look like the only things that we really have to care about when using cassandra first session storage, or at least things that we have to write for shiro or the session dao session id generator and the the data store is already there. It's cassandra. We don't have to do anything with that.

A

We don't need a session cache right s. Cassandra is so blazingly fast that you don't really need an intermediate second layer cache in between your application and cassandra, the read and and rights of cassandra, depending on your deployment environment and the hardware you're using is typically extraordinarily fast such that you know you don't here. Here's a good example in our storm path uses a cassandra in production. We also use hazelcast for certain things, for things unrelated to persistence, but a typical read and we're all in an amazon ec2.

A

By the way, a typical read from hazelcast will take around two milliseconds, maybe maybe a little bit more or less, depending on the on. What's going on that day, a typical read from cassandra might take three or two and a half depending on what's going on with the network, so because the the the speed is so quick, you don't really need this additional infrastructure in place um just for the purposes of of second level cash for in front of cassandra.

A

So we're going to we're going to ignore this entirely we're not going to write one we're not going to use one. The only thing that we have to worry about are these two components um and, as I said before, we'd probably write like a time. Uid generator just to insert things into cassandra that way, and then, if you wanted to insert or create your own secondary indices, you could then filter by time you ids of when sessions were created and kind of run reports and all that fun stuff, but it doesn't really matter.

A

This could be a randomly generated string. What whatever works but we'll use a time, uid generator and the session dao, and those are the only two things that we have to configure so we'll we'll implement our own again we're not going to be uh using the cache and session dao. So all the things that we have to do are just override these methods do create. Do read, update and delete.

A

There's this thing get active sessions that is necessary for many session dio implementations and that is used by the um the validation scheduler to make sure that all non-stop or deleted sessions make sure they're still alive and if they're not alive, it'll close out the session and then delete it you'll soon see that we're not actually going to need this with cassandra, though, so this will be kind of a no op implementation, it'll return null or an empty collection. Every time and all you have to do is update or override those those methods.

A

So how do we enable this stuff in shiro? You know: we've got a session dao we've. We got an implementation and I'll show you guys the implementation a little bit later. If we have time, um but now we just need to tell shiro to use it, so we've got a session manager again we're running in a web container, so we'll use the web session manager and we're going to tell shiro's security manager. This is again it's global construct. It's used in every there's, only always one of these in every application.

A

We're going to tell the security manager that hey the session manager that I want you to use for the app. I want you to use this web-based one because we're in a web context, so now that we're using a web session manager.

A

The next thing to do is configure the session or the session dao for that session manager, and this is pretty easy. So, as you guys can kind of see this poor man, dependency, injection is kind of kicking in right. We've got on the left hand, side of the equal sign, the identifier and then a class to instantiate, and then these are just kind of name value pairs that are set via setters and we can do maps and other things. But it's hopefully it's easy to understand.

A

What's going on here, this object is just being assigned a reference that we've defined up here under the hood. When we see some code a little bit later, you'll see that all of this is using the cassandra. Excuse me the datastax java driver, and so this cluster factory is just a wrapper that creates a datastax driver cluster object and returns it to shiro such that it can be injected into other methods for use later, so we've acquired a datastax.

A

Excuse me cluster object. We were, we have a key space called shiro sessions and we're going to stick it in this column. Family called sessions.

A

And so once we've defined the dao. All we have to do is just plug it into the session manager to tell the session manager hey. I want to use this object to perform the crot operations for me, so anytime, a crud operation occurs. It's going to be hitting cassandra now because of this uh very simple dao implementation.

A

So we've got, we've got the dao defined. How does what does sessions actually look like in cassandra, and this is a sequel or cql3 kind of example?

A

You know thrift works the same way, but since we're using the data stacks java driver in our code base, uh which will be it's on github and I'll, give you guys a link afterwards, um we'll stick to cql3 to define the table. So there's a sessions table there's a a prime, a primary key, which is the time uid in this case. There's a start and a stop time stamp. There's all access lacks last access timestamp.

A

um What the time out of that session is? You know 30 minutes or one hour. You know stuff that you configure in your application. um There's there's a expired boolean that can be set.

A

If you want to retain the session for a long period of time, maybe you want to do historical reporting, so maybe you don't delete the sessions, maybe maybe they stay in there, but you keep this expired flag to determine whether or not it's actively used the host from where the session was initiated and the serialized value here is really just serialized representation of the attributes map. So when you have a session like on in a web tier, you call session.getattribute.set attribute.

A

This is really just a very efficiently serialized um version of that map, and so shiro does a lot of the serialization itself via bit masks and other kind of optimizations. But um this is just the raw output of that of that attribute map and, of course, because because it's a blob, um you want to make sure that this isn't too big. um You know you can store pretty large objects as column values and cassandra, so cassandra's not going to really have that much of a problem with it.

A

But the bigger your session is the more I o overhead it's going to cause, and this is not. This is a general statement for all servant containers, but sessions should be very lightweight in general right. You should only kind of have pointers to objects and sessions.

A

You shouldn't store entire object graphs in the session themselves, because if you do it's going to be serialized out to this blob, so you know instead of storing a user object, you would store the user's primary key in the session and then you get the key out and then look up the user and then maybe that's in second level cache. But the idea is: keep this minimal your I o overhead will be low and it will be very fast with cassandra.

A

So this is how this is kind of the layout of how our our session objects are going to be persisted.

A

But if you guys remember back here, there's no session validation scheduler here, that's you. That's, usually a pretty important thing for all session managers, but one of the cool things about cassandra is that we don't need one.

A

The reason why we don't need one is because scandal has built into it. This really cool concept of ttl time to live, and so we can use cassandra's time to live to perform the same operations that accession validation, scheduler would do right. Auto expire, auto delete all the records, or at least have the make. The client have the perception that they're automatically removed just based on the ttl setting. So you don't need the session validation scheduler. It doesn't need to be operating as a background process or a daemon thread.

A

It doesn't need to be querying the data store every so often to uh to reap orphans. We can just let cassandra handle all that for us, which is super cool, so all we have to do in this case is tell shiro's session manager. I don't want the session validation scheduler to be enabled because the data store is going to handle this automatically.

A

This is a very risky thing to turn to false. Unless the data store supports ttl expirations. So because cassandra can we can turn that off.

A

And so now that that's off, how do we insert data or update sessions in the data store? And it's just a very simple cql3 statement right cassandra? We call this an upsert because there's really no difference between an insert and an update, and so you can use a single query to satisfy both insert and update scenarios.

A

So here's what the query looks like we're going to update the sessions, we're going to use a particular ttl- and this is your session timeout value- also note that this is not dollar sign notation, it's not part of cql if you're not familiar with the cql syntax.

A

This is just me kind of illustrating where, in the query, you would put the timeout technically. This should be another question mark, so it can be specified as part of a prepared statement, but the idea here is that we're going to update the sessions using the particular ttl. In this case the timeout is going to be whatever the number of seconds is times 30 minutes by default. So this is a this is the number of seconds and 30 minutes by default. You can change it to be whatever you want, and this can change per session.

A

So maybe some sessions are 30 minutes by default, but other sessions are five minutes or an hour. So just like the server container you can you can control which sessions timeout when and then it updates all the fields that it needs to update serialized value, of course, and then the id is your auto generated identifier.

A

I should also kind of note that this last access, timestamp thing is pretty important, so in all servlet containers, whenever a request comes in, we've got to update the session to let this or to let the infrastructure know that this session's still alive, it's still active, and so what that means for cassandra- and this is really important. It's like one of the most important parts of this talk is with regards to how you set up and maintain cassandra.

A

That means every request that comes in is going to read from the session, because it has to pull it from somewhere right and also every single request is going to result in a write to the session table, because you've got to now update that session entry to reflect the last access timestamp all right.

A

This has to happen all the time and so with, because with shiro you can have a level one cache that kind of does the reading and the writing multiple times in a single thread, but at the end of that thread, these updates have to go out to cassandra to make sure that the data store reflects the current persistent state of that of that session.

A

What this means, essentially, is that there's a one-to-one correlation between reads and writes for session activity using cassandra.

A

So this might you know if you've used cassandra for a while or you've kind of been to some of the other talks. This might you know, bring up a question for you: what about tombstones every time that you do a modification of a column, value or you? You know you kind of delete something it creates a tombstone in cassandra. How many people here know what tombstones are?

A

Okay, almost every how many people do not know what tombstones are there's a few people. So when cassandra persists data into ss tables, it's it's kind of under the hood representation of physical data.

A

It never modifies that data in place, ss tables are read only or excuse me append only you can append data to them, but you're never allowed to change the data. That's in there, so because it's an append only data form or a format or data storage mechanism.

A

Whenever you delete something, it's actually going to write append to that ss table and say hey this. Actually, this value has changed or it's removed so reflect that whenever the customer or the client rather queries the data store, and so because there's tombstones there's additional. I o overhead every time that you delete or update something. So if you have an append, only kind of write log cassandra is amazing for that, because there's almost no tombstones being generated, but when you do have tombstones generated as a result of an update or a delete.

A

This causes overhead for cassandra because when it goes through compaction to consolidate its ss tables, it's got to resolve these tombstones or these deletion or these update markers um and that there's io overhead in all of that and there's processing overhead and doing that. So the more tombstones you have in a cluster, the more overhead you're going to have so there's a caveat with this. How do we? How do we handle tombstones in a relatively efficient way, so that things are that performance still stays uh reasonable if you're going to be?

A

If you're going to be doing all these one-to-one reads and writes the answer is: is that we want to use a level compaction strategy so, um and you want to change your gc grace to make sure that the number of tombstones doesn't fill up your data store or doesn't fill up cassandra over time. Gc grace is by default 10 days, which is probably way too long for a session store mechanism, so here we've actually cut it down to 24 hours. It's actually probably even better to go lower than this.

A

If you can just realize that the lower the gc grace the higher the opportunity that data might be lost, if you, if you don't, go through a repair process, so that's usually not really a problem in practice with shiro the sessions usually last much much.

A

Their lifespan is much smaller, much shorter than the gc grace. So it's almost never really an issue, but you want to keep this number relatively low. A day is probably good. I've talked to some of the cassandra experts that the cassandra expert meetup talks at the um the summit over the summer, and they said that this was probably more than fine for for kind of this type of io operations, and it's really important to use level compaction strategy here, instead of size, tiered compaction.

A

How many people here do not know what the difference between those two strategies are? Okay, we got a couple couple folks, so level compaction strategy will actually compact ss tables into larger ss tables. That can be more efficiently read over time and it does it based on a tiered strategy, so um you'll have like level level. Zero ss tables will get compacted into a level. One ss table that's ten times the size.

A

I think this is five megs by default, so the next level below that is 50 megs, the next one below that's 500 megs, and what that allows cassandra to do is kind of compact. All the data into ss tables that are read in in order, so it'll check, l0 first before it hits l1 before it hits l2 l3 and what that really amounts to is in a very efficient read result. So if I have my data tiered like this in multiple levels, I can check out one first.

A

um The probability distribution on that is that there's a 90 chance that I'm gonna find my data there because of it's most frequently acts or frequently used. If it's not there I'll fall back to l1, there's a 90 chance, it's going to be there, and so you get this really efficient, read distribution to make sure your data is accessible and very very quick reads are very, very fast because tombstones, however, being created due to the right, this will incur a higher. I o overhead for cassandra. So you have to be aware of this.

A

Your io io overhead is at least doubled. If not more than what size tiered compaction is all about size tiered will compact ss tables based on size, so if they hit a certain file, size limit it'll start going through the compaction process. The key pro the key issue here with tombstones and changes with size tiered, is those changes. Those markers can be spread out over multiple files.

A

So, in order to read a single record, cassandra might have to read two three four ten different ss tables right and that's a very inefficient process for things that have to be read on every request, so do not use size tiered compaction if you're using sessions for for for cassandra, definitely stick to leveled.

A

Just again be aware that your I o overhead is going to be doubled, that of normal workload.

A

So- and this is kind of true in general- not just for sessions, the more that your application reads from a table, the more it will benefit from from level compaction, the more that it writes or appends the more size tiered will benefit from it. Just be aware, the fragmentation that occurs on on size tiered, what about row? Caching, people ask you know: hey. Should I enable row caching, in order to make my session reads even faster and the answer? Is you don't you don't really need it?

A

It's not really that beneficial ss table your s's table again via all the off-heap storage. That's in in use uh for the later later uh editions of cassandra odds are very high or the ss table is already going to be in operating system. Page cache. You don't need a row cash. It's there's only very very specific situations under which you would probably ever need to to enable a row cash and if you are debating whether or not you need to do that. My best answer for you is to talk to the data stacks team.

A

You know there's so many minutiae involved as to whether or not it's a good idea that they're really the only people that are qualified to answer that question. The vast majority of time you're never going to need row. Cash key cash, however, is very important right to cash. The row keys and that's enabled by default in 1.2 and above so you don't really have to worry about that.

A

11 40, okay, I got this is actually the final slide. I wanted to give you guys a demo. Apache shiro has a demo web app and I've already configured it to point to cassandra.

A

Here it is it's on github. You can go to my page check it out. It's literally you get clone you change into the directory fire up cassandra.

A

Then you run mvn jetty, run and you'll see a full web application. That's running on top of cassandra using apache shiro for session clustering.

A

Everything works beautifully. I also did some performance testing on this. I can't again, I don't have time to demo, but it's extraordinarily fast, you can. You can easily handle concurrent operations for a hundred thousand different sessions, um doing nothing but io. I think it only takes like maybe 60 seconds to run a full test. I have eight threads running. Each thread will create a session.

A

Read the session update the session, delete the session and then do another read to make sure that it's gone. It does five of those those query operations against cassandra.

A

Each five of those operations are executed by eight different threads, concurrently, 100 or 100 000 times, and the whole thing takes less than 60 seconds so, like the ability to support hundred thousand or five hundred thousand or a million concurrent sessions is extraordinarily easy. You could probably do it on a single box if your uh your your scheduler was, was efficient enough for the cpu that you had was efficient enough. So um highly recommend doing this storm path. Obviously we run cassandra in production and we do this.

A

We support all the sessions that we have to support with no problem at all so um check out the code. If you have any questions hit me up at uh on twitter or access our website, we are most definitely hiring cassandra engineers if you're interested in a job working with cassandra and storm and real-time processing with kafka, um give us a shout out and and or come talk to me, but that's it I'm out of time. Thank you so much for uh for attending. If you have any questions, maybe I've got time for it.