Apache Cassandra Cassandra Day Silicon Valley 2014, 8 May 2014

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Netflix's Distributed Computing Strategies: Optimistic Design for the Eventual Consistency Model

Description

See the full post here: http://www.hakkalabs.co/articles/netflixs-optimistic-design-for-eventual-consistency-model

In this talk, Christos Kalantzis (Cloud Persistence Engineering Manager, Netflix), explains why he believes companies should adopt an Optimistic Software Design model that is similar to what is used at Netflix. This is a practical talk with strategies for implementing the design model, testing for eventual consistency, and convincing your organization that it's the right move.

This talk was given at Cassandra Day Silicon Valley 2014.

ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.

FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520

A

So I thought me to something like we're: gonna talk about eventual consistency which is the and why you know race, optimistic design. You know so be moral discussion.

A

My name is christos.

A

Pretty much anything that's we extend.

A

So this is a quick recap: replication and consistency onset, so the sender's and eventually consistent. This means you're. All your data will get there eventually and eventually is Noriko. Second, in almost all cases, it is milliseconds, it's not micro seconds from so if for those who've ever used.

A

You know you'll see, sometimes you know seconds or even if you seven counting the.

A

Senators not like, but it's never going to take very long to get replication.

A

Now that being said, eccentric does have time to get assistance. If you say.

A

There's more quorum is take a replication factor divided by two plus one. So you know at least half of all.

A

Okay, we're getting better. You can also do a replication factor.

A

That's that's really. No one people say people or is sweet spot. It's really not you're. Still waiting for these two nodes to respond.

A

So during the year in 2000 about rights going to a master, and then you scale down your reads with with some slave databases.

A

This was good because you're able to scale out breeds scaling and especially signals the Internet of Things website wanting to millions of things a second, it's the right scale, really really looking.

A

But a lot of people.

A

Even when we had that single master, the replication for the slaves always wasn't always guaranteed I've seen many of a slave didn't get a particular less, and you know you have this confidence that your slaves got it. They did in the industry playing States for whatever reason: Network pickups, whatever you can have steak and get lost, and now you've got slaves which are inconsistent with your master.

A

Well, Cassandra this coming to Center to, but Cassandra at least can repair this cluster. Now.

B

What that does is.

A

It will now start comparing records.

A

So, even if you've got.

A

B

A

Out there to do.

A

A

An iron of PAS to build those kind of tools, Zappos. Does that iStockphoto Samantha you send it.

A

They also, they also have this.

A

A

But yeah some people, that's what they'll do to just replay on that and to the slave as opposed to trusting that but nothing school. And then the point is that even.

A

A

Everyone says: I want consistency with my reason: rights just like I haven't like bnsf. Well, we saw before that there was no guarantees so so having a single master, multiple slaves, whether you go through internally, there's always a way of possibly losing I, want my DVD to catch integrity issues. So I'm going to use Ruby on Rails here or has used it before rails.

A

If you actually look so they also don't know cradles Grails. These are all MVC frameworks that allow you to prototype monopolies, quick and, as we all know, prototypes and going to production, and then you got to scale them.

A

The issue would not the issue, but if you actually look at the code generator underneath there are creating any forms and as a matter.

C

Of fact, one of the tricks.

A

I used to use DBA was, you know, remove foreign keys because then you're not checking the referential integrity you're increasing through yes, you know one of the problems. Is you don't catch.

A

The site fast, as opposed to you, know trying for perfection and the other question is, can I trust that the sender will replicate my data even when I'm writing a scale of 1. So let's address that last. So we didn't experiment actually do go back there of the appendix, so we created a multimedia Center cluster at the time it was one one seven and it was 48 nodes in each.

A

Read operations per second, when I mean biggest center, I mean so it give us easy to us was to so we wrote a million records and one of the data centers and what we knew the blue records. We read them.

A

A

Interestingly enough, we saw all our million records there. I shouldn't have seen there. The point I'm trying to make is there will be a small percentage that will make it over to the other side and that's ok, that's what we're going to talk, but in our particular case all the records got coffee. On the other side, this was and then again we were trying to need any failure scenarios. It was, you know both healthy.

A

So so you can trust the center of that the majority. In my case, all the majority of your.

A

So, even if my million records didn't make it to the other side, let's say.

B

A

B

A

Let's say 0.1% didn't make it on the other side or 0.1%. So what's gonna, be the knee jerk reaction. Oh increase the consistency, because I want my stuff to be well.

A

What would that do is immediately slow down your rights.

A

If you really want to be consistent across multiple data, centers you're, not even going to do core across the rim and.

A

That's how gonna be not in the microseconds, but possibly in.

B

The honey microseconds.

A

If not one second, if you're, if you need to go across a wire to another data set, you really want to polish 99% of your users and their experience for that one percent.

A

It's nothing for me! It's unfathomable! Then you want to punish your users that indeed 99% of your users for Mitch cases. So so you know, higher consistency is, is higher latency and obviously diminished user experience when, when, when you're striving for higher resistance.

A

In an optimistic design, I want to I'm going to trust that my data stores trust like a Sandra to get the majority of everything on the other side and then nothin, and if, even if it doesn't it eventually will, because it has built-in tools to then repair. The cluster.

A

Notes to make sure what is there, what is the latest record? Oh I, don't have that I'm not going to get into the mechanics, but it's there it's very it's very sophisticated and, as I've studied a lot other day of stores, I'm surprised how well how well it works at the center.

A

So trust me we'll get there eventually, no business application always ask yourself. Is it really that important now I'm, not here to say, forget it? Consistency is not good. There will be use cases. You know your app, you know your product managers, you know, you know your use case. You need to make that judgment. Whatever we want to get to here is don't need your two full consistency think about their problem, so you can use it so.

A

As you know, you better play this right, so you add movies to your what we used to call it. You now call the play list and used to your list and.

A

Well, what happened is one percent of the time when you chose that and will be to your list. It wasn't there you'll refreshing and it will be fun right, so that happened. One percent of the time would you cancel your Netflix subscription for that anyone know.

A

That important, so so in that particular use case I can afford a little more inconsistency there and you know next time I'm ever maybe he reads this record that there's also these two weeks in December called read, prepare so so a lot of times that you read records, Italy circuitously, fix stuff. In the background or there's an annual repairs, you could run on a specified interval and that's what we mean by eventually eventual isn't just the fact that it's asynchronous.

A

It's also the fact that there are tools within database to get the whole clusters consistent again. So is it that important you only you that only you can and alleged cases through contingency plans and not all the entities. This contingency plans need to be covered. So let me give you an example: Amazon Amazon, so some people would say: hey, Norman retail.

A

You know my inventory system has to be consistent, like I, can't not give the right amount of widgets that sell well, Amazon, believe it or not, has a bracelet eventual consistency, even in their inventory system. Sometimes.

B

Amazon will sell you.

A

Something that they don't actually have an inventory, because at that moment you you were that one percent that bought but along the water- and it wasn't any good. But what does Amazon do they say very next? Polite email, hey sorry who sold you this, but we don't have it anymore, oh and by the way, here's the 10% credit towards your next purchase. So even in retail, through a contingency.

B

A

Necessarily through code, you can handle these kinds of use cases and, in.

D

Some cases you can actually.

A

Make money with the division of assistance? Thanks banks have been, and eventually consistent since the beginning of time. Anyone ever write a check. You write your paycheck. You cited there's no code there to make sure that the money is in your ear in your Italian and so on. Give.

D

It check to somebody.

A

And one in two things will happen immediately. The optimistic thing is I believe the check and the check representative money, your widget or your service, or or whatever, or thank you for your point and but the check events. If there's not enough funds in the account, this is mrs. eventual consistency and where's the bank do: will they actually chargeable people over the back of the check bounced.

C

Actually, that's true, that's true, because, because.

A

So why they were ferment for a long time. I worked in the bank when I was studying through college back in the morning and every bank ever every branch had it's only been three, so all that all transactions were there at the end of the night or batches sent out really really checking. If even the money we gave out.

A

That's why sometimes bank accounts are negative because they will give money out without necessarily without necessarily making sure at a hundred percent consistency level that there's enough funds I have I have seen bank accounts for to my account too sometimes.

A

So the hurdles you'll face, so you know now I'm giving you you know, hopefully apprentice you up on eventual consistency and I've sold you on it. You know what are the hurdles you're going to face? Some of the hurdles are gonna face. Are that you know try something that to an engineer, an engineer. He knows one thing: one plus one equals two, not eventually two so a lot of times, even in systems where it's not that important. You know the engineers like how much papa you know I've gotten to the test. Here it doesn't equal.

A

What I need it to equal that it's failed and it doesn't get deployed. So so engineers I lovingly include leather, stubborn, I'm, an engineer as well, but.

B

You're gonna have.

A

To sell them on it, they're gonna have to adopt new philosophies to the ways of thinking about our technical applications. The.

B

A

Look a lot of people, you know.

C

A

Organizations you know management is graded on uptime and an error that low error rates and stuff like that they're scared is. This is what I'm grading on so so they're going to be often be roadblocks to be more open to adopting the more optimistic design and eventual consistency.

A

Engage in product non-engineers to get this better than engineers, especially the use case with with the list right by like convincing a product manager that paid one percent of the time you might get at this, but at the next refresher within a couple hours it'll all be fixed, but that'll do 90% of all the other users, this great experience, because it can be super fast and so on for the best. That's that's a no-brainer! That's that's again! Let's do it! You know what percent we can add to that.

A

So, overcoming those hurdles PLC's show them that show them the percentage of time. It'll actually it'll actually go wrong. You know we did one. We did one. We showed that there was a hundred percent of the right Scott over there. It's unfortunate that that happened. I would prefer not to percent get there to prove my point, but I Sandra is fairly resilient and will replicate.

A

You know we're with the pianist to show the benefit to the user, because at the end, every company's moving to this thing, where it's all about the user experience even Microsoft this week, everyone see what they were talking about the last week and Bill they, you know everybody's, getting it's about the interviews. We want to give them the great experience, the best experience possible and, of course, disco if you're working for the wrong.

D

D

Questions yes,.

D

A

Because some clusters would embrace it.

D

A

D

Self-Service between equal one, you can leave the devices in the water. Well, that's the consistency. Yeah.

D

And then and then clean up so that they thought would be seat by.

A

So so the question is correct: you're ready, that's sealed and that no one goes there, but with your records so see what doesn't, he only were writing to one. No, obviously, it's always writing to all your don't waste. It's just waiting for that first note to come back and acknowledge it up, right and and then and then the application moves spoke. So I trust, I trust that Sandra will actually have. That will has has sent that data to more than one place, and so, even if I was that, no other reads will end.

A

Remember the replicas other other nodes in the replicas said.

D

D

We couldn't be any obviously knows yes, but you can read them to know that you Johnny yeah you'll, get that motivate. Are you.

A

Gonna stay out behind it up most reason: okay, yeah, that's right, that's very possible and that's where you need to understand your application can absorb, tended absorb them under sales, so anybody would have been living with the exam alive at the place. You just removed you just you just added something to your playlist. So the worst case scenario there you won't get. You know, Avengers your list until the whole repair stuff, hanger application in that.

B

A

A

So on that particular note and that's very possible, can your application you know so.

B

I must say: never.

D

A

I'm just saying analyze, your application see what the worst case scenario is and then I just start from the bottom up, as opposed to going from fully consistent.

D

Well then, it's gone to.

A

Three other.

A

Well, they might not have enough time, it's it doesn't go to. One then replicate to others. They would say it's a coordinator that coordinator sends it to all everyone's synchronous it just it's.

D

D

A

Our right pattern is you go to the coordinator, you give it a right, it's going to send it to everyone when the consistency level does is you're defining how many nodes need to respond before I can then go back to the application and declare that the right was successful.