►
Description
Speaker: Christos Kalantzis, Engineering Manager of Cloud Persistence Engineering at Netflix
Slides: http://www.slideshare.net/planetcassandra/c-summit-2013-eventual-consistency-hopeful-consistency-by-christos-kalantzis
This session will address Cassandra's tunable consistency model and cover how developers and companies should adopt a more Optimistic Software Design model.
A
Well
this
afternoon,
we'll
change
pace
a
little
bit
and
have
more
of
a
philosophical
discussion
about
you
know,
embracing
optimistic
design
in
your
persistence
layer,
which
I
called
eventual
consistency,
doesn't
equal
hopeful
consistency.
So
who
am
I?
My
name
is
christos
kalansis
I
work
in
netflix
I
manage
the
cloud
persistence,
engineering
team,
which
is
a
fancy
word
for
the
cassandra
team.
My
twitter
handles
chris
callan.
A
A
A
You
can
do
all
which,
in
a
replication
factor
of
3
means
all
three
nodes
need
to
get
the
the
right
or,
if
you're
reading
you
have
to
read
from
all
three
nodes:
that's
not
that
great
on
latencies
quorum
quorum
has
kind
of
been
accepted
as
the
de
facto
you
know
standard
it's.
You
know
good
balance
between
availability
and
latencies.
Personally,
I
think
it's
meh.
A
So
let's
take
a
trip
back
in
time
back
in
the
early
2000s,
we
had
one
master
database
and
you
know
we
needed
to
scale
reads
out.
So
what
did
we
do?
We,
you
know
we
we
slay,
we
had
a
bunch
of
slaves,
either
piggybacking
or
replicating
right
off.
The
master,
as
you
can
see
in
the
diagram
rights
went
to
the
master
updates,
went
to
the
slave.
An
application
would
read
from
the
slaves
and
write
from
the
master.
Well,
that's
an
eventual
con,
eventually
consistent
architecture.
A
Sometimes
we
didn't
even
know
it.
There
wasn't
a
repair
function,
so
I
tried
that
yesterday,
in
the
latest
version
of
my
sequel
and
repair
database
didn't
work.
So
it
didn't.
It
didn't
repair
my
cluster
of
masters
and
slaves,
so
we
trusted
it
and
there's
a
lot
of
big
name
companies
out
there
that
still
trust
it.
A
I
want
high
consistency
in
my
reads
and
writes
just
like
I
have
in
my
rdbms.
You
know
master
slave
cluster.
Well,
you
know
you
never
really
had
it.
So,
if
you're
concerned
about
it
in
cassandra,
you
should
really
be
concerned
about
it
in
your
current
architecture,
want
my
db
to
catch
integrity
issues.
A
As
a
matter
of
fact,
modern
mvc
frameworks
like
rails
and
grails
when
you
use
their
internal
object,
domain
creation
tools,
they're,
not
creating
foreign
keys,
even
if
you're
creating
some
kind
of
relation
between
the
objects
and
the
final
one
which,
which
is
one
of
my
favorite
and
we'll
you
know
touch
upon
for
a
little
bit
is.
Can
I
trust
cassandra
that
cassandra
will
replicate
my
data
when
writing
at
cl1?
A
Well,
I
had
that
question
asked
about
a
month
ago.
Actually,
within
netflix
one
of
the
application
designers
said,
you
know,
I
write
a
quorum
and,
and
I
read
a
quorum
because
do
I
really
trust
it
and
in
a
replication
factor
of
three?
Would
you
know
if
I
write
at
one?
Will
it
actually
go
to
to
all
three
replicas?
A
A
So
you
know
I
wasn't
always
hitting
just
one
note,
so
you
can
trust
it,
and
so
well,
no
test
is
is
good
if
you
only
run
it
once
so,
we
ran
it
five
times
and
each
time
we
had
the
same
result,
all
records
were
read
back
successfully,
so
we
can
trust
it
so
now
we
know
we
can
trust
cassandra.
Now
we
know
that
you
know
our
current
architectures
aren't
as
as
high
a
high
consistency
as
as
we
thought
they
were.
A
That's
that's
a
service
oriented
architecture
where
one
web
page
may
call
five
or
six
services
and
each
one
is
making
a
database
call
underneath
so
embrace
optimistic
design,
trust
your
data
store.
I
just
showed
you
that
cassandra
can
be
trusted,
know
your
business
and
your
application.
I
mean
yes,
okay.
This
is
not
the
answer
for
every
application,
but
think
about
it.
Can
what
can
your
application
live
without?
A
You
know.
High
consistency
level
in
your
reads
and
rights,
I'll
give
you
an
example.
So
netflix,
as
you
know,
you
can
watch
a
movie,
you
can
pause,
go
to
another
device
and
you
know
continue
where
you
left
off
and
that's
being
done
in
the
background
by
sending
by
sending
signals
to
a
cassandra
database
and
we're
actually
writing
at
a
consistency
level
of
one
99.99
of
the
time.
It's
fine.
Sometimes
it's
not
sometimes
the
movie
is
going
to
start
five
minutes
earlier,
because
that
may
be
the
last
the
last
ping
we
recorded
into
the
database.
A
Well,
we
know
our
business,
so
we
don't
believe
I
mean
who
in
this
you
know
audience
is
going
to
dump
their
netflix
subscription
because
once
in
a
million
you
know
you,
you
resumed
on
another
device
and
it
was
a
couple
of
minutes
earlier
and
not
exactly
where
you
left
it.
Well,
I
don't,
I
don't
think
anyone
and-
and
if
you
do
maybe
you
should
be
our
customer.
A
So
really
you
need
to
know
your
application.
You
need
to
know
what
problem
you're
actually
solving,
and
you
know
and
in
certain
cases
you're
going
to
have
to
handle
edge
cases
through
contingency
plans,
and
I've
got
a
couple
of
examples
of
of
companies
that
aren't
netflix
and
how
they
handle
contingency
plans
in
a
in
a
in
a
eventual,
consistent
architecture.
A
Sometimes
you
buy
something
on
amazon
and
it's
just
not
not
there.
It's
not.
It
says
it's
in
stock,
but
it
was
it's
not.
You
know
when
they
go
fulfill
the
order.
You
know
either
the
robots
in
the
warehouse
can't
find
it
or
or
or
if
it's
a
warehouse
with
humans,
the
human
can't
find
it,
and
so
what
do
they
do?
They
cancel
the
order.
A
They
cancel
the
order
and
their
contingency
plan
is
they'll
credit
you
10
towards
a
future
purchase.
Okay,
I
mean
it
kind
of
sucks.
I
wanted
it,
you
know
we're
in
the
generation
that
wants
everything
right
away,
but
you
know
it
happens
and
you
know
they
they
not
reward
us,
but
you
know
they
have
a
nice
contingency
plan
when
that
happens,
so
it
doesn't
hurt
that
much.
A
So,
okay,
here's
rita,
that's
an
example
of
retail
being
able
to
live
in
a
low
consistency,
low
consistency
architecture,
but
what
about
financial
I
mean?
Surely
that
needs
to
be.
You
know
high
highly
consistent
right,
I
mean
you
know
eventually
consistent,
doesn't
work
in
the
financial
world.
Well,
banks
are
the
most
eventually
consistent
system
out
there
you
know
I
can.
I
can
write
a
check.
Actually,
I'm
gonna
write
you
a
check
for
a
million
dollars,
okay,
one
or
two
things.
A
A
A
A
You
know
how
consistent
you
need
to
make
your
application
so
you're,
going
to
go
back
to
your
company
and
you're,
going
to
face
hurdles
right,
you're,
going
to
be
a
little
pawn
against
the
whole
chessboard
of
of
engineers
and
managers
who
who
probably
think
you
you
drank
something
silly
at
the
data
stacks
conference,
and
you
know
you
don't
know
what
you're
talking
about.
Well,
engineers
are
stubborn.
You
know
one
plus
one
equals
two,
not
eventually
two
so
you're
gonna
have
to
manage
down.
A
Middle
management
is
scared.
I
mean
you
know
good
on
you
for
those
using
cassandra
that
you
convinced
them
to
even
be
using
cassandra.
Now
you
know
you're
gonna
have
to
convince
them
to
to
accept
a
low
low,
consistent,
low
consistency.
Architecture.
That's
not
gonna
be
easy.
You're
gonna
have
to
manage
up
as
well
you're
going
to
have
to
engage
the
product
team
a
lot
of
engineers
they
like
give
me
the
specs
go
away
I'll,
build
it
for
you
and
you
know
just
hand
a
nice.
A
A
Wow.
We
really
breezed
through
this.
So
how
to
overcome
these
hurdles,
you
can
prove
it
through
a
poc
that'll
help
manage
the
downstream
that'll
help
manage
the
engineers
rupa,
and
I
the
db
engineer
that
run
that
test.
We
really,
you
know,
we
really
changed
one
of
those
engineering
managers
minds
by
by
showing
him
the
results
of
that
test
and
they
accepted
to
reduce
their
their
consistency
level.
A
You
need
to
show
the
benefits
of
improved
user
experience.
I
mean
if
it's
going
to
take
five
seconds
to
load
a
page.
You
know
telling
telling
a
product
guy
hey.
I
can
cut
that
latency
in
half.
Well,
you
know
they're
going
to
be
happy
about
that
and
and
say,
okay
cool.
Let's
do
it
and
finally,
you
may
be
working
for
the
wrong
company.
A
Netflix
is
hiring
jobs.netflix.com
and-
and
you
know,
I'm
looking
for
devops
engineers
if
you're,
if
you're
such
a
person
come
see
me
after
the
session
and
and
we'll
have
a
chat,
so
we
really
blew
through
this.
So
I
want
to
open
this
up
to
questions
and
and
whoever
I
pick
please
please
go
to
the
microphone.
A
We
were
like,
I
said
the
de
facto
has
been
quorum
right,
so
people
have
written
and
read
in
quorum
so
quorum
for
those
who
don't
know
when
you,
when
you
read
or
write
quorum,
you're
writing
to
two
nodes
and
you're
or
you're
reading
from
two
nodes
and
you're
waiting
for
a
coordinator
to
actually
finish
writing
to
those
two
nodes
before
it
releases
the
write
or
it's
reading
from
two
nodes,
then
then
really
they're
really
checking
which
one's
got
the
latest
timestamp
and
sending
you
back.
So
we
cut
it
by
two.
A
B
A
C
A
A
I
think
there's
new
features
in
in
1.2
that
can
catch
possible,
dropped,
drop,
replication
or
statistics
on
that.
However,
there's
a
repair,
if
you
just
want
to
blindly
you
know,
heal
the
whole
cluster.
If
you
think
you've
had
dropped
operations,
there's
a
repair
command
in
cassandra,
which
will
then
make
sure
everyone's
got
all
the
data
they
need
to
thanks.
Thank
you.
A
Yes,
absolutely
so,
and
that
ties
into
that
repair
command
as
well,
so
we
did,
we
did
kill
some
nodes
while
we
were,
we
were
doing
that
test
and
I
think
that
was
in
our
third
or
fourth
run,
that
that
we
killed
some
nodes
and
we
still
continued
reading
from
it.
However,
after
an
event
like
that,
we
will
run
a
repair
at
that
point.
A
A
B
A
F
A
Yeah,
so
so
the
replication
factor
of
the
cluster
was
three:
we
wrote,
it
got
replicated
successfully
local
locally
and
at
the
other
data
center,
and
the
reads
happened
at
the
other
data
center
and
we
were
randomly
hitting
nodes.
So
it's
not
like
we
were.
We
chose
the
node
that
should
have
that
token.
We
just
you
know
we
just
let
the
coordinator
do
its
thing
and
randomly
chose
nodes
and
we
read
everything
back.
Yes,.
F
We
run
some
clusters
extended
clusters
and
I
noticed
that
they're
always
a
problem.
When
you
do
repair,
it
takes
a
very
long
time
to
repair
that
cluster.
We
have
the
16
casino
nodes.
So
would
that
be
a
case
that
we
need
to
consider
when
we
run
accenture
clusters,
for
example,
for
the
repair
situations?
A
D
A
That's
in
the
point:
zero,
zero
one.
You
know
off
that
that
you
know
a
particular
right
didn't
make
it
in
so
repairs.
Take
a
long
time.
Yes,
absolutely
repairs
do
take
a
long
time,
but
you
know
it's
it's
that
safety
net.
It's
that
absolute
okay!
You
know
we're
drawing
a
line
from
now
on.
It's
it's
repaired
and
everything's
everywhere.
G
A
E
I
I
E
D
H
So
I
like
this
idea
of
having
the
applications
compensate
essentially
for
inconsistency,
but
I'm
curious
what
you
would
recommend
in
terms
of
strategies,
both
from
a
operational
perspective
and
when
you're
talking
to
application
developers
about
detecting
and
reasoning
about
these
inconsistencies.
For
instance,
do
people
say
well
as
long
as
I
have
an
sla
of
99.9,
that's
good
enough
or
do
they
need
to,
or
do
they
often
ask
for
maybe
like
well,
I
mean
one
aspect
of
optimism
is
a
is
a
callback
or
a
notification
when,
when
things
go
wrong,
yeah
I'm
curious.
A
It's
kind
of
annoying
to
give
you
the
answer
and
say
it
was
the
wrong
answer.
Absolutely
so
I'm
going
to
give
you
the
cop-out
answer.
It
depends
on
your
application,
but
I'll
give
you
an
example.
So
you
know
in
certain
things
we
might
not
catch
it
in
the
application,
we'll
catch
it
by
customer
service
calls-
and
you
know
at
that
point:
it's
it's
a
human
contingency
plan.
It's
not
necessarily
code
that
that's
doing
it.
There's
there's
going
to
be.
You
know
a
combination
of
both
depending
on
your
application.
A
Leverage,
so
cassandra
does
have
mechanisms
actually
to
fix
it.
There's
read
repair
sure,
so
you
can
turn
on
read
repair.
So
if,
if,
if
you
return
it
once
and
then
read,
repair
runs
and
and
depending
on
what
percentage
of
repair
you
have
on
second
time,
you
read
it,
it
will
actually
be
repaired.
Thanks
you're
welcome.
F
So
here
we're
talking
about
the
consistent
level
of
cassandra,
but
we
always
face
the
case
where
we
have
different
systems.
For
example,
we
want
to
have
a
search
over
the
data,
so
we
have
some
index
we
create
built
on
top
of
data.
So
how
do
you
guarantee
the
consistencies
between
these
different
systems?
Because
that's
a
problem
I'm
facing
for
our
gotcha
group
and
I
think
netflix
provide
the
same
like
search
infrastructure
on
top
of
the
sender,
yeah.