►
Description
Apache Cassandra is singular in its ability to provide a single logical database dispersed over multi-region datacenters. Cassandra databases can lose entire datacenters due to network outages or natural disasters and the database will remain available for application users. This talk will discuss how Cassandra is able to achieve this feat, and dive deep into the programming realities of distributed systems and consistency models. It turns out that the consistency model is the critical feature of successful distributed systems. When an application and a database can have an honest conversation about the consistency needs of each, a powerful system is born. This honesty is at the core of Cassandra's dominant position in the world of distributed databases.
Matt Kennedy is an architect at DataStax. Matt has been a Cassandra user and occasional contributor since version 0.7 and is a co-organizer of the Cassandra meetup in the Washinton DC area. At the European Cassandra Summit in October of 2013, he was recognized by DataStax as a Cassandra MVP. Matt has been working with distributed systems his entire career and kinda wishes he was one. Because while Cassandra is partition tolerant, Matt is not.
A
Hi,
I'm
matt
kennedy,
I'm
a
partner
architect
at
datastax,
a
little
bit
about
me,
I'm,
not
from
around
here
I'm
based
in
Washington
DC
I'm,
a
co-organizer
of
the
Cassandra
DC
meet
up.
I
am
an
occasional
contributor
to
the
Cassandra
project
and
in
case
it's
not
obvious.
I
am
the
offspring
of
the
unholy
union
of
Zach
Galifianakis
and
Nick
Offerman.
A
A
Let's
just
change
the
words
and
because
you
know,
once
you
really
get
to
understand
the
capabilities
of
Cassandra,
you
start
to
actually
feel
like
there's
something
harmonious
about
how
it
enables
different
kinds
of
consistency
capabilities,
and
so
we
came
up
with
this
harmonious
consistency
idea
and
you
know
I.
It
sounds
a
little
bit
like
marketing
BS,
but
the
idea
is
not
that
this
is
like
some
kind
of
slogan
or
you
know
doublespeak.
The
idea
is
that
this
is
a
framework
for
a
conversation
about
how
we
can
communicate
to
people
that
don't
necessarily
understand
Cassandra.
A
So
cassandra
has
kind
of
an
image
problem
and
forgive
me
because
I'm
going
to
anthropomorphize
here
for
a
little
bit,
but
this
is
kind
of
this
image,
because
Cassandra
is
this
eventually
consistent
database
that
she's
some
kind
of
slacker
college
student
that
gets
around
a
pain
or
mobile
phone
bill
and
you
know
doing
her
term
papers
and
doing
the
laundry
you
know.
Eventually,
when
it's
convenient
and.
A
Unfortunately,
this
is
an
inaccurate
and
really
wrong
impression.
Cassandra
is,
in
fact,
the
world's
greatest
pragmatist,
and,
while
these
rumors
of
her
being
eventually
consistent,
are
true,
they
don't
that
doesn't
really
tell
the
whole
story
right.
There's
there's
much
more
to
this
story
that
really
gives
her
an
unparalleled
worldliness
and
an
unquenchable
thirst
for
data.
A
Speaking
of
unquenchable
thirst
for
data
I
wanted
everybody
in
the
room
to
be
aware
that
there
is
in
fact
a
big
data.
Drinking
game
and
I
did
try
to
avoid
this
talk,
triggering
any
drinking
events
from
the
Big
Data
drinking
game.
So
watch
it
on
video
later
and
tell
me,
if
I'm
honest,
so
to
solve
this
image
problem.
We
in
the
Cassandra
community
can't
just
swap
out
the
term
eventual
for
harmonious
and
call
it
a
day
that
doesn't
do
the
job
of
helping
the
world
to
better
understand
Cassandra.
A
A
The
first
problem
with
labeling,
Cassandra
and
eventually
consistent
database
is
that
it
excludes
other
classes
of
consistency
for
which
it
is
capable
and
we'll
get
to
those
later,
and
I
also
want
to
table
the
discussion
of
what
exactly
consistency
means
for
a
minute.
Let's
look
at
this
word
eventual
so
in
the
context
of
distributed
computing,
eventual
you
know
is
a
term
we
apply
to
the
synchronization
or
timing
patterns
of
a
particular
system.
A
So
like
it
or
not,
these
connotations,
the
negative
connotations
we
inherently
attached
to
the
term,
even
when
we
should
be
thinking
like
logic,
driven
engineers
and
this
behavior
can
lead
to
impress
ice
conclusions
about
the
sort
of
problems
to
which
cassandra
is
suited
and,
worse,
I,
think
inadequate.
Research
can
lead
decision-makers
to
choose
strongly
consistent
systems
simply
because
they
sound
more
rigorous,
so
Cassandra's
eventual
consistency
is
sometimes
regarded
as
a
liability,
which
is
very
strange
because
at
the
same
time
someone
will
criticize
cassander
for
being
eventually
consistent.
A
So
it
makes
little
sense
that
eventual
consistency
should
be
derided
as
a
weakness
when,
in
fact,
the
features
that
are
considered
core
strengths
are
completely
dependent
upon
that.
So
we've
said
consistency
a
lot.
Let's
talk
a
little
bit
about
what
that
means,
because
in
any
cassandra
conference
or
anytime,
people
are
talking
about
Cassandra.
This
word
gets
used,
it's
not
possible
to
avoid
it.
A
So
because
cassandra
is
a
database,
a
lot
of
people
think
oh
well,
we're
talking
about
the
word
consistency.
It
must
be
the
same
thing
as
the
sea
in
acid
right,
because
that's
the
consistency
term
that
people
are
familiar
with
in
the
database
world,
but
it's
actually
not
quite
the
same
definition.
The
seein
acid
has
to
do
with
the
state
of
the
database
being
coherent
at
this.
A
A
It
doesn't
necessarily
matter
to
client
applications
that
all
of
the
nodes
that
make
up
a
database
agree
on
the
state
of
the
data
as
long
as
the
application
has
the
ability
to
insist
that
it
sees
a
coherent
state
of
the
data,
and
that
is
how
we
enable
high
availability
and
multi
region
data
centers
in
Cassandra.
So
this
notion
of
an
application
or
a
client
insisting
to
a
database
that
it
has
preference
on
how
its
data
is
handled
is
sometimes
this.
A
So
by
taking
advantage
of
this
facility,
applications
can
tune
their
consistency,
demands
according
to
the
logical
tolerances
of
the
application,
and
they
can
even
achieve
immediate
consistency
of
their
view
of
the
data
for
right,
then
read
operations
without
sacrificing
high
availability.
This
is
a
strong
distinction
from
strongly
consistent
systems.
So,
more
importantly,
for
multi
data
center
applications,
clients
can
achieve
immediate
consistency
with
their
local
data
center
and
let
the
eventual
consistency
capability
work
on
the
other
data
centers,
so
that
you're
not
paying
Geographic
Layton
sees
for
local,
immediately
consistent
operations.
A
So
how
does
this?
How
does
this
immediate
consistency
work
we'll
go
through
a
detailed
example
in
a
minute,
but
in
order
to
understand
the
concept,
we
need
to
be
aware
of
two
terms
that
are
fundamental
to
Cassandra,
the
replication
factor
and
the
consistency
level,
and
these
are
things
that
are
sometimes
conflated
by
newcomers.
It's
important
to
understand
that
the
replication
factor
is
an
aspect
of
a
key
space.
It's
set
at
the
data
side,
database,
side
level
and
the
consistency
level
is
the
per
client
operation
setting
here
right.
A
So
the
two
relate
to
each
other,
but
they're
not
the
same
thing
and
that's
really
important
to
kind
of
internalize
for
Cassandra.
So
to
look
at
what
immediate
consistency
looks
like
this
is
a
Cassandra
ring
and
we
have
rights
come
into
a
coordinator
node
and
we
successfully
right
to
two
nodes
out
of
our
three
and
four
hour.
A
So
let's
look
at
kind
of
a
crazier
example,
because
I
think
it
helps
to
internalize
exactly
what's
going
on.
So
that's
a
little
washed
out,
but
this
is
our
client.
This
is
our
Cassandra
cluster,
so
we're
going
to
do
a
quorum
right
and
it
gets
to
the
coordinator
node
and
the
coordinator
notes
is
okay.
I'm
the
coordinator
node,
for
this
request.
A
I'm
going
to
send
this
right
to
all
five
of
the
nodes,
but
oh
two
of
them
are
down,
but
that's
okay,
because
I
get
acknowledgments
from
the
other
three
nodes
right
and
so
now
I
got
acknowledgment
from
a
quorum
of
my
nodes
because
a
quorum
of
5
is
3
right.
So
the
coordinator
says
I
saw
an
acknowledgement
of
the
right
from
three
nodes:
okay,
I
can
tell
the
client
that
I
succeeded.
A
So
let's
look
at
the
read
side.
So,
let's
say
some
time
later,
and
this
could
be
very
briefly,
but
we
go
and
read
that
data
that
we
just
wrote,
and
we
see
that
from
the
last
example.
These
three
guys
have
that
data
that
we
want
and
we
want
to
get
it
back
and
we
want
to
make
sure
we
get
the
right
data
back.
But
oh,
we
seem
to
be
having
some
kind
of
serious
problem
with
the
cluster
two
of
the
nodes
that
were
involved
in
that
quorum
right
have
now
failed.
A
So
that's
not
great,
but
what
happens
when
we
attempt
the
read
so
we
get
to
our
coordinator,
node
we're
doing
a
quorum
read
and
he
sends
the
read
request
to
three
of
the
nodes.
Now
two
of
these
guys
don't
know
about
the
data,
yet
they
haven't
gotten.
There
hinted
handoff
replays
yet,
but
we
got
one
guy.
That
does
so.
He
sends
back
the
data,
and
now
our
coordinator
node
says
all
right:
I
see
what
you
see
and
now
I
get
the
other
responses
back
from
the
other
guys,
and
they
don't
have
the
same
data.
A
Maybe
they
have
an
old
ver,
maybe
I
don't
have
it
at
all,
but
because
I
have
achieved
a
quorum.
I
had
an
overlapping
note
from
both
operations.
I
have
a
time
stamp
that
says
this.
Guy
wins
and
now
I
can
return
the
correct
data
to
my
client.
So
this
is
powerful.
The
client,
if
it
chooses
to
never,
has
to
see
an
inconsistent
state
in
the
database
that's
optional.
A
A
Those
nodes
need
to
become
consistent
right,
so
eventual
consistency
helps
us
get
back
to
a
healthy
state
when
we
do
have
failures,
but
we
also,
we
might
not
care
about
immediate
consistency,
and
this
is
especially
true
in
save
Internet
of
Things
applications
where
we
are
doing
a
heavy
right
workload
into
a
database
to
capture
things.
That
are
events
that
are
happening
on
things,
and
we
might
never
read
that
data.
A
A
But
it's
also
worth
talking
about
the
performance
of
eventual
consistency.
How
official
is
eventual-
and
this
is
really
one
of
the
things
that
I
think
inserts
people
right
if
I
have
it
eventually
consistent
database?
Does
that
mean
that
my
data
will
get
replicated
daughter
knows
in
two
hours
to
get
the
switch?
Oh,
so
my
good,
no.
A
A
Netflix
perform
this
experiment,
and
this
was
a
sander
summit
last
year.
I
believe
it's
entitled
eventual
consistency
is
not
equal
hopeful
consistency
and
essentially
what
they
did.
You
know
this
is
important
end
their
big
cassandra
users.
They
run
in
clouds
that
built
necessarily
animals
reliable
and
worse.
A
They
said
we
want
to
know
what
happens
if
we
have
to
cassandra
clusters
in
data
centers
across
the
country
and
we're
going
to
put
a
load
on
we're
going
to
do
fifty
thousand
operations
per
second
and
then
we're
going
to
do
a
million
consistency
level.
One
writes
followed
by
a
million
consistency
level.
One
reads
now:
I
think
everybody
should
go
watch
the
video,
because
it's
pretty
good,
I'm
sure
if
you,
google,
netflix
eventual
consistency,
experiment
before
the
slide
to
go
out.
A
The
link
to
the
talk
is
at
the
bottom,
but
I
want
to
spoil
it,
but
they
were
able
to
read
all
the
data
in
a
very
short
amount
of
time.
So
its
eventual
consistency
is
not
something
that
says
your
data
is
not
going
to
get
where
it
needs
to
be.
Eventual
consistency
allows
us
the
flexibility
to
be
able
to
deal
with
conditions,
and
the
database
knows
how
to
behave
under
those
troubles
from
circumstances.
A
So,
in
our
estimation,
we
wouldn't
really
have
the
ability
to
talk
honestly
about
this
notion
of
harmonious
consistency.
If
is
we're
not
for
Cassandra
2
point
0
having
linear
icicle
consistency
leave
her
just
talked
about
a
little
bit
today.
This
is
the
paxos
capability.
Also
know,
there's
comparison
check
and
said:
datastax,
we
call
lightweight
transactions,
but
essentially
these
are.
These
are
the
cql.
A
A
So
might
not
be
immediately
obvious,
why
that's
useful
so,
let's
walk
through
an
example,
so
suppose,
there's
this
game
called
Monster
Mash,
where
players
create
monsters
that
compete
to
see
who
can
mash
potatoes
the
fastest.
So
the
game
allows
players
to
create
monsters
with
different
attributes
to
help
them
mash
faster.
Each
monster
has
a
name,
but
it
needs
to
be
unique
because
that
saying
that
this
is
the
most
popular
game
in
the
world,
so
it
absolutely
has
to
use
Cassandra
for
its
database.
A
So
now
suppose
two
players,
Alice
and
Bob-
want
to
create
a
monster
named
stinky
right,
so
they've
designed
stinky,
they
figured
out.
You
know
how
stinky
is
going
to
mash
potatoes,
the
fastest
and
now
well,
so
this
is
prior
to
lightweight.
Transactions
will
show
lightweight
transactions
next,
so
they
need
to
check
with
the
database.
Say:
hey
can
I
create
stinky
the
stinky
exists.
They
both
try
to.
They
both
asked
the
database
at
roughly
the
same
time.
Database
says
no
stinky
doesn't
exist,
you
can
go
ahead
and
create
stinky.
A
So
now
Alice
does
a
quorum
insert
to
create
stinky
and
Bob
doesn't
from
insert
to
create
stinky
and
Alice
for
a
second,
their
own
stinky.
She
was
happy
she
got
to
create
her
monster
with
her
great
name,
but
Bob
now
came
along
afterwards
and
squashed
Alice's
monster
so
now,
Bob,
Owens,
stinky
and
Alice
doesn't
know
why
she
can't
see
the
monster
she
just
created.
So
let's
look
at
the
alternative,
so
we've
got
Alice
and
Bob.
They
create
stinky.
The
stinky
exists,
no
insert
stinky
with
a
lightweight
transaction
insert.
A
So
what
alice
is
doing
is
saying,
insert
my
values
into
my
table
where
you
know,
or
if
does
if
this
row
does
not
already
exist
so
insert
stinky.
If
stinky
does
not
exist.
Basically
and
Bob
does
the
same
thing,
he
says
also
insert
stinky
if
stinky
does
not
exist,
but
since
Alice
already
succeeded
with
the
lightweight
insert
for
stinky.
Bob
gets
a
sad
face
and
he
does
not
get
to
name
his
monster
stinky,
but
we
don't
have
the
confusion
created
in
the
prior
example.
So
this
is.
A
A
A
Let's
look
at
some
cool
things
that
you
can
do
because
of
these.
You
know
consistency
capabilities.
So
this
is
a
Cassandra
data
center
data
centers
can
go
offline
at
any
time
for
a
variety
of
reasons,
and
we
need
to
be
able
to
deal
with
the
fact
that
data
centers
go
offline
right.
What
happens
if
I've
got
three
data?
Centers
and
I
lose
one
well,
your
applications
can
immediately
route
to
the
other
data,
centers
and
you're
good
to
go.
A
But
oh
I
was
in
New
York
I
messed
up,
I
changed
my
slides
around,
because
I
didn't
want
the
lightning
bolt
to
be
going
on.
New
York,
while
I'm
in
New
York
I
figured.
That
was
a
bad
omen
but
forgot
to
change
all
the
slides.
So
anyway,
so
let's
say:
we've
lost
a
data
center
and
our
application
is
able
to
go
to
the
other
data
centers,
but
we
maybe
we're
seeing
some
higher
Layton
sees
now
right,
where
we
need
extra
capacity.
A
But
what's
even
more
interesting
with
clouds
is
you
know
we
can
span
cloud
availability
zones,
but
we
can
also
span
cloud
to
cloud
which
is
a
pretty
interesting
capability.
In
case
you
don't
want
to
get
tied
into
a
particular
cloud
vendor,
but
then
the
other
thing
that
seems
to
be
really
popular
is
the
ability
to
go
cloud
to
on
prem.
A
The
consistency
mechanisms
in
Cassandra
make
Cassandra
what
it
is
all
of
her
core
strengths
are
dependent
on
the
consistency
model,
high
availability,
tolerance
and
node
failures,
data
center
failures,
network
outages
and
a
network
outage
doesn't
have
to
be
like
a
transatlantic
cable
cut
right.
It
can
be
there's
a
lot
of
traffic
and
my
ping
times
across
the
country
are
too
high
from
iris.
A
My
request
to
finish
within
the
time
outs
right,
network
partitions,
take
on
all
different
forms
and
having
the
flexibility
deal
with
bad
network
weather
is
really
powerful,
so
essentially
we're
making
the
argument
that
we
can
help
people
understand
the
reality
of
eventual
consistency
as
a
not
necessarily
scary
term.
If
we
can
clearly
explain
everything
that
goes
in
behind
that
right,
and
so
that's
why
I
think
that
harmonious
consistency
is
a
useful
framework
for
that
conversation.