►
From YouTube: Coursera: Coursera's Adoption of Cassandra
Description
Speaker: Daniel Chia, Software Engineer
Like many startups, Coursera began its data storage journey with MySQL, a familiar and industry-proven database. As Coursera's user base grew from several thousand to many millions, we found that MySQL provided limited availability and restricted our ability to scale easily. New product initiatives and requirements provided a perfect opportunity to revisit our choice of core workhorse database.
After evaluating several NoSQL databases, including MongoDB, DynamoDB and HBase, we elected to transition to Cassandra . Cassandra's relative maturity, masterless architecture (for availability), tunable consistency, and stable low-latency performance made it a clear winner for our needs.
Learn more about what it takes to transition from SQL to Cassandra in this talk.
A
Hi
everyone
I'm
going
to
talk
about
casares
adoption
of
Cassandra,
so
to
start
off.
Let
me
just
tell
you
a
little
bit
about
myself,
I'm
daniel.
I
work
at
Coursera
as
a
software
engineer
on
the
infrastructure,
team
and
I
mainly
build
core
services,
as
well
as
data
access
layers,
and
we
also
take
care
of
the
databases
and
service
architecture,
of
course
Sarah.
So
what
I'll
be
covering
today
is
I'll.
Do
a
brief
introduction
of
what
Coursera
does
just
so
that
you
understand
what
our
data
needs
at
use.
A
So
Coursera
is
a
company
that
aims
to
provide
universal
access
to
walls,
best
education,
and
we
do
this
through
a
combination
of
lectures,
quizzes
discussions,
programming
assignments
and
peer
review,
and
we
are
available
on
web
iOS
and
Android.
So
we
are
triple
platform
web
service,
and
what
do
you
want
from
our
database
is
the
following
features.
A
First,
is
we
want
consistently
fast
Layton
sees
from
our
database
here
and
the
reason
is
on
any
single
page
to
provide
a
full
and
rich
learning
experience.
We
actually
need
to
do
many,
many
queries
so,
for
example,
on
this
page
where
we
are
showing
the
lecture,
video
they're,
actually
many
components
that
need
to
be
retrieved
before
we
can
successfully
serve
this
page
to
the
user.
For
example,
in
the
upper
left
hand
side
you
can
see,
we
have
the
structure
of
the
course
that
says
this
is
week
one,
and
this
is
a
lecture
name.
A
We
need
to
retrieve
basically
information
about
the
course
and
then
to
show
the
actual
lecture.
Video
need
to
make
another
metadata
call
to
figure
out.
Where
is
this
lecture
for
this
we're?
Sorry?
Where
is
the
video
file
for
this
lecture
where's
it
stored
with
CDN
to
use
and
then
just
below
this,
the
start
of
the
discussions
tab?
So
we
need
to
retrieve
discussions
relevant
to
this
particular
lecture
item
and,
in
the
upper
hand,
side.
A
This
is
actually
the
users
overall
progress
in
the
course,
and
this
consists
of
many
different
items
that
are
then
aggregated
together
in
a
different
query.
So,
even
though
just
a
simple
looking
page
like
this,
we've
made
at
least
like
five
or
six
separate
sub
component
calls
and
if
any
of
these
components
start
to
slow
down
that
can
start
to
affect
the
user
experience,
because
the
page
will
load
slower
and
so
having
a
database
where
we
can
get
consistently
fast
queries,
it's
actually
very
crucial
to
us
to
achieve
a
good
user
experience.
A
The
second
thing
we
care
about
its
availability,
so
for
a
company
of
our
size
of
about
200,
engineer,
200
employees,
it
turns
out
that
we
actually
have
a
pretty
high
proportion
of
users
that
are
not
from
the
US,
in
fact
over.
Seventy
percent
of
our
users
that
use
the
site
actively
every
month
outside
the
US
and
as
such,
there
isn't
really
a
good
time
where
we
have
like
a
blackout
period
where
we
can
just
say
it
like.
Okay,
we're
going
to
go
down
for
like
an
hour
to
do
upgrades
like
any
other
Internet
company.
A
We
are
basically
a
24-7
type
web
servers,
so
Cassandra
or
any
database
that
we
use
ideally
will
be
able
to
give
us
the
flexibility
to
do
operations
through
migrations
all
without
taking
down
time
thoroughly.
We
care
about
scalability.
As
time
goes
on,
we
acquire
more
users,
we
acquire
more
content.
We
enrich
the
site
with
more
features
such
as
tracking
your
learning
tracking
discussions,
letting
people
avoid
discussions
and
all
these
things
have
a
tangible
cost
in
terms
of
database
performance,
and
so
we
want
a
database
that
can
scale
with
our
usage.
A
A
A
So
increasingly,
this
became
a
problem
because
someone
would
deploy
a
piece
of
code
that
has
a
really
bad
query
in
it
and
you
Anthony
start
calling
it
a
lot
and
the
database
will
go
down
and
we'll
spend
a
lot
of
time
figuring
out
why
this
disk
we
banned.
How
can
we
fix
it?
So
that's
not
quite
ideal,
but
that's
not
the
only
problem.
There
are
two
ways
to
scale
my
sequel
one
is
you
keep
getting
a
bigger
box
and
then
eventually
your
ball.
A
This
is
something
that
other
databases
can
do
transparently,
but
not
a
sequel
like
database,
and
the
last
limitation
is
that
because
sequel
databases,
a
single
master,
every
time
you
have
to
do
a
failover
involves
taking
down
time
and
that
usually
factories
between
one
to
five
minutes,
depending
on
whether
your
databases
set
up
across
different
data
centers
and
things
like
that.
So
it's
not
very
ideal,
as
you
run
more
machines,
because
every
time
you
have
to
do
an
upgrade
or
to
do
a
table
change,
you
might
result
in
downtime.
A
So
eventually
we
decided
that
we
wanted
to
try
Cassandra,
so
Cassandra
column
on
model
appeal
to
us,
because
it
was
pretty
flexible
in
particular,
is
tunable.
Consistency
was
a
very
good
feature
for
us.
A
lot
of
no
sequel
stores,
don't
have
a
very
good
story
around
consistency,
for
example.
Some
of
them
are
just
eventually
consistent
and
you
cannot
control
what
kind
of
consistency
or
getting.
This
is
a
little
bit
problematic
for
some
of
our
use
cases,
because
there's
an
educational
company.
A
We
set
people's
homework
submissions
effectively
and
I'm,
not
sure
you
ever
had
your
homework
lost
before.
But
people
get
really
angry
when
you
lose
a
submission
so
having
the
choice
to
be
consistent
or
not,
is
actually
very
important
for
us,
cassandra
is
fast
and,
as
I
mentioned
earlier,
having
a
consistent
performance
is
very
critical
for
us
and
it's
horizontally
scalable,
and
the
last
thing
to
not
overlook
is
that
cassandra
has
a
good
community
like.
A
At
some
point,
you're
going
to
run
into
this
weird
edge
case
situation
and
with
my
sequel,
if
you
go
online,
you'll
probably
find
a
solution
to
it,
and
the
same
thing
probably
holds
true
for
Cassandra,
which
is
great
because
otherwise
you
might
have
to
start
reading
the
code
and
digging
deep
into
internals
and
that's
not
really
a
situation.
You
want
to
find
yourself
in
very
often
so.
A
So
some
of
the
initial
pain
points
we
face
is
that
Cassandra
doesn't
let
you
execute
arbitrary
queries
in
my
sequel?
You
can
filter,
you
can
soar,
even
if
you
don't
have
an
index
on
the
column
seco.
I
happily
do
a
full
table
scan
for
you
to
get
back
all
your
data,
not
that
I
recommend
doing
this
for
production
use
cases
or
it
turns
out.
A
C
A
Basically
are
restricted
to
reading
a
partition
you
can
scan
over
the
whole
database
or
you
can
read
a
partition
that
isn't
really
in
middle
ground,
where
you
can
ask
the
database
to
aggregate
things
for
you.
It's
changing
a
little
bit
with
user-defined
functions,
but
at
a
high
cassandra
is
designed
to
be
an
OLTP
database.
So
it's
designed
to
touch
very,
very
few
roles
at
one
time
and
so
not
being
able
to
use
it
as
an
OLAP
database
can
be
a
little
problematic
give
your
company's
smaller,
and
your
data
is
in
a
medium
size.
A
Scale
where
you
could
probably
get
away
with
using
my
sequel
is
an
OLAP
database
for
some
of
your
dashboards
inquiries,
because
in
Cassandra
you
no
longer
have
this
option
and
you
have
to
resort
to
what
it
means
like
spark
or
loading
it
into
a
different
database,
for
example.
So
this
was
a
little
bit
of
a
mindset
shift
for
people
too,
and
the
last
one
is
developers
who
are
worried
about
this
term.
Eventual
consistency
to
call
someone
else.
People
often
confuse
the
eventual
consistency
with
hopeful
consistency.
A
We're
hopeful
means
that
at
some
point
my
data
may
or
may
not
show
up.
Cassandra
is
much
better
than
that
when
they
eventually
is
that
the
data
will
get
there.
It's
just
that
it
might
take
a
little
bit
of
time
depending
off
if
there
are
no
failures
and
weird
Network
scenarios
like
that,
but
in
most
cases,
if
you
are
reading
from
coram
writing
from
coram,
unless
you
have
machine
failures
or
weird
network
partitions
you're
going
to
get
back
the
latest
data.
So
this
really
isn't
a
problem.
A
It's
more
about
educating
people
that
this
is
not
what
you
think
it
is
and
getting
them
to
understand
and
be
comfortable
with
that.
So
this
is
something
that
I've
been
alluding
to
in
the
previous
slide.
But
basically
cassandra
is
a
no
sequel
database,
but
a
lot
of
people
are
used
to
using
a
sequel
database
and
they're,
inherently
different
from
the
top
to
the
bottom.
So
you
need
to
start
designing
your
application
differently.
You
need
to
model
your
data
differently
and
even
the
way
you
do.
A
Your
queries
and
some
of
the
use
cases
will
be
handled
differently
and
so
for
a
lot
of
people
who
are
used
to
using
sequel.
This
requires
a
pretty
big
mindset
shift
because
they
no
longer
can
use
the
patterns
that
they're
used
to
and
have
become
accustomed
to
or
have
learned
in
school.
So
in
particular
what
was
very
helpful
for
us
is
we
had
a
core
group
of
people
who
had
very
good
Cassandra
expertise,
not
just
on
the
operations
side,
but
also
on
how
to
model
your
data.
A
In
particular,
we
knew
how
all
student
data
models
might
affect
performance
on
the
cluster,
as
well
as
how
student
data
models
are
considered.
Anti-Patterns,
for
example,
it's
really
easy
to
end
up
creating
a
data
model
that
resembles
a
queue
which
is
explicit
anti-pattern
in
Cassandra,
and
the
problem
doesn't
show
up
immediately
in
development,
but
once
you
start
running
it
in
production,
you
start
running
into
operational
issues
such
as
compaction,
problems
and
tombstones.
A
So
it's
really
important
to
get
the
data
model
right
for
Cassandra
and
more
so
than
in
sequel,
because
the
data
model
will
affect
your
performance
and
it
will
also
affect
what
queries
you
can
do,
which
leads
to
my
last
point
in
which,
unlike
sequel,
where
the
promise
is
once
you
put
in
the
data,
you
can
get
the
data
back
out
in
any
way
shape
or
form.
You
want
using
some
really
long,
sequel
query
if
necessary
in
Cassandra.
You
really
need
to
know
the
queries.
A
You
want
to
ask
of
your
data
beforehand,
and
this
is
really
one
of
the
biggest
issues
that
people
have
when
transitioning
from
sequel
to
no
sequel,
and
that
is
they're
not
used
to
thinking
about
all
the
queries
they
want
to
know
of
their
data
beforehand.
So
you
really
need
to
sit
them
down
and
think
about
the
application
requirements
and
figure
out
what
the
queries
you
want
to
us
and
then
think
about
the
data
model
for
your
application.
A
That
will
allow
you
to
do
these
queries
without
doing
without
knowing
the
queries
is
not
possible
to
design
the
data
model
for
Cassandra
effectively.
So
this
is
pretty
much
like
the
most
important
thing
when
transitioning
to
Cassandra,
which
is
you
need
to
know
your
queries
first,
so
that
you
can
determine
the
appropriate
data
model
for
your
application
and
the
last
part
is.
There
is
a
lot
of
benefit
in
knowing
very
intricate
details
about
Cassandra
such
as
compaction.
A
We
read
repair
normal
repair
and
how
the
data
models
later
on
this
and
how
that
affects
performance,
because
this
knowledge
will
enable
you
to
advise
people
the
best
and
I'll.
Just
leave
you
with
a
little
quote,
which
is
that.
But
if
you
judge
a
fish
by
its
ability
to
crime
climb
a
tree,
it
will
live
its
whole
life,
believing
that
it
stupid.
A
You
need
to
fundamentally
change
how
you
work
with
the
database
in
order
to
maximize
its
potential
so
to
help
illustrate
at
this
point:
I'll
walk
you
to
a
simple
example
of
enrolling
learners
into
a
course.
So
on
Coursera
we
have
learners
and
we
have
courses
and
we
need
to
keep
track
of
which
courses
a
particular
learner
has
enrolled
it.
So
if
you
think
about
it,
learnin
seven,
many
too
many
relations
with
courses
and
we
need
to
keep
track
of
this
membership.
So
in
the
sequel
wall,
you'll
have
a
table
that
looks
like
this.
A
You
create
table
with
an
idea
course:
ID,
learn
your
ID
and
you
index
the
learner,
and
course
ID,
so
that
you
bill
to
efficiently
check
this
membership.
So
this
schema,
if
you
translate
it
directly
to
Cassandra
that
actually
quite
a
few
problems
with
it.
So
the
first
problem
is
the
ID
field
is
auto
increment
view,
which
is
a
very
convenient
feature
in
sequel.
A
But
such
a
thing
doesn't
exist
in
Cassandra
and
the
reason
is
in
a
digital
system,
it's
very
hard
to
have
a
consistently
incrementing
integer,
because
you
would
then
need
to
coordinate
among
all
the
nodes
to
be
able
to
create
this
sequence.
So
an
easy
fix
for
this
is
to
just
use
a
time
uuid,
which
is
a
unique
ID
generated
by
combination
of
methods,
and
you
can
use
this
effectively
to
replace
your
primary
key.
A
A
So
here's
what
the
same
table
would
look
like:
kickin
Sandra,
so
we've
gotten
rid
of
the
primary
key
ID
and
instead
we've
decided
to
use
a
compound
key.
That
represents
this
relationship.
So
I
hear
the
partition.
Key
is
the
learner
ID
and
we've
made
the
clustering
key
the
course
ID
and
both
of
them
are
you
IDs,
which
are
generated
on
the
application,
server
side
or
using
a
sequel
function,
but
they're
basically
unique
at
that
point
in
time,
and
we
use
them
to
represent
this
entity.
A
Now
see
some
of
the
things
that
we
found
very
helpful
in
our
move
to
Cassandra
is
having
a
team
that
part
time.
Job
is
to
consult
with
people
on
trickier
use.
Cases
of
cassandra
is
very
helpful
because
then
we
can
unblock
the
application
developer
from
having
to
know
everything
about
Cassandra
before
they
start
working
on
their
code,
and
they
also
make
sure
on
the
operation
side
that
people
are
not
writing.
A
A
So
we
have
a
library
that
lets
people
use
cassandra
as
a
key
value
store
and
we
handle
the
serialization
and
deserialization,
and
using
this
we
can
do
all
the
best
practices
like
using
prepared
statements
in
cql
setting
the
right
consistency,
level
and
the
application
developer
doesn't
have
to
care
about.
All
of
this,
all
they
need
to
do
is
to
just
use
our
library
for
more
complicated
things
like
storing
embroidered
indexes
or
the
normalized
tables.
We
also
experimenting
with
having
a
is.
That
is
basically
what
we
call
data
accessed
here
where
you
declare.
A
This
is
my
data
model.
These
are
the
fuse
that
I
want
to
denormalize.
Please
do
it
for
me
when
I
give
you
the
data,
so
as
the
infrastructure
team
we
can
handle
all
the
nitty-gritty,
Cassandra
details
and
the
application
developers
are
free
to
do
your
job
and
right
think
about
the
correct
data
model
for
the
application,
and
we
will
take
care
of
the
Cassandra
side,
so
the
indexing
and
the
querying
setting
all
the
correct
parameters,
and
so
far
it's
working
out
pretty
well
for
us,
it
allows
people
to
work
faster
without
having
to
learn.
A
A
So,
to
recap,
some
of
the
gotchas
we
face
in
the
last
few
years
are
having
lots
of
truly
ad-hoc.
Queries
is
really
hard
still
so,
for
example,
if
you
have
a
discussion
forum
that
needs
to
sort
by
time
or
by
vote
count,
and
you
need
to
filter
out
post
a
tag
with
a
certain
tag,
but
not
some
other
tag.
A
Cassandra
is
probably
not
the
correct
data
store
to
use
for
this,
in
fact,
like
all
the
sorting
and
filtering
is
very
hard
to
express
natively
in
Cassandra,
like
even
if
you
Dena
mais
your
view,
a
form
view,
for
example,
that
might
have
many
sort
orders
in
many
ranking
orders,
and
it
would
require
many
denominations
to
get
correct
on
mutable
fuels,
which
is
basically
a
nightmare.
So
for
us,
what
we've
done
is
for
this
particular
use
case.
We've
used
solas
to
do
our
forum
discussion,
data
store,
and
in
future
we
have
more
of
these.
A
Like
really
ad-hoc
queries,
we
need
to
think
about.
You
know,
which
is
the
right
data
store
for
it,
because
in
this
case,
Cassandra
might
be
useful
as
a
source
of
truth
for
the
individual
entities,
but
for
all
the
sorting
and
filtering
ranking.
You
probably
want
to
look
at
some
other
system
and
the
last
thing
is
you
don't
want
to
use
Cassandra
to
directly
explore
your
data,
so
things
that
developers
often
have
asked
us
is
that
here
I
just
want
to
check
there.
A
You
know
for
transactions
that
happen
in
this
state
that
the
Rose
came
out
correct.
They
come
to
me
on
sight,
so
it's
jesse
cox
that
you
can
use
to
do
this
and
I'll
be
like
actually.
No,
so
you
need
to
have
some
sort
of
system
in
place
to
let
them
explore
that
data
and
in
our
case
we
have
red
shift,
which
is
Amazon's
massively
parallel
database
for
like
data
warehousing,
and
we
tell
them
to
use
that
instead
to
explore
their
data
and
we
have
a
pipeline
that
laws
data
from
Cassandra
into
edw
every
night.
D
A
So
when
we
are
looking
to
switch
will
basically
deciding
between
doing
Charlie,
my
sequel
ourselves
as
our
storage
system,
we
also
looked
at
each
base
and
dynamo
dB
and
in
these
were
the
three
that
we
consider
most
seriously.
We
use
MongoDB
in
the
past
before
as
a
key
value
store
and
that
didn't
go
off
very
well.
So
we
didn't
want
to
look
at
that
again.
A
So
I'll
go
through
this
three
like
very
quickly
so
the
first
one
is
HBase.
Hbase
is
actually
quite
difficult
to
set
up
correctly,
especially
in
Amazon,
because
HBase
requires
HDFS.
So
you
need
to
set
up
hdfs
on
amazon,
which
doesn't
work
out
very
well,
because
HDFS
needs
to
know
every
single
notes
like
IP
address
and
because
we
have
a
very
small
ops
team.
We
didn't
want
to
be
constantly.
A
Hbase
also
has
a
couple
of
other
problems,
which
is
it's
not
really
a
very
highly
available
system,
because
there
are
a
lot
of
single
masterpieces
in
it,
and
so,
if
you
lose
any
one
of
them,
your
system,
I
start
going
down
and
I
think
this
is
evidenced
by
the
fact
that
Pinterest
uses
HBase
for
some
of
your
storage
needs
and
their
solution
to
high
available.
You
have
a
backup,
HBase
cluster
and
they
have
a
dynamic
router
that
routes
between
the
clusters,
depending
on
whether
the
primary
cluster
is
up.
A
A
So
at
that
time,
I
think
you
could
not
store
our
documents
that
were
bigger
than
64
kilobytes,
which
is
pretty
generous,
but
we
do
have
like
a
couple
edge
cases
where
the
document
is
bigger
than
64
kilobytes,
and
so
we
didn't
want
to
tie
ourselves
into
having
this
like
restriction.
They've
listed
it
by
that
time,
it
didn't
seem
so
suitable.
A
The
second
thing
is,
at
a
time
the
only
hair,
locally
local
secondary
indexes,
with
very
strong
restrictions
on
data
size
there
since
introduced
global
secondary
indexes,
so
there's
more
in
line
with
something
like
Cassandra's
materialized
views.
So
that's
pretty
nice,
so
I
wouldn't
say:
there's
a
big
like
loss
in
using
dynamic
like
DynamoDB
and
Cassandra
are
relatively
comparable.
In
that
point,
in
fact,
diamond
we
deem
I
have
a
couple
more
features.
The
last
week
thing
to
think
about,
depending
on
whether
this
affects
you
is.
A
There
is
very
big
vendor
lock-in
using
dynamodb
using
the
animal
DB.
Basically,
guarantees
that
you
can
leave
amazon
leaves
unless
you
replace
this
entire
data
store,
which
is
a
little
bit
worrying
so,
depending
on
your
company's
take
on
such
things,
and
I
think
they
now
have
dynamodb
local,
which
is
this
thing
you
can
install
on
your
laptop
to
use
DynamoDB
in
the
time
they
didn't
have
that.
So
it
was
also
quite
a
big
problem
for
us
and
the
last
one
was
so.
A
B
B
A
Right
now
so
the
course
information
was
stored
in
a
separate
table,
and
there
are
two
big
motivations
for
that
is
the
course
information
often
changes
out
of
sync
with
your
membership,
so
you
might
leave
and
go,
but
the
course
data
is
changed
by
the
professors
and
if
we
change
the
professor
is
a
professor
changes,
the
course
and
we
had
to
normalize
the
course
into
membership
and
every
time
of
course,
data
has
changed.
You
have
to
go
through
every
single
member
and
updated,
which
is
a
pretty
painful
thing
to
do.
Okay,.
A
And
it
kind
of
ties
into
the
fact
that
your
service
aren't
it,
but
we
have
a
single
service
that
serves
all
the
learning
material,
including
the
course
so
at
the
apt.
You
actually
wouldn't
even
go
to
the
database
for
the
course
material
or
there's
a
service
that
controls
memberships
and
users
and
that's
not
a
service.
It
does
all
the
learning
material
stuff,
so
the
service.
On
top
that's
doing
both
the
membership
look
up
and
the
cross
look
up,
we'll
go
to
separate
services
for
these
entities
right.
C
E
I
would
like
to
shed
some
light
upon
the
fall
of
in
case.
Imagine
that
we
already
created
some
data
structure
and
it's
already
using
production
and
they
have
huge
amount
of
data.
So
it's
really
big
data
and
all
of
a
sudden
business
requirements
change
and,
for
example,
we
need
to
introduce
new
tables,
your
tables
for
additional
views
and,
of
course
we
need
to
copy
data
into
do
those
additional
tables.
E
So
we
need
to
start
some
process
of
copying
the
data
which
is
distributed
across
different
nose,
so
this
process
process
gonna
take
long
time
and
during
that
time,
to
the
original
data
to
the
search
data
we
are
cutting
from
gonna
happen.
A
lot
of
changes,
a
lot
of
data
as
a
lot
of
rows
will
be
marked
as
deleted,
modified,
etc.
How
it's
been
resolved
in
Cassandra,
right.
A
A
But
still
do
your
reads
only
from
the
old
table
and
the
second
step
is
to
start
a
background
process
to
copy
over
the
all
data
to
the
new
table
and
here's
where
you
can
use
a
cassandra
trick,
which
is
why
it's
useful
to
know
about
Cassandra
internals.
So
when
you
copy
the
data,
you
can
see
that
all
this
data
was
written
at
time
X
and
when
you
write
the
data
into
the
new
table,
you
can
tell
Cassandra.
A
Please
use
the
right
time
stem
X
instead
of
the
current
timestamp
and
if
you
for
those
not
familiar
Cassandra's
conflict
resolution
between
two
rights,
is
it
the
right
with
the
time
standards
later
we'll
win?
Alright.
So
when
I
right
into
the
new
table
since
I
used
all
the
time
stem
any
future
updates
from
the
double
writing
process
at
apt.
Here
will
supersede
the
background
copy,
and
so
the
new
value
will
be
reflected
even
the
least.
B
E
Imagine
if
I
don't
want
to
write
anything
in
that
new
table
and
kill
the
initial
copy
is
done,
so
how
can
I
make
sure
that,
finally,
when
copy
is
done,
the
data
is
consistent,
because,
while
it
was
being
copied,
the
the
data
changes
happened
to
the
two
day
original
tables,
from
which
data
record
from
so.
If.
A
You
don't
write
to
the
new
table
before
you
start
copying,
then
there's
no
way
to
make
sure
it's
consistent,
which
is
why
the
first
step
is
always
to
start
writing
the
new
data
first
or
what
you
can
do
both
simultaneously
but
there's
always
a
period
of
time
where
you're
writing
to
both
tables
but
you're,
reading
from
the
old
table
and
then.
Finally,
when
you're
pretty
sure
that
the
new
table
has
all
the
data
you
want,
then
you
switch
the
reeds
to
the
new
table
and
you
can
drop
the
whole
table.
A
I
see
I,
think
there's
the
upside
and
enters
the
developer
side,
so
on
the
developer,
side
is
really
the
mind
shifts
that
I
told
you
about
right.
People
like
I
need
to
think
about
my
queries.
First,
I
no
longer
can
query
my
data
anyway.
I
want
the
worries
about
eventual
consistency
and
like
a
new
way
of
structuring
our
data.
I
think
that
was
the
biggest
hurdle
that
we
are
still
working
through
even
now,
because
nobody
learns
about
Cassandra
and
university
or
or
in
any
job
training.
A
A
When
we
were
using
my
sequel,
we
used
amazon's
manage
like
RDS
service,
so
you
click
a
button
and
then
poof.
You
have
a
database
with
cassandra.
We
had
to
set
it
up
ourselves
and
set
up
the
ring
and
get
them
also
talked
together
and
so
there's
a
little
bit
of
a
learning
curve
there.
But
that
took
like
you
know
a
couple
weeks
of
research
and
understanding
to
fully
understand
like
how
to
run
the
cluster
correctly
and
then,
after
that,
it
was
pretty
good.
I.
Think.
A
Colin
families
I
think
we
have
somewhere
in
the
region
between
fifty
to
a
hundred
publico
set
of
50
I
have
encountered
recently.
You
probably
don't
want
to
exceed
500
on
one
cluster,
but
because
we
are
not
that
big,
a
deployment
we
try
and
collocate
all
our
use
cases
as
much
as
possible
into
one
like
really
powerful
cluster.
Are
we
using
the
I
22
excels
on
amazon
and
they're,
pretty
big
machines,
but
a
very
good
for
latency,
so
classifies?
A
A
There
are
two
reasons
for
that,
so
when
we
want
one
that
to
do
a
lot
of
problems
with
virtual
nodes,
I
think
that's
better
from
200
and
201
onwards.
The
second
reason
is:
we
actually
use
the
netflix
tool
preamp,
which
is
a
co
process
to
manage
our
deployment,
and
so
it
is
science
to
talk
into
our
node
and
with
virtual
nodes.
You
need
to
assign
many
tokens
and
pm's
just
not
set
up
to
do
that.
The
main
reason
we
use
p.m.
A
D
A
A
Okay,
so
the
general
philosophy
is
services
on
the
data
store
using
alright,
so
certain
services
might
be
using
my
sequel.
Certain
services
might
be
using
Cassandra
and
a
unifying
in
a
sense
that
all
services
speak
the
same
RPC
language.
So
any
of
the
mid-tier
is
that
interacting
with
each
other?
Basically
just
talk
to
the
service
at
the
RPC
level,
so
they
don't
need
to
care
of
is
Cassandra,
my
sequel,
because
it's
all
done
to
our
PC
and
using
the
service
oriented
architecture.
A
A
Yet
right
for
the
new
things
where
we
want
to
use
Cassandra's
such
as
possible,
then
we
try
as
much
as
possible
to
use
Cassandra
if
the
data
model
really
can't
fit,
for
example,
the
filtering
and
sorting,
and
all
that
kind
of
things
I
told
you
about,
then
we
can
consider
like
solar
or
really
need
be.
We
can
go
back
to
Mexico
all
right,
but
because
its
service
owns
this
database,
you
are
free
to
change
it
out
later
on
in
time.
If
we
find
that
it's
not
working.
A
F
F
A
So
we
are
data
sets
enterprise
customer,
and
that
was
pretty
intentional
because
in
our
first
year
we
basically
had
no
Cassandra
expert
in
house,
and
we
wanted
to
have
a
venir
to
like
ask
people
questions.
So
I'll
show
you
like
two
bugs
that
we
ran
into
so
the
first
one
is.
I
saw
that
the
concentric
heap
kept
going
up
and
up
and
up
as
that
we
spent
on
by
and
then
eventually
a
nodal
crash
because
it
ran
out
heap
about.
A
It
was
a
very
slow
process
and
when
you
force
a
fool
GC
that
people
actually
drop
again,
which
is
really
weird
and
it
turns
out
the
incident
versions
of
the
JDK
we
are
using,
it
doesn't
collect
jmx
classes,
and
so
our
monitoring
system
was
creating
all
these
classes.
They
are
not
getting
GC
and
then
the
node
would
die.
If
you
add
this
magical
flag,
I
think
it's
like
collect
all
classes
or
something
it
won't
like
start
working
again.
A
So
that's
one
case
where,
if
we
like
head
support
and
that
really
helps-
and
now,
if
you
search
on
internet
you'll,
find
like
people
talking
about
the
same
thing,
the
second
mark
we
have
is,
we
were
using
columns,
maybe
a
little
bit
too
eagerly
in
the
life
cycle
of
the
introduction.
And
so
it
turns
out
that
if
you
reverse
the
sort
order
and
you
have
static
Collins,
the
static
column
will
disappear.
A
It
wouldn't
show
up
in
the
result,
and
so
we
filed
a
bug
and
we
look
to
the
gyro
and
then
we
found
the
bug,
and
so
we
knew
that
it
will
be
fixed
in
like
two
versions,
time
or
something
right.
So
there
I
feel
like
all
these
minor
edge
cases,
and
usually
I
did
a
mailing
list.
So
the
gyro
of
was
like
asking
a
support
would
be
able
to
answer
the
question
and
then
in
like
very
odd,
like
eccentric
age.
F
Okay,
so
I
have
one
more
question,
so
you
said
you
are
using
both
my
sequel
and
Cassandra.
So
have
you
encountered
any
case
where
that
data
that
you
that
is
residing
in
my
sequel
and
data
residing
in
Cassandra?
Basically,
that's
basically,
is
there
a?
Is
there
any
intersection
in
both
of
them
right
Cassandra,
my
signal,
like
services,
those
operate
off
of
Cassandra
need
the
same
data
that
sort
of
for
that
services
that
operate
off
of
my
sequel
needs.
Is
there
any
kind
of
intersection
in
your
case,
yeah.