►
From YouTube: C* Summit EU 2013: Being Closer to Cassandra at Ok.ru
Description
Speaker: Oleg Anastasyev, Lead Platform Developer at Odnoklassniki.ru
Slides: http://www.slideshare.net/planetcassandra/c-summit-eu-2013-being-closer-to-cassandra-at-okru
Odnoklassniki uses cassandra for its business data, which doesn't fit into RAM. This data is typically fast growing, frequently accessed by our users and must be always available, because it constitute our primary business as a social network. The way we use Cassandra is somewhat unusual - we don't use thrift or Netty based native protocol to communicate with Cassandra nodes remotely. Instead, we co-locate Cassandra nodes in the same JVM with business service logic, exposing not generic data manipulation, but business level interface remotely. This way, we avoid extra network roundtrips within a single business transaction and use internal calls to Cassandra classes to get information faster. Also, this helps us to create many small hacks on Cassandra's internals, making huge gains on efficiency and ease of distributed server development.
A
A
A
A
A
A
A
On
it,
you
will
have
you
written
next
to
the
counter,
and
if
you
mouse
over
on
it,
you
will
have.
You
can
see
the
friends,
the
friends
of
you
who,
like
the
same
item
so
light
widget,
is
everywhere.
We
have
it
on
every
page.
Thousands
of
this
light
vision,
I
counted
20
more
than
20
widgets
on
the
main
page
of
the
site.
On
the
on
one
page.
A
We
have
it
on
feeds,
feed
timeline
fees.
So
this
is
the
main
consumer
of
of
white
pigeons.
People
mostly
used
from
their
timelines
in
these
like
regions,
and
we
have
it
on
third-party
websites,
probably
like
button
webmasters,
can
integrate
this
like
button
to
their
sites.
So
it's
everywhere
and
it's
on
everything.
B
C
A
Widget
renders
for
every
second
kind
of
much
smaller
number
of
rights,
but
still
these
rights
are
making
more
more
than
100
million
like
records
every
day,
and
the
data
set
grows
in
more
than
15
gigabytes
in
storage
early
so
and
for
cassandra.
It
is
hard
to
load
a
couple
of
profile
because
it
is
read
most
not
only
because
most
most
is
commonly
not
good
for
cassandra.
It
has
a
long
tail
with
many
reeds,
not
cacheable
at
all,
and
because
the
system
is
used
from
so
many
parts
of
the
site.
A
Three
terabyte
datasets,
since
she
seems
not
very
big,
but
it
is
big
enough
not
to
choose
in
memory
only
storages
like
radius,
for
example.
Something
like
that.
Let's
see
how,
but
let's
discuss
how
it
could
be
implemented
in
sql
trying
to
solve
this
in
sql.
We
will
have
this
simple
table
in
our
database
having
reference
entity
id
entity,
type
user
collects
the
entity
and
when
he
liked
the
entity
so
to
render
like
widget
a
single
widget,
we
must
execute
these
three
prayers.
A
A
A
Answer
is
a
lot
on
each
of
these
queries.
You
have
disk
accesses.
It
will
have
at
least
one
logical
read
to
check
existence
of
the
record.
For
the
first
query,
you
will
have
some
waterfall
records.
Sometimes
it
has
to
be
too
much.
Some
of
entities
have
more
than
a
million
likes,
but
the
real
problem
is
filtered
by
friends,
because
friendship
information
is
actually
stored
in
separate
clusters,
so
you
have
to
pull
all
the
friends
from
the
separate
cluster
and
execute
first
query
against
every
friend
from
from
this
against
every
friend
id.
A
A
A
A
A
A
A
A
It
has
more
network,
but
the
real
problem
is
that
it
multiplies
partitions
count
by
more
than
10
times
so
partition
keys
have
overhead
in
both
in
memory
and
disk.
So
in
the
end
you
have
twice
as
much
data
as
you
had
so.
Instead
of
quick
solution,
we
decided
to
extend
cassandra
and
what
we
implemented
is
by
alarm
bloom
filter,
in
addition
to
participation,
keys,
it's
towards
pairs
of
partitioning
key
and
clustering
key
into
remains
a
stable
filter
file
so
now
to
check
the
existence
of
the
whole
primary
key.
We
don't
have
to
push
this.
A
A
A
Data
server
flights,
so,
instead
of
separated
application,
servers
and
cassandra
nodes
will
have
a
single
live
server.
It
has
a
business,
remote
interface,
exposing
methods
like
get,
which
returns
all
the
information
necessary
to
render
a
single
widget
in
a
single
router
and,
for
example,
the
method
add
like
which
does
all
the
necessary
invitations
to
file.
A
This
remote
interface
uses
the
one
neo
remoting.
This
is
our
internal
development
invention.
Now
it's
open
source
environment,
plastic
camera,
it's
available
on
our
github.
It
is
implemented
for
linux
and
makes
much
faster
and
much
less
overhead
io
than
java
has
so
any
framework
you
will
use
on
java
will
have
more
overhead.
A
A
A
This
is
in
memory
of
fast
memory
on
the
operation.
We
select
check
it
with
blue
memory,
boom
filter
and
select
some
id's
much
more
fitter,
and
we
stop
as
soon
as
we
have
double
of
the
double
of
n
double
of
the
number
of
friends.
We
want
to
return
as
soon
as
we
have
double
all
the
these
friend
ids.
We
do
reads,
and
we
read
until
we
have
10
friends.
A
Also,
this
allows
us
to
implement
custom
caches
which
are
tuned
for
application
and
implement
custom
data,
merge
logic,
because
you
have
your
business
logic
right
in
sight
of
microsoft,
but
this
was
not
quite
possible
until
we
made
a
small
hack
to
cassandra
code,
we
made
a
we
made
a
listing
for
mutation.
We
made
we,
we
implemented
this
simple
interface,
which
has
a
single
pre-applied
method,
and
you
have
to
register
this
in
your
local
cassandra
instance.
Just
this
is
looks
like
just
goals
of
the.
A
Can
start
listening
for
notations,
we
changed
the
original
mutation
applying
code
to
call
this
listener,
so
we
will
have
your
listener.
You
have
the
notification
pass
to
your
listener
for
every
original
mutation,
like
the
triggers
do
in
cassandra
2.0.
But
beyond
that
you
have
ability
to
extend
radiological
reputation.
Mutations
hints
read
repairs
all
the
data
that
actually
remove
this.
You
can
check
exam
and
do
some
additional
classes.
A
A
A
A
A
Normally,
cassandra
does
not
reproduce
notice
raw
cache
states.
Now
they
can
place
to
its
cash,
only
rows.
It's
recently
read
from
the
local
sustainable
as
a
part
of
creative
press
processing.
This
is
not
what
we
wanted,
because
when
we
read
only
from
local
nodes
cache,
a
cache
of
other
nodes
will
contain
no
code
data,
so.
A
A
A
A
A
A
When
you
request
data
from
your
cardinality
mode
of
choice
of
choice,
it
actually
performs
these
steps.
It
chooses
one
node
for
data
and
other
nodes
for
to
satisfy
consistency
level.
It
chooses
from
digest
when
it
sends
the
rig
request
to
all
the
nodes.
One
node
will
receive
the
data
request.
Other
others
will
receive
the
digital
press,
it
waits
for
data
and
footages.
A
A
A
A
A
A
And
later
after
we
had
several
load
failures
during
operation,
we
learned
that
this
approach
has
the
one
additional
important
benefit.
The
traffic
and
work
of
the
cluster
remains
constant,
no
matter
all
nodes
of
cluster
are
up
or
reoperating
cautiously,
so
this
greatly
simplifies
capacity
and
failure
planning.
Actually,
because
if
you
your
cluster
is
working
now,
it
will
work
even
in
case
of
massive
failure.
A
A
A
This
is
from
client
side
even
less
for
from
from
10
times
less
from
the
service.
So
that's
all
about
the
fast
cluster.
Let's
talk
about
the
fact,
so
the
fight
cluster
is
about
charts.
Everyone
knows
charts,
you
have
one-on-one
charts
as
well
as
you
can
invite
more
people
to
it.
When
you
chat
with
someone,
a
new
chat
with
unique
your
id
is
created.
A
When
you
write
messages,
wait,
it
has
to
be
saved
for
a
chat
to
the
messenger's
custom.
Obviously,
messages
are
being
added
most
of
the
time,
but
sometimes
they
are
updated
right.
The
user
has
edited
the
message
or
anti-spam
system
kicks
in
to
modify
the
text.
Also,
the
special
part
of
this
workload
is
that
the
freshest
messages
page
are
accessible.
A
A
A
Have
some
properties
retry
to
their
management
and
rendering
all
chat
messages
are
stored
in
a
single
partition
which
key
is
ideally
charged
inside?
All
the
partition
messages
are
sorted,
so
freshest
messages
are
going
first,
having
so
much
data.
We
wanted
the
size
of
the
data
stored
to
be
as
small
as
possible,
so
we
placed
all
known
key
properties
into
a
single
block,
because
columns
itself
have
overhead
of
both
memory
and
storage.
A
Of
concurrent
modifications
to
the
sink
alarm
made
from
separate
servers
and
subsystems,
we
didn't
wanted
to
use
some
kind
of
block
coordinators
or
something
like
that.
Fortunately,
there
are
very
few
updates.
Most
most
of
rights
are
in
sorts
of
new
messages
and
we
have
very
few
updates,
so
we
implemented
lightweight
conflict
resolution.
A
A
Of
the
same
data
stored
in
column.
Finally,
so
let's
see
how
read
modify
right
scenario
would
work
with
this
structure.
Consider
having
two
clients?
First
and
second
example,
both
executing
the
read
modified
write
on
the
single
message:
storage.
The
first
client
pulls
data
from
the
storage
and
receives
current
data
d1
with
the
version
of
the
data
attached.
A
Storage
applies.
These
changes
by
deleting
the
old
data
for
ts1
is
used
in
new
version
of
the
data.
This
new
version
is
actually
an
intersect
microsecond
precision,
monotonic
timestamp
of
the
local
storage
and
storage,
writes
new
data
under
this
just
issued
version
number
concurrently.
Second
client
client
executes
same
steps.
Pulling
data
from
cluster
modifying
it
and
writing
back
to
storage
storage
will
apply
the
changes
the
same
way
relating
already
existing
ts1
version
of
all
data
using
new
version,
which
is
now
ps3
thanks
to
monotonic
timestamp.
A
B
A
So,
in
addition,
we
build
a
specialized
cache
for
this
system
as
well,
because
we
can
so
it's
so
again,
so
keep
it
caches.
Only
freshest
chat
page,
it
saves
its
state
to
local
family.
Like
like
system
column,
family
doesn't
introduce
custom,
it's
needed
to
fast,
populate
manually,
start
mode.
We
use
stuff
with
cold
with
empty
cache,
so
we
have
to
reduce
some
from
somewhere
and
we
read
it
sequentially,
which
is
much
much
much
faster
than
reading
keys
and
reading.
The
data
from
column
frameworks
like
assam,
will
do
so
cassandra
and
it.
A
A
C
A
A
A
A
A
A
But
some
distances
become
overloaded
by
original
quest
and
somewhere.
I
don't,
but
they
are
filled
equally.
What
happens
investigating
this?
We
found
that
this
is
because
some
disks
have
more
newer.
Sustainables
than
others
consider
just
finished
compaction.
Big
file
is
removed,
disk
has
a
lot
of
free
space,
so
all
main
table
flashes
go
to
a
single
disk,
so
but
newer
data
are
being
read
more
often
so
you
have.
This
summaries
are
overloaded
and
some
items.
C
A
Around
robin
and
his
collocation
works
better
in
this
case,
but
finally,
we
got
into
the
generational
policy
each
disk.
This
works
in
a
way
that
each
disk
has
the
same
number
of
same
same
generations
and
tries
to
keeping
them
saying
this.
It
doesn't
look
at
the
size
of
the
free
spacing,
so
that's
all
about
the
fat
pasta,
let's
go
to
other
one.
A
I've
been
thinking
a
lot
about
which
cassandra
cluster
the
ugliest
front
in
stream
in
our
freak
show,
and
so,
ladies
and
gentlemen,
need
the
chat,
summaries
cluster.
This
cluster
is
about
messaging
too,
but,
unlike
the
fat
class
construct
messages,
this
cluster
stores
index
of
active
charts
for
the
user.
A
You
can
see
it
on
the
left
side.
This
index
shows
number
of
unread
messages
or
not.
It
has
a
small
data
set
only
about
a
couple
of
gigabytes
and
the
hotset
having
only
five
persons
of
misses
going
to
disks.
The
car
part
about
this
index
is,
is
that
it
reorders
itself
too
often,
for
example,
if
you
have
a
new
message
from
someone
we
want
from
someone,
we
want
that
someone's
name
to
be
on
top
of
the
list.
A
If
you
have
several
charts
open
with
unread
messages,
we
want
chance
of
your
close
friends
to
be
on
the
top
of
the
list
and
so
on
and
so
on.
So
when
you
read
the
message
from
the
chat,
he
also
can
review,
so
it
is
being
reordered
on
almost
every
right
of
a
message
to
a
child
and
rid
of
a
message
from
each
other.
A
Carrying
many
tombstones
in
our
family
is
not
a
good
idea.
It
hits
performance,
so
we
chosen
to
store
it
in
a
single
block
column
and
because
we
chosen
it
to
sort
into
a
single
block
alarm.
We
call
we
have
almost
no
inserts
and
a
lot
of
updates
to
existing
lists
to
existing
values
which
bring
potential
conflicts
from
any
of
27.
A
A
These
are
the
old
removed
version
columns
just
too
much
updates
using
some
kind
of
distributed
low
carbonator.
It
was
not
an
option
again
because
number
of
potential,
but
actual
conflicts
is
not
so
high,
so
pessimistic,
looking
distributed.
Low
carbonation
beneath
berlin
is
is
an
overkill
taking
in
account
that
merge
algorithm
for
conflicting
updates
is
doable
for
this.
For
the
chat
summary
list
it's
somewhat
complicated
but
still
doable.
A
A
It
does
not
have
any
structure
like
column
phenols
or
so
it
maps
curved
binary
keys
to
binary
values,
but
we
don't
need
a
structure
which
is
just
summary:
voldemort
have
distributed
coordination,
logic
pushed
to
client
sides,
so
clients
could
be
implemented
only
in
java.
Language
java
is
perfect,
fit
for
us,
and
voldemort
has
plugged
all
storage
changes
into
architecture.
So
you
can
implement
your
own
storage.
A
If
you
wish,
we
did
tested
some
storage
changes
available
for
voldemort
out
of
the
box
and
available
on
the
internet
and
found
that
their
performance
at
scale
is
not
very
good
for
us.
So
after
a
couple
of
performance
tests,
we
implemented
a
new
storage
engine
for
voldemort
based
on
cassandra,
which
is
based
on
cassandra
persistent
player.
The
only
thing
we've
done,
we
developed
our
our
own
cash
flow
data,
hello
development,
by
the
way.
A
So
you
might
ask
why
to
give
life
to
such
an
early
frankenstein.
Okay,
the
answer
is
simple:
it
performs
for
a
test
cluster
bit
of
three
cheap
commodity
servers
available
today,
marking
eight
by
keys,
which
is
chat
id
to
one
kilobyte
binary
value,
which
is
list
of
summaries.
We
reached
the
total
performance
of
90
000
operations
per
second,
so
six
nodes
cluster
can
handle
all
the
load
of
chart
summaries
of
our
social
network.
A
A
B
A
Cassandra
2.0
is
efforts
going
in
in
another
way.
Currently
we
don't
plan
to
replace
all
those
clusters
which
is
developed
it.
It
has
no
much
reasoning
instead,
we're
trying
we
have
internally,
not
in
our
infrastructure.
We
have
microsoft,
skill
servers
which
do
have
some
transactional
data
and
we
want
to
replace
the
goals
because
cost
and
reliability.
A
A
A
All
the
data
of
the
social
graph
are
stored
in
memory.
This
is
in
memory
only
storage.
There
are
about
60
60
machines,
with
more
than
100
gigabytes
of
memory
which
serve
problems
stored
in
the
social
graph
and
the
interesting
part
that
they,
when
they
receive
updates
of
the
to
the
social
graph.
They
write
these
updates
to
their
cassandra
persistence
today
and.