►
From YouTube: C* Summit EU 2013: Playlists at Spotify - Using Cassandra to Store Version Controlled Objects
Description
Speaker: Jimmy Mardell, Senior Software Engineer at Spotify
Slides: http://www.slideshare.net/planetcassandra/c-summit-eu-2013-playlists-at-spotify-using-cassandra-to-store-version-controlled-objects
All systems at Spotify have to deal with huge amounts of data. Playlists in particular is a unique challenge. We need to store more than one billion playlists, and make them accessible for not only the playlist owner but also subscribers. Furthermore, we need to handle concurrent changes to collaborative playlists and offline scenarios. The devised solution treats every playlist as a versioned object. We use Cassandra to store these objects in an efficient way, allowing fast read- and write queries. The road there was not pain free however. I will talk about the data model we ended up using, and lessons learned along the way.
A
A
A
I
used
to
work
on
our
playlist
system,
which
is
what
I
will
talk
about
today
and
during
that
time
I
learned
a
lot
about
cassandra.
So
please
interrupt
me
with
a
question.
If
something
is
unclear,
I
know
I
have
a
tendency.
Sometimes
I
talk
a
bit
too
fast
when
I'm
excited
about
something
and
kind
of
like
our
playlist
system,
so
it
could
happen
so
spotify.
I
trust
most
of
you
in
here
know
about
spotify.
A
We
have
a
lot
of
machines
in
different
places
in
the
world
and
we
have
a
lot
of
playlists
that
our
users
have
created
one
billion
on
them
more
than
one
billion
playlists,
and
so
that
is
what
I'm
going
to
talk
about
today
and
at
spotify
we
use
cassandra
quite
a
lot.
The
cassandra
cluster
that
stores
our
playlists
are
is
by
far
the
largest
cluster.
A
A
I
will
talk,
then
I
dive
into
how
we
use
cassandra
for
storing
our
playlist,
about
which
data
model
we
use
and
some
cool
stuff
we
can
do
with
it
and,
finally,
some
lessons
we've
learned
along
the
way.
The
road
to
where
we
are
now
has
not
been
pain-free.
We've
done
errors
mistakes,
and
so
I
want
to
share
with
some
of
them
alright,
so
version
control,
most
people
really
associate
version
control
with
standalone
software.
Like
subversion
or
git,
and
things
like
that,
they
work
slightly
differently.
A
They,
some
of
them,
use
a
centralized
server,
some
used
more
distributed,
but
they're
really
basic
the
same
thing.
Your
version
control
files
on
disk
some
features
of
these
programs
is
that
you
can
create
branches,
you
can
merge
branches,
you
can
send
changes
back
and
forth,
but
it's
quite
manual
process.
Little
branches,
especially
this
is
not
the
kind
of
version
control
I'm
really
talking
about
today.
A
What
I'm
we'll
be
talking
about
is
rather
more
similar
to
perhaps
where
it's
embedded
version
control
and
an
example
of
a
system
using
embedded
version
control,
which
I'm
sure
most
of
you
have
been
using,
is
google
docs,
so
google
docs
is
pretty
much
office
in
the
cloud.
You
have
a
spreadsheet,
you
have
a
word
processor
and
more,
and
you
can
just
write
documents
and
you
can
collaborative
write
documents.
A
You
can
multiple
people
editing
the
same
document
at
the
same
time
without
things
just
work
and
you
can
revert
to
change,
you
can
do
undo
stuff
for
a
very
long
time
in
a
way
and
the
way
it
that
works
is
that
it
uses
it
really
uses
version
control
internally.
It
doesn't
use
git
or
anything,
but
it's
still
washing
control.
Everything
you
do
are
changes
that
just
keep
piling
up.
You
don't
manually
branch
stuff
when
you
write
the
word
processor,
but
that
can
pretty
much
happen
if
you're
right,
two
people
write
the
same
document.
A
If
you
start
writing
what
you're
really
doing
is
that
you
temporarily
can
create
small
branches
that
are
automatically
merged.
So
this
is
really
what
we'll
talk
about
today,
embedded
version
control,
so
yeah.
I
will
mention
some
of
the
advantages
of
using
embedded
version
control.
Another
advantage
could
be,
for
instance,
performance.
A
So
let's
say
we
have
a
document
that
we
want
to
synchronize
across
several
devices,
and
this
document
keeps
getting
updated.
It
might
actually
be
more
performant
to
just
synchronize
the
changes
rather
than
synchronize
the
entire
document,
depending
on
the
size
of
the
document
and
the
size
of
the
changes.
A
Another
reason
could
be
that
you
have
business
logic
that
depends
on
the
history
of
the
document
and
that
kind
of
depends
on
your
business
and
notice
that
when
I
said
when
I
say
document
there,
I
really
mean
any
kind
of
object
that
mute
takes
over
time.
So
it
could,
for
instance,
be
a
shopping.
Cart,
that's
a
quite
a
short-lived
object
or
it
could
be,
as
I
will
talk
about
today,
playlists,
which
is
another
kind
of
document
that
mutates
over
time,
which
brings
us
over
to
the
spotify
universe.
A
I
won't
talk
so
much
about
the
spotify
architecture
in
general,
except
that
I
would
say
that
we
are
really
service
oriented.
We
have
many
small
services
doing
one
thing
and
one
thing:
only:
we
try
to
keep
our
services
as
separate
or
as
decoupling
from
each
other
as
possible
and
independent
of
each
other,
and
if
a
service
needs
just
storage
well,
it
use
has
its
own
storage.
A
A
The
playlist
clusters
is,
as
I
said,
the
largest
one
and
the
playlist
service
in
itself
is
also,
I
would
say,
the
most
complex
of
our
backend
services,
and
this
is
because
we
have
this
version
control,
which
is
it's
still
not
too
complicated.
But
it's
it's.
We
try
to
keep
things
really
simple
as
spotify
in
order
for
us
to
be
able
to
scale
but
at
the
same
time,
still
able
to
add
new
features
without
risk
of
breaking
existing
features.
A
So
the
playlist
system
that
I
will
talk
about
today
is
called
playlist
four
and
the
reason
for
that
is
that
all
our
systems,
that
spotify
have
names
like
the
name
of
the
systems
and
a
version
number
and
most
systems
are
at
version
one
or
two
for
playlists.
However,
we've
screwed
up
three
times
already
well,
actually
the
first
two
times
were
very
early
prototypes
and
then
we're
never
out
in
a
while.
So
it's
really
the
second
iteration
playlist
and
hopefully
we
won't
see
a
playlist
file,
but
you
never
know
so.
A
This
is
a
playlist
in
spotify
and
this
is
a
screenshot
from
the
desktop
client.
It's
just
to
illustrate
what
a
playlist
release
it's
an
ordered
list
of
tracks
really
and
it's
important
that
it's
ordered
it's
not
just
a
set
figure
to
use.
You
can
change,
move
around
tracks
plain
different
order.
Each
track
is
just
an
id,
so
we
don't
have
to
care
so
much
about
that
and
playlist
may
contain
duplicates.
A
It
has
metadata
such
as
name
or
playlist,
and
when
you
added
a
track
and
things
like
that,
and
so
this
is
what
you
need
to
store
in
the
playlist
system.
And
so
what
are
the
challenges?
This
sound
doesn't
sound
too
hard
right.
So
some
challenges
yeah.
This
is
a
large
scale.
User
generated
content
system.
As
I
said,
we
had
1
billion
playlists.
A
It
receives
a
lot
of
requests
more
than
40
000
per
second
at
peak,
and
some
features
that
we
want
that
we
wanted
early
on
is
possibility
to
offline
edit,
your
playlist.
That
means
that
if
you're
offline,
I
can
still
modify
my
playlists
and
when
I
get
online
again
it
just
syncs
and
everything
works.
If
I,
if
there's
some
other
someone
else's
myself
or
someone
else,
just
modify
the
same
playlist
while
I
was
offline,
it
should
still
has
to
work.
We
shouldn't
overwrite
each
other
and
yeah.
A
Concurrent
changes
is
almost
the
same
thing
to
you:
can
hey
in
spotify
you
can
have
collaborative
playlists,
which
means
that
multiple
people
can
change
the
same
playlist
just
like
in
google
docs,
and
that
should
just
work.
People
should
just
be
able
to
add
tracks,
remove
tracks,
move
around
it
shouldn't
or
we
shouldn't
overwrite
each
other's
changes.
A
A
So
in
the
simple
answer.
The
short
answer
is
that
every
playlist
at
spotify
is
a
version,
version
control,
object
or
you
can
see
it
as
a
repository
of
something.
So
we
have
one
billion
repositories:
small
repositories
of
objects,
that
we
have
keep
track
of
the
entire
history
and
each
playlist
is
totally
independent
from
each
other.
A
So
we
only
have
to
focus
about
one
playlist
at
the
moment,
so
the
playlists
are,
of
course,
the
spotify
service
stores,
the
playlist,
that's
the
central
repository
when
the
client
login
it
just
fetches
the
latest
version
it
syncs
its
own
local
copy.
It
has
a
local
copy
of
the
playlist
on
the
disk
and
when
you
log
in
again
it
synchronizes
from
the
server,
if
you
you're
offline,
you
can
still
have
the
playlist.
Since
it's
cached
local,
you
can
still
use
the
playlist.
A
But
when
you
log
in
you
get
online
again
it
syncs
just
like
just
like
they
do
with
a
standard
version
controller.
You
have
a
git
pull.
For
instance,
then
you
think
you
synchronize
with
what
you
have
on
the
server.
So
that's
pretty
much
what
happens
and
when
you
make
a
change
to
a
playlist
in
your
client.
The
change
is
immediately
applied
and
this
allows
for
a
really
snappy
ui
and
which
is
good
because
it
feels
like
everything
works
at
once.
A
Our
users
didn't
really
notice
it.
They
could
still
modify
the
playlist
as
long
as
they
didn't
bring
up
their
mobile
phone
and
wonder
why
the
changes
didn't
appear
there
as
well,
but
this
was
when
we
had
when
we
were
really
saved
by
this.
Like
few
years
ago,
there
were
not
so
many
mobile
clients.
Most
people
just
had
their
desktop
client
and
changed
the
playlist,
so
everyone
was
happy
even
though
our
half
a
service
were
down
that
that
happened,
but
this
was
many
years
ago
right
now,
everything
is
very
fast
and
working,
hopefully
all
right.
A
So
let's
go
into
how
we
store
this,
how
we
represent
the
playlist
on
the
server
side.
I
want
you
could
talk
a
lot
about
how
we
do
this
on
the
client
side.
It's
not
it's
actually
non-trivial
summer
stuff,
but
I
will
focus
this
talk
on
how
we
store
this
in
the
service
side.
Since
this
is
like
a
center
conference,
I
want
to
go
into
how
we
use
cassandra,
so
this
is
a
as
with
every
version,
control
object.
A
You
have
a
graph
of
versions,
and
this
is
an
example
how
it
can
look
with
playlists
and
so
at
the
bottom
we
have
an
empty
playlist
that
really
doesn't
exist,
and
then
the
user
has
made
three
changes
to
the
playlist
and
he
starts
by
adding
three
tracks
at
the
first
position
index
zero.
He
then
moves
one
track,
and
finally,
he
deletes
one
track
to
the
right.
Is
the
snapshot?
How
the
playlist
looked
like
after
east
operation-
this
is
just
for
your
for
the
convenience
here.
It's
not
really
part
of
the
graph.
A
It's
just
to
show
you
easy
to
see
what
happens
and
in
the
left.
Part
of
this
picture
is
the
revision
number
so
at
the
bottom,
every
vision
number
zero,
nano
revision,
number
one
comma,
a
hash
revision,
number,
two
common
hash
and
so
on.
I
will
go
back
come
back
to
later,
exactly
what
this
just
revision
number
means,
but
essentially
every
change
you
make
receives
a
unique
revision
number.
A
So
whenever
you
modify
playlists
you
just
append
to
this
graph
and
and
this
graph
is
really
the
only
persistent
storage,
you
would
need
in
theory,
because
if
you
want
to
know
how
the
playlist
looks
like
well,
you
can
just
get
all
the
changes
and
replay
them
and
you
will
see
the
final
snapshots.
This
might
not
be
terribly
efficient
if
you
have
a
really
large
graph,
which
is
why
we
also
store
one
snapshot
of
the
playlist,
and
that
is
snapshot
is
usually
the
latest.
A
Yes,
so
we're
restoring
both
the
entire
anti-graph
of
changes
and
the
snapshot,
so
one
might
think.
Okay,
that
seems
to
be
wasting
some
storage
space
right.
Well,
it
at
least
doubles
the
amount
of
space.
Obviously,
but
this
space
is
cheap.
Today,
the
graph
in
itself
is
actually
not
much
more
than
the
snapshots,
because
most
users
just
add
the
tracks
to
their
playlists.
The
majority
used
to
just
add
the
tracks,
so
storing
all
the
changes
for
such
a
playlist
or
storing
the
snapshot
is
pretty
much
the
same
amount
of
space.
A
So
that's
not
a
problem,
so
we
have
used
double
as
much
storage
to
be
able
to
store
more
both
graph
and
snapshot.
And
why
do
we
do
this?
Why
can't
we
just
store
the
snapshot
only,
and
that
is
because,
of
course,
because
of
concurrent
concurrent
edits
so
concurrent
changes,
so
you
can
have
playlists
of
spotify
that
are
collaborative.
A
Multiple
people
can
change
it
at
the
same
time
you
can
be
offline
and
make
changes
and
then
at
the
same
time
can
actually
be
a
very
long
time
because
they
were
offline
and
then,
when
you
get
online
again,
it
turned
out
that
some
other
had
made
a
change
to
the
same
playlist
and
how
what
this?
So
this
is
pretty
much
what
happens
then,
the
graph
splits
into
two
branches,
as
I
said,
with
a
more
traditional
version,
control
software-
you
have
to
manually
create
a
branch
here.
A
It
happens
automatically
the
user,
don't
notice
it
some
nomenclature
here,
each
of
these
branches.
So
we
have
two
branches
and
the
most.
The
latest
revision
on
each
branch
is
what
we
call
a
head.
This
is
the
same
terminology
used
by
git,
for
instance,
so
when
the
service
detects,
oh,
there
are
two
heads
in
this
in
this
playlist.
I
have
to
do
something
because
we
can't
have
it
a
playlist
in
this
split
state.
A
So
what
it
does
it
does
it
that
automatically
it
automatically
fixes
this.
So
the
end
result
becomes
like
this
and
the
algorithm
used
to
merge.
This
is
based
on
operational
transformation,
which
is
something
you
can
read
about
on
wikipedia
it's
slightly
beyond
the
scope
of
this
talk.
But
what
you're
essentially
doing
is
that
you
crosswise
apply
the
changes
but
modify
them,
so
they
work
depending
on
the
previous
change
and
that
this
is
now
we're
getting
really
domain
specific
here.
So
what
does
that
mean?
A
If
you
add
a
track
at
some
position-
and
you
know
the
other
branch,
you
had
a
track
at
a
different
position.
If
you
crosswise
apply
them,
you
have
to
modify
the
indexes,
so
they
end
up
in
the
correct
places.
Also,
and
this
this
can
get
a
bit
messy,
but
it's
doable
and
it's
as
I
said,
there's
research
done
on
this
topic
and
but
the
end
result
is
pretty
nice.
You
get
a
one
head
again
and,
what's
even
better,
is
that
the
clients
are
completely
oblivious
that
the
playlist
has
branched.
A
So
the
client,
the
client
that
did
the
change
a
in
the
left
branch
when
it
syncs
to
playlist
again
it
will
receive
change
b,
prime,
which
you
think
happened
after
my
change,
and
it
will
be
happy
and
the
client
that
made
this
b
change.
But
when
it
sinks
the
playlist,
it
will
receive
change
a
prime
and
be
happy.
So
both
we
end
up
with
the
same
result.
The
clients
thought
they
were
a
linear
history.
A
At
the
server
side
we
actually
stored
in
the
entire
branching
to
be
able
to
handle
these
conflicts
and
well
if
there
is
a
client
that
is
even
further
down
here,
it
will,
when
it
sinks
it
will
receive
just
one
of
these
paths.
It
doesn't
really
matter
which
one
and
in
the
client,
you
don't
see
this.
You
just
see
the
end
result
all
the
time,
all
right,
everything
clear
so
far,
awesome,
because
now
this
is
kind
of
the
general
thing
thing.
A
A
A
We
actually
used
it
partly
to
evaluate
cassandra
as
well
and
it
went
well
so
now
we
use
cassandra
for
a
lot
of
things
we
have,
as
I
said,
I
think
I
said
more
than
30
clusters
using
cassandra
and
we
started
back
way
back
in
version
0.7,
when
a
lot
of
features
today
didn't
really
exist,
which
is
also
why
we
today
still
not
using
so
very
many
features
because
we're
not
using
image
sql.
Although
that
will
change.
I
assure
you
there
are
other
things
we're
not
using
like
super
columns.
A
For
instance,
we
never
tried-
which
I
would
guess
was
a
good
thing
since
they're
almost
obsolete
now,
all
right,
we
we're
not
only
using
customers,
we're
also
using
other
databases,
but
cassandra
is
our
first
choice
of
database
when
we
need
a
new
storage,
so
playlists
cassandra
and
data
modeling.
So
this
is
a
general
rule.
When
you
plan
a
data,
modeling
cassandra,
you
really
should
start
with
the
queries
and
for
playlists
and
version
control
system
in
general.
There
are
three
kinds
of
queries:
really
that
are
the
important
ones
it's
synchronizing
your
playlist
get.
A
That
is,
you
have
one
copy
of
the
playlist
and
you
want
to
receive
the
latest
changes.
You
might
also
want
to
get
a
completely
fresh
copy
of
the
playlist.
If
you
only
if
you
want
to
get
a
new
playlist
you
haven't
seen
before,
and
of
course
you
might
want
to
update
a
playlist,
so
these
are
sync
get
an
append.
I
just
call
this
different
kind
of
queries,
so
we
will
start
with
the
sync
sync
query,
and
we
first
need
some
way
to
represent
this
entire
graph.
A
So
then
we
have
a
column.
Family
called
playlist
change
where
each
row
is
is
a
playlist.
So
there's
one
billion
rows
in
our
database.
The
the
primary
key
is
simply
the
uri
of
the
playlist.
That's
the
yeah,
the
unique
identifier.
Every
column
is
a
change.
So
every
time
you
make
a
change
to
a
playlist,
you
add
a
new
column.
A
The
name
of
the
column
is
the
revision
number.
The
relation
number
is
assigned
by
the
server
automatically
and
the
actual
value
is
the
change
which
is
just
a
protobuf,
encoded
blob
of
the
change
operation,
including
a
reference
to
the
previous
version.
So
you
can
follow
the
links
back,
so
you
can
walk
back
in
the
graph.
You
start
from
the
head
in
the
graph.
You
can
just
follow
the
links
back
up
until
the
root
and
some
words
about
the
version
number
here.
A
This
is
how
it
would
look
like.
Then
we
on
the
bottom.
This
is
the
same
example
as
I
showed
before.
Actually
so
the
bottom
row
here
we
have
a
short
branch
which
was
a
merge
attribution
four,
which
is
the
rightmost
division,
and
that
particular
revision,
of
course,
has
two
parent
pointers,
since
it's
a
bit
special
all
right.
So
this
is
pretty
much
the
only
only
kind
of
data
you
need
to
store
to
represent
the
entire
graph.
A
So
you
can't
do
that
really.
So,
instead,
we
need.
We
use
a
separate
column
family
to
store
only
keep
track
of
the
latest
revision
of
each
playlist,
and
we
call
this
one
playlist
head.
So
it's
it's
really
a
subset,
a
very
small
subset
of
the
playlist
change
column
family,
where
each
again,
each
row
is
a
playlist
and
each
column
is
again
the
revision
number
of
a
head.
So
in
almost
all
playlists
there
is
only
one
column,
and
that
is
the
latest
revision.
We
don't
store
any
value
in
it.
It's
empty.
A
This
is
only
to
be
able
to
quickly
find
the
latest
revision
and
the
reason
we
had
we're
not
using
a
fixed
column
name
here
we
could
have.
If
you
have
a
fixed
column,
name,
that
we
kept
where
the
value
would
be
revision,
we
would
again
overwrite
if
the
concurrent
changes.
So
that's
why
we
need
the
revision
number
in
the
value
in
the
in
the
name
of
the
column.
A
So
this
is
what
would
happen
if
you
would,
if
there
would
be
two
concurrent
concurrent
changes
to
the
same
playlist
playlist
will,
for
a
short
time,
have
two
columns
and
the
service
when
it
wants
to
find
out,
which
is
the
latest
revision
of
the
playlist.
It
reads:
the
entire
row
in
playlist
said,
which
is
fast.
It's
just
one,
sometimes
two
columns.
It
could
be
three
if
you're
really
unlucky,
it
sees
that
there
are
more
than
one
and
then
it
will
start
this
merging
operation
using
operational,
transform
and
quickly
change
it
into
this.
A
So
playlist,
so
we
need
two
tables
two
column:
families
playlisted
is
a
really
small
column
family.
It
fits
entirely
in
ram.
It
turns
out
that
95
of
all
the
requests
only
needs
to
read
from
this
table,
which
is
in
ram,
which
is
awesome,
and
the
reason
for
this
is
quite
simple.
Is
that
when
you
log
in
to
spotify
client,
you
have
your
a
lot
of
playlists.
Most
of
these
playlists
haven't
changed
since
the
last
time
you
logged
in
they
just
we
have
sent
to
the
server.
I
have
this
playlist.
A
That
is
a
distribution
and
services.
Yep
checks,
playlist,
said
yep.
That
is
indeed
the
latest
tradition.
No,
more
no
need
to
do
anything
more,
but
if
you
play,
if,
if
it
sees
that
okay,
I
have
revision,
seven
comma
something
of
a
playlist
and
the
latest
tradition
is
eight.
Then
I
just
do
a
slice
query
on
that
playlist
change
draw
from
from
the
reverse
the
revision
I
have
to
the
end
of
the
row
and
return
the
data.
The
service
will
have
to
return
the
data
nicely
in
just
a
single,
so
you
don't
have.
A
A
So
this
is
how
we
do
that
we
have
a
separate
table.
Column.
Family
called
plainly
snapshot
it
that
one
has
a
single,
fixed
column
name,
it's
not.
It
does
not
have
a
column
name
that
is
based
on
revision.
I
will
come
back
later
to
why
and
we
have
stored
one
snapshot
and,
of
course,
this
we
store
which
version
of
the
snapshot,
but
that
that
is
part
of
the
the
blob
that
is
inside
inside
this
column
value.
A
So
so
what
happens?
Is
that
when,
if
a
client
wants
to
delete
the
snapshot
or
playlist,
it
just
simply
looks
first
in
the
head
table
again
check,
which
is
the
latest
version?
Okay,
it
find
that
version.
Nine.
Something
is
the
latest.
Then
I
check
the
playlist
snapshot
value.
Is
that
the
version
that
we
have
cached
and
then
usually
that
is
indeed
the
case,
because
whenever
a
playlist
is
updated,
we
make
sure
the
cache
is
the
latest
one,
but
it
can
happen.
A
For
instance,
in
case
again,
there
are
concurrent
changes
that
the
snapshots
might
be
slightly
outdated.
It
stores
only
the
stores,
one
of
these
paths
or
something
or
something
then
the
get
the
guest
qr
will
actually
have
to
update
that
snapchat
cache
and
what
it
does
is.
Okay,
I
picked
existing
snapshot.
I
see
that
this
is
some
old
version,
I'll
load
the
relevant
changes
from
playlist
change.
I
might
even
have
to
go
back
in
history.
Sometimes
I
have
to
refine
the
snapshots.
There
are
cases
for
that
and
then
update
it.
A
So
I
can
just
rewind
or
replay
changes
to
get
from
one
revision
of
the
object
to
another,
and
that
is
quite
cool.
So
but
again,
the
majority
request
is
simply
just
checking
playlist
head.
What
is
the
latest
revision
check
the
cache,
the
persistent
cache
and
find
it's
indeed
the
right
one,
and
then
we
have
to
return
it
and
for
updating
playlist,
that's
quite
simple:
when
you
still
need
an
update,
we
pretty
much
just
add
it
to
the
playlist
change
column,
family.
A
We
need
to
fetch
a
snapshot
of
the
version
we,
the
user
appended
it
to
just
to
make
sure
that
his
operation
is
valid.
So
it's
not
trying
to
exceed
the
maximum
number
of
items
in
a
playlist,
for
instance,
but
in
general
you
have
to
update
the
playlist
change
column,
family.
You
update
the
snapshot
and
importantly
last
you
update
the
playlist
head
column
family.
A
This
is
because
you
don't
want
to
first
update
the
playlist
send
column,
because
if
it
crashes,
then
you
would
have
playlist
head
point
to
reference
to
a
revision
that
does
not
exist
in
playlist
change,
and
that
is
bad.
But
if
you
do
it
other
way
around,
you
will
have
a
reference
in
plain
exchange
that
which
no
one
will
see
so
that's
kind
of
fine.
So
it's
play
ever.
Every
request
really
starts
by
checking
playlist.
So
that's
that's
the
one
you
want
to
update
last.
A
So
this
is
a
nice
way
to
something
that
looks
like
you
would
need
a
transaction
to
make
sure.
But
it
doesn't
you
if
you
just
make
sure
to
update
the
tables
in
the
correct
order
it
will,
it
will
work
out.
Fine,
the
data
data
will
be
consistent
and
everyone
will
be
happy
and
final
thing
about
this
data
model
is
that
it
has
some
interesting
when
it
comes
to
consistency
levels.
A
We
use
replication
factor
three
on
all
our
clusters.
When
you're
reading
playlist
said
to
make
sure
we
really
have
the
latest
version.
We
use
quorum,
both
read
and
write,
read
and
write,
and
this
is
in
ram.
So
it's
pretty
fast,
no
matter
what,
but
for
the
plate
exchange
column,
family,
which
is
really
big.
We
only
need
to
use
consistency
level
one,
and
this
is
because
we
never
mutate
data.
We
never
change
data.
You
can
never
read
an
old
value.
A
You
can
do
a
read
on
something
and
and
expect
the
value
to
be
there,
because
the
playlist
has
said
there
should
be
a
value
there.
And
then
you
read
nothing,
but
then
you
know
that
okay,
let
me
just
retry
that
read
using
quorum
or
or
retry
on
another
node
or
whatever,
but
we
use
quorum
because
the
data
really
should
be
there,
and
so
we
never
we.
So
that
means
that
the
majority
of
requests
can
be
done
with
cl1.
A
If
a
node
was
temporarily
down
or
something
a
few
of,
the
requests
may
have
to
fall
back
to
quorum
before
we
have
this
invention.
Consistency
kicking
in
same
thing
goes
with
the
playlist
snapshot.
It's
just
a
crash,
it's
mostly
all
already
contained
in
the
latest
value,
but
the
service
doesn't
rely
on
that.
So
we're
using
cl1
again
to
read
the
snapshot.
A
A
All
right.
I
have
a
few
minutes
left
here,
so
I
will
quickly
go
into
some
lessons.
We
learned
while
you,
while
trying
out
cassandra,
spotify
and
particular
playlist.
So
cassandra
is
super
robust
we've
used
this
since
versus
0.7
for
playlist,
we've
upgraded
it
all
the
way
to
well
it's
1.1
now,
but
yeah.
Some
other
cluster
is
at
1.2.
A
A
Some
optimizations!
We've
done
level
compaction
wasn't
when
we
started,
we
added
it
worked
wonders
since
we
had
so
many
read,
requests
compressions
turned
out
to
yeah.
It
wasn't
us
not
from
not
from
a
performance
perspective.
It
didn't
help
so
much,
even
though
it
reduced
this
usage.
Obviously-
and
that's
mostly
because
of
internal
crc
checks,
which
the
compression
compression
code
used
some
other
optimizations,
so
playlist
head,
we
thought
that
one
was
would
be
entirely
in
ram
and
we
trusted
the
linux
disk
cache
for
this
turns
out.
This
was
not
a
good
idea.
A
It
didn't
work
as
well
as
we
thought
we
tried
to
use
the
road
caching
cassandra
that
one
is
unfortunately
not
terrible
efficiently
implemented.
It
uses
a
java
heap,
for
instance,
just
that
adds
a
lot
of
overhead
to
things.
We
quickly
run
out
of
ram,
so
instead
we
used
mlok
which
fixes
files
in
memory,
and
that
did
wonders
so
we
had
this
fantastic
small
batch
script
running
on
some
nodes
for
a
time
and
it
improved
performance
a
lot.
So
what
it
does
is
it
takes
all
the
ss
tables
for
the
playlist
head
column.
A
Family
keeps
them
in
memory
forces
it.
The
memory
and
then
updates
every
now
and
then
this
has
since
been
replaced
with
a
slightly
more
robust
python,
python
thingy
and
it's
it's
a
really
good
way.
If
you
have
a
if
you
have
spinning
disks
when
you
really
want
to
fix
some
some
stuff
in
ram,
no,
it
turns
out
now
that
we
don't
really
need
this
anymore
in
our
playlist
system,
and
this
is
because
we
switched
to
future
nio
disks,
which
helped
us
reduce
the
amount
of
disk
machines.
A
This
amount
of
machines
in
the
cluster
from
60
to
30
in
this
case,
and
we
could
have
probably
reduced
it
even
more,
but
there
are
other
bottlenecks
that
we
run
into
now,
like
cpu
and
just
memory.
So
it's
a
lot
of
data
last
tombstones,
I'm
not
sure
you've
heard
about.
I
think
I
already
heard
mention
here
several
times
today
and
yeah.
A
So
the
problem
with
plainly
said
is
that
you
probably
realize
this,
but
every
time
you
make
a
change
to
playlist,
you
delete
that
one
value
in
it
and
you
add
a
new
and
we
delete.
We
don't
overwrite,
we
actually
write
new
column
names,
and
so
we
have
a
lot
of
tombstones,
but
we
thought
well.
They
should
go
away
after
a
while.
We
have
one
week
before
we
compact
the
things
and
then
the
gc
time
and
things,
but
it
shouldn't
be
too
bad.
Perhaps
a
thousand
tombstones.
Perhaps
that
should
be
doable
right
turn
out.
A
A
So
the
solution
we
did
for
this
I
mean
we
could
probably
have
enabled
level
compaction.
It
might
help
the
problem,
but
not
sure
how
much,
but
instead
we
use
major
compaction,
which
is
something
you
probably
should
avoid
in
general,
but
for
this
one,
which
is
a
fairly
small
column,
family
of
future
ram
running
regular
scheduling,
major
compaction
weekly,
it's
actually
not
that
bad.
It
just
takes,
I
don't
remember
normally
15
20
minutes
or
so
to
do
that
and
then
get
rid
of
all
the
tombstones
awesome.
A
So
those
are
the
prob
some
of
the
problems
we've
encountered
and
solved.
We
have
some
other
problems
that
we
have
not
have
encountered
and
not
fully
solved.
They
are
in
progress
of
being
sold,
for
instance,
we
might.
We
have
still
had
this
really
wide
road
and
played
this
change.
We
have
used
it
at
users
that
automatically
update
playlists
every
three
or
four
minutes
based
on
music
played
on
radio
station.
A
So
we
have
these
really
large
graphs
that
we
really
want
to
truncate
and
you
then
you
have
the
tombstone
problem
again
sort
of
except
you
can't
do
major
compaction,
so
there
are
all
kind
of
different
things
that
still
needs
to
be
solved.
So
if
you
think
this
is
interesting,
just
give
us
a
call
so
questions.
A
This
might
be
an
obvious
question,
but
how
do
you
figure
out
how
many
tube
sets
you've
got?
Oh,
we
copied
yeah.
No,
it's
not
an
obvious
question.
We
copied,
since
it's
not
exposed
in
the
api,
so
we
copied
ss
tables
to
a
local
computer
and
then
we
ran
there's
a
tool
ss
table
to
json,
or
actually
we
wrote
our
own
tool
based
on
that
which
actually
just
iterates
through
the
column.
Yes
prints
out,
so
we
wrote
some
old
two,
but
you
can
use
ss
table
to
json.
I
think
I
think
it
exposes
tombstones.
A
Correctly,
clusters
of
having
many
cassandra
classes
versus
having
one
huge
yeah,
so
that's
all
about
the
coupling
we
want
to.
We
have
when
we
add
new
new
features
to
the
product.
We
really
want.
We
don't.
We
might
be
a
bit
unsure
about
how
they
will
perform.
So
if
they,
if
they
underperform,
they
shouldn't
affect
any
other
parts
of
spotify.
That's
really
our
we
try.
This
driver
is
very
very
hard.
We
have
some
of
our
systems
that
have
failed,
have
failed
because
they
didn't
really
decouple
properly.
A
A
Most
likely
that
user
has
very
many
playlists,
I
would
guess,
and
then
it's
because
we
have
a
we
don't
synchronize
all
playlists.
We
don't
synchronize
all
players
at
once.
We
want
to
synchronize
them
in
some
orderly
fashion.
So
you
don't
we're
not
bashing
the
system
when
you
log
in
I.
I
would
guess
that
anyway
could
be
other
things
but
seems
like
that.
A
Given
you
have
some
very
large
version,
history,
graphs.
Are
you
looking
at
doing
some
garbage
collection?
Yes,
that's
what
I
mentioned
in
the
last
slide:
cleaning
up
old
history,
truncating
it.
Yes,
that
is
something
we
actually
do
already.
It's
it's
not
fully
automated!
Yet
because
we
don't
have
the
need
for
that
really.
Quite
yet,
but.
A
A
Yeah,
so
the
reason
it
doesn't
compact
away,
you
mean
it's
because
how
the
way
compaction
works
is
that
you
can
only
remove
a
tombstone
if
you
compact,
at
the
same
time,
all
the
ss
tables
that
contain
data
for
that
row.
So
if
you
have,
if
you
have,
if
you
size
the
compaction,
you
usually
compact
four
tables
at
once.
A
If
you
have
a
piece
of
data
that
exists
in
five
five
tables
that
that
no
tombstone
in
that
will
ever
go
away
slightly
simplified
answer,
but
in
general
and
that's
if
you
start
thinking
about
it,
it
all
has
to
do
with
data,
not
reappearing.
So
there
are
a
lot
of
safety
measures
to
make
sure
that
data
don't
reappear,
which
makes
that
you
can't
remove
the
tombstones
there
are.
There
are
ways
being
worked
around
this.
This
was
valid
a
year
or
so
ago.
I
think,
but
yes,
tombstones
is
a
problem
with
the
level
compaction.
A
Have
you
tried
it?
No,
we,
the
thing
is
we
we'll
use
level
compaction
on
the
playlist
change
and
play
the
snapshot
and
not
playlist
said
in
that
was
we
never
did
that?
That
would
indeed
probably
help
I'm
not
sure
we
didn't
try
to,
because
we
tried
immediately
made
a
compaction
and
it
worked
so
well.
We
never
went
back
and
tried
to
do
level
compaction
again.
That
would
probably
help
unless
I'm
not
sure,
I'm
just
speculating
that.
A
A
A
In
most
cases
we
use
cassandra,
so
each
system
playlist,
is
one
but
pretty
much
all
the
features
you
see
in
spotify
is
a
separate
system
and
each
system
has
its
own
storage,
its
own
cassandra
cluster.
If
it's
cassandra
it
might
be
posgress,
it
might
be
something
else,
and
this
is
just
to
make
sure
everything
is
decoupled
so
nicely
as
possible.
So
each
of
our
clusters
really
only
have
one
key
space,
one
two
or
three
column
families.