►
Description
Speaker: Mick Semb Wever, Programmer at Finn.no
FINN.no is the leading classifieds website in Norway and the country's busiest website. This session will go through various product development where Cassandra has shown to be the best choice. Focusing on the primary use case: a tracking solution that collects raw time-series data in C* and aggregates it in near-real-time using Hadoop into various new datasets from advert-centric statistics to user-centric behavioural analysis. Mick will cover the final technical design chosen after three years of development iterations, touching on technologies: scribe, thrift, kafka, hadoop, pig, mahout; the hurdles faced along the way, integration improvements done between Cassandra and Hadoop, and the throughput and performance of today's systems.
A
Little
bit
about
the
company
Finn,
it's
Norway's
busiest
website,
it's
a
classifieds
website.
It
has
the
vast
majority
of
classifieds
and
traffic
in
every
single
vertical
of
classifieds.
So
we're
talking
about
real
estate,
job
listings,
cars,
boats,
motorcycles,
bits
and
pieces,
cheapest
flights
and
chips,
hotels,
services,
professional
and
amatuer
wanted
in
on
for
sale.
Only
other
country
in
the
world,
which
has
that
dominant
position
on
all
verticals
on
11
website
is
in
New
Zealand.
You
can
see
we're
in
quite
a
unique
position
there
that
we're
basically
sitting
on
the
country's
online
buying
and
selling
experience.
A
The
story
behind
Finn
is
also
quite
interesting
in
that
in
99.
It
was
started
up
by
the
country's
largest
media
corporation,
so
it
was
a
start
up
with
the
intention
of
taking
the
classifieds
away
from
the
newspapers
and
putting
them
online
and
cannibalizing
its
own
profits.
There
was
a
hugely
successful
adventure
and
has
led
to
two
things:
one
that
media
corporations.
Shipstead
is
much
the
envy
of
media
corporations
around
the
world
at
the
newspaper,
profits
plummet
and
two
it
is
led
to
ship
stirred
heavily
investing
in
classifieds
throughout
Europe.
A
A
Now
these
statistics,
previously
just
for
collecting
the
numbers
in
their
totals,
was
already
taking
up
fifty
percent
of
the
databases
right
CPU
time,
in
fact,
during
peak
traffic.
If
there
are
any
problems
in
production,
the
first
thing
operations
would
do
would
be
to
turn
off
that
right
procedure.
So
when
users
were
getting
the
most
traffic
and
the
most
statistics,
we
had
to
stop
counting.
A
So
we
evaluated
different
scalable
databases
and
when
we
looked
around,
there
were
three
things
that
we
looked
for.
We
looked
for
the
feature
list
of
those
databases.
We
looked
at
the
quality
of
the
code
in
those
open
source
projects
and
we
looked
at
the
quality
of
community
in
those
open
source
projects
and
at
that
time
there
are
really
only
two
choices
based
off
those
criterias
and
that
was
Cassandra
and
HBase
for
that
project.
A
A
A
Once
we
now
that
module-
and
it
was
quite
a
tough
module
for
us
to
nail-
it
wasn't
just
new
technology
Cassandra-
it
was
a
new
technology
Hadoop
and
scribe.
We
were
also
dealing
with
brand
new
paradigms
in
our
development.
A
synchronicity,
scalability
fault
tolerance,
circuit
breakers
will
still
very
new
at
the
micro
services
platform
and
trying
to
change
whole
organization
to
adapt
to
it
once
we
nailed
it.
We
found
that,
with
this
established
Cassandra
platform
in
the
company,
it
wasn't
just
our
big
data
platform.
A
It
was
also
incredibly
suited
to
all
these
other
simple
use
cases
that
we
thought
lived
in
the
SQL
world
and
especially
in
the
last
12
months,
we've
seen
a
explosion,
a
doubling
of
use
cases
coming
into
Cassandra,
often
quite
simple,
small
things,
but
they
just
work
so
much
better.
On
top
of
Cassandra.
A
A
Improvement
and
write
performance
reads:
well
we're
usually
bringing
over
the
schemas,
which
makes
sense
in
Cassandra
and
when
they
make
sense,
everything
improves
and
reads
can
too
in
some
use
cases.
So
we've
gone
from
a
sybase
or
postgres
over
to
Cassandra.
We've
seen
a
20-fold
improvement
in
reading
performance
this
year
as
well.
We've
seen
one
third,
the
number
of
serious
the
production
incidences
compared
to
our
SQL
databases.
A
Norway
is
a
little
country
in
the
other
room.
You
have
apple
talking
about
10,000
nodes,
and
here
we're
talking
about
eight
nodes.
Part
of
this
talk
is
to
show
you
how
much
you
can
do
with
a
small
cluster
and
I
think
it's
also
a
relevant
journey
to
share,
because
this
is
where
people
start.
You
start
with
a
small
cluster
and
the
things
that
you
learn
on
a
small
cluster
stays
with
you.
As
your
cluster
gets
bigger
and
bigger.
A
Now
the
cassandra
recommendation
is
to
have
separate
data
centers
for
offline
and
online
use
cases.
We
don't
have
enough
nodes
to
properly
follow
that
recommendation,
so
we've
got
all
of
our
offline
and
online
applications
in
the
one
cluster,
but
to
ensure
low
latency
times
for
the
online
applications.
A
We've
got
these
two
nodes
that
don't
have
a
dupe
on
them
and
to
do
that,
we
ensure
that
they're
each
sitting
in
their
own
virtual
data
center,
so
that
online
applications
has
a
replica
always
on
those
nodes,
and
then
we
leave
it
up
to
the
clustered
snitch
and
cross
node
timeout
and
the
rapid
read
protection
to
ensure
that
the
requests
will
always
go
to
those
online
nodes.
If
Hadoop
is
really
bringing
the
other
nodes
down
to
its
needs,
knees.
A
2015
we
have
a
budget
to
move
towards
something
like
this.
I
did
forget
to
mention
that
this
is
dedicated
hardware.
It's
quite
beefy
machines.
That's
what
another
reason
we
have
a
smaller
cluster
and
it
also
takes
time
for
us
to
get
new
nodes.
We
have
like
a
three-month
turnaround
for
Africa.
New
server
in
2015
will
be
moving
towards
a
larger
cluster
like
this.
We
really
need
it.
A
This,
of
course,
has
opened
up
what
we
can
do
with
statistics
here.
A
number
of
screenshots
that
we've
got
top
left
hand
corner
is
statistics
for
one
particular
ad
over
days
and
all
the
different
types
of
statistics,
pageviews
emails,
sent,
etc.
The
middle
in
the
top
is
pageviews
for
one
particular
ad
by
a
day,
in
a
column,
stacked
columns,
we
show
which
variant
of
the
website
was
used
to
the
desktop
or
the
mobile
website
and
against
which
client
device
was
used.
A
Android
iOS
desktop
tablet,
underneath
that
is
the
same
graph,
but
this
time
broken
up
by
referring
domain
in
the
bottom
left-hand
corner
with
the
horizontal
bars
you've
got
repeat
visitors,
so
the
bottom
horizontal
bar
is
how
many
people
have
seen
your
ad
once
the
bar,
above
that
how
many
people
have
seen
your
head
twice
above
that?
How
many
people
have
seen
it
three
times
it
gives
you
a
good
intent
to
the
seriousness
of
the
buyer.
A
On
the
very
right
hand,
side
you've
got
a
screenshot
which
puts
a
lot
of
these
statistics
together
from
a
mobile
app
that
we've
written
in
the
line
graph.
You
see
the
page
views
for
your
ad
overlaid
onto
average
traffic
pattern
and
that
average
traffic
pattern
line
you
can
choose.
For
example,
if
you're
selling
a
house
you
can
say,
what's
the
average
traffic
since
the
day
got
since
the
ad
got
published
for
a
house
in
Norway
or
for
house
in
this
price
range
in
Norway
or
for
a
house
in
his
price
range
in
this
suburb.
A
A
Hak
me
went
with
the
aggregation
approach
and
there's
a
number
of
reasons
for
that.
First
of
all,
when
we
started
34
years
ago,
these
technologies
were
still
quite
new
and
we
wanted
to
open
up
our
horizon
and
we
wanted
to
work
with
just
Cassandra
and
to
do
we
knew
if
we
introduced
to
many
new
technologies
would
not
end
up
with
a
solution
which
was
inflexible
rather
than
opening
up
a
horizon.
A
A
So
the
events
which
are
created
are
very
simple:
in
our
micro
services
platform
we
either
have
refs
services
or
we
have
api's
/
thrift.
Everything
that
goes
into
our
message.
Bus
system
is
defined
by
thrift.
So
here
we
have
a
very
simple
thrift
bean
three
values,
pipe
subcategory
properties.
So,
for
example,
a
pageview
to
an
ad
is
type
ad
sub
category
page
view
and
the
properties
are
going
to
have
the
idea
to
add
the
idea,
the
user,
the
browser
type,
the
device
type
except,
etc.
A
Message
about
system
was
originally
scribe.
Scribe
hit
the
sweet
spot
for
this
project.
It's
a
very
resource
efficient,
but
we've
moved
away
from
scribe
and
to
Kafka
for
a
number
of
reasons.
One
scribe
was
pretty
much
a
dead
project
too.
It
didn't
give
us
a
combination
of
fault,
tolerance
and
load,
balancing
that
we
expected
it
to.
At
the
same
time,
in
our
micro
services
platform,
Kafka
was
chosen
as
of
the
message
bus
solution,
and
it
certainly
comes
with
a
lot
more
options.
A
For
example,
now
we
can
also
do
direct
streaming
of
those
events,
but
Cathcart
uses
a
lot
more
resources,
while
scribe
demons
are
originally
just
almost
invisible
processes
running
on
every
machine.
We
now
have
a
five
node
Kafka
cluster.
You
compare
that
to
a
eight
node
Cassandra
Hadoop
cluster.
We
were
actually
putting
Hadoop
jobs
in
and
running
them.
There
there's
kind
of
an
odd
comparison
to
efficiency
of
resources.
Also
caprica
uses
zookeeper,
which
is
kind
of
an
unfortunate
when
the
asynchronous
component
to
your
service.
A
A
A
A
The
other
thing
in
the
real
world
is
that
there
is
in
fact
two
timestamps.
There
is
the
real
time
stamp
for
when
the
event
happened,
and
there
was
the
time
stamp
that
got
collected
in
Cassandra.
So
all
of
those
incremental
aggregating,
Hadoop
jobs,
they're
interested
in
what
are
all
the
events
which
are
coming
to
Cassandra
since
the
last
time
I
executed.
Whereas
if
I'm
writing
an
ad
hoc
query,
for
example,
how
many
bicycles
were
sold
in
Oslo
for
200
kronos
on
the
seventeenth
of
May
I'm,
working
with
real
time
stamps.
A
A
Things
he
have
come
a
long
way
with
sequel.
Sequel
has
really
changed.
This
picture.
For
us,
the
sequel
driver
will
a
favor
localhost
coordinator
and
with
a
consistency
level,
local
one
always
be
reading
data
from
the
local
data
center,
and
that
helps
a
lot.
But
the
problem
can't
be
solved
there,
because
it's
about
the
capacity
schedule
in
Urdu
and
how
it
sends
out
tasks,
each
task
that
comes
with
a
list
of
locations
where
its
data
can
be
found
where
the
replicas
set
and
the
job
tracker
will
try
and
send
those
tasks
to
those
task
trackers.
A
A
For
example,
if
you
set
it
to
three,
what
it
does
is,
instead
of
sending
that
task
to
the
free
any
free
task
tracker,
it
will
instead
put
the
task
to
the
back
of
the
queue
and
try
and
send
the
next
task
somewhere,
and
once
the
task
has
been
through
the
queue
three
times.
It
will
then
just
give
up
and
send
it
anywhere
with
that
solution.
We
seed
our
locality
rates
well
over
ninety
nine
percent,
and
we
see
at
times
the
difference
means
a
throughput
four
times
greater.
A
A
Each
day
we're
running
over
5,000
continuous
Hadoop
jobs,
those
who
do
jobs
run
in
about
10
to
15
seconds,
so
you
can
kind
of
see
that
we're
breaking
down
the
technical
separation
between
what
is
streaming
and
what
is
aggregation,
the
streaming
solutions.
They
also
do
batch-
and
this
is
just
continuously
running
job
yeah.
A
But
these
jobs
can
catch
up
after
a
period
of
downtime
or
maintenance,
far
quicker
than
streaming
solutions
can
they
can
tack
tackle
days
or
weeks
worth
of
data
very
effectively
or
just
the
last
minute
of
data.
We
have
numerous
nightly
jobs.
We
have
ad
hoc
jobs
each
day,
even
when
we
only
had
four
Hadoop
nodes.
We
were
reading
about
a
billion
records
a
day
out
of
Cassandra
and
writing
back
about
150
million
records.
A
Average
is
interesting
one
because
it's
an
example
of
where
you
need
larger
data
sets
to
calculate
the
data,
for
example,
here
in
the
schema
that
we've
got
here,
group
is
like
those
groups
I
mentioned.
Single
houses
are
houses
in
norway
or
houses
in
this
price
range
or
houses
in
this
price
range.
In
this
little
Valley
and
the
day
is
how
much
traffic
on
average
or
the
max
traffic
or
the
men
traffic
it
got
on
the
first
day
since
it
got
published
at
the
second
day
since
it
got
published.
So
you
get
these
nice
graphs
of.
A
A
So
what
we
do
to
make
it
incrementals
we
have
this
TS
timestamp
clustering
key,
and
so
when
we
run
the
job,
we
run
it
over
the
last
couple
of
weeks,
and
that
goes
into
a
data
set
with
a
pacific
time
stamp
and
when
that's
finished,
we
then
fold
it
back
into
the
original
data
set
which
has
x
f0
and
because
we
have
sample
size,
we
can
do
the
weighted
folding
for
all
those
statistics.
There's
another
clustering
key
here
called
re,
adding
sense!
That's
because
we
also
do
sliding
windows
of
averages.
A
In
our
micro
services
platform
we
use
Zipkin.
If
you
haven't
heard
of
zipkin
it's
distributed
tracing,
it
came
from
Twitter
and
it
uses
architecture
very
much
the
same
as
ours.
It
sends
all
in
all
its
messages
or
the
traces
through
scribe
and
then
go
into
Cassandra,
and
it
provides
you
add,
distributed
tracing
like
this
makes
it
very
easy
to
performance
profile,
cross
microservices
architecture.
You
can
also
get
this
graph
in
the
browser
with
a
zip
can
plug,
in
extension,
sit
at
the
bottom
of
the
screen.
A
We
have
a
Hadoop
job
written
in
cascading,
which
runs
once
a
day
and
analyzes
all
of
those
traces
and
builds
up
for
us
an
overview
of
our
micro
services
platform
and
also
timestamps
it.
So
we
can
see
how
the
Micra
services
platform
is
evolving
if
there
are
connections
being
created
which
we
don't
want,
or
what
we
should
be
working
on
separating
out.
A
A
Yarn
has
been
a
fantastic
improvement,
has
changed
a
lot
of
things
for
us.
We
were
already
using
pig
a
lot,
but,
like
you've
heard
in
this
conference,
a
lot
already
I'm
sure
spark
is
just
fantastic
spark
integrates
really
nicely
with
her.
Dupes
spark
isn't
good
enough
yet
for
us
to
solve
all
I
use
cases,
so
we're
still
kind
of
steady
on
Hadoop,
but
the
trend
is
more
and
more
with
spark
code
being
written
for
our
recommendation
system.
A
We
have
a
number
of
pre-processing
jobs,
generating
data
sets
that
we
need,
and
then
we
run
spark
ALS
to
generate
a
wreck
that
seems
to
generate
the
best
model,
but
we
write
that
model
back.
In
my
help
format
and
in
the
recommendation
module,
we
use
my
house
to
deliver
recommendations
that
combination
between
the
two
hits
a
sweet
spot
for
us
again.
This
is
pretty
much
all
the
spark
code.
That's
used
to
generate
the
LS
model.
It's
a
refreshingly
simple
and
brief.
A
That
wraps
up
the
code
run
through
I've
got
two
more
slides.
The
first
one
is
a
compliment
to
the
Cassandra
community.
Like
I
said
it
just
keeps
on
coming
leaps
and
bounds,
some
of
the
things
which
have
really
meant
a
lot
for
us
date,
tier
compaction
strategy,
we're
already
running
in
that
we
have
been
for
two
weeks
in
production
on
two
different
key
spaces.
One
of
them
didn't
work
out
so
well,
so
we
went
back
to
levelled
another
one
works
really
well,
just
with
the
default
settings.
A
Secondary
indexes
meant
a
lot
for
us.
Compression
meant
a
lot
for
us.
Cql
3
is
fantastic,
especially
the
integration
in
MapReduce
and
spark
automatic
pagination,
even
though
it
sounds
so
simple,
it
really
simplified
a
lot
of
code.
For
us,
the
fluent
api
to
the
sequel
java
is
brilliant.
You
know
you
can
have
this
hardcore
SQL
fans
sitting
next
to
you.
Looking
over
your
shoulder,
you're,
writing
your
persistence
layer
and
Cassandra
damn.
A
Have
a
solid
monitoring
go
overboard
with
monitoring
we
have
off
center,
we
have
muninn,
we
have
stats
d
and
graphite
in
a
client
pools.
We
put
it
from
the
jmx
pools.
We
put
that
through
stats
d
and
graphite,
we've
seen
from
Colonel
upgrades
latency
on
SSD
disk
go
up
to
50
millisecond
average,
and
it
takes
us
months
to
see
it
because
Cassandra
sorry
bastard
just
keeps
on
working
and.