►
From YouTube: C* Summit EU 2013: From CQL to Time-Series Event Tracking and Aggregation Using Cassandra and Hadoop
Description
Speaker: Mick Semb Wever, Programmer at FINN.no
Slides: http://www.slideshare.net/planetcassandra/c-summit-eu-2013-from-cql-to-timeseries-event-tracking-and-aggregation-using-cassandra-and-hadoop
FINN.no's is a classifieds website and Norway's busiest website. This session will go through various product development where c* has shown to be the best choice, focusing on our primary c* use-case: our in-house tracking solution that's collects raw time-series data in c* and aggregates minute-by-minute it using hadoop into various new datasets from advert-centric statistics to user-centric behavioural analysis. I'll cover the final technical design chosen after a number of development iterations touching on technologies: scribe, thrift, kafka, hadoop, pig, mahout; the hurdles faced along the way, and the throughput and performance of today's systems.
A
The
talk
this
talk
is
going
to
be
a
run-through
of
use
cases
we
have
in
production
on
Cassandra,
it's
not
going
to
be
a
particularly
clever
or
innovative
talk,
and
we're
certainly
not
going
to
try
and
sell
you
anything
that
we've
written
on
top
of
Cassandra
that
we
think
you
should
be
using
to
it's
really
gonna.
Be
plain:
sell,
look
Cassandra!
A
If
there's
a
reason
for
me
to
want
to
do
this
talk,
it's
the
hope
that
I
can
inspire
confidence
to
bring
Cassandra
to
production,
to
bring
production
faster
or
to
use
Cassandra
more
than
you
are
already
using
today,
a
little
bit
about
the
company
Finn,
it's
a
classifieds
website
and
it
has
very
broad
monopoly
on
the
classifieds
in
the
country.
Here.
A
We're
talking
about
real
estate,
job
listings,
Trading
Post
vehicles,
bits
and
pieces
travel,
cheapest
flights,
chips,
hotels,
cheapest
car
hire
the
only
other
country
in
the
world
which
has
everything
on
one
website
and
is
monopoly
or
dominant
player,
is
New
Zealand
and
being
in
Australian
I
couldn't
possibly
go
work.
There
there's
another
interesting
story
behind
fin
and
that's.
It
was
started
up
in
the
late
90s
in
year,
two
thousand
and
it
will
start
up
by
the
country's
largest
newspaper
corporation
with
the
intention
of
cannibalizing,
their
own
profits.
A
I've
stolen
here
a
slide
from
Doug
cutting
and
a
Hadoop
presentation,
but
I
think
it's
equally
appropriate
for
Cassandra
or
any
new
big
technology.
It's
certainly
an
experience
of
story
which
is
applicable
to
us.
Any
big
established
company
in
their
right
mind
doesn't
jump
on
a
big
paradigm
shifting
technology
without
hitting
a
use
case
that
simply
can't
be
solved
any
other
way,
and
it
can
be
quite
challenging
to
get
that
new
technology
into
production
and
on
top
of
that
you're
working
with
a
challenging
use
case
and
bringing
that
into
production.
A
And
so
now
that
we're
using
Cassandra
more
and
more
for
simple
stuff,
we
have
more
teams
coming
to
us
in
the
company
and
asking
hey.
We
think
we've
got
a
use
case,
that's
going
to
work
well
on
Cassandra,
and
this
is
a
rough
checklist
that
we
have
do
you
have
a
higher
write
throughput.
You
have
a
scheme,
Alice
design,
or
do
you
have
a
doll
model
with
a
lot
of
joins
because
we
don't
do
joins
anymore
volume
of
data
up
time,
etc.
A
I've
made
this
talk
a
little
bit
difficult
for
myself,
because
I
wanted
to
go
through
some
of
those
simple
use
cases
and
also
go
through
our
initial
advanced
use
case,
which
is
going
to
presume
some
knowledge
of
Hadoop.
Either
way.
Please
bear
with
me
before
I
go
into
the
simple
use
cases
I'd
like
to
go
through
our
production
cluster,
we're
not
Netflix
we're,
not
ebay.
We
don't
have
a
particularly
large
cluster
I
still
wanted
to
share
this
story
because
I
think
it's
applicable
for
a
lot
of
people.
A
The
first
use
case
is
our
users
search
history.
This
is
just
like
with
Google.
You
go
into
your
profile
page.
You
can
see
all
the
searches
that
you've
done
for
the
last
six
months
or
three
months
for
us
exactly
the
same
years
case.
It
works
really
well
on
Cassandra.
It
uses
the
time-to-live
feature,
fantastic
feature
Cassandra.
It
is
right
heavy
supposed
to
reeds
and
it
has
wide
rows.
Each
user
has
their
own
petition
key
and
the
searches
just
for
each
user
attacked
on.
A
A
I'm
a
bit
worried
about
time,
so
I'm
going
to
jump
over
one
of
those
use
cases,
but
you
can
download
download
slides
up
to
its
amber.
Look
if
you
want
third
use
case,
is
fraud
detection
that
is
like
any
classifies
website.
We
have
a
real
problem
with
with
fraudulent
ads
and
general
misbehavior
and
it's
never
the
bad
guy.
That
gets
the
blame.
Who
gets
the
blame?
We
get
the
blame,
so
we
spend
a
lot
of
time
having
to
manually
vet
ads
as
they
come
in
to
us.
A
A
A
A
A
This
system
has
two
components
to
it:
one
is
for
the
inbox
data
and
one
is
for
the
email
aliases
and
we
don't
simply
provide
one
email
alias
for
each
user.
Instead,
for
each
users
add
and
potential
buyer
conversation,
we
have
a
separate,
alias
email,
alias
so
we're
collecting
tens,
50,000
email
aliases
each
day
around
all
these
ads.
This
allows
us
when
an
email
comes
in
to
us
to
check.
Is
this
an
email
from
someone
who
is
part
of
established
in
that
conversation?
A
This
is
one
of
only
two
use
cases
that
we
have
in
fin,
which
require
strong
consistency
when
an
email,
alias
email
comes
in,
that
alias
must
match
to
an
email
address,
because
of
that
this
was
sitting
inside
base.
First,
until
we
were
110
percent,
confident
that
we
could
run
it
in
Cassandra
and
what
we
found
when
we
put
it
into
Cassandra
performance,
improved
20
to
40
times,
even
with
strong
consistency.
A
A
A
You
to
to
our
use
case,
which
got
a
stellar
with
Cassandra
time,
series
event
tracking
and
aggregation
or
event
statistics
as
we
call
it.
This
is
all
of
statistics
that
we
show
to
the
user
on
the
website.
We're
not
talking
business
intelligence
here
or
analytics.
This
is
what's
in
our
product.
Each
user
gets
lots
of
different
statistics.
How
many
people
have
shown
how
many
people
have
seen
the
ad?
How
many
emails
around
the
added
being
sent?
A
In
fact,
in
peak
traffic,
if
there
were
any
signs
of
problems
with
the
database
operations,
would
the
first
thing
they
would
do
would
turn
that
right
procedure
off?
That
is
when
users
were
getting
the
most
traffic
on
their
ads,
we
stopped
counting,
so
we
had
to
come
up
with
something
new
and
we
looked
for
a
asynchronous
fault,
tolerant
and
linear
scaling
solution.
So
we
came
up
something
like
this.
It's
come
a
long
way
in
the
last
two
years
is
more
what
it
looks
like
today,
but
we
have
an
event
collection,
part
of
this
system.
A
We
have
where
events
come
in,
they
come
into
Cassandra
into
a
raw
events,
column
family.
Then
we
have
Hadoop
jobs,
running
every
minute
or
every
day,
depending
on
the
statistics
and
then
upgrade
I'm
updating,
separate
column
families
in
Cassandra,
and
then
we
have
a
separate
module
which
reads
those
statistics
and
serves
and
back
to
the
product,
those
events
they
can
be
happening
from
inside
our
own
platform,
server
side
events.
They
can
also
be
client
side
events,
so
we've
kind
of
got
our
own
tracking
ocean
here,
but
slightly
different
purpose
before
inside
base.
A
A
So,
looking
into
the
system,
the
first
I
will
look
into
the
event
collection
side
of
things.
Our
events
are
thrift,
beans
and
they're.
Very,
very
simple.
We
have
a
type
which
is
like
an
ad
something
happened
to
an
ad.
We
have
a
sub
category
which
is
like
page
view,
door,
email
sent
or
marked
as
favorite,
and
then
just
a
map
of
values
that
will
hold
the
ID
of
the
add
the
user,
a
germs
referring
to
Maine,
etc,
etc.
I
mentioned
described
network
we
have
since
installed
calf,
go
into
production
the
last
two
years.
A
We've
lost
too
many
buffers
after
a
period
of
downtime
and
its
options
are
just
archaic.
Some
options,
don't
work
out
or
they're,
just
relics
or
Facebook
code.
Other
options
work
with
only
in
Pacific
combinations,
with
with
other
options
and
there's
no
documentation
and
the
community
seems
to
be
dead.
Kafka,
on
the
other
hand,
is
very
much
alive.
A
It's
a
rather
than
just
hollered
of
individual
processes
on
different
machines.
It's
actually
a
cluster,
so
you
can
send
to
it
synchronous
synchronously,
which
means
you
don't
run
the
risk
of
losing
messages
in
memory.
Keep
in
mind
that
the
big
picture
is
still
an
asynchronous
system.
A
most
important
Kafka
is
stream
processing
a
lot
of
these
events,
which
are
coming
in
a
valuable
pieces
of
information,
and
there
are
consumers
within
our
own
platform
that
want
to
listen
to
those
events
in
real
time.
A
So
the
schema
for
the
events
when
they
come
into
Cassandra.
Look
something
like
this.
We
keep
one
petition
key
of
one
you
want
to
eat
for
each
minute,
so
the
events
are
just
going
into
minute
rose.
We
have
the
times
exact
time
stamp.
The
type
the
subcategory
in
the
map
the
map
will
keep
is
a
JSON
string.
A
This
part
of
the
system
wouldn't
perform
if
we're
using
the
map
data
type,
and
then
we
take
the
row
key
and
we
duplicate
it
again
into
a
column
so
that
we're
going
to
put
a
secondary
index
on
that
before
secondary
indexes
in
Cassandra.
We
had
to
use
in
order
possess,
preserving
petitioner
if
there's
one
thing
that
I
can
say-
and
that
is
please
for
the
love
of
God-
don't
use
an
order,
preserving
petitioner,
it's
a
complete
nightmare.
A
Of
course.
The
real
world
isn't
that
simple.
First
of
all,
our
minute
rose
we
manually
petition
by
adding
an
extra
random
digit.
We
don't
want
hot
rows
in
the
system.
Also,
an
event
has
two
timestamps:
it
has
the
real
time
stamp
when
it
happened
and
it
has
the
collected
timestamp
when
it
came
into
andhra
the
minute-by-minute
Hadoop
jobs,
they're
interested
in
all
the
events
with
a
collected
timestamp
within
the
last
minute,
whereas
hone
do
a
more
analytic
type
job
of
how
much
referring
traffic
that
I
get
from
facebook.
A
Yesterday,
all
the
events
with
a
real
time
stamp,
obviously,
but
for
the
rest
of
the
talk
I
want
to
focus
just
on
the
simple
schemer
okay.
So
the
next
section
is
the
Hadoop
MapReduce
side
of
things
before
I
start
talking
about
that.
I
want
to
just
show
what
the
aggregated
data
is
going
to
end
up.
Looking
like
here.
This
relates
to
the
statistics.
A
A
A
This
system
was
originally
written
with
super
columns
if
you're
still
using
super
column,
stop
take
one
guess
at
where
we
switch
to
a
composite
columns
on
this
graph,
okay,
so
to
Hadoop
when
Hadoop
job
starts,
it
does
to
queries
to
the
events
table.
The
first
is
to
get
the
list
of
splits.
Each
split
becomes
a
task,
and
the
jobtracker
sends
each
task
to
a
task
tracker.
A
A
Is
a
really
important
part
to
talk
about
here,
and
that
is
data
locality,
the
performance
and
throughput
over
who
you,
your
Hadoop
jobs,
depend
on
data
locality
and
unfortunately,
because
Cassandra
only
supports
for
do
zero
point
20,
you
can't
really
enforced
our
locality.
You
can
give
it
the
hints
when
you
create
those
splits,
you
give
it
the
locations
of
where
that
data
is
found,
but
it's
only
a
hint.
You
find
that
when
the
job
tracker
is
handing
out
the
tasks,
if
all
the
task
track
is
where
that
data
would
be
locally
found
are
currently
busy.
A
But
there
are
other
tasks:
trackers
free
job,
Jagger
won't
wait
it
ships
them
straight
off.
It's
only
in
Hadoop,
zero
point
22
with
this
property,
an
affair
scheduler
that
you
can
say
job
tracker,
wait
for
one
of
those
task,
trackers
to
free
up
to
ensure
that
you
get
done.
Locality
I,
put
a
link
here
to
a
github
project
which
takes
Cassandra's
integration
classes
to
a
dupe,
and
it's
a
very
simple
patch.
A
The
MapReduce
algorithm
is
very
simple:
it's
just
a
glorified
word
count.
Each
of
those
event
pick
up
the
ID
that
you're
interested
in
count
as
one
combine
it
all
together.
We
have
this
extra
dimension
for
those
subcategories
and
then
in
the
reduced
job,
for
each
roll
up
put
it
in
so
at
the
end
of
the
job
you
get
all
of
those
combined
numbers
feature
that
I'd
each
of
the
ad
IDs
and
each
of
the
roll-ups,
and
it
just
gets
dumped
onto
the
existing
counters
in
Cassandra.
A
In
production
we
have
about
five
and
a
half
thousand
minute
jobs
each
day
about
six
full
day
jobs
during
the
night.
That's
around
300
million
records
into
the
system
and
50
million
records
out
of
the
system
each
day,
one
of
those
day
jobs
is
it
taking
about
18
million
or
100
million
records
in
and
writing
out
half
a
million
23
million
records
out,
and
we
have
seen
those
jobs
run
in
as
little
as
three
to
four
minutes,
so
those
performance
that
we're
really
happy
with
and
we're
maintaining
web
latency
on
the
system.
A
At
the
same
time,
the
graph
here
is
really
important
because
those
Peaks
are
not
our
peak
traffic.
Our
Cassandra
graphs
are
different
to
all
the
other
performance
graphs.
We
have
all
of
our
performance.
Graphs
go
up
a
little
bit
as
we
hit
peak
traffic.
Cassandra
remains
dead
flat
through
our
peak
traffic.
A
Those
Peaks
are
the
reeds
in
the
middle
of
the
night,
when
we
run
the
full
day,
jobs
and
those
six
full
day
jobs,
we
throw
it
all
on
at
once.
We
hit
Hadoop
hard
and
heavy.
We
want
to
see
how
the
things
how
everything
works
when
Hadoop
is
doing
its
best
to
bring
it
to
its
knees,
and
so,
if
you
go
back
and
look
at
all
the
other
performance
graphs,
you'll
see
the
same.
Peaks
they've
been
caused
from
this
those
Peaks
on
the
other
graphs.
A
A
We
use
pig,
we
use
pig
for
debugging,
very
valuable
and
for
ad-hoc
jobs
you
don't
want
to
have
to
write
code
and
deploy
it
to
run
your
heart
ad-hoc
job
I
found
pig
to
be
a
bit
difficult
to
learn.
One
thing
is
to
learn
it,
but
it's
got
some
peculiarities
there
too,
but
once
you've
got
a
little
bit
under
your
belt
it.
It
helps
a
lot.
A
A
This
is
the
approach
that
we
wanted
to
start
with.
We
wanted
something
that
we
could
understand
that
we
could
easily
easily
measure
the
results
of
before
we
go
forward
to
some
of
the
more
complicated
solutions
out
there.
For
example,
the
taste
recommendation
system
from
mahou,
which
is
today
further
developed
in
Mirax.
A
A
B
A
A
Originally,
our
events,
column
family,
was
storing
those
thrift
beans
in
the
civilized
state.
They
were
civilized
over
thrift
and
byte
array
and
we're
just
putting
them
straight
in
two
events:
Colin
family-
that
was
disk
space,
mainly
from
of
that,
is
that
no
one
can
actually
read
that
read
it,
and
transparency
is
gold
for
adoption
of
new
technologies.
A
Counters
really
important,
but
if
there's
anyone
from
data
snacks
here
we
really
want
the
Second
Amendment
implementation
of
counters
that
give
you
petition
tolerance,
cql,
three,
fantastic,
we're
doing
everything
in
c
QR.
Three
now
and
being
able
to
use
sequel
and
MapReduce
has
been
a
huge
step
forward
to.
A
Avoid
order
preserving
petitioner,
avoid
customs
here:
alliance
data
avoid
skinny
rose
on
non
commodity
machines,
I,
don't
know
if
this
is
still
the
case
for
Cassandra
to
butt
in
Cassandra
one
and
one
point
one
I
believe
if
you
were
getting
billions,
billions
and
billions
of
rows
on
each
node.
Your
bloom
filters
were
getting
large.
All
of
this
all
this
stuff,
which
was
previously
in
the
heap,
was
just
getting
larger
and
larger
and
larger
and
was
causing
problems.
A
A
Can
you
imagine
if
that
was
happening
to
your
traditional,
my
squirrel
or
sybase?
It
was
locking
up
on
you
twice
a
day.
It
would
be
a
business
disaster
because
Cassandra
wasn't
a
problem
except
for
the
job
tracker
and
then
a
name
node
whenever
it
happened
to
that
node,
the
dude
went
down
so
we're
since
we
have
moved
job
tracker,
a
name
node
to
a
very
small,
separate
virtual
server.
A
A
The
other
thing
that
we've
learnt
is
pay
attention
to
disk
latency
and
utilization
find
that
if
it
goes
about
twenty
percent
for
us,
if
it
goes
about
twenty
percent,
that's
usually
you
should
look
into
it.
You
could
be
asking
for
trouble.
Some
of
the
tuning
that
we
have
done
is
use
the
noppe
scheduler
for
the
colonel
I
know
that
Patrick
mentioned
using
the
CF.
This
fq
scheduler,
which
is
a
default
on
Linux
I,
believe
that's
what
we
were
using
and
for
us.
A
B
A
A
A
A
A
A
B
A
Yes,
yes,
we
would
like
to
have
moved
forward
where
they
do
more
I
can't
give
you
an
exact.
So
the
question
was
why
we're
using
Cassandra
instead
of
HDFS
I,
can't
give
you
the
exact
reason.
I've
just
can't
be
coming,
bring
it
to
mind
when
we
went
through
it,
but
what
I
can
say
is
that
we've
been
a
lot
more
happy
with
Cassandra
and
with
Hadoop.
A
Cassandra
has
been
fantastic
product
to
work
with
and
suddenly
I
think
that
cassandra
is
your
big
data
platform,
because
it
can
do
a
lot
of
things
in
real
time
for
you.
Well,
Hadoop
is
a
useful
tool
to
have
on
top
of
it,
because
it
does
the
more
batch
processing
side
of
stuff
hdfs.
We
keep
that
on
the
SSD
disk
and
it's
a
volatile
file
system.
So
if
you
stop
purdue
you're
always
free
to
wipe
it.
So
it's
only
really
two
for
the
hadouken
tonals.
B
A
A
A
A
A
Yeah
be
careful
upgrading
to
the
major
versions
and
going
crazy
with
settings.
We
got
bitten
quite
hard
by
we
upgraded
quickly
to
Cassandra
one
because
it
offered
compression.
We
could
see
what
that
that
was
a
huge
win
for
us
and
we
jumped
in
and
tweak
the
whole
of
the
settings
which
we
thought
would
be
good
for
us
and.