►
Description
Check out the full post for more information: http://www.hakkalabs.co/articles/spark-shark-cassandra-data
Evan Chan (Software Engineer, Ooyala), describes his experience using the Spark and Shark frameworks for running real-time queries on top of Cassandra data. He starts by surveying the Cassandra analytics landscape, including Hadoop and HIVE, and touches on the use of custom input formats to extract data from Cassandra. Then, he dives into Spark and Shark (two memory-based cluster computing frameworks) and explains how they enable often dramatic improvements in query speed and productivity.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
Been
working
on
Apache,
Cassandra
and
spark
for
well
Cassandra
for
a
number
of
years
and
here
to
talk
about
this
really
interesting
fusion
of
two
two
technologies.
So
we
will
tell
you
a
little
bit
about
how
we
use
Cassandra
at
ariela
and
how
we
are
integrating
Cassandra
and
sparking
a
little
bit
about
how
you
can
use
spark
with
Cassandra.
A
So,
who
are
we?
This
is
a
really
brief
introduction.
We
are
go
with
the
power
personalized
video
experiences
across
all
screens.
That
means
in
a
pipeful.
Unless
stated,
you
are
watching
video
on
one
of
our
customer
sites
like
ESPN
and
whenever
you
watch
a
video,
we're
getting
events
we're
about
a
seven
year
old
company
and
since
the
very
beginning,
we've
been
very
focused
on
analytic.
A
So
as
of
today,
we
get
maybe
two
and
a
half
billion
analytics
pings
a
day
so
that
from
our
customers,
customers
and
our
customers
has,
you
know
pretty
big
list
of
some
some
of
the
top
names.
Whenever
you
watch
the
video
under
sites,
we
are
getting
data
and
we're
trying
to
help
them
figure
out
like
who's
watching
video
and
how
are
they
watching
it
so
that
they
can
improve
everyone's
experiences
with
video,
we
are
fairly
large,
Cassandra
user.
A
We
are
hoping
to
do
at
some
point.
We
eventually
try
out
the
newer
versions
like
20
to
that
one,
and
we
are
also
starting
to
become
a
big
smart
user.
We
have
multiple
teams
using
spark,
and
it
seems
that
especially
recently
spark
has
gone
a
huge
amount
of
momentum.
It
is
now
a
top-level
paci
project.
A
This
quick
show
of
hands
like
how
many
people
have
heard
of
a
spark,
so
fair,
okay,
so
pretty
pretty
well
known
and
how
many
people
are
actually
using
it
in
their
internet,
other
companies,
okay,
so
so
fairly,
small,
okay,
so
so
I
think
this
is
about
the
right
level
of
talk,
we'll
we'll
go
a
little
bit
into
what
spark
is
and
hope.
You
guys
see
the
use
case
for
for
using
sparkles
Cassandra.
A
So
what
so?
What
problem
are
we
trying
to
solve
them?
And
this
is
this
applies
not
just
to
us,
but
to
most
people.
I
imagine
is
that
we
get
a
lot
of
data
right.
This
is
I
mean
it's
not
quite
that
bad
right,
but
the
general
raw
data
is
pretty
useless
in
the
sense
that
you
know
you
have
a
lot
of
it,
but
you
don't
quite
know
how
to
deal
with
it.
You
can't
quite
present
it
to
the
your
customers
right,
so
we
want
to
go
from
mountains.
A
We're
all
data
to
to
these
nuggets
that
you
can
present,
and
you
want
to
do
this
quickly
and
painlessly
as
easy
as
possible,
and
you
want
to
do
it
at
scale
and
no
doubt
you
guys
are
here
because
of
Apache
Cassandra,
because
under
helps
what
whole
bunch
of
these
things
with
the
quickly
in
that
scale.
But
let's
dive
into
the
problem
space
a
little
bit
further.
A
So
what
we
have
before
using
spark
is
that
we
have
pre
computed
aggregates.
So
for
us
our
data
is
defined
along
several
dimensions,
for
example
geo
such
as
country
city
device,
type
things
that
you
get
from
the
user
agents
and
was
browser
this
kind
of
thing.
So
what
we
did
was
that
we
would
precompute
and
in
Cassandra
this
is
some
of
you
guys
have
probably
been
over
at
the
other
talk
with
data
modeling
and
cassandra
is
very
popular
to
denormalize
your
data.
A
So
if
we
had
data
that
could
be
sliced
nine
different
ways
we're
just
going
to
compute
it
nine
different
ways
and
store
it,
you
know,
storage
is
cheap
and
and
what
what
you
get
with,
that
is
that
you
get
very
fast
lookups.
So
for
any
one
of
those
nine
breakdowns,
we
can
get
very
fast
lookups.
We
read
the
data
back
right
away
and
everyone's
happening
right.
A
The
problem
with
this
is
that,
though,
that
it
is
so
it's
a
little
bit
inflexible
right,
so
inflexible
in
terms
of
the
way
you
curry
it
they'll.
Also
for
us,
it
was
a
bit
hard
to
change.
So
if
we
add,
if
we
needed
to
do
a
new
kind
of
query,
that
meant
that
you
had
to
add
a
new
column
family
and
you
to
change
your
entire
right
pipeline
to
write
things
in
your
way
right.
A
The
other
problem
is
that
it's
hard
to
know
like
you
need
to
break
down
things
a
lot
of
different
ways,
but
in
fact
most
of
the
data
that's
written,
never
gets
read,
because
it
turns
out
that
people
care
about
certain
queries
a
lot
more
than
others
right
and
what
happens
if
we
need
to
do
more
dynamic
queries.
For
example,
I
one
of
the
dimensions
is
geo,
for
example,
cities.
Another
one
might
be
the
type
of
device
you
are
using,
such
as
mobile
versus
tablet.
A
Whatever
wanted
more
complex
queries
such
as
I
wanted
to
cross
I
want
to
find
out
the
top
mobile
users
in
France,
or
something
like
that,
so
that
involves
a
cross
part.
Now,
if
I
had
to
materialise
all
the
different
variations
that
people
could
want
that
this
would
be
like
very.
It
would
be
really
really
hard
to
do.
Oh
and
you,
with
this
run
out
of
space
right
and
there
might
be
more
interesting
kind
of
queries
that
you
want
to
query
so.
A
The
big
question
is
you
want
to
use
concern
for
a
scale,
but
how
can
you
serve
more
complex
queries
quickly
right,
so
you
can
imagine
that
there's
kind
of
a
there's,
a
continuum
on
one
side
of
the
continuum.
Let's
see
this
is
actually
oh
cool.
Okay,
so
on
one
side
of
the
continuum
is,
let's
say:
I've
we
compute
everything
which
is
what
I
was
just
describing
you
get
fast
lookups,
but
it's
pretty
inflexible.
On
the
other
hand,
you
can
imagine
totally
other
end
of
the
spectrum.
A
Is
that
my
data,
my
competition,
is
dynamic,
in
other
words,
imagine
that
if
I
always
like
random
every
ducharme
to
get
the
restart
at
one,
so
I
have
a
lot
of
flexibility,
but
it
is
really
slow.
So
that's
not
going
to
work.
If
you
want
to-
and
you
know
answer-
queries
like
interactively
right
and
I-
think
what
we
really
want
is
we
want
something
in
the
middle
that
is
partly
dynamic
and
where
you
can
have
enough
pre
aggregation,
but
it's
also,
but
it's
also
flexible
and
fast
at
the
same
time.
A
So
now
why
spark
so
a
really
quick
introduction
for
folks
that
are
not
aware
spark
is
an
in-memory
distributed
computing
framework.
It
actually
does
support
spilling
over
results
to
disk.
So
it's
not
strictly
in
memory
it's
created
by
came
out
of
the
UC
Berkeley
and
blab
about
34
years
ago
and
initially
targeted
problems
that
mr
was
really
bad
at.
So
just
iterative
algorithms,
where
you
need
to
keep
spinning
on
the
same
data
set
again
and
again
and
data
mining.
A
It
is
more
general
purpose
than
then
who
do
in
that
it's
being
applied
to
all
sorts
of
problems
and
right
now
it
is
getting
a
huge
amount
momentum.
It
is
just
graduated
the
Apache
Incubator,
so
it's
now
a
top-level
project
projects.
It's
getting
a
lot
of
contributions
like
the
rate
of
change
in
the
project.
This
is
actually
like,
accelerating
right
now
so
yeah
a
lot
of
interest
and
just
a
really
quick
recap
of
the
difference
in
model
between
spark
and
Hadoop.
A
In
Hadoop
we
start
out
with
data
and
ex
gfs,
and
what
you
do
is
everything
is
broken
out
into
MapReduce
faces,
so
we
read
data
out
of
HDFS.
Does
some
map
on
it?
Then
you
do
a
shuffle,
and
after
you
reduce
the
data,
then
you
write
back
to
HDFS.
Now
most
jobs
don't
take
more
than
just
one
stage,
so
you
end
up
having
multiple
stages,
which
is
reading
data
out.
Writing
it
back
reading
data
out
waiting
it
back
right
so
that
that's
how
that's
how
it
works,
spark
is
a
little
bit
different.
A
A
This
is
all
happening
in
memory
and
then
the
magic
is
that,
instead
of
having
to
write
it
back
to
HDFS,
you
can
write
it
back
to
a
traverse
if
you
want,
but
you
also
have
the
option
to
cache
the
intermediate
result
set
in
memory,
then,
once
this
is
done,
you
could
then
do
more
transformations,
and
this
is
read
out
of
memory,
add
10
to
100
x,
SP
de
compared
to
do,
and
what
we
find
is
that
throughput
and
the
in-memory
is
really
a
huge
benefit.
So
I
have
three
use
cases
here.
A
One
is
reading
out
data
out
of
Cassandra
when
the
caches
code.
Another
is
this
report
I
get.
When
I
read
out
of
a
warm
cash
now
warming
a
cash
does
make
a
huge
difference
photo,
as
most
of
you
know,
cash.
It's
really
hard
to
control
like
some
of
the
dl
to
come
in
and
do
somewhere
to
read
it
and
your
cash
is
kind
of
thrown
out
of
whack
right
when
I
cash
stuff
in
spark
I
get
about
a
10
to
50
x,
speed
up
depending
on
you
know,
cash
and
other
kinds
of
things.
A
We
find
that
our
developers
really
like
spark
they're
able
to
get
the
API
really
quickly,
because
it's
based
on
data
high
level
data
transformations
and
it's
compared
to
dupe
the
API
is
much
cleaner
and
you
have
an
interactive
shell
oops.
You
have
an
interactive
shell
that
allows
you
to
experiment
where
quickly.
This
is
a
fun
slide.
I
like
showing
this
on
the
right
side,
is
the
the
Java
MapReduce
program
for
a
word
count
I,
to
shrink
this
to
about
four
point
font:
to
get
it
to
fit
on
a
slide
on
the
left.
A
One
thing
that
is
cool
about
spark
is
that
people
have
been
building
components
on
top,
so,
for
example,
one
component
that's
been
built
on
top
is
called
that's
just
coming
out
is
called
Quebec's.
It
is
a
graph
computation
library,
kind
of
like
RAF
lab
or
other
things
like
that.
Other
things
that
people
have
done
shark
is
a
project
for
high
bun
spark.
You
can
run
your
SQL
SQL
queries
and,
finally,
there's
something
called
spark
streaming,
which
is,
basically,
you
can
think
of
it.
It's
like
storm
at
spark.
A
You
can
basically
it
does
like
a
tiny
commutations
and
people
are
using
this
for
a
stream
processing,
like
second
with
say.
Second
latency
is
that
kind
of
thing,
and
what
is
really
neat
is
that
you
can
build
all
this
stuff
on
one
platform
so,
instead
of
having
to
manage
there's
to
do
but
under
storm
and
inter
scrap
lab
and
then
there's
like
under
whatever
else
you
use
right,
you
have
one
platform
that
is,
you
only
have
to
learn
to
maintain
it
once
and
every
time
that
the
base
platform
improves.
A
It
improves
the
performance
of
all
these
different
modules
and
the
other
thing
that
is
really
cool
is
that
you've
got
really
good
integration.
So
it's
possible
in
spark,
for
example,
to
run
an
SQL
or
hive
QR,
query
writer
and
you
can
take
the
results
of
that
and
you
can
massage
it
and
scala
java
feed
it
into
graph
la
graphics
or
the
machine
learning
library,
and
it
do
like
modeling
on
it.
A
I'm
not
going
to
spend
much
time
on
this,
but
for
those
of
you
guys
that
use
hi.
How
many
people
here
use
hive
so
fair
bit
shark
is
like
a
hundred
percent
hive
ql
like
compatible
layer.
The
reason
why
is
compatible
is
because
it
actually
includes
a
fork
of
high,
and
so
we
can
reuse
hive
udfs
30s
and
that
kind
of
thing,
and
if
you're
using
DSE
it
can
actually
use
just
like
DSE,
has
high
on
Cassandra.
A
A
A
A
A
It
is
basically
it
reads
all
the
roles
from
one
column,
family
which
is
ok,
but
not
as
helpful
for
most
people.
There
is
the
cql
paging
input
format
for
Cassandra,
one
or
two
and
up
which
is
great.
It
allows
you
to
use
your
secondary
indexes
and
gives
you
a
lot
more.
Flexibility
gives
the
automatic
paging
you
can
roll
your
own.
We
actually
wrote
all
in
perform
at
because
we
have
data
that
spread.
A
Finally,
there
is
the
co
NHO.
Cassandra
has
a
native
jdbc
driver
now
and
there
is
someone
did
write
some
integration
with
jdbc,
so
you
can.
You
could
go
through
that
route
as
well
and
I.
Think
I
left
this
thing.
I
left
this
off,
but
there's
also
some
third-party
integrations.
There's
a
company
in
India
called
to
poor
chump
that
wrote
a
cassandra
and
spark
integration
which
looks
very
slick,
I,
yea.
Sorry
I'll,
add
this
back
to
my
slides
at
some
point.
It
must
have
accidentally
tooken
it
out.
A
If
you're
using
the
column
family
input
format
it
will
it
translates
it
directly.
Basically,
every
role
in
Cassandra
gets
slammed
into
one
record
in
spark,
so
it's
pretty
straightforward
and
thence
Marcus
operators
for
operating
and
then
you
can
take
each
roll
data
and
manipulated
and
spark
using
your
join
and
your
other
operators.
A
A
It's
marks
place,
a
concept
that
you
have
data
that
is
distributed
and
you
can
manipulate
it
like
a
a
collection
of
data,
so
basically
all
datasets
RTD.
So
these
are
divided
into
partitions.
So
you
can
think
of
a
partition.
Is
a
partition
has
to
live
on
one
node?
So
if
you
were
to
divide
your
data,
/
10
notes
that
you
would
have
at
least
10
predictions.
You
could
have
a
lot
more
than
10
partitions.
A
And
yeah
different
partitions
live
on
different
nodes
in
general.
You
don't
have
to
worry
about
partitions
too
much
other
than
as
a
sizing
kind
of
thing
in
general,
wet
and
smart
you're
working
with
individual
records.
So
these
partitions
would
have
I,
don't
know,
say
you
know,
let's
say
that
you
have
a
user
database
and
so
each
user
is
a
person,
so
usually
you're
working
on
the
user
level,
you're
kind
of
doing
filters
on
users,
but
sometimes
it
is
helpful
to
know
the
topology
underneath.
A
So
I
just
had
a
quick
slide
about
if
you
were
to
choose
between
reading
about
data
using
and
if
perform
at
verses
and
writing
using
sparks
api's
directly
so
I
guess.
The
idea
is
that
one
of
Cassandras
built-in
in
performance
work
for
you
like
the
c
ql
paging
in
perform
at
that
works
and
that's
the
easiest
route.
A
But
if
you
had
to
develop
any
perform
at
versus
doing
it
natively
in
spark,
we
just
want
to
throw
up
some
corners
for
comparison
and
if
the
format
is
really
flexible
because
you
could
use
it
in
Hadoop
as
well
as
spark.
On
the
other
hand,
I
don't
know
if
anyone's
tried,
implementing
an
input
format,
it's
it's
kind
of
a
big
pain.
There's
like
multiple
Hadoop,
api's
and
they're
very
clunky.
It's
not
the
easiest
thing
to
do
so.
A
Personally,
like
the
lesson
that
we've
learned
is
that
if
we
have
to
go
back
and
do
it
again,
we
would
probably
have
this
gone
straight.
We
would
have
skipped
the
input
format,
route
and
just
done
it
entirely
in
spark.
It
would
be
much
easier.
Department
probably
saves
us
about
release
a
month,
maybe
more.
A
A
You
can
treat
a
list
of
rockies
as
a
distributed
data
set
and
spark
and
there's
multiple
ways
of
creating
that
list.
But
maybe
you
start
off
with
an
index,
and
your
index
gives
you
a
set
of
rookies.
So
then
what
you
do
is
that
you
have
a
distributor
set
of
rookies
now
I'm
going
to
do
something
called
a
flat
map.
So
if
flat
map
takes
basically
one
item
like
a
rocky
and
it
transforms
it
into
one
or
more
items
and
then
all
those
results
get
become
one
big
data
set.
A
So
what
I'm
doing
here
is
that
I'm
doing
we,
let's
say
you
have
a
function
called
three
columns.
Basically,
this
takes
a
row
key
or
in
NC
coil
three.
It
would
be
like
your
primary
key
or
whatever,
and
then
you
can
read
in
all
the
data
from
from
new
row
and
then
now
you
have
a
distributed
data
set
that
has
all
the
data
from
those
girls.
The
reason
why
I'd
like
to
think
in
terms
of
Rho
keys
is
because
we
know
that
rose
are
distributed
across
the
cluster
in
Cassandra.
A
So,
if
you
think
of
it,
this
way
this
maps
pretty
well
to
Cassandra
slay
out,
because
different
role
keys
will
go
to
different
nodes.
So
this
this
is.
This
is
a
parallel
operation.
You
can
even
do
if
you
want
to
be
very
aggressive
about
this.
You
can
even
try
to
like
figure
out
what
which
will
keys
are
in
which
replicas
and
therefore
try
to
get
a
note.
A
A
So
our
Cassandra
SPARC
architecture.
So
what
we
do
is
that,
as
these
events,
video
events
are
coming
in,
they
get
let's
say
to
get
written
to
wall
logs
somewhere.
Let's
say
they
are
HDFS,
so
we
process
the
raw
logs
and
we
write
semi
aggregated
OLAP,
like
tables
into
Cassandra,
as
collinar
storage,
now
go
into
like
what
kind
of
schema
we
use
for
this,
then
what
we
do
is
that
we
load
data
from
Cassandra
into
memory
and
spark,
or
we
can
do
very
fast
queries
and
we
can
query
it
directly
using
spark
for
certain
predefined
queries.
A
This
one
to
like,
explain
a
little
bit
why
it's
not
enough
just
to
read
and
write
directly
from
Cassandra,
so
what
we
have
is
that
every
day
we
have
about
30
million
multi-dimensional
fact
table
rows,
and
so
these
are
imagine
that
you
have
every
combination
possible
of
your
different
dimensions,
your
geo
and
your
other
things
about
a
user,
and
there
are
about
30
million
different
combinations.
Now
for
you
to
answer
a
query,
materializing
every
possible
answer
isn't
really
possible
because
for
us,
the
user
can
group
by
multiple
dimensions.
A
At
the
same
time,
let's
say
that
you
want
to
group
by
country
as
well
as
you
know,
browser
or
something
like
that,
which
is
not
uncommon
or
more
common.
It's
like
I
want
to
group
by
video
watch
and
something
else
like
country.
So
so
you
end
up
having
a
lot
of
cross
processing
and
the
number
of
combinations
that
are
possible
from
like
grouping
all
these
different
dimensions
as
well
as
filtering
and
indexing.
It's
like
into
many
billions.
So
it
is
this
not
possible
to
materialize
that
many
answers
in
Cassandra.
A
So
for
that
reason
we
we
take
a
different
approach.
The
other!
Well,
so
the
other
approach,
besides
trying
to
materialize
every
answer
is
that
you
could
store
your
fact
tables
in
Cassandra.
This
is
possible.
Let's
say
that
I
stored,
I
have
30
million
rows
in
Cassandra
or
I
could
learn
million
columns
or
something
like
that.
That's
possible
to
the
reason
why
that
doesn't
really
work
either.
A
It's
because,
depending
on
the
type
of
query
you're
reading
from
basically
every
for
every
query,
reading
from
random
subset
of
these
30
million
rows,
so
you're
talking
about
a
a
read
that
is
extremely
random
in
nature,
I
think
with
SSDs.
It
might
be
fast
enough,
but
at
that
scale
it
becomes-
and
this
is
for
one
day
when
you
start
going
beyond
like
three
million
like
and
you're
reading,
random
data
it
becomes
like
not
I
mean
you
can
do
it,
but
it
will
take
a
long
time
in
Cassandra.
A
A
So
this
is
what
our
skimmer
looks
like.
We
have
one
color
family
for
indexing,
all
of
the
different
OLAP
tables.
We
have
a
metadata
family
that
essentially
stores
these
are
the
type
of
columns
that
we
have
and
what
what
types
of
columns
they
are
they
are,
are
they
integers
or
a
whatever
and
our
main
color
parameter
stores
data?
This?
Basically,
we
store
one
row
per
column
of
data
and
the
reason
why
we
do
that
is
so
that
we
can
selectively
load
columns.
A
The
other
thing
that
we
do
is
why
we
store
things
by
column
is
so
that
you
can
get
very
good
compression,
so
you
can
use
a
lot
of
techniques
such
as
dictionary
compression
other
things.
Most
most
columns
have
a
much
lower
cardinality
of
values
compared
to
the
number
of
roles
that
you
have
so
we're
able
to
achieve
very
good
compression
using
techniques
like
that,
and
we
store
whole
bunch
of
roles
together
that
we
can
read
them
in
big
chunks
in
it.
It's
very
fast.
A
A
Okay
right,
I
guess
I
could
good
through
this
quickly.
This
is
going
to
go
over
really
quickly
like
how
this
works
in
spark.
Basically,
you
have
records
that
are
distributed
across
multiple
nodes
and
in
each
partition,
your
multiple
records,
and
then
you
can.
You
can
convert
all
these
records
into
a
let's
say,
a
sharp
table
and
run
SQL
over
it
or
you
can
for
for
known
prairies.
You
can
basically
use
whole
bunch
of
operators
like
basic
map,
store
group
join
other
things
like
that,
there's
also
a
top
k
operator.
A
Just
a
really
quick
note
on
fault
tolerance:
there
is
so
if
this
data
leaves
the
lips
in
the
heap,
you
might
ask
well
what
happens
if
it
goes
down.
So
a
spark
has
a
traditional
fault,
tolerance
mechanism
called
lineage,
and
what
lineage
says
is
that
I'm,
going
to
remember
the
exact
sequence
of
operations
hope
to
compute
this,
and
so
something
fails.
What
spark
is
trying
to
do
is
to
we
appealed
stuff.
In
this
case
you
don't
necessarily
want
to
go
that
route,
because
a
reading
stuff
rosanna
might
be
slow.
A
So
instead
one
well
there's
different
ways.
So
one
way
you
can
take
advantage
of
is
that
spark
and
replicate
the
memory
data
set
across
different
notes.
So
then,
that
way
that
survives
anyone
failure
we're
also
not
terribly
worried
about
losing
data,
because
in
the
worst
case
we
can
always
veto
back
from
Cassandra
and
our
schemas
optimized
for
faster
or
put
so
it's
it's
not
so
that
hasn't
been
so
having
cassano
de
reserve
is
a
pretty
big
relief,
I
guess,
cool
I!
Think
I,
don't
really
have
time
for
demo.