►
Description
SlideShare: http://www.slideshare.net/planetcassandra/olap-with-spark-and-cassandra
How do you rapidly derive complex insights on top of really big data sets in Cassandra? This session draws upon Evan's experience building a distributed, interactive, columnar query engine on top of Cassandra and Spark. We will start by surveying the existing query landscape of Cassandra and discuss ways to integrate Cassandra and Spark. We will dive into the design and architecture of a fast, column-oriented query architecture for Spark, and why columnar stores are so advantageous for OLAP workloads. I will present a schema for Parquet-like storage of analytical datasets onCassandra. Find out why Cassandra and Spark are the perfect match for enabling fast, scalable, complex querying and storage of big analytical data.
Evan Chan is a Principle Systems Engineer at Socrata. In his own words: I love to design, build, and improve bleeding edge distributed data and backend systems using the latest in open source technologies. I am a big believer in GitHub, open source, and meetups, and have given talks at conferences such as the Cassandra Summit 2013 and will be presenting at Cassandra Summit 2014.
A
A
So
tell
you
a
little
bit
about
us
or
Prada
where
we're
based
close
to
pioneer
square,
but
we
go
software
to
make
data
useful
to
people
out
more
people
and
it
specifically
public
data
for
citizens.
So
there
there's
a
lot
of
websites
that
we've
done.
That
opens
up.
Data
sets
from
government
and
organized
really.
Citizens
learn
more
about
government
to
help
governance
more
accountable
and
see
how
spending
money
are
quickly
released
on
me
and
poking
improve
on.
A
How
do
we
create
and
analyze
it
efficiently
right?
My
last
job
at
we
Allah
we
were
getting
about.
You
know
three
billion
events
today,
even
if
you
were
to
take
that
create
a
role
table
that
works
out
to
about
30
million
rows
the
day
and
people
wanted
one
reason
like
year
make
it
one
right.
So
how
do
you
do
that
and
it's
a
Prada
we
have
like
tons
of
data
sets,
each
of
which
are
millions
of
rows
so
again
like
in
the
aggregate
with
the
problem
where
this
could
use
the
scale
to
abilities.
A
There's
a
big
question
that
once
my
eyes,
how
do
we
allow
people
to
pray
a
lot
of
data
quickly,
you've
heard
John
gave
an
excellent
talking
here
about
make
soccer
and
I
can
want
to
do
you
no
lies,
but
sometimes
you
can
only
do
normalize
so
much
if
you
want,
if
you
want
people
to
be
able
to
either.
If
you
want
to
support
more
complex
types
of
queries
in
a
joint
different
things
like
that,
it
is
a
little
really
hard
to
materialize
or
maybe
a
lot
of
times
impossible
to
materialize
all
different
answers.
That's
them!
A
A
A
You
start
you
finally
start
hitting
limits
and
various
points,
I.
Think
for
us.
You
know
when
you
get
to
like
volcanoes.
You
start
what
you
know:
it
depends
on
exactly
what
hardware
use
but
cerca
de
limits
in
terms
of
X
and
because
of
the
type
of
prey
as
well,
for
you
tried
to
like
or
comply
to
revise
that
limit
market
right.
The
other
thing
with
our
dbms
is:
is
that
I'm
probably
preaching
requiring
it
but
I?
A
Did
there
isn't
really
any
good
solution
for
at
least
an
expensive
solution
for
charting
and
curry
you
can
buy
or
but
no
one
wants
what's
the
country,
hopefully
no
one
with
you
were
to
run
back
there.
There
isn't
any
robust
way
to
scale
it
horizontally.
There
are
some
new
things
that
are
coming
out
and
post
grass
as
a
pixie,
Excel
puzzle,
kind
of
thing
we've
tested
it
found
that
it
is
who
there
are
these
stages.
A
A
Huge
country
circuit
occurs
for
doing
these
kind
of
things
which,
if
it
works
well,
because
what
we'll
have
to
do
is
then
they
materialize.
A
piece
of
the
puzzle
answer
combinations
in
a
way
that
you
can
get
out
quickly.
So
this
is
a
time-proven
approach.
Only
problem
is
that,
like
as
the
size
you
didn't
grow,
the
size
of
a
cube
grows
exponentially,
or
you
know,
with
regard
to
how
many
dimensions
are
in
libya
and
quickly
at
the
levels
of
the
size
of
the
data
we're
talking
about.
A
They
started
yet
untenable
and
nobody
really
knows
how
to
work
there.
Usually
it's
great
proprietary
technology,
a
lot
of
papers,
we've
been
a
non-event
like
it's
one
thing
to
have
a
paper:
how
home
runs
divided
production
so,
and
there
are
very
few
solutions
for
from
CubeSat
scale
works.
Actually
they
they're
probably
argument
again
the
great
sweetie.
So
so,
what
do?
What
do
we
do?
Right?
A
You
guys
honored
work
on
top
sites
and
he
said
when
adult
use
brute
force.
No,
no
is
this
software
developers
we
probably
are
trained
to
think
this
is
a
very
bad
idea.
Right,
boot
boot
forest
like
go
to
programming
interviews.
We
always
taught
you
know,
use
the
best
algorithm,
but
there's
actually
a
you
know.
A
A
really
good
grain
of
truth
is
that
sometimes
the
best
way
to
solve
problems
is
to
keep
things
simple,
actually
a
lot
of
times
and
to
work
on
a
reason
right,
so
I
introduce
you
to
two
technologies
that
guess
Lou
Cassandra
was
smart.
That
Thunder,
you
probably
all
familiar
with,
but
we'll
talk
about
both
how
we
can
you
still
together
to
help
solve
this
problem.
A
So,
as
you
all
know,
cuz
I'm,
very
wonderful
technologies,
horizontally
scalable
can
model
your
data
in
many
different
ways
and
it's
relatively
easy
to
operate,
but
relatively
it.
What
I
mean
is
that
everything
is
like
same
type
know
you
want
to
worry
about.
Well,
do
I,
have
a
worker,
node
and
then
I
have
got
some
kind
of
master
for
certain
files.
This
kind
of
that
kind
of
thing
right.
The
only
thing
is
that
the
Sun
works.
A
A
Anything
is
yet
right,
just
the
Ducks,
that's
what
we're
going
to
talk
about
yeah.
So,
let's
hang
on
so
I
would
like
to
introduce
you
Apache
smart.
Many
of
you,
guys,
probably
heard
of
or
on
a
plane
with
it,
but
I'll
just
run
through
this,
for
spark
spark
is
essentially
like
an
event.
We
can
do.
It's
always
sound,
really
scalable
they
can
cash,
so
they
do
best
hdfs
spark
and
worth
make
sure
you
invest
with
sparking,
also
cash.
A
A
The
really
neat
thing
about
Marcus
is
how
much
momentum
it
has
scattered.
I
also
said
the
sparks
limit
Utah
in
July.
This
is
the
seconds
of
it
as
like
over
a
thousand
people.
The
first
one
was
last
December.
My
lesson
coverage
people
and
it
is
amazing,
like
every
single
vendor
of
group,
has
at
its
mark
as
an
add-on
and
as
an
unbelievable,
like
it
has
ten
times
more
activity
than
any
other
Big
Data
project,
pretty
much
it's
developers,
we
love
it.
A
There
is
now
it's
called
data.
Data
brings
theta
prints,
they
receive
40
million
dollars
in
the
first
round.
Anyway,
you
got
you
want
details,
you
can
remember.
What's
really
neat
about
spark.
Is
that
so
traditionally
in
the
Duke,
you
had
a
do
bit
selfish
and
never
use.
Then
you
have
a
separate
project,
which
is
wrapped
in
position.
Get
high,
which
is
again
in
separate
projects
and
typically
moving
day
between
these
project
is
not
that
easy.
What's
more,
you
have
a
lot
of
components
that
are
built
on
top
squad.
A
You
have
an
esco
component
machine
learning,
an
instrument
component,
and
these
are
all
I
pretty
tightly
together.
Oh
I'm,
sorry
yeah
yeah
sure
so
so
spark
has
a
lot
of
components
that
are
tied
very
tightly
together.
There
is
a
human
component
and
SQL
component,
a
graph
component
machine
learning
component.
So
an
example
of
something
that
you
can
do
very
easily
expired
is
that
I
can
take
data
load.
A
A
So
the
question
was
in
terms
of
memory:
does
smart
leverage
memory
across
to
do
love,
notes
and
certain
conclude
that
still
spark
is
actually
you
can
think
of
it
as
smarter?
She
doesn't
want
to
talk
to
do
it's
visited
it.
Peggy
I
can
write
as
far
without
consulting
you
at
all.
Does
the
first
thing
we're
not
in
Cleveland
you
can
create
a
smart
customer
kakuzu.
In
fact,
that's
where
spark
and
concern
here,
because
we're
interesting
as
you
can
set
it
out
without
setting
up
all
of
do,
which
I
think
increasingly
makes
more
sense.
A
A
A
High
is
completion
engine
it
can
wreak
tables
from
Cassandra
spark
can
again,
you
can
read
data
from
a
design
material,
but
I've
and
smart
are
both
mutation
engines,
they're,
not
databases,
right,
Cassandra,
actually,
story
to
clean
up
for
you,
but
I'll
get
to
that
in
two
minutes.
So
basically,
what
what
I'm
trying
to
say
is
that
spark
provides
the
missing
vast,
deep
analytics
that
people
always
monitor,
which
is
one
reason
why
database
put
away
I,
don't
work
lyrics,
but
I
think
what
they're
doing
is
brilliant,
like
partnering
with
datastax
is
partnering
with
lyrics
to
offer.
A
I'm
going
to
talk
really
quickly
about
integrating
sparking
Cassandra
and
I'll
go
on
to
what
we've
been
talking
about.
So
there's
a
couple
ways:
you
can
bring
data
from
Thunder
into
spark
very
easily
datastax.
Is
this
looking
source
soldered
writers
park
project
which
uses
the
new
data
stack,
cql
driver
underneath,
and
so
you
can
pull
data
out
and
in
what
Peter's
quarter?
There's
other
project
as
well
I'm
a
scholar
person
so
I
only
research
in
stellar
ones.
I
think
they're,
probably
farther
along
than
the
java
and
python
once
so,
but
I'm
not
sure.
A
But
smart
can
actually
pull
from
any
video
format.
Now
there
are
actually
multiple
different
forensic
work
with
Cassandra
there's,
a
sequel
patient
in
format
where
you
can
give
it
a
sequel.
Prairie
will
get
out
for
you
and
in
pages,
which
is
pretty
nice
and
there's
older
ones
that
we
hired
table
or
what
we
just
called
our
families,
but
working.
Those
is
going
to
be
a
bit
working,
the
format's
of
generals
and
lot
more
info.
You
can
also
just
give
you
a
flavor.
A
What's
partly
do
you
can
also
be
directly
from
Cassandra
using
some
lightness,
so
this
is
scholars
mark
pooh
go
through
this.
There's
really
only
two
lines
that
mattered
here.
The
first
one
is
you
see
the
SC
go
parallel
eyes.
What
it
does
is
smart
can
take
any
list
and
paralyze
it
destroy
the
cross
clustering.
So
what
we're
doing
is
first
we're
saying
we
have
with
a
bunch
of
rookies.
Rookies
are
difficult
to
listen
karaoke.
Is
that
one?
We
know
we're
going
to
paralyze
that
this
rule
across
this
one.
A
Second,
like
that's
interesting
this,
where
I
think
I'm
doing
a
essentially
a
transformation,
I'm
mapping,
a
rookie
data
containing
materials
by
using
sdn
x3y,
the
youth
center
library
to
do
any
get
on
that
primary
key
from
the
economy
to
table,
and
so
so
I
start
with
a
bunch
of
rookies.
Then
I
transform
it
in
smart
into
the
actual
data.
So
that's
something
that
gives
you
a
flavor
of
the
kind
of
pattern
that
you
typically
do.
Its
part
is
that
I
sort
of
some
data
and
our
informant
is
going
them
to
just
anyone.
A
A
A
A
A
So,
let's
talk
about
their
architecture
for
doing
sparking
the
Sun
ruler,
it's
the
exciting
stuff.
So
one
of
the
principles
is
that
once
you
separate
the
storage
in
freedom,
so,
unlike
the
traditional
database
market
X,
you
think
of
only
store
David
here
and
romantic
pre-owned.
The
idea
is
I'm
going
to
use
the
best
of
wheat
storage
platform
which
the
sound
room
is
for
modern
uses.
A
One
of
the
benefits
that
you
get
from
doing
this
operation
is
that
your
concurrency
for
praying
can
have
a
different
level
of
you
can
scale
your
weak
currency
separating
from
storage.
Let's
say
that
you
have
data
in
one
table,
that's
very
like
pocket
right
and
you
really
want
to
serve
as
many
people
as
possible.
Well
as
far
what
I
can
do
is
I
can.
I
can
populate
that
into
memory
of
multiple
sparknotes
and
therefore
increase
my
parents.
A
A
A
A
What
yes,
I'll
preface
this
by
saying
that
this
is
really
optimized
for
data.
That's
mostly
append
only
like
I'm
not
going
to
be
imitating.
Like
you
know,
some
rolls
come
our
way.
I'm
going
to
be
I'm,
get
the
data
once
and
I'm,
proposing
an
add
to
it.
There's
like
very
small
percentage
engine.
So
it's
so
guess
you're
right!
It
does
depend
on
that.
What
I'm
talking
about.
A
A
That's
kind
of
puppy,
paws
I
want
to
get
rid
of
honest.
It's
far
me
too
won't
work
on
the
run.
Here's
the
thing
after
I
do
some
competition,
its
mark.
I'm
gonna
put
it
back
in
time
right,
so
it
depends
on
your
workflow
like
so
I
could
store
what
I'm
talking
about
the
story
more
likely,
Rawdon
or
data
that
you're
going
to
do
something
together.
Dad
has
some
things,
my
music,
what's
a
guy,
I
use
the
villains.
A
So
that
my
access
yeah,
so
I
think
that's
a
very
different
kind
of
pattern
right,
like
that's,
more
etl
they're,
taking
source,
Davis,
USB
thing
and
then
doing
competition
spark
and
then
putting
Cassandra
where
I
can
do
point
queries
on
it.
What
I'm
talking
about
is
little
different
is
more
like
I've
already
done.
The
etl
right,
I
have
I,
have
the
data
I'm
going
to
vary,
but
is
really
make
and
I
can't
do
primaries
on
now
we'll
go
to
the
spark,
so
can
run
interesting,
preys
on
Monday.
A
A
What
I
mean
is
I'm
going
to
up
I'm
going
to
architect
this
architecture
go
to
maximize
for
want
to
call
those
scans
me,
which
is
instead
of
building
a
lot
of
fancy
indexes
I'm
going
to
try
to
go
over
as
many
rows
as
I
can
as
quickly
as
a
channel
and
I'm
going
to
look
for
games
in
place,
such
as
using
a
column,
recognition
presentation
to
make
things
fast
and
as
well
as
things
like
the
memory
Nexus.
So
it's
like
out
the
roles
that
I
need
that
meds,
probably
about
it.
A
Now
you
can
do
it
this
way.
You
can
also
do
it.
The
other
way
we're
like
muscle
sensibly
I,
could
have
influenced
part
that
I
could
do
a
bike
through
cash
back
in
Sondra.
We
haven't
even
won
that
theory
as
possible.
So
that's
why
I'm
I
like
it
do
depends
on
the
more
flow,
but
typically
libreta
stomach.
First,
revisits
pretty
fast
yeah,
listen
it
pretty
much
latency,
like
utilities,
there's
a
slight
variation
on
this.
It's
marc
has
a
hundred
contacts
which
is
cached
HD
offense,
and
what
that
does
is
it
accelerates?
A
Hdfs
treats
rights,
but
it
allows
you
to
plug
in
different
backends,
so
techy
our
stores
data
mafi.
So
the
advantage
of
this
is
that
you're
caching,
data
from
Sondra
are
into
you
and
our
peeps,
or
so
that
we
can
process
knives.
No,
doesn't
you
can
restore
the
process
and
not
have
to?
Instead
of
you
know,.
A
This
is
that
support,
doesn't,
is
not
there
yet
by
the
way.
So
hopefully
someone
will
do
it.
So
can
you
guys
see
this?
Ok,
ok,
let's
try
and
it
doesn't
work
then.
So,
unfortunately,
since
rescaling,
it
seems
like
that
is
the
most
weekend
with
you.
So
so
you
guys
can
probably
like
I'll
just
go
over
this.
What
what
are
the
first
few
lines
is
basically
defining
data
de
seis,
Val
Rose,
secretive
sequence
are
and
then
suitors
is
this
a
message:
sequence,
each
life:
besides
people
grow
there
and
each
value
is
basically
at
home.
A
So
this
was
an
internal
temp
and
what
basically
important
things
to
notice
that
each
value
and
different
type
you
could
have
to
train.
You
see
those
numbers
and
then
a
theatrical
code.
The
first
line
says
SC
no
parallel
lines,
so
there's
that
paralyzed
again
so
we're
thinking
arose
and
we
aren't
distributing
it.
A
The
number
of
times
for
prayer
mountain
by
type
of
super
group
are
so
first
I
would
map
the
value
of
such
that
I'm.
Getting
a
deep
hole
with
the
prime
type
is
the
first
knowledge
me,
then,
is
that
the
third
lines
of
groupon
key,
which
is
named
distributed
grouping
and
so
now
I
have
all
the
values
come
from
wherever
your
medicine,
that's
together
my
place,
then
the
final
Isaac
reviews,
which
adds
up
all
of
the
different.
So
this
is
pretty
basic
map.
Reviews
of
you
have
basically
learned
in
two
lines,
so
this
works.
A
The
problem
is
that
well
by
the
way,
this
is
this.
You
could
do
today,
I'm
using
one
can
start
it.
It's
wonderful
is
Mary,
so
we
didn't
really
use
it
about.
The
problem
with
this
approach
is
that
it's
very
you
see
at
a
time
it's
very
well
oriented,
so
I
need
to
pull
out
values
that
I
need
to
create,
and
also
especially
for
our
morning
rare
conditions,
and
so
in
Java
like
if
you
want
to
store
a
collection
that
can
store
any
type.
Can
we
get
to
do?
A
A
Did
there
actually?
No
it's
it's
getting
forwarded
it
to
is
quite
signals
so
I
sure.
So
the
question
was
a
smart
sequel
going
to
kill
shark
shark
is
a
project
that
Sicilian.
The
answer
is
yes,
it's
a
really
quick
thing
about
colander,
like
story
things
and
Collins
rose.
You
guys
know
who's
Michael
school
records,
the
founder
of
those
grass
like
one
of
the
world's
foremost
interviews
experts.
What
do
you
said
recently
is
that
to
their
traditional
role
based
data
storage
is
facebook.
A
This
might
partly
because
she's
found
several
startups.
We
had
color-
depictions,
but
I
think
he's
actually
right.
On
god,
sorry,
I
want
to
hold
off
on
a
question
because
I
think
we're
running
out
of
time
and
I
still
have
material
so
I'm
circle
that
but
I'll
be
around.
So
you
can
ask
questions
the
sort
of
did
traditionally
a
story
that
rose
both
in
memory
and
on
disk.
A
So
let's
say
you're
two
rows:
they're
the
two
columns
are
being
stored
together
and
so
the
Hillary
and
66
always
come
after
the
first
role
in
memory
and
on
this,
when
you
store
stuff
in
the
comic
program,
what
you're
doing
is
I'm
separating
out
the
stuff
in
each
column
so
that
all
the
names
get
scored
first
and
then
you
know
the
pizza
from
the
second
column
and
SportsCenter.
It's
get
get
stored
together
and
you
can
do
the
same
thing
in
Cassandra
and
I'm.
Sorry.
This
is
in
the
old
casado
renewal
business.
A
If
you
can
basically
translates
EF,
which
is
called
Emily
to
table
and
ro
kheili
to
so
basically
imagine
if
I
stored
on
all
of
the
data
you
can
think
of
it
as
a
traditional
table,
but
flipped
sideways.
So
basically
all
of
the
data
in
one
column
gets
stored
in
one
physical
bro,
perdition
Lucas.
On
that
way,
it
was
the
market
continues
that
disk
and
can
be
accessed
right.
A
And
the
advantage
of
total
column
graph
format
is
that
we
are
able
to.
First
of
all,
you
can
save
my
own,
because
you
only
need
to
read
in
the
columns
that
need
for
a
caring
and
that's
what
all
of
the
work
on
in
a
do.
Parque,
plies
compile
a
lot
of
them
or
not
have
gained
lots.
We
just
from
this
single
thing
alone.
It's
reducing
my
own.
A
You
also
get
a
huge
amount
of
compression
that
is
much
easier
achieve
when
you
store
stuff
column
wise
because,
for
example,
like
let's
say
that
explore
bunch
of
names
well,
I
know
that
they're
there
are
only
so
many
names
and
so
I'm
going
to
use
dictionary
to
put
a
number
instead
of
the
string.
So
I
can
get
a
lot
of
savings.
Just
that
way.
Right
and
in
terms
of
the
jvm
for
numbers,
you
gotta
get
a
big
benefit
because
I
can
sort
of
return.
A
A
So
now
that
we've
talked
about
doing
a
lot
of
cassandra
and
spark
no,
no,
we
do
that.
The
party
started
first
of
all,
mention
is
again,
but
datastax
has
a
conservative
squawk
integration,
so
they
I
mean
there's
the
previous
driver
that
I
prefer
to
for
the
community.
Cassandra
there's
also
data
spec
datastax
enterprise,
which
we
rolling
on
smart
support
and
some
of
the
things
that
I
know
they've
added
is
that
they
have
added
an
action.
A
spark
master
rates
with
sentence
sounds
pretty
nice.
A
A
He
has
an
in-memory
on
Cullen
Ursula,
so
it
actually
store
stuff.
Like
I
talked
about
intolerant
with
which
minimizes
the
amount
of
space
you
need
in
memory
to
store
stuff,
although
recurring
is
not,
is
still
roaming,
but
it
is
definitely
a
step
in
the
right
direction
and
I
know
that
cyber
integration
work
is
coming,
which
I've
heard
from
several
sources.
A
Right
now,
there's
no
indexes
yet,
so
it's
basically
doing
two
table
scan
every
time,
although
it's
still
pretty
fast,
are
the
two
exciting
things
about
its
mark.
Sql
is
that
one
is
that
we
need
custom
functions,
can
actually
write
them
in
scholar,
Java
and
Anna
do
stuff
in
your
SQL,
which
is
which
is
pretty
cool.
It's
much
easier
than
using
high
mediums
a
devious
times,
and
the
output
will
invert
anyway
variously
with
the
machine
learning
library,
Raph
awesome.
A
A
Some
I
think
a
work
that
is
still
needed
from
my
spark
is
for
something
right
now
in
some
Texas.
So
if
you
know,
if
there's
a
different
way
to
add
indexes,
you
can
definitely
speeds
things
up
for
certain
kinds
of
queries
and
to
be
able
to
read
in
a
more
efficient
either.
There's
a
lot
more
work,
but
you
know
I
think
those
are
clean
things.
A
So
sorry
I'm
a
bit
out
of
time,
but
I
wanted
to
summarize
that
I
think
this
architecture
of
Cassandra
it's
park
together
can
give
you
very
fast,
complex,
praying
for
our
certain
use
cases
that
the
data
doesn't
change
much.
You
have
it's
mostly
independently
kind
of
data
set
and
you
focus
on
the
America
I
patients,
then
I
think
that
it
it
works
pretty
well
spark
this
is
that
one
thought
already
like
it
is
getting
lesson.