►
From YouTube: C* Summit EU 2013: One Million Books: Adventures in Discoverability with Cassandra and Solr
Description
Speaker: Patricia Gorla, Systems Engineer at Opensource Connections
Slides: http://www.slideshare.net/planetcassandra/patricia-gorla
For any venture, storing your data is just the first step in making sense of it. How do you make your system discoverable? How do you tune your relevancy to accommodate real-time updates? In this session, we explore pairing Cassandra with Solr using Datastax Enterprise Search, and look at different search paradigms to help your users find patterns in your data.
A
Thank
you
all
for
being
here
today,
so
I'm
I'm
here
to
talk
about
cassandra
and
solar
and
and
really
what
you
can
get
out
of
pairing
the
two
and
what
what
benefits
that
solar
adds
to
any
system
that
you're
looking
for
so
a
little
bit
a
little
bit
about
me.
So
I
work
with
solar
and
cassandra
and
information
retrieval.
I
work
for
a
consultancy
called
open
source
connections.
We
build
search,
applications
and
discovery
platforms,
and
and
really
when,
when
clients
come
to
us,
what
they're
looking
for
is
how
do
I
find
my
thing?
A
How
do
I
find
what
I'm
looking
for?
Excuse
me,
what
I'm?
What
I'm
looking
for
and
and
this
can
this
can
be
any
number
of
questions
this.
This
can
range
from
the
simple
to
the
complex
and
I
guess
just
to
just
get
a
little
get
to
know
a
little
bit
about
each
other.
How
many
of
you
have
have
ever
worked
with
solar?
A
Oh
great,
great,
how
many
of
you
have
worked
with
cassandra
excellent,
so
so
with
cassandra
when,
when
you're
starting
off
to
answer
these
questions,
what
it
comes
down
to
is
is
that
that
cassandra
really
was
not
built
for
for
full
text
search.
So
if
you're,
if
you're
asking
simple
questions
like
who
is
aristotle,
where
was
he
born?
What
are
the
coordinates
of
this
place?
Then,
then
the
answer
is,
the
answers
are
quite
simple.
A
All
you're
doing
is
just
a
straight
select
and
and
again
that's
that's
what
cassandra
was
designed
for
when
you
want
to
start
looking
at
more
interesting
questions
such
as
give
me
give
me
the
list
of
all
the
greek
philosophers
or
talk
to
me
about
about
topics
related
to
philosophy
and
thoughts
and
and
any
other
tangents
coming
from
that,
then
you
can.
You
can
create
an
index
on
your
column.
You
can
search
through
the
column,
but
this
this
comes
up
with
with
a
number
of
problems,
one.
A
If
you
have
any
spelling
mistakes,
those
will
not
match
those
will
not
make
a
match.
Two.
If
you
have
a
word.
That's
similar
to
what
you're
looking
for,
but
isn't
it
isn't
exactly
it
then
you're
going
to
fall
through
and
and
if,
if
you're
looking
for
for
any
geospatial
support,
if
you
want
to
be
able
to
say,
I
want
to
see
all
of
the
cities
within
100
kilometers
of
this
of
this
region,
then
you're
you're,
basically
at
a
loss
and
you're
going
to
have
to
find
other
things.
A
Fortunately,
there
there
is
a
way
search
engines
make
all
of
these
make
all
of
these
questions
very
answerable
and
very
approachable
to
where
you
can
pull
out
birth
places.
You
can
pull
out
locations
and
you
you
can
start
to
make
inferences
from
all
of
those,
and
so
so
now
we're
going
to
talk
about.
How
do
we
even
approach
search
to
begin
with?
How
do
we
go
after
what
we're
looking
for?
How
do
we
find?
A
But
this
is
not
going
to
give
you
the
customization
that
you're
looking
for
this
isn't
going
to
provide
you
with
the
level
of
detail
and
granularity
over
your
data.
There's
always
more
nuance
than
than
just
adding
on
a
a
self-made
solution
or
people
will
go
with
like
statements
using
their
their
mysql
database,
and
this
will
work
up
until
a
certain
point
which
which
maybe
anywhere
from
an
hour
to
a
day
or
even
a
week
at
most.
A
But
you
see,
you
actually
see
a
lot
of
companies
doing
this,
where
they'll
have
these
these
hacked
together
solutions
to
get
them,
something
that
that
really
isn't
working
for
them,
and
so
so
I'm
here
to
talk
to
you
about
solar
and
what
what
solar
can
provide
for
you,
and
also
why
this
is
a
good
match
for
your
data
in
cassandra.
So
so
solerp
gives
you
a
lot
of
power.
A
Just
straight
out
of
the
box,
you
get
full
text
search,
you
get
tokenization,
you
get
fascinating
facet
fastening
and
you
also
get
a
wide
variety
of
aggregation
functions.
You
can
join.
You
can
run
sums
you
can
you
can
even
run
your
own
queries,
your
own
functions
over
your
queries
when
you're
pulling
back
your
data
and
a
lot
of
people
use
solar.
So
this
is.
This
is
just
a
hand
from
a
handful
from
a
list
of
hundreds,
it's
a
it's
a
top-level
apache
project
and
it's
based
on
the
lucine
search
search
engine.
A
A
It's
basically
the
index
from
from
the
back
of
the
book,
and
this
makes
lookups
very,
very
simple
and
very
straightforward,
and
another
very
powerful
tool
with
solar
is
that
you
can
define
your
own
field
types.
So
when
information
comes
into
solar
or
when,
when
a
request
comes
to
solar,
it's
it's
either
one
of
two
types:
you're,
either
indexing
data
into
solar
or
you're
you're,
sending
up
a
query
to
extract
information
from
solar
and
so
you'll
notice.
A
A
So
your
queries
will
get
transformed
in
a
certain
way
and
your
your
index
will
also
get
transformed
another
way
and
the
the
most
common
example
is
that
that
with
when
you're
indexing
data,
you
tokenize
it
you
chop
it
up,
just
how
just
how
you
want
it,
and
then
you
put
that
into
solar
and
really
what
you're
looking
for
is
to
distill
all
the
information
down
into
its
key
components.
So
you
want
to
store.
You
want
to
keep
your
index
as
small
as
you
can,
and
not
not
bloated
out.
A
So
you
can
see
you
can
see
through
this
when,
when,
when
a
piece
of
information
comes
in
to
be
indexed,
punctuation
is
stripped,
stop
words
are
removed
and
then,
and
then
each
word
is
distilled
down
to
to
the
initial
meaning.
Whereas
when,
when
you're
querying
for
for
this
same
phrase,
you
go
through
the
same
steps
except
at
the
end,
you're
you're,
also
adding
synonyms
and
again
this
in
this.
This
improves
this
improves
the
recall.
This
improves
the
number
of
documents
that
you'll
get
back
from
solar
and
in
search
engine
land.
A
When
we,
when
we
talk
about
what
is
a
good
result,
what
we're
looking
at
is
precision
versus
recall,
and
it's
always
a
trade-off
between
the
two
precision
refers
to
is
this
document
actually
relevant?
Does
this?
Does
this
actually
matter
to
me
and
recall,
recall,
is
the
the
amount
of
documents
in
your
entire
space
and
whether
or
not
you're
actually
getting
getting
a
sufficient
course
portion
of
of
those
of
those
documents
and
another
really
powerful.
A
Another
really
powerful
tool
that
you
get
with
solar
is
the
ability
to
facet.
You
can
facet
on
any
field
that
you
index
and-
and
this
is
this-
is
just
a
basic
grouping
that
that
really
offers
powerful
flexibility
to
be
able
to
create,
create
unique
relationships
and
show
and
show
unique
relationships
among
among
the
pieces
of
your
data.
A
But
there
are
some
drawbacks:
what
happens
when
you
want
to
scale
out
what
happens
when
you
want
to
distribute
your
data?
What
happens
when
you
need
to
look
at
fault
tolerance?
What
happens
when
you
need
to
have
a
system?
That's
constantly
available?
What
happens
when
you
have
a
real
world
application
that
people
are
using
and-
and
this
is
getting
this
is
getting
hammered
from
from
all
different
sides?
Well,
there
are
a
number.
There
are
a
number
of
different
steps
that
you
can
take.
A
So
if
you're,
if
you're
using
google
site
search,
you
pay,
google,
more
money
and
and
some
magic
happens
behind
the
scenes.
I
if,
if
you're
like
the
rest
of
us,
however,
you've
gone
with
something
else.
If
you,
if
you're
using
mysql,
you're
you're
going
to
shard
out
your
database-
and
I
don't
need
to
tell
you
why
sharding-
why
searching
over
a
sharded
distributed
database
is
is
a
bad
idea
exactly
and
with
with
solar.
You
also
have
a
few
different
options.
You
have
master
slave
replication.
A
You
set
up
you
set
up
one
or
two
or
five
five
different
nodes
that
all
point
to
their
own,
their
own
supplicants,
and
so
all
the
data
is
replicated
out,
and
this
works
great,
except
for
the
fact
that
when
one
node
fails
when
the
master
node
fails,
there
is
no
way
to
know
that
that
node
has
failed,
except
for
your
own
custom
monitoring
that
you
have
over
the
system
and
when,
when
that
node
fails,
there's
also
no
automatic
switching
around.
So
you
you'll
have
to
go
you'll
get
that
3
am
call
to
go.
A
It's
it's
still
relatively
young,
a
relatively
young
project
and
there
is
still
there's
still
a
lot
of
a
lot
of
hiccups
and
getting
getting
solar
clouds
set
up
and
setting
up
your
own
zookeeper
ensemble
to
keep
track
of
of
all
of
the
indexes.
So
it's
promising,
but
but
they're
this.
This
all
points
to
the
problem
that
that
this
is
a
difficult
solution.
No
one's
really
figured
this
out
quite
yet,
and
and
search
is
becoming
more
and
more
important
in
our
society.
We're
a
search
driven
society.
A
When
you
pull
out
your
phone
and
you're
looking
for
information,
you
are
you're
either
typing
into
the
search
box
into
google
into
yelp
into
google
maps
into
into
whatever
other
application.
You
have,
there's
always
going
to
be
a
search
box,
and
so
and
so
what
I'm,
what
I
am
looking
at
and
what
what
we
are
really
interested
in
is
datastax
enterprise
search
which
which
takes
cassandra
and
solar
and
pairs
them
together.
A
And
so
not
only
do
you
get
the
full
power
of
of
solar
with
the
aggregation
capabilities
with
with
fastening
with
tokenization,
but
you
also
get
the
fail
safety
of
cassandra
and
you
get
you
get
the
the
ability
to
sleep
well
at
night,
knowing
that
your
node
in
your
cluster
is
your
cluster
is
up
and
running
and
and
a
little
bit
about
solar
and
cassandra.
So
so
this
started
off
as
leucandra,
which
was
originally
leucine
with
cassandra,
and
this
was
an
open
source
project
by
jake
luciani,
and
he
so
he
eventually
moved
that
into
solandra.
A
Also
an
open
source
project
before
leaving
for,
for
data
stacks
to
start
working,
working
on
data
stack
search
and
the
premise
the
premise
behind
that
was
that
cassandra
was
essentially
just
writing
out
to
solar
every
single
time,
whereas
here
with
data
stack
search,
what
you
have
is
is
you
have
solar
is
the
secondary
index.
The
leucine
index
is
a
secondary
index
in
cassandra,
so
so
next
we're
going
to
look
at
we're
going
to
look
at
a
little
demo
that
I
had
bought
that
I
had
brought
up.
A
The
demo
isn't
online
right
now,
but
we're
going
to
walk
through
some
of
the
steps
that
I
had
taken
and
and
basically
what
you
know:
you're
you're,
just
looking
at
a
straightforward
data
set
of
people
and
places
and
then
joining
them
up
together
at
indexing
time
and
aristotle,
of
course,
was
talking
about
the
inherent
duplicity
of
data.
How
do
we
represent
hierarchy
in
a
flat
system?
A
How
do
we
combine
different
pieces
of
information
to
get
a
rich
data
set
and
when
we,
when
we
put
these
into
when
we
put
these
into
solar,
we
we
get
the
ability
we
can
run.
We
can
run
our
straight.
We
can
run
our
straight
searches,
we
can.
We
can
query
for
for
whatever
we're
looking
for
and
we
can
get
back
dates.
We
can
get
back
places
and,
and
one
thing
that
that
solar
does
really
well
is
date.
Ranges
date
range
fasting,
so
you
can
set
facets
based
off
of
a
particular
date.
A
A
What
you
have
right
now,
is
basically
the
ability
to
combine
location
and
metadata
and
and
say
I
want
to
know
where
philosophy
was
contained
in
the
past
hundred
years
and
where,
where
it
has
popped
up-
and
you
can
even
see
that
progression
throughout
time,
but
of
course,
with
with
any
system
with
combining
of
any
systems
there's
always
there
are
always
compromises.
There
are
always
there
are
always
decisions
that
need
to
be
made.
A
So
the
the
schema,
the
schema
that
I
used
for
this
was
just
just
a
straightforward
schema
and
for
those
of
you,
familiar
with
solar,
you'll,
see
that
that
I'm
using
a
lot
of
dynamic
fields-
and
this
is
because,
when
you're
pairing,
when
you're
pairing,
solar
and
cassandra,
you
don't
get
multi-valued
fields,
and
these
these
are
exactly
as
they
sound.
This
is.
A
This
is
just
having
having
multiple
values
in
a
single
field
in
solar
really
useful
for
for
lots
of
things,
but
we
can,
we
can
mimic,
we
can
mimic
multi-valued
fields
with
dynamic
fields
and
the
the
process
to
to
set
this
up
is
very
straightforward.
You
upload
your
schema
and
your
solar
config
into
into
the
cluster,
and
what
happens
here
is
that
that
these
actually
are
stored
in
cassandra
itself,
and
so
so
all
of
your
configurations
are
managed
by
cassandra.
A
And
and
when
you,
when
you
upload
this
schema
you're
also,
you
also
create,
at
the
same
time
a
schema
in
cassandra.
So
you
get
you
get
all
of
the
fields
that
you've
been
that
you've
been
looking
for,
along
with
along
with
a
few
more
and
then
indexes
created
on
each
field
and
you'll
notice,
you'll
notice
a
few
a
few
other
fields
so
doc
boost.
So
so
that's
keeping
track
of
what
you
are,
what
you
are
boosting
in
in
solar.
A
So
so,
when
you're,
looking
for
when
you're
looking
for
a
term
in
solar
and
when
you're
when
you're
setting
up
your
search
engine,
you
can
say
that
the
title
field
is
much
more
valuable
than
say
the
author
or
the
tag
field,
and
so
you
give
that
a
particular
boost-
and
this
is
just
recording
that
and
the
dynamic
field
behind
me
is-
is
also
recording.
Similarly,
just
which
dynamic
fields,
you've,
created
and
you'll
notice,
something
interesting,
you'll
notice,
a
solar
query
in
this
column,
family
and
and
what
this
is
is.
A
A
You
can
also
look
up
information
using
cql
by
specifying
a
where
clause
with
solar
query
equals,
whatever,
whatever
your
term
is,
and
this
way
you
get,
you
get
the
matching
you
get,
you
get
the
ability
to
be
able
to
look
through
the
tokenized
information.
That's
in
that
sorting
cassandra
from
solar
and
and
again
a
few
things
have
changed.
So
in
solar.
You
have
no
multi-valued
fields,
and
you
also
don't
have
any
joints
and
there's
there's
a
little
caveat
here,
because
you
you
never
had
joins
and
distributed
search
to
begin
with.
A
But
when
you're
running
solar
on
a
single
core
or
on
a
single
on
a
single
index,
you
can
you
can
join
across
course
or
cross
tables,
essentially
and
in
cassandra.
You
don't
have
composite
columns
and
you
also
don't
have
counter
columns
as
well,
and
so
so,
ultimately,
this
so
data
stacks
data
stack
search,
provides
a
real,
easy
way
of
of
making
your
solar
fault
tolerant
and
making
your
solar
highly
available.
B
A
B
C
B
A
A
D
A
So
those
are
excellent
questions,
so
the
the
questions
were:
why
have
compact
storage?
Isn't
this
being
phased
out
of
cassandra
to
begin
with,
and
then
the
second
question
was
why
why
are
there
so
many
secondary
indexes
this
to
the
second
question,
I'm
not
sure
I'm
not
sure
why
there
are
secondary
indexes
on
it
on
on
everything.
My
my
hypothesis
is
that
the
the
way,
the
way
that
this
is
structured
is
that
that
any
request
that
comes
to
solar,
that
solar
is
looking
for.
It
looks
up
in
cassandra.
A
It
has
everything
indexed,
but
then
it
will
go
and
make
a
direct
request
to
pull
back
that
information.
That's
stored
in
cassandra.
So
that's
that's
my
guess
as
to
why,
as
to
your
first
question
of
why
compact
storage,
I'm
actually
not
sure
about
this
as
well,
I'm
not
sure
if
anyone
anyone
else
here
would
have
an
idea,
and
I
I'm
also
not
sure
that
it's
being
phased
out.
C
A
A
It's
it's
not
so
so
this
way
is
actually
there.
There
are
a
few
instances
in
which
you'd
actually
want
to
direct
your
questions
into
cassandra,
as
opposed
to
solar,
solar
query
is
there
to
to
provide
the
benefit
of
being
able
to
access
the
same
features,
but
you
you'll
get
the
same
thing
from
solar.
A
A
Here
we
go
to
to
this
to
the
inverted
index,
and
so
so
a
really
good
example
of
tf
idf
is.
The
concept
of
stop.
Words
is,
is
where
you
have
where
you
have
words
that
occur
frequently
in
in
a
single
document,
and
they
also
occur
frequently
in
every
other
document.
And
so
you
have.
You
have
a
very
a
very
high
term
frequency,
but
you
have
a
low
inverse
document
frequency,
because
this
is
a
word
that
appears
everywhere.
A
So
it's
it's
kind
of
meaningless
at
this
point,
whereas
if
you
have,
if
you
have
a
word
that
that
occurs
in
like
a
couple
times
in
one
document
and
then
a
few
more
times
here
and
there
that'll
have
a
higher
that'll
have
have
a
higher
tf
idf,
because
this
is
a
word.
That's
been
keyed
on
as
important.
E
E
A
A
I
think
you
might,
if
you
want
to,
if
you
want
to
get
away
from
that
completely
you'll
have
to
you'll,
have
to
create
a
separate
core
for
for
the
different
language,
but
but
what
you
can
also
do
is
you
can
also.
You
can
also
split
up
split
up
each
language
into
a
different
field
type.
So
you'll
have
text
for
english
text
for
for
japanese
and
then
when,
depending
on
what
you're
searching
over
you'll
get
you'll
get
the
relevant
results
back
based
on
that
particular
language
and
again
relevancy
and
the
the
scoring
itself.
A
B
C
B
A
Well,
so
so,
actually
so
this
is.
This
is
a
really
fair
point,
so
one
of
one
of
my
last
projects
was
actually
with
with
the
pattern
and
trademark
office
and
they
they
were
pulling
in
data
from
from
the
chinese
government
that
had
been
ocr'd
and
machine
translated.
It
was.
It
was
interesting,
it
was
an
interesting
experience,
but
but
here
again
you
have,
you
have
two
different
languages
that
that
people
need
to
be
able
to
index
and
people
need
to
be
able
to
search
over
simultaneously
and
so
so
what
we
did.
A
B
A
And
and
again
the
relevancy,
the
relevancy
score.
So
that's
something
that
that
we
affected
that
we
that
we
changed
in
the
configuration
itself.
So
so
the
english
would
get
a
much
higher
score
on
certain
fields,
whereas
chinese
I'm
also
much
higher
score
on
other
fields.
Relevancy
with
chinese
is
a
little
hard
to
to
determine
if
you
don't
actually
speak
mandarin,
but
that's
that's
a
different
story.
Yes,.
A
A
A
The
way
atomic
updates
work
in
solar
is
that
that
you
they're
not
actually
atomic
you're
putting
in
one
piece
of
information
into
solar,
but
first
you
pull
back
the
entire
document,
merge
the
two
and
then
re-index
it
so
you're,
actually
not
saving
any
time.
I
wouldn't
see
why
that
would
be
a
problem
with
with
cassandra
solar.