►
From YouTube: Stratio: Advanced Search and Top K Queries in Cassandra
Description
Speakers: Andres De La Peña & Daniel Higuero, Big Data Architects at Stratio
This presentation introduces the open sourced Lucene based implementation of the Cassandra secondary indexes developed by Stratio. It allows users to make complex queries in Cassandra using CQL3, including full text search, top-k queries and free multivariable search. Relevance queries and filters can be combined to make searches such as “give me the 100 tweets that best matches this phrase of those written in a certain date range”. Cluster-wide relevance search allows retrieving the N more relevant results that meet a given condition. It’s done through a modified version of Cassandra’s storage proxy in which the coordinator node requests the N best results of each node in the cluster in parallel and combines their partial results to get the N best of them. Stratio’s index is fully compatible with Cassandra’s Hadoop support because it supports all the key/token restrictions in the CQL3 statements. Filters are a powerful help when analyzing the data stored in Cassandra with MapReduce frameworks such as Hadoop or, even better, Spark. Filtering the job input avoids full data scanning, dramatically reducing the amount of data to be processed. Any cell in the tables can be indexed, including primary keys as well as collections. CQL3 wide rows are also supported.
A
Thanks
everybody
for
coming,
my
name
is
Daniel
I,
my
colleague
is
Andres
and
we
are
going
to
talk
about
advanced
search
and
top
key
queries
in
Cassandra.
So
before
we
begin,
we
would
like
to
present
ourselves.
So
we
work
in
a
company
called
strata
and
he's
a
big
data
company
that
was
founded
in
2015
and
became
public
in
2014.
We
have
like
70
employees
in
Madrid
and
we
also
have
an
office
in
San
Francisco
and
we
have
one
of
the
certified
part
distributions.
A
We
are
quite
focused
on
spark,
so
the
contents
of
this
talk
will
be.
First,
we
are
going
to
briefly
review
the
different
way
method
that
already
sees
in
Cassandra.
Then
we
are
going
to
talk
about
our
implementation
of
a
secondary
index,
and
then
we
are
going
to
talk
about
how
these
secondary
index
can
actually
work
with
a
spot.
So
that
will
be
the
contents
of
representation
so,
first
to
briefly
overview.
What
are
we
talking
about?
A
We
have
three
main
ways
of
green
nodes
in
Cassandra,
so
we
can
query
what
primary
key
we
can
play
by
secondary
index
and
we
can
query
right.
Talking
ranges:
the
difference
between
the
different
selections
is
actually
whether
you
want
to
be
more
press
it
and
be
able
to
specify
in
more
detail
which
type
of
data
you
want
to
retrieve
from
the
database
or
you
want
to
be
efficient
and
you
will
like
to
have
more
throughput.
So
the
first
obvious
point
is
queries
by
primary
key,
so
we
have
her
consistent
hashing
in
Cassandra.
A
We
know
where
to
wait
to
go
for
a
specific
query.
So
once
a
query
arise
to
a
Cassandra
know,
it
becomes
the
coordinator
for
the
query
and
we
can
define
and
we
can
identify,
which
will
be
the
note
that
is
turn
the
data.
So,
as
in
the
sample,
we
can
just
issue
a
query:
go
through
the
partition
key
then
go
through
the
clustering
key
which
of
the
rules
that
I
want
to
strut
and
then
divide
the
results
to
the
client.
That
is
fine,
but
we
need
primary
case.
A
So
what
then,
when
we
want
to
make
some
queries
that
are
not
part
of
the
primary
key
or
we
cannot
do
it
by
themselves,
then
we
need
to
rely
on
secondary
indexes
secondary
indexes,
basically
allow
you
to
create
in
vertex
index,
so
you
can
create
index
for
columns
are
not
part
of
the
primary
key
and
then
you
will
go
and
retrieve
the
data.
The
problem
that
you
have
is
that,
as
has
been
said
in
the
keynote
once
the
client
is
a
security
to
the
coordinator.
A
Then
the
coordinator
is
to
forward
that
query
on
all
the
nodes
in
the
cassandra
ring,
because
you
don
no
longer
know
where
to
go
for
a
specific
part
of
data.
It's
in
this
its
store
locally
in
each
node.
So
a
note
maintains
the
index
for
the
data
that
is
stored
locally.
So
you
need
to
go,
find
the
results
in
this
node
and
then
you
get
bad
the
results.
A
A
This
is
especially
nice
for
other
queries.
What
you
want
to
explore
your
data
analyze,
an
aspect
of
your
date.
The
third
part
is
that
we
can
mix
things
in
between.
We
already
have
some
secondary
indexes.
We
would
like
to
make
some
spark
job.
Sometime
of
them,
and
we
can
mix
them
to
strut
data
locally
using
the
index,
so
we
are
going
to
first
query
the
local
index.
Then
this
data
is
going
to
bring
back
to
the
spark
worker,
and
then
we
can
perform
the
MapReduce
job
as
usually
so
there
are.
A
Some
problems
are
those
things
some
things
that
we
are
missing
for
this
instance
secondary
index
implementations,
actually
what
we
would
like
to
have
its
Morse
blessedness
users
come
and
say:
ok,
I
would
like
to
have
this
particular
query.
Solving
Cassandra,
and
sometimes
we
reach
a
point
where
expressive
expressiveness
is
actually
the
problem.
We
would
like
to
be
able
to
make
some
range
queries
like
I
would
like
to
start
data
between
two
given
date.
I
would
like
to
do.
A
Multivariable
search
I
would
like
to
have
some
index
that
actually
indexes
different
columns
at
the
same
time,
so
my
query
can
use
the
index
selecting
different
columns.
In
one
query,
I
will
also
would
like
to
have
some
filter
cells
capabilities
like
I
would
like
to
hire
to
search
for
the
half
stacks
insider
to
enmesh
it.
I
will
also
like
to
have
some
sorting.
A
I
would
like
to
solve
the
fields
the
results
once
they
come
back
from
the
index,
so
there
will
be
easily
analyzing
the
spark,
and
we
will
be
also
interested
in
solving
some
dubcake
queries.
I
would
like
to
have
the
10
most
visited
websites
in
the
last
hour,
for
instance,
but
there
are
things
that
we
actually
like
from
the
index
and
in
particular,
is
this
architecture
characteristics.
A
So
it's
note
indexes
is
some
data
we
have
said
before,
and
that
gives
you
the
ability
to
think
as
the
index
been
a
local
entity
and
avoiding
thinking
about
a
distributed
environment.
So
you
can
focus
actually
on
implementing
and
quite
quite
nice
index
that
it
only
needs
to
think
about
the
local
data.
There
is
no
internet
between
the
different
exits.
Is
this
a
national
natural
extension
point
for
Cassandra's?
You
can
already
implement
different
indexes.
A
A
B
So
what
we
like
from
the
scene?
Well,
it's
a
proven,
stable
and
fast
in
texting
solution.
He
hosts
the
query
in
dr.
Corey
types
we
want,
such
as
wooden
queries,
range
queries
and
relevant
search,
etc.
There
are
several
successful,
distributed,
search
products
built
on
top
of
the
scene
and
also
we
like
the
leucine
is
a
library
and
not
an
installable
service,
so
we
can
embed
it
directly
in
the
code.
Additionally,
it's
an
Apache
project
like
Cassandra
with
a
large
user
community.
Oh
sorry,.
B
B
B
Then
we
create
the
index
insecure,
specifying
the
strategy
index
class
and
indexing
properties.
We
specify
our
index
replace
of
one
minute
and
we
specify
the
default
language
which
will
be
English.
We
specifies
the
date.
Format
is
jared
montz
and
a
and
message
body
will
be
a
token
Isaac
test
in
English,
and
you
said,
ID
and
username
will.
C
B
Once
we
have
created
the
index,
we
can
start
this
inquiries.
For
instance,
we're
going
to
retrieve
10
tweets
containing
the
word
Cassandra
in
the
message
body.
The
strategy
in
syntax
is
very
similar
to
that
of
elastic
search
to
use
your
index.
We
touched
the
query
to
tell
you
seen
extra
column.
This
column
will
be
used
to
retrieve
the
score
in
that
data
back
from
the
index
and
when
using
relevance.
Quizzes
as
this
query
does
not
involve
relevance,
the
Koala
distribution
is
done
using
the
standard
secondary
index
process
shown
before
first.
B
The
query
requests
in
turn,
results
arrives
to
the
coordinator
node,
then
the
coordinator
searches
on
its
own
index
for
the
ten
required
rose
and
imagine
it
only
found
six
rows.
Much
in
the
query,
then
the
coordinator
as
the
nest
node
in
the
cluster
to
search
for
the
remaining
four
in
the
nest
note
finds
the
remaining
four.
Then
the
quad
is
done
and
we
don't
need
to
continue
searching
on
the
remaining
nodes.
B
Now
imagine
we
would
like
to
find
the
five
best
matching
tweets
containing
the
word
Cassandra.
As
a
top
kek
query,
it
will
involve
us
in
relevant
features
before
the
query
arrives
to
the
coordinator
and
then
the
coordinator
distributes
the
query
to
all
nodes
in
the
cluster.
As
we
don't
know,
in
which
notes
the
more
relevant
results
are.
B
B
B
This
way
or
changes
are
generic
and
can
be
used
for
other
possible
index
implementations
phase.
We
have
faded
two
additional
methods
to
the
Cassandra
secondary
index
searcher
and
one
indicates
that
the
quality
required.
The
query,
sorry,
it
requires
relevance
and
the
other
specifies
how
to
combine
the
partial
results.
According
to
the
query,
then,
to
repeal
methods
have
been
added
to
the
cassandra,
astra
brains,
common
and
finally,
some
additional
changes
in
Gator
ranch
slice
method
in
the
class
stress,
proxemics,
edge
of
possible.
B
B
B
Document
ratings
are
committed,
the
sin
index
at
the
same
time
as
the
Cassandra's
column,
families
are
flashed
and,
finally,
that
you
see
the
index
segments
merge
our
synchronize
it
with
Cassandra
column,
family
comp
actions
and
a
future
is
that
or
loosen
indexes
are
fully
embedded
in
Cassandra,
so
no
maintenance.
For
the
point
of
view,
the
user
is
needed
as
compaction.
Our
losses
are
automatically
performed.
B
B
B
B
No
we're
going
to
focus
on
how
can
we
use
the
indexes
from
spark
it's
very
important
for
us
to
combine
or
indexes
with
MapReduce
frameworks
as
Hadoop
or
spark
to
do
it?
We
have
two
requirements.
First,
queries
involving
indexes
must
be
compatible
with
token
reg
restrictions.
Mr.
Conrad
restrictions
have
used
by
spa
reverse
to
partition
the
data.
The
data
install
small
splits
little
partitions
can
be
large,
so
Cassandra
reads
them
using
paging.
B
So
the
second
restriction
for
the
index
is
to
support
page
reads:
the
last
rocky
of
a
page
becomes
the
starting
point
of
the
next
page.
So
from
the
point
of
view,
the
index
this
translates
into
returning
result
in
the
same
order
as
Cassandra
das,
starting
from
a
particular
key
satisfy
the
requirement.
We
need
to
store
the
road
token
and
the
clustering
key.
B
Since
our
index,
XR,
but
expressive
in
integrator,
is
its
integration
with
the
sparkle
06
/
jobs
to
retrieve
from
Cassandra.
Only
the
information
that
you
need.
It
avoids
unnecessary
reads
produced
by
the
usual
systematic
full
table
scan
and
it
greatly
reduced
the
amount
of
data
to
be
processed.
Speeding
up
the
jobs.
B
B
If
you
want
to
process
a
relative
small
fraction
of
the
total
data,
this,
as
you
know,
is
the
general
rule
to
use
indexes,
in
any
case,
for
example,
when
we
want
to
quantify
only
the
data
of
a
specific
time
range
in
a
database
that
has
been
storing
information
from
several
years.
Additionally,
you
will
want
to
filter
this
data
for
the
specific
region.
B
In
this
case,
we're
going
to
process
say
less
than
the
term
percent
of
the
total
store
data
and
if
we
will
do
it
in
a
regular,
full
scan,
imaginate
worsted
time
reading
the
ninety
percent
of
unwanted
data.
So,
let's
put
it
on
prospective
here.
We
have
a
graph
taken
from
a
small
batch
mark
and
it
compares
the
performance
of
spark
use
in
Allison
index
or
using
full
scan
with
the
successive
queries
requesting
from
the
one
percent
to
the
one
hundred
percent
of
the
storage
data
on
the
x-axis.
B
We
show
the
number
of
return
arrows
on
the
y-axis.
We
saw
the
query
response
time
in
seconds.
We
can
see
that
the
full
scan
time
in
blue
is
constant
and
independent
of
the
amount
of
filter
data.
On
the
other
hand,
we
can
see
a
high
performance
for
the
index
in
orange
for
the
coolest
requesting
strongly
filter
data.
However,
the
performance
the
case
in
less
restrictive
queries
and
as
the
number
of
records
returned
by
the
core
increases,
we
reach
a
point
where
the
index
becomes
is
lower
than
the
fools.
B
D
D
C
B
D
A
So
yep
so
we're
going
to
talk.
We
are
going
to
present
other
tools
that
we
have
built
on
top
of
these
indexes,
so
it
may
be
easier
for
end
users
to
use
them.
The
first
one
is
called
strata
deep
and
it's
an
integration
with
a
spa
is
a
different
aggression
with
a
spar
than
one
of
the
of
data
stats.
We
start
to
go
in
a
different
path,
the
very
beginning
of
this
year
actually,
and
it
provides
you
with
a
commercial
obstruction
for
every
dds.
So
you
can
also
get
data
from
our
data
stores.
A
It
maintains
compatibility
with
the
system
spare
operations
and
it
uses
the
classical
addition
of
another
type
of
spark
context
and
another
job
comfy
and
it's
compatible
with
the
listening.
This
is
that
we
have
presented
next.
We
have
a
sequel
layer
on
top
of
everything,
on
top
of
spark
and
Allison
indexes,
they
tries
to
unify
button
and
streaming
queries
in
a
single
sequel
like
query,
so
we
have
a
common
language
that
is
sequel
like
at
least
sequel
and
it's
compatible
with
the
use
of
different
data
store
technologies.
A
At
the
same
time,
so
you
can
effectively
make
a
joint
query
between
two
tables
that
are
stored
in
different
databases
or
in
the
same
database,
even
though
the
database
may
not
support
the
joint
operate
or
as
Cassandra,
for
instance,
it's
based
on
defining
different
connectors,
so
you
can
define
the
connector
you
want
for
the
database
of
your
choice.
It
has
this
ability
and
it
complements
its
non-native,
supported
query
by
the
means
of
using
sparkles.
So
the
user
doesn't
need
to
think
about
whether
it
actually
uses
power
or
not.
A
We
respect
to
the
indexes
that
what
we
are
talking
today
about,
we
have
made
some
small
changes
in
the
same
tax,
so
we
can
facilitate
the
creation
of
this
type
of
indexes
and
avoid
the
you
have
to
write
all
the
JSON
files.
So
you
can
issue
I,
create
full
text
cinders
and
you
can
pass
the
different
columns
that
will
be
part
of
the
index.
A
This
is
going
to
be
automatically
processed
and
the
types
are
going
to
be
automatically
identify
and
match
to
the
to
those
of
the
scene,
or
you
can
specify
your
Jerome
maintained
and
then,
for
the
point
of
view,
the
queries
what
we
have
done
is.
We
have
introduced
a
new
operator
in
the
work
loss
of
a
select
statement,
call
Matt
and
much
is
used
to
say
that
whatever
comes
after
the
Matt
is
going
to
be
interpreted
as
losing
syntax.
A
So
some
conclusions,
we
have
added
a
new
quay
method.
We
have
support
for
different
multivariable
queries
with
different
connectors
and
or
not.
We
support
range
queries
in
this
type
of
indexes,
regular
expressions
when
you
try
to
find
full,
you
try
to
make
folder
searches,
different
type
of
filters,
queries
like
match
phrase
fussy
and
we
support
top
kek
queries
by
means
of
adapting
the
coordinator
logic,
so
we
can
order
the
results
before
returning
them
back
to
the
user.
A
We
have
make
the
solution
compatible
with
the
system
MapReduce
framework.
So
if
you
would
like
to
make
a
spiral
job,
you
can
already
use
this
solution
and
we
preserved
the
cassandra
functionality
and
compatibility
with
secure
both
of
our
solutions
and
everything
we
develop.
It's
open
source.
We
have
a
fork
from
Cassandra
that
contains
these
modifications
to
the
cassandra
logic
and
you
can
find
it
in
github
strategy
of
strata.
Cassandra
is
everything
is
published
and
then
under
an
apache
license,
so
you
can
feel
free
to
go.
A
A
A
Yeah
this
is,
it
is
the
system
design
tool,
so
there
is
no
modification
from
the
leucine
bar.
The
work
is
done,
integrating
both
both
systems.
So
what
we
have
done
is
willing.
The
different
operations
in
Cassandra
will
loosen.
So
if
you
define
an
index-
and
you
make
an
insert,
then
that
insert
will
be
passed
to
listen,
so
the
colonies
indexes
and
everything
is
updated.
C
A
A
Yeah
I
would
say
that
this
is
highly
motivated
by
the
fact
that
we
would
like
to
perform
some
spark
jobs.
So
what
we
like
to
do
is
to
as
much
as
possible
filter
the
data
that
we
are
uploading,
let's
say
or
treating
in
as
part,
so
the
ability
to
just
loosen
locally
in
each
node
give
you
the
ability
to
reduce
the
amount
of
data
that
you
start
moving
through
a
glass.
Sir.
That's
why
we
choose
this
type
of
integration
and
not
go
in
with,
let's
say
external
service
or
sternal
integration.
D
A
A
Yeah
I
mean
what
this
part
is
basically
already
supported
by
Cassandra.
So
do
you
just
connect
to
the
Sistine
index
API
and
you
will
be
notified
that
this
is
a
chain
or
the
compaction
is
going
to
start,
so
you
can
basically
attach
your
operations
to
that
different
events
in
the
system
and
perform
accordingly.
If.