►
Description
Speaker: Richard Low, Analytics Tech Lead at SwiftKey
Slides: http://www.slideshare.net/planetcassandra/c-summit-eu-2013-mixing-batch-and-realtime-cassandra-with-shark
Everything Cassandra does is designed for a real-time workload of high volume inserts and frequent small queries. Cassandra has Hadoop and Hive integration, but performing long running ad-hoc queries with these tools is difficult without impacting real-time performance or requires duplicate clusters. This talk will explain how I'm integrating Cassandra with Shark, a drop-in Hive replacement developed by Berkeley's AmpLab. It's designed to give fine grained control over all resource usage so you can safely run arbitrary ad-hoc queries on your existing cluster with controlled and predictable impact.
A
All
right,
good
afternoon,
everybody
thanks
for
coming
so
today
I'm
gonna
talk
about
a
little
bit
of
research.
I've
been
doing
on
mixing
batch
and
real
time
using
cassandra
and
shark.
So
to
begin
with
a
tiny
bit
about
me.
So
currently
I'm
the
analytics
tech
lead
at
swiftkey,
so
swiftkey
we
make
the
number
one
selling
android
app,
which
is
a
keyboard.
We
do
various
other
things
as
well
and
I
deal
with
the
analytics
infrastructure.
A
I
also
do
a
bit
of
casana
freelance
stuff.
You
may
see
me
write
blog
posts
or
answer
questions
on
the
mailing
list
or
various
other
things,
and
and
today's
talk
fits
under
the
freelance
cassandra
banner.
This
isn't
this
isn't
some
some
work
at
swiftkey.
This
is
some
work.
I've
done
in
my
in
my
spare
time
and
before
I
I
did
cassandra
and
analytics
development
at
okunu.
A
So
what
am
I
going
to
talk
about
today?
So
yeah?
The
subject
is:
is
batch,
analytics
and
real
time,
so
I'm
I'm
going
to
sort
of
work
through
a
particular
use
case
that
I
have
in
mind
for
this,
and
I
feel
that
this
use
case
is
kind
of
in
between
the
tools
that
are
available,
and
you
can't
quite
do
it
today,
so
I'm
I'm
sort
of
doing
it.
I've
done.
I
came
up
with
an
idea
and
I'm
trying
to
show
that
it.
It
does
something
useful
so
yeah.
A
A
So
what
do
I
mean
by
batch
analytics
and
real-time
databases?
So
if
you're
using
something
like
cassandra,
you
probably
have
some
real-time
requirement.
You
want
to
you,
know,
power
your
website
or
power,
your
stock
picking
system
or
something
where
you
care
about
how
how
quickly
you
get
answers
back.
A
If
you
don't
care
about
that
speed,
then
you
probably
shouldn't
be
using
cassandra,
because
it's
doing
extra
work
that
you
don't
need,
but
whenever
you
have
a
database
there
are
always
queries
to
do
on
it
which
are
slow
or
you
didn't
think
of
in
advance,
or
that
you
didn't
even
know
you
you
could
possibly
do,
but
someone
else
will
come
and
say.
Can
you
run
this
query
for
me?
Please.
A
So
if
you're
running
a
real-time
system,
you'll
say
no.
You
can't
do
this,
because
if
you
run
this
query,
then
our
performance
is
going
to
tank.
You
know
requesting
in
a
timeout,
customers
are
going
to
complain,
stuff's
going
to
break
so
you
say
no,
I
can't
do
it,
but
this
query
may
be
immensely
valuable.
It
may
be.
You
know
the
difference
between
you
may
affect
a
marketing
campaign,
for
example
that
may
make
make
loads
more
sales.
A
You
know
it
could
be
a
really
valuable
thing
so
so
they
won't
take
no
for
an
answer,
but
then
you
think
well,
actually,
maybe
I
can't
even
do
this
query.
You
know
cassandra
doesn't
support
every
query.
It
doesn't
support
a
lot
of
queries.
Actually,
you
can't
issue
a
query
which
is
going
to
scan
through
all
of
your
data
and
and
find
something
else.
You
know
my
sequel
or
or
you
know
most
sql
databases
could
do
that,
but
they
they
will.
A
A
This
is
you
know,
a
list
of
fairly
boring
user
accounts,
but
every
operation
that
your
website
does
hits
this
at
least
to
verify
someone's
password.
So
this
this
sort
of
underpins
your
entire
website.
So
if
you
have
timeouts
or
latency
spikes
here,
then
everything
slows
down
pages
low,
page
load
slows
down
things
time
out,
people
get
unhappy
and
you
know
maybe
stop
using
your
site.
So
you
know
you
do
care
about
the
latency.
Quite
a
lot
here.
A
You
don't
just
have
user
accounts,
you
have
other
stuff,
you
know,
maybe
you're
doing
you
know,
messaging
or
or
whatever
you
know,
you've
got.
You've
got
other
things
in
your
in
your
database
in
your
cassandra
database.
Even
that
are
ongoing.
So
you
know
you've
got
you've
got
this.
Balance
of
these
reads:
you've
got
some
rights
in
some
other
column.
Families
you've
got
all
this
stuff
going
on
in
in
in
cassandra,
but
let's,
let's,
let's
carry
on
focusing
on
these
user
accounts.
A
So
let's
say
you've
got
this
schema.
You've
got
some
user
id
a
bit
of
stuff
about
the
user.
Actually,
the
stuff
that
I'm
going
to
focus
on
for
this
example
is
the
country
and
the
last
visited
so
country
is
just
there.
Just
remember
that
last
visited
is
the
only
thing
in
this
table
which
isn't
static.
Pretty
much
you
know,
maybe
you
can
change
your
username.
Maybe
you
can
change
your
email
or
your
password,
but
that
doesn't
change
very
often
last
visited
will
change
every
time
you
hit
the
website.
A
A
A
Please
can
you
find
all
users
from
brazil
who
haven't
logged
in
since
july
and
their
email
is
at
yahoo.com
and
I
need
the
answer
on
monday,
so
you
know
you
can
imagine
they're
going
to
launch
some
advertising
campaign
on
yahoo
in
brazil
to
say.
Please
come
back
to
our
site,
it's
great,
but
they
want
to
know
who
it
is
so
they
can
say
to
yahoo.
A
Please
put
these
adverts
up,
so
you
know
it's
pretty
important
that
you
get
this
right
because
otherwise
they're
going
to
waste
all
this
advertising
money-
and
maybe
you
won't
get
your
any
of
your
users
from
brazil
to
come
back,
but
it's
also,
you
know
you
don't
have
to
do
it
particularly
quickly.
You
know
it's
thursday.
Today
you
could
run
the
query
tomorrow
and
then
it
could
run
over
the
weekend.
You
get
the
answers
on
monday
morning.
A
So
a
few
sort
of
thoughts
about
this
query.
We've
got
500
000
users
from
brazil.
I
just
picked
that
number.
That's
how
many
users
we
have
so
that's
60
megabytes
of
raw
data,
something
like
that
for
our
our
our
accounts
database.
So
it's
not
a
huge
amount
of
data,
but
how
do
we
get
that
data?
So
we
we
didn't
put
index
on
country?
I
I
didn't.
I
didn't
show
you
any
indexes
there.
There's
no
way
of
selecting
by
country
know
also
that
most
of
this
data
is
unchanging.
The
timestamp
is
changing.
A
That's
the
only
thing
that
is
and
that
we
do
actually
care
about
that
for
our
query,
because
we're
saying
people
who
haven't
come
since
july,
but
if,
if
you
know
if
we
miss
people
who
haven't
been
since
july,
but
came
in
the
last
hour,
maybe
we
don't
care
too
much.
So
basically
we
could
work
on
a
on
a
snapshot
of
the
data
or
you
know
the
data
from
yesterday
maybe
would
work
as
well
it
doesn't.
It
doesn't
have
to
be
the
current
totally
current
data
yeah.
A
We
can
take
hours
running
this
query
that
doesn't
matter
we.
I
guess
we
could
take
the
weekend
to
run
it,
but
it
would
be
nice
if
you
had
the
result
before
you
went
to
the
pub
tomorrow
evening.
So
you
know
if
it
took
hours,
you'd
be
you'd,
be
happier
and
note
also
that
we
probably
won't
have
to
rerun
this
query
again.
Maybe
we'll
have
to
run
something
related
to
that,
but
we
don't
have
any
any
knowledge
of
that.
A
So
we
don't
really
want
to
build
some
complicated
system
or
add
an
index,
or
things
like
that
that
may
you
know,
require
maintenance
in
the
future,
because
we
probably
only
have
to
run
this
once
so
before
I
explain
more
technical
stuff
just.
Why
am
I
interested
in
this?
Why
do
I
want
to
do
this?
Well,
I
think
I
think
this
sort
of
use
case
is
is
a
bit
underrepresented
in
the
in
the
tools
that
are
available,
and
I
have
seen
people
spend
days
writing
code
messing
around
with
databases
to
answer
stuff
like
this.
A
So
if
there
are
tools
there,
then
you
know
it
would
help.
So
I
thought,
let's
see
what
we
can
do.
So
what
can
you
do?
What
what
can
you
download
off
and
and
run
so
so
you've
got
cassandra?
Well,
you
can
run
hive
on
cassandra,
so
so
we've
seen
talks
today
with
people
who
have
been
running
hadoop
jobs
on
cassandra.
Well,
you
can
stick
hive
on
that
and
it
will
read
data
added
cassandra.
A
You
can
run
arbitrary,
sql
queries
that
will
work,
but
it
may
cause
you
some
problems,
so
so
you're
you're
going
to
compete
with
cassandra
on
all
of
the
resources,
and
you
don't
know
what
what
hive's
going
to
do.
You
know
it's
going
to
run
a
few
map-reduced
jobs.
Some
will
be
io
intensive.
Some
will
use
lots
of
memory.
Some
will
use
lots
of
cpu
at
various
different
times
and
you'll
probably
find
your
latency
will
go
through
the
roof
and
you'll
get
some
timeouts
and
people
will
complain.
A
You
may
find
that
because
you're
you're
reading
data
through
the
cassandra
query
interface,
you
know
hive
is
just
another
thing.
That's
querying
cassandra.
You
may
get
some
extra
gt
pressure
on
cassandra
which
again
hurts
your
your
latency,
and
you
may
also
find
that
you
flush
your
your
cache.
You
pollute
your
cache
so
that
your
hot
set
of
user
accounts
is
no
longer
in
in
cache.
A
We
could
write
some
code.
We
could
extract
the
data
set
up
a
mysql
instance
or
something
you
know.
It's
only
60
megs
of
data
to
extract.
We
could
write
some
code
to
read
through
it
and
do
that.
That's
not
not
pretty.
That's,
not
very
good.
We're
going
to
have
to
write
some
code,
we'll
probably
get
it
wrong.
It
will
be
single
threaded,
which
means
it
will
be
slow.
A
So
it
probably
won't
impact
our
cluster
too
much,
but
it
may
take
a
very
long
time
because
you
know
just
one
thread
reading
through
your
data
and
if
that
fails
after
10
hours,
then
how
do
you
restart
it?
Things
like
that
and
you've
still
got
the
caching
problem,
although
because
it's
so
slow
it
that
may
not
matter
so
we
don't
really
want
to
do
that.
We
could
also
clone
the
cluster.
We
could
say
right.
A
We've
got
this
cassandra
cluster
here,
let's,
let's
copy
the
the
the
ss
tables
off
disk
into
another
one,
and
then
we
can
do
what
the
hell
we
like
there,
because
it's
not
it's
not
powering
our
system,
but
that's
going
to
generate
the
worst
possible
network
load.
It's
going
to
copy
all
your
data
out
and
it's
a
manual
import.
So
I
mean
I
said.
I
said
that
we're
not
going
to
run
this
query
again,
but
maybe
we're
going
to
want
to
run
some
different
kind
of
query
next
week.
A
A
The
final
thing
is
quite
a
common
way
of
solving
this
problem
is
to
add
a
sort
of
logical
data
center
which
which
you
don't
read
and
write
from
in
your
real-time
application,
but
you
run
you
run
your
hadoop
hive,
jobs
on
which
does
work
well,
but
for
us,
that's
not
great,
because
we'd
have
to
bring
a
new
data
center
online
before
monday,
which
again
involves
copying
all
the
data
out
and
yeah.
We
need
to
get
some
more
hardware,
maybe
we're
on
the
cloud.
A
So
that's
okay,
but
if
we're
not,
then
we're
probably
not
going
to
get
some
more
boxes
by
monday,
and
this
keep
keeping
this
this
extra
cluster
maintained.
Well,
if,
if
it's
really
busy
running
your
query,
then
it's
still
going
to
drop
right
and
get
out
of
sync,
maybe
that's
okay,
maybe
that
isn't
I
don't
know
so
so
this
this
is.
This
is
a
reasonable
solution
to
this
problem
in
in
some
cases,
but
when
you're
doing
sort
of
the
occasional
high
value
ad
hoc
query,
it's
probably
not
not
not
the
right
thing.
A
So,
let's
talk
about
spark
and
shark.
The
two
tools
that
I'm
I'm
going
to
try
and
use
to
to
solve
this
problem.
I
couldn't
find
a
shark
logo,
but
shark
is
hive
on
spark.
That's
where
the
h
comes
from,
I
think
so
spark
so
it's
developed
by
amplab
in
in
berkeley.
It's
it's
a
distributed,
computation
tool.
It's
it's
a
bit
like
hadoop,
so
you
can.
A
You
can
replace
hadoop
with
with
spark,
but
the
main
reason
to
me
why
it's
interesting
is:
is
it's
designed
for
iterative
algorithms
by
iterative
I
mean
that
make
multiple
passes
through
the
data,
so
most
sql
queries.
You
do
actually
have
to
read
through
the
data
more
than
once,
and
this
is
just
like
a
simple
filter.
A
But
what
spark
is
designed
for
is
to
store,
rather
than
the
intermediate
data
it
stores
the
intermediate
computation.
How
do
you
get
from
one
step
to
another
so
that,
if
you,
if,
if
something
fails
in
between
you,
can
rebuild
it
from
the
from
from
the
just
by
applying
the
computations
to
that
little
bit
of
data
rather
than
pulling
it
back
in
which,
which
means
that
you
can
store
this
in
memory?
A
Like
almost
all
queries,
you
do
the
amount
of
the
amount
of
data
you
know
rapidly
decreases
with
each
iteration,
so
in
our
case
we're
we're
reading
through
all
of
our
data
to
extract
60
megs.
So
we
can
obviously
store
that
in
in
memory
you
know
distributed
or
not.
So
so
that's
that's
a
pretty
nice
feature.
So
you
find
that
once
you've
done
your
first
pass
through
the
data
hive,
queries
on
spark
will
run
many
times
faster
than
hive.
You
know
people
have
reported
more
than
100
times
in
speed
improvement.
A
So
spark
is
used
by
many
people.
I
just
picked
a
few
of
their
website
that
you've,
probably
heard
of
so
you
know
it's
it's
fairly
new,
but
it
is
being
used
in
production,
so
shark
yeah.
This
is
hive
on
spark.
It
looks
like
hive,
it's
the
same
query
language.
You
can
use
the
same
user-defined
functions.
A
You
can
use
the
same
storage
handlers,
the
thing
that
the
the
pull
data
out
of
other
systems
into
hive
so
so
yeah
it
looks.
It
looks
pretty
much
the
same
to
you,
but
the
really
nice
feature
of
it
is
that
you
can
cache
tables.
So
you
can
write
something
like
this
create
table.
User
accounts
underscore
cached,
that's
giving
shark
the
hint
that
it
needs
to
cache
this
as
select
style
from
user
accounts
where
country
equals
brazil.
A
So
so,
if
you
ran
that
query,
what
it
would
do
is
it
would
read
through
all
of
your
data.
Only
keeping
the
data
from
brazil
store
it
in
memory
across
your
nodes,
and
then
you
can
run
really
complicated
queries
on
it
ridiculously
quickly.
You
know
it
will
read
through
you
know:
100
000
records
a
second
per
core
kind
of
speed
once
you've
got
it
in
memory
like
that.
A
So
I'm
not
you
can't.
You
can't
do
that
in
in
hive
on
on
hadoop
at
all.
So
I
think
I
think
that's
that's
kind
of
the
killer
feature
of
spark.
For
me
it
means
that
if
you're,
if
you're
an
analyst
you
can
you
can
come
in
in
the
morning,
you
can
run
this
cache,
go
away
and
have
some
meetings
and
make
some
coffee
or
whatever.
And
then
you
come
back
at
lunch
time
and
everything's
in
memory,
and
now
you
can
do
interactive
queries
on
your
on
your
data
set.
You
know
that.
A
A
So
now
we
get
to
shark
on
cassandra.
So
you
can
do
this
now.
There's
the
cql
storage
handler,
which
you
can
download.
It's
not
part
of
cassandra.
Yet
there's
a
there's
a
ticket
to
sort
of
integrate
it,
but
you
can
find
it
on
github.
I
don't.
I
don't
think
it's
properly
released,
yet
there's
probably
a
few
bugs
in
it,
but
it
works
pretty
well
and
actually
there
was
a
talk
at
cassandra
san
francisco.
A
So
what
this
does
is
you
run
spark
nodes
on
your
cassandra
nodes,
but
instead
of
pulling
the
data
through
the
cassandra
cql
interface,
you
pull
it
through
the
raw
ss
tables
on
disk.
So
there's
various
advantages
to
that.
You
have
a
bit
more
control
over
the.
I
o
that's
going
on,
so
you
can.
You
can
do
have
less
impact
you're
not
going
through
cassandra.
A
A
If
you
run
shark
through
the
cql
storage
handler,
you
know,
you've
got
disk
memory.
Cassandra
then
you've
got
your
two
processors.
You've
got
your
spark
worker.
That's
doing
your
hive
query
and
you've
got
your.
This
is
your
client,
your
website,
that's
doing
the
things
you
really
care
about,
and
then
maybe
you
find
you
you,
you
get
some
latency
spikes.
So
what
the
ss
table
storage
handler
does.
Is
it
skips
straight
from
ssd
was
on
disk
to
the
spark
worker?
So
we
skipped
the
cache
and
we
skipped
the
the
cassandra
jbm.
A
There
are
a
few
disadvantages
of
this,
which
I
should
say
before.
I
show
you
it
working,
so
we
can't
do
any
reads
from
other
nodes
to
find
out.
If
we've
got
the
latest
value
or
not,
we
just
we
just
read
the
file
and
take
it.
That's
the
latest
value,
so
that's
equivalent
to
the
cql
storage
handler
reading
at
consistency
level,
one.
It
just
reads
from
one
node,
that's
not
too
bad.
A
This
second
point
always
runs
task
like
all
the
data
is,
is
actually
mostly
an
advantage,
but
it's
it's
a
disadvantage
in
in
some
cases,
because
you
don't
have
the
option
of
running
non-local
tasks.
So
you
know
maybe
one
node
is
running
slow
because
its
cpu
is
doing
something
you
can't
sort
of
allocate
that
task
to
another
node,
because
we
can't
copy
that
data
across.
We
don't
have
you
know
htfs
or
anything
to
do
that
for
us.
A
A
A
A
These
have
two
virtual
cores,
7.5
gigs
of
memory
and
two
ephemeral
disks.
I
was
using
the
local
storage
on
them,
so
you
need
one
spark
master.
That's
a
little
bit
like
the
job
tracker
in
hadoop.
If
you're,
if
you're
familiar
with
hadoop
yeah,
I
set
up
the
the
spark
workers
on
the
four
cassandra
nodes
and
I
limited
each
each
worker
process
to
only
use
one
core
and
one
gig
of
ram.
A
So
you
can
tell
you
can
tell
spark
you
know:
you're
only
allowed
to
use
these
these
resources
on
on
these
nodes,
and
what
I
wanted
to
do
was
compare
the
cql
storage
handler
with
the
ss
table
storage
handler.
So
I've
got
some
results
from
that
just
a
little
bit
of
stuff.
On
the
on
the
configuration,
so
I'm
using
the
latest
version
of
cassandra
1.2
three
gig
heap,
so
that's
almost
half
my
memory
is
on
the
the
heap
in
cassandra,
I'm
using
virtual
nodes,
replication
factor.
A
A
But
apart
from
this,
I
didn't
do
any
optimization
or
tuning.
Let's
say
that
you
know
the
person
who
who
deployed
this.
You
know
they
just
set
it
up
and
they
they
didn't.
They
didn't
want
to
be
bothered
with
any
any
tuning,
a
few
tools
in
case
you're
interested.
So
so
I
used
the
the
metrics
library
from
coda,
hale
and
well.
I
think
it
originally
came
from
yammer.
It's
absolutely
fantastic,
really
good
for
instrumenting
your
code
to
get
some
results,
so
so
a
good
tool
to
use.
A
So
I
actually
used
that
to
pipe
the
results
through
to
ganglia.
So
I
could
see
what
was
going
on.
As
I
was
running
running
the
tests,
I
wrote
some
load
generators
using
the
data
stacks
java
driver
and
google
spreadsheet
is
surprisingly
useful
tool,
especially
for
looking
at
time
series
data.
It's
got
this
really
nice
visualization,
where
you
can
like
zoom
in
on
different
times,
which
I
didn't
realize
existed.
So
that's
a
good
tool.
Okay.
A
So
let's,
let's
look
at
the
first
test
I
ran
so
this
is
cassandra
idle
cassandra's,
not
doing
anything
at
all,
just
a
sort
of
baseline
and
the
query
I'm
actually
going
to
measure.
Is
this
caching
query
the
one
I
said
before
read
through
all
the
data
filter
out
only
the
ones
from
brazil
and
store
in
memory,
so
this
took
33
minutes
through
cql
storage,
handler
and
13
minutes
through
the
sstable
storage
handler
which,
which
equates
to
about
130
000
records
a
second.
A
So
it's
about
two
and
a
half
times
faster,
and
in
fact
this
is.
This
is
slightly
unfair
on
on
the
ss
table
storage
handler
because
the
cql
will
not.
We
can't
actually
limit
it
to
one
cpu,
because
it's
talking
to
cassandra
cassandra's
using
some
cpu.
It
turns
out
that
if
you
only
give
it
one
cpu,
it
doesn't
use
much
more
than
one,
but
it
does
it
uses
about
60
rather
than
50.
So
that's
that's
a
little
bit
unfair,
but
it
doesn't
matter
so
so
yeah.
A
So
it's
faster
right,
but
you'd
expect
it
to
be
faster
because
we're
not
serializing
deserializing
shifting
stuff
over
the
network
and
other
things.
So
now
we've
done
that.
Let's
just
show
that
we
can
actually
run
the
query.
We
want
to
run
so.
We've
got
the
data
in
cache.
So
so
we
can
run
our
query.
So
this
is
this.
Is
the
the
shark
query
or
the
hive
query?
The
only
thing
that
looks
a
bit
weird
is
the
email
I
randomly
generated
emails.
A
I
didn't
have
the
yahoo.com,
but
that
one
gives
a
reasonable
amount
of
of
data.
So
so
this
is
the
query
you
can
run
and
it
took
18
seconds
right
that
that's
the
really
nice
thing.
So
this
read
through
yeah
the
500
000
records.
Actually
the
randomly
generated
one
turned
out
to
be
400
000,
but
you
know
a
reasonable
number
of
records
and
the
answer
was
282.
If
you
care
so
so
yeah,
so
you
can,
you
can
run
your
result
really
fast.
A
Now,
okay,
so
this
is
kind
of
the
the
baseline
thing,
but
actually
we
don't
have
an
idle
cassandra.
We
have
lots
of
people
visiting
our
website,
so
I
added
a
read
load
of
200
reads
per
second,
but
remember
when
we
do
when
we
actually
log
into
our
website
we're
updating
the
last
visited
time.
So
actually,
what
I'm
doing
is
I'm
I'm
reading
in
the
the
the
user
account
and
then
I'm
writing
out
the
last
visit
at
a
time
saying:
yep.
I've
just
come
so
I'm
measuring
the
latency
of
that
process.
A
Because
that's
the
thing
I
really
care
about.
You
know
if
if
this
latency
is
high,
then
my
users
are
going
to
time
out
and
or
and
get
unhappy,
so
so
for
the
test.
I
I
I
ran
the
cql
storage
handler
and
then
worked
out
how
long
it
took
and
then
rate
limited
the
ss
table
storage
handler
to
that
speed
to
compare
the
latencies.
A
A
And
this
is
what
happens
so,
the
the
different
lines
here
so
the
the
yellow
is
the
mean
latency
for
ss
tables
and
the
green
is
the
95th
percentile
latency
for
ss
tables,
and
then
the
blue
and
the
red
are
the
same
for
the
cql.
So
you
can
see
the
mean
latency
is
lower,
but
probably
more
interesting
is
the
the
95th.
Percentile
is
much
more
consistent.
A
So
there's
those
two
big
spikes
when
reading
through
the
the
cql
loader,
but
you
know
there's
too
much
smaller
spikes,
otherwise,
so
just
just
for
comparison
when,
when
you're
not
running
a
query,
the
the
the
mean
latency
is
about
20
milliseconds
and
the
and
the
90
percentile
is
about
60..
So
we
do
have
an
impact
on
latency.
Of
course
we
do
we're
using
some
io.
You
know
we're
doing.
200
random
reads
a
second
across
our
cluster
of
four
nodes.
So
you
know
we're
using.
A
You
know
a
reasonable
amount
of
rio,
but
now
we're
running
a
query
on
it.
So
you
know
we
are
going
to
have
some
impact.
You
know
the
question
is
how
much
and
and
how
predictable
so
yeah.
So
this
shows
the
the
latency
impact
is:
is
less
on
the
on
the
for
the
mean
latency,
but
the
the
the
95th
percentile
latency,
the
peaks
are
much
lower.
It's
much
more
predictable.
A
Well,
I
didn't
spend
too
much
time
looking
at
it,
but
you
know
I
expect
the
license
the
the
average
latency
changes,
because
we're
we're
using
one
core
out
of
two
there's,
just
less
less
calls
available
to
do
your
stuff,
and
maybe
the
cql
storage
handler
has
some
some
some
of
those
latency
spikes
do
some
gc,
because
there's
lots
of
data
going
through
the
sql
interface.
I
don't
know
I
didn't.
I
didn't
measure
it.
So
I
don't
know
I
don't
know
what
what
was
what
was
happening
there.
A
So
the
final
result
was
that
was
trying
to
say.
Well.
Okay,
we've
got
some
other
tables
on
our
cassandra
cluster,
doing
loads
of
inserts.
What
happens
there,
but
I'm
still
measuring
the
read
latency.
You
care.
You
normally
care
more
about
read
latency
than
than
rights
latency,
so
I
added
two
and
a
half
thousand
random
inserts
a
second
into
another
table,
and
I
got
this
horrible
graph,
so
this
doesn't
really
show
anything
useful.
A
I
so
so
it's
like
so
just
just
to
explain
what
this
what
is
showing
so
the
beginning
is
just
the
load.
The
read
load
and
the
right
load
then
add
the
cql
query.
I
didn't
actually
let
it
complete
just
ran
for
half
an
hour.
Then
I
killed
it
left
it
for
a
bit
then
run
the
ss
table.
One
and
you
know
there's
loads
of
latency
spikes.
Even
when
I'm
not
running
any
queries,
so
this
is
basically
showing
my
my
insert
load
is
is
harming
my
my
read
load.
A
So
I
guess
it's
showing
my
cluster
isn't
really
big
enough
to
do
what
I'm
trying
to
give
it.
It
was.
It
was
quite
late
at
night
by
this
point,
so
I
didn't
run
any
more
queries.
But
if
I
had
more
time,
what
I
would
have
done
is
run
it
with
a
bit
less
insert
load
to
try
and
get
you
know
a
flat
latency
without
any
queries,
and
then
maybe
we
get
some
interesting
comparison
there,
but
I
didn't
have
time
to
do
that.
A
So
this
this
doesn't
really
show
show
anything
particularly
useful,
yeah
lots
of
latency.
There
is
anyway,
so
so
yeah.
So
what
do
we
find
from
the
performance?
So
it's
two
and
a
half
times
faster,
which
means
we're
doing
we're
using
less
resources
to
do
the
same
thing,
which
is
always
a
plus,
but
that's
not
the
main
excitement.
The
main
excitement
is
that
the
the
latencies
are
are
more
predictable
when,
when
running
at
the
same
speed,
so
so
it
shows
that
you
know
this
approach
of
bypassing
cassandra
to
read.
A
The
data
does
work
so
so
actually
what
you'd
probably
want
to
do,
though,
is
limit
it
further.
You
know
those
latency
spikes
may
still
be
too
much
so
you'd
actually
run
it
run
it
even
slower
than
that.
So
I
guess,
if
you
were
going
to
run
this
on
your
website,
you
probably
have
a
graph
of
speed.
You
can
run
it
out
versus
latency
impact.
You
can
say
okay,
so
today
I
need
to
get
the
answer
in
this
time.
Oh
I'm
going
to
have
this
lazy
latency
impact.
Is
that
okay?
A
Yes,
if
it's
not,
then
you
run
it
a
bit
slower.
So
so
I
guess,
if
I,
if
I
had
this,
what
I
would
do
is
I'd
run
it
I'd
I'd
work
out,
so
I
need
the
result
by
monday.
Actually
I'd
prefer
to
have
it
friday
evening.
Let's
kick
it
off
now,
estimate
it's
going
to
take
24
hours
to
run,
work
out
that
speed,
which
will
be
really
slow.
You
know
it'll
be
like
100k,
a
second
or
something
rate
limited
to
that,
and
then
you'd
expect
to
have
very,
very
little
latency
impact.
A
A
Finally,
what
am
I
going
to
do
with
this?
Well,
an
easy
name.
If
you
have
any
suggestions,
please
tell
me:
I
need
to
push
it
to
github,
so
people
can
use
it
but
yeah
if
you're
interested,
then
please
come
and
talk.
You
know
it's
just
it's
just
a
research
project
that
I've
done.
If
you
want
to
use
it,
if
you
think
it
could
do
something
or
couldn't
do
something,
then
please,
let
me
know
thanks
very
much.
Do
you
have
any
questions.
A
A
So
it
feels
like
the
main
thing
that
you
want
to
be
optimizing
for
in
this
case
is
to
minimize
impact
on
latency,
and
I
guess
one
of
the
things
that
you
did
do
was
you
took
you
had
this,
where
clause
in
your
in
your
query,
which
was
bringing
the
data
into
memory
for
just
looking
at
the
people
who
were
in
brazil,
it
might
be
interesting.
I
was
thinking
to
have
a
continual
process
where
you
can
bound
the
the
the
I
guess,
the
sort
of
out
of
date.
A
A
A
No
so
I
didn't
set
up
miso
yarn,
you
can
use
spark
without
miso
or
yarn
if
you've
only
got
one
job.
So
my
one
job
is
my
shark
client,
but
but
then
then
what
it
does
is
it
just
grabs
all
the
resources
and
if
you
try
and
run
another
one,
it
just
waits
until
you
finish,
whereas
if
you
have
measles
or
yarn,
then
it
will
say
you
know
oh
you're,
not
using
this
at
the
moment.
Someone
else
can
use
it
and
would
like
schedule
between
them.
So
I
didn't.
A
Oh
compactions
are
ongoing,
so
what
what
I
basically
do
is
is
is
start
and
get
a
reference
to
to
the
files.
You
know
open
the
files
so
that
I
can
read
them
so
that
there's
some
sort
of
race
conditions
where
stuff
may
or
may
not
exist.
So
I
just
if
I
get
an
error
I
retry
so
yeah.
So
so
when
there's
ongoing
compactions,
you
will
find
some
stuff
gets
deleted,
but
I
I
think
I
think
I've
worked
around
that
so
it
so
it
works.
A
Yeah,
I
didn't,
I
didn't
run
hive,
so
I
guess
the
first
stage
of
hive
would
be
to
read
through
all
the
data
which
would
be
fairly
similar.
I
mean
the
reason
why
shark
is
better
is
because
we're
not
writing
stuff
out
to
hdfs.
So
I
didn't
you
know
I
didn't
have.
If
I'd
done
that,
then
I
would
have
had
data
nodes
running
on
my
sonic
cluster
as
well
and
loads
of
other
stuff,
or
maybe
I
had
a
separate
hdfs
cluster.
B
A
Yes,
yes,
so
so
for
my
case
with
60
megs
of
ram
that
doesn't
really
matter,
but
it
means
that
you
can.
You
know
I
had
one
gig
of
memory
allocated,
so
I
could
actually
have
cached
four
gigs
of
data
and
run
it
very
quickly
on
multiple
you
could.
But
you
would
have
to
set
up
mesosoian
to
be
able
to
do
that
and
then
it
would
schedule
between
them.
A
B
A
I
I'm
not
sure
I
think
I'm
certainly
preferably
local
reads
like
hadoop.
Does,
I'm
not
sure
if
you
can
say
only
do
local
reads.
I
mean
in
in
my
case,
if
you
try
and
do
a
non-local
read,
it
will
throw
an
exception
and
say
can't
do
it.
So
I'm
not
sure
if
you
can
force
spark
to
say
look.
You've
just
got
to
wait
until
a
node
with
the
with
the
data
is,
is
up
or
all
free.