►
Description
Speaker: Robbie Strickland, Director of Software Development at The Weather Channel
The state of analytics has changed dramatically over the last few years. Hadoop is now commonplace, and the ecosystem has evolved to include new tools such as Spark, Shark, and Drill, that live alongside the old MapReduce-based standards. It can be difficult to keep up with the pace of change, and newcomers are left with a dizzying variety of seemingly similar choices. This is compounded by the number of possible deployment permutations, which can cause all but the most determined to simply stick with the tried and true. But there are serious advantages to many of the new tools, and this presentation will give an analysis of the current state--including pros and cons as well as what's needed to bootstrap and operate the various options.
A
So
thank
you
for
joining
my
talk
called
the
new
analytics
tool
box,
the
plan
for
today
and
if
you're
not
here
for
the
new
analytics
tool
box,
then
you
should
probably
find
the
other
room.
The
plan
for
today
is
to
talk
about
the
state
of
big
data
analysis
and
and
how
that's
changed
over
the
last
couple
of
years.
A
A
My
name
is
Robbie
Strickland
I'm,
not
British,
I'm,
still
getting
used
to
the
the
feeling
that
I'm
saying
everything
wrong
so
you'll
have
to
to
forgive
me
I
work
on
the
platform
team
at
the
weather
channel,
which
is
maybe
not
as
well
known
here
as
it
is
in
the
US,
but
the
reality
is
most
of
you
use
our
data
every
day.
If
you
use
things
like
Google
now
and
iOS
and
Yahoo
and
most
of
the
the
data
originates
with
us.
So
that's
that's.
A
What
I
do
that's
the
team
that
I
managed
we
we
power,
weather.com,
the
TWC
mobile
apps
and
most
of
like
I,
said
the
the
weather
data
that
exists
in
the
world
today.
So
the
first
question
you
might
ask
is
what's
wrong
with
who
do
so
by
a
show
of
hands.
Let
me
see
who
is
working
with
Hadoop
today,
so
just
a
small
number
of
people,
maybe
that's
good,
so
who's
working
who's,
running
Cassandra
in
production,
okay
and
who
is
using
or
planning
to
use,
spark
okay,
interesting
all
right.
A
Well,
so
hopefully
this
won't
be
too
elementary,
we'll
see
so
what's
wrong
with
the
do,
it
works.
Fine
right,
you've
invested
all
this
time
and
money
into
it.
So
you
know:
what's
the
big
deal
well
for
starters,
it's
11
years
old,
which
is
very
hard
to
believe
some
of
you,
the
Hadoop
people
in
the
room
might
be
thinking
well,
I,
guess
not
that
do
people
the
people
who
didn't
raise
their
hands
for
either
might
be
thinking
that
you're
late
to
the
game.
A
But
I
would
say
to
that
that
this
is
actually
good
news
for
you,
because
you're
unencumbered
by
your
existing
technology
investment
like
those
who
do
people
in
the
room
so
and
while
it
was
groundbreaking
at
its
introduction,
it's
actually
pretty
slow
and
inefficient
by
modern
standards,
and
this
is
primarily
because
everything
gets
written
to
disk,
including
all
the
interim
steps,
and
in
many
cases
you
actually
need
a
chain
of
jobs.
To
perform
your
analysis,
so
you
can't
just
one
single
MapReduce
job
isn't
going
to
do
the
job.
A
You
need
multiple
chain
if
you've
actually
written
any
MapReduce
code,
then
you
know
it's
not
the
most
intuitive
API
we
sort
of
put
up
with
it
because
it's
been
what
has
been
there
to
deal
with.
It's
easy
to
get
things
wrong
and
it's
pretty
hard
to
test,
and
actually
the
cost
of
getting
it
wrong
can
be
very
high,
which
is
kind
of
a
bad
combination.
A
You've
also
seen
that
it
requires
quite
a
bit
of
code
to
perform
even
the
simplest
of
tasks,
and
this
is
largely
a
result
of
the
significant
amount
of
boilerplate
code,
and
maybe
the
fact
that
it's
in
Java
this
is
also
improved.
This
is
somewhat
improved
by
tools
like
cascading,
pig,
hives
and
those
sorts
of
things.
A
But
to
me,
that's
really
just
more
evidence
that
the
core
API
is
inherently
burdensome,
and
if
that
weren't
enough,
it
can
be
difficult
just
to
get
the
thing
working,
I,
don't
know
if
you've
tried
to
do
this
to
roll
your
own,
but
it
doesn't
really
do
anything
out
of
the
box.
It's
like
a
box
of
tools
that
you
don't
know
how
to
use.
You
have
to
do
a
lot.
A
A
Fortunately,
we
are
seeing
a
new
crop
of
tools
designed
to
solve
many
of
these
problems.
This
is
a
partial
list
of
some
of
the
more
popular
choices
you
may
have
seen,
but
the
hadoo
key
Hadoop
ecosystem
is
already
suffering
from
an
overabundance
of
Bolton
components,
which
is
part
of
the
problem.
So
it's
easy
for
the
really
revolutionary
stuff
to
kind
of
get
lost
in
the
shuffle.
So
let's
quickly
break
some
of
these
options
down
to
see
what
they
offer.
A
First,
we
have
cloud
errors,
Impala
and
apache
drill,
which
are
both
massively
parallel:
query,
processing
tools,
they're
designed
to
run
against
your
existing
Hadoop
data.
Basically,
they
give
you
fast,
SQL
style,
queries
for
HDFS
and
HBase,
with
the
theoretical
possibility
of
integrating
with
other
input
formats.
Although
I'm
actually
not
aware
of
anyone
having
done
this
successfully
with
cassandra,
maybe
someone
has,
but
I'm
not
aware
of
it,
and
then
we
have
other
proprietary
vendors
that
offer
a
variety
of
analysis
tools
such
as
spunky
and
I.
A
So
if
you've
used
hide,
then
you're
familiar
with
the
construct,
it
uses
your
existing
hive
meta
store,
but
it
executes
the
queries
on
your
spark
cluster.
However,
at
spark
summit
a
few
months
ago,
it
was
announced
that
shark
will
be
ultimately
replaced
by
spark
SQL.
So
it's
effectively
deprecated
so
which
tool
or
tools
should
you
ultimately
choose?
Well?
Actually
the
good
thing
is
you
don't
have
to
choose
one,
because
most
of
the
major
Hadoop
vendors
have
embraced
spark,
as
has
datastax.
A
So
this
actually
makes
it
an
ideal
Hadoop
replacement.
So
and
if
you
haven't
started
using
Hadoop
at
this
point,
it
may
make
sense
for
you
to
simply
redirect
your
efforts
towards
spark.
Instead,
there's
been
a
lot
of
investment
in
Hadoop
connectors
as
well,
and
if
you
know
they're
called
input
formats,
and
these
can
all
be
leveraged
in
spark.
So
you
can
still
use
your
Hadoop
connector,
which
I'm
sure
we're
all
employing
inside
spark,
even
though
it
was
designed
for
a
Duke
and
even
though
hadoo,
even
though
it
makes
use
of
the
Hadoop
components.
A
You
may
call
it.
You
know
something
you
have
to
learn,
but
I
think
it's
fantastic
and
it's
arguably
better
suited
for
data
analysis
than
Hadoop's,
MapReduce,
API
and
certainly
better
suited
than
than
Java
ahem.
So
I'll
show
you
the
difference
in
a
minute
and
since
I
assume
you're
all
interested
in
how
well
this
works
with
Cassandra.
The
good
news
is,
you,
don't
even
have
to
use
the
Hadoop
input
format
like
we
early
adopters
had
to
use,
because
datastax
has
built
an
excellent
direct
spark
driver
and
open
source.
A
I
will
add
so
I
assume
since
many
we,
some
of
you
raised
your
hands,
saying
you're
using
Hadoop
now
so
or
you're
planning
to
use
it
soon.
Let's
quickly,
look
compared
it
with
spark
for
those
who
are
unfamiliar
with
what
it
looks
like
to
write
MapReduce
and
let's
not
jump
ahead
for
those
unfamiliar
with
what
it
looks
like
to
write,
MapReduce
and
maybe
to
cause
a
little
post-traumatic
stress
episode
for
those
who
do
have
experience.
A
Let's
look
at
a
simple
MapReduce
job,
so
the
use
case
for
this
job
is
our
social
weather
feature
in
the
weather
channel
app
and
basically,
what
this
allows
you
to
do
is
a
user.
You
can
report
that
you
disagree
with
our
current
your
art.
What
we
report
is
weather
conditions
or
simply
agree
that
and
just
because
you
want
to
click
on
it
and
it
reports
your
current
weather
conditions.
The
way
you
see
them,
this
data
is
stored
in
Cassandra
and
based
the
basic
idea.
A
Is
we
want
to
create
a
simple
histogram
of
check-ins
by
hour,
which
is
nothing
more
than
just
bucketing
them
by
a
rounded
timestamp?
The
first
thing,
you'll
notice,
is
all
the
imports.
That's
a
lot
of
imports
for
a
rudimentary
job
right.
We're
not
really
doing
a
whole
lot
here.
There's
no
third-party
api's
we're
bringing
in
it's
just
the
basic
importance,
there's
a
lot
of
stuff
going
on
there,
and
then
we,
you
know,
start
with
our
our
configuration
here.
A
So
we
continue
start
configuring
the
job
starting
by
parsing,
some
command
line
arguments,
and
then
we
finish
off
the
config
with
another
50
lines
or
so
short
50
lines
of
boilerplate
code
there.
That
are,
that
that
we
see
in
every
single
MapReduce
job
after
that,
we'll
have
to
create
a
mapper
which
groups
all
the
input
keys
by
our
and
I
won't
bother
to
to.
A
Let
you
linger
too
much
on
this,
but
you
can,
if
you're
familiar
with
it,
you'll
understand
if
you're
not
you'll,
get
the
gist
of
it
and
then,
of
course,
you
have
to
have
a
combiner
or
who
knows
you
know
if
you've
not
written
a
combiner
in
something
like
this.
You
know
that
the
job
will
take
forever.
You
write
a
combiner
and
cuts
the
time
down
exponentially,
so
we
have
to
do
that.
A
This
is
a
this
essentially
pre
aggregates
the
counts
before
we
ship
it
off
to
the
shuffle
and
finally,
we
aggregate
all
the
hours
and
add
up
the
counts
in
the
reducer
and
then
output
them
to
Cassandra
130
lines
of
code.
Later
we
have
grouped
check-ins
by
hour
and
in
case
you
weren't
counting
only
five
lines,
some
of
which
had
to
be
repeated
for
the
combiner
of
the
130
were
actually
devoted
to
creating
the
histogram.
The
rest
of
it
was
just
Hadoop
cruft
and
now
spark.
A
Let's
have
a
moment
of
silence
as
we
mourn
the
death
of
MapReduce
okay
morning
over.
As
you
can
see,
we've
reduced
our
code
by
a
hundred
and
seven
lines.
We
have
to
imports
two
lines
to
configure
the
job,
then
we're
right
into
our
logic.
There
are
no
byte
buffers
to
parse
and
the
logic
is
easy
to
understand
so
now
that
everyone
wants
to
use
it.
What
is
it
exactly
so
spark
is
actually
an
in-memory
cluster
computing
framework.
As
I
said
it's
built
in
Scala,
which
means
it
runs
on
the
JVM.
A
A
It
has
the
native
language
binding
for
Scala,
Python
and
Java,
and
there's
a
separate
project
called
spark
r,
which
is
designed
to
enable
our
on
spark,
and
this
is
actually
a
fantastic
way
to
let
your
data
scientists
play
with
spark
if
you're
using
RR
is
really
really
good
for
data
exploration.
It
offers
stream
processing
capabilities
using
the
same
collections
API.
A
So
you
only
have
to
learn
the
one
paradigm
as
opposed
to
having
to
learn
to
write
MapReduce
code
or
pin
code
or
hive
code
and
then
turn
around
and
learn
how
to
write,
storm
topologies
and
all
those
sorts
of
things.
You've
got
one
API
to
rule
them
all
and,
as
I
mentioned
earlier,
it
also
supports
any
existing
Hadoop
connectors.
A
So
you
can
use
spark
also
offers
some
interesting
additional
features,
including
a
native
graph
processing
library
called
graphics
and
as
well
as
a
machine
learning
library
called
ml
lib,
which
is,
if
any
of
you
have
used
to
put
mahout.
It's
really
what
mahout
always
wanted
to
be
on
Hadoop,
but
never
was
and
spark
SQL
gives
you
the
ability
to
run
SQL
queries
over
any
spark
collection.
A
So
there's
no
additional
bolt-on
or
query
built
on
tool
or
query
framework,
that's
required
and
if
you're
a
fan
of
elastic
MapReduce
you'll
be
happy
to
know
that
you
can
enable
spark
out
of
the
box
on
a
DMR
cluster
as
well.
So
I
will
say
they're,
usually
a
version
or
so
behind.
But
you
know,
if
you're
not
into
bleeding
edge,
then
it
works.
Fine.
One
of
my
favorite
features
of
spark
is
the
ability
to
join
datasets
across
multiple
disparate
data
sources,
which
we
do
have
at
The,
Weather
Channel
imagine
doing
your
joins
across
Cassandra
react.
A
A
So
datastax,
as
I
said,
is
recently
open
sourced
a
fantastic
spark
driver,
that's
available
on
their
github,
and
it
gets
rid
of
all
the
job.
Config
cruft
that
early
adopters
of
spark
on
Cassandra
had
to
deal
with
and
there's
actually
no
longer
a
need
to
go
through
the
spark
Hadoop
API,
which
was
pretty
annoying.
A
And
but
this
is
sort
of
a
low
overhead
usage
personally
I
find
it
useful
to
go
ahead
and
have
HDFS
running
so
that
I
can
throw
unstructured
data
into
my
analysis
cluster
alongside
the
cassandra
data.
So
what
does
this
all
look
like
on
a
physical
cluster?
A
lot
of
people
get
this
wrong,
which
is
why
I
throw
this
in
so
let's,
let's
look
at
what
an
analysis.
Data
center
looks
like
running
cassandra
with
hadoop,
because
a
lot
of
people
are
doing
this
already.
A
You
can
see
that
in
the
canonical
setup
that
you
have
your
Hadoop
master
with
the
name
node
the
secondary
name,
node
and
the
jobtracker,
then
you
have
your
cassandra
ring
with
co-located
data
nodes
and
task
trackers.
This
is
this:
is
the
canonical
set
up
so
that
this
allows
the
local
Hadoop
task
trackers
to
understand
where
your
data
is
and
pull
it
from
the
local
node
spark
actually
looks
very
very
similar
to
this.
A
A
The
spark
application
will
then
load
data
from
a
data
source
and
serialize
it
into
a
spark.
Our
DD
or
resilient
distributed
data
set
will
talk
about
what
that
is
in
a
minute.
A
next
will
perform
a
series
of
operations
on
the
data.
These
operations
come
in
the
form
of
transformations
and
actions
and
finally,
we'll
save
or
display
the
resulting
data
or
calculation.
A
So
what
is
an
RDD?
Well,
it's
a
distributed
collection
of
items.
That's
partition
the
clock
across
the
cluster.
You
can
sort
of!
Imagine
it
like
the
way.
If
you
write
a
file
into
the
dupe,
it
gets
blocked,
it's
split
into
blocks
and
written
across
the
cluster.
It's
similar
a
similar
sort
of
concept
in
a
resilient
distributed
data
set.
A
It
has
an
API,
that's
very
similar
to
that
of
the
Scala
collections
API,
as
I
showed
you
earlier.
It
allows
the
transformations
and
actions
operations
on
the
data
and
all
the
operations
are
actually
performed
lazily
across
the
cluster,
so
they're
not
actually
executed
until
you
have
to
materialize.
The
data
set.
A
Our
db's
are
also
the
key
component
to
spark
fault
tolerance
because
they
can
recalculate
values
from
any
step
in
the
job
in
case
of
a
worker,
slow
down
or
node
failure
and
spark
does
a
very
good
job.
Keeping
track
of
that
spark
will
automatically
detect
the
failure
and
redistribute
the
work
to
other
nodes
in
the
cluster.
A
So
let's
look
at
transformations
versus
actions,
transformations
produce
new,
our
db's
and
that's
actually,
if
you're
looking
at
the
API,
how
you
can
tell
the
difference.
If,
if
the
result,
the
return
value
is
an
RDD,
you
know
it's
a
transformation
on
by
contrast,
actions
actually
require
full
materialization
of
the
records
to
produce
the
value.
This
is
obviously
more
costly
than
the
lazy
lazy
transformations,
and
you
can
tell
the
difference
with
these
because
they
produce
use
depending
on
which,
which
language
binding
you're,
using
they
produce
a
language,
specific
collection.
A
So
some
examples
of
transformation
include
things
like
filter
or
map.
Flatmap
collect
those
sorts
of
operations
and
you'll
recognize
those
if
you're
familiar
with
scala
or
other
functional
type
paradigms,
and
then
actions
by
contrast
are
things
that,
like
count
and
fold
and
reduce
which
actually
require
you
to
sort
of
ship
data
around
the
cluster.
A
So
here's
an
example
of
a
transformation
at
an
action.
In
this
case
we
apply
a
filter
to
produce
an
RDD
of
persons
above
the
age
of
17.
Next,
we
materialized
the
rdd
to
obtain
account.
So
the
first
part
the
filter
is
a
transformation.
The
second
part,
the
count
is
an
action
and
here's
another
example
of
more
detailed
looking
at
code
here
of
how
we
might
how
Cassandra
and
spark
actually
integrate
using
the
spark
connector
driver.
A
Next
we
filter
all
persons
over
the
age
of
17
and
then
finally
save
the
rdd
directly
back
to
Cassandra,
using
the
save
to
Cassandra
method.
So
we
actually,
in
this
case,
never
do
never
materialized
anything
until
we
write
it
to
Cassandra.
So
there's
no
actions
here
for
those
new
to
sparking
Cassandra.
This
is
quite
powerful
and
skips
a
lot
of
the
steps
previously
required
through
the
older
drivers,
and
here
is
an
example
similar
to
the
one
we
just
showed,
except
that
we
use
spark
SQL
to
filter
the
person's
rdd
and
like
before.
A
We
load
the
data
from
Cassandra
into
a
spark
our
DD
first,
and
then
we
immediately
use
the
register
as
table
method
to
create
a
table
named
persons.
This
will
allow
us
to
issue
SQL
queries
against
the
person,
our
DD,
we
filter
out
all
persons
of
us
17
using
SQL
and
then
print
the
results,
and
so
it's
worth
mentioning
that
this
is
not
Cassandra
specific.
This
actually
works
against
any
RTD.
A
So
while
this
isn't
directly
related
to
kiss
the
Cassandra
integration,
let's,
let's
look
at
sparks
training
as
well,
because
it's
one
of
its
key
points
of
differentiation
with
it
with
the
spark
streaming
context.
We
can
now
process
streams
just
like
we
can
any
other
data,
spark
slices
up
the
stream
and
produces
our
db's
on
a
defined
interval.
A
So
it
supports
a
variety
of
sources.
Things
like
Kafka
cues,
a
twitter
which
twitter
is
mostly
just
for
playing
around
you
know.
Obviously
there
are
some
people
who
might
use
that
for
your
real
analysis,
but
as
mostly
playing
around
things
like
flume,
if
you're
doing
log
aggregation,
you
know
you
can
send
your
your
data
to
spark
instead
in
this
example,
here
we
want
to
filter
tweets,
we
think
may
be
happy
tweets,
and
so
how
do
we
define
happy
word?
How
do
we
define
happy
tweets?
A
We
set
a
create
a
set
of
happy
words,
and
then
we
use
to
it
the
Twitter
utils
library,
which
is
part
of
the
spark
streaming
to
obtain
the
stream
of
tweets
and
begin
to
produce
a
d
stream
of
tweet
texts.
We
then
define
a
method
that
searches
the
text
looking
forwards
in
the
happy
words
set,
and
then
we
use
that
method
to
filter
tweets,
we
believe,
are
happy
and
print
them
to
the
console.
So
pretty
straightforward
I
mean
from
a
streaming
analysis
perspective.
A
If
you
can
imagine
doing
something
like
this
with
even
a
pretty
modern
API
like
storm,
this
is
substantially
more
straightforward.
So
then
we
get
an
output.
That
looks
something
like
this
I
originally
had
this
talk
as
a
demo,
but
the
35-minute
time
window
doesn't
allow
for
a
demo
and
I'm,
not
I'm,
not
brave
enough
to
do
it
in
front
of
this
many
people.
A
So
we've
seen
some
simple
examples
that
show
how
easy
it
is
to
get
started
with
spark.
But
let's,
let's
talk
about
how
you
structure
a
spark
application
for
real-world
task
distribution,
one
of
the
things
you'll
find
if
you
try
playing
with
spark,
is
that
things
that
you
would
think
work
don't
work
because
you
know
like
just
like
cql
makes
abstracts
a
lot
of
the
things
going
on
underneath
the
hood.
The
same
thing
is
true
here,
whereas
when
we
write
MapReduce,
we
have
to
sort
of
manually,
do
it
and
understand
everything.
A
That's
going
on
spark
abstracts
that
so
there's
some
some
need
to
kind
of
understand.
What's
happening.
For
starters,
rdd
operations
are
similar
to
the
Scala
API,
but
your
choice
of
operations,
in
the
order
in
which
they
are
applied,
is
really
critical
to
important
to
performance.
You
must
organize
your
processes
with
task
and
with
the
task.
Distribution
and
memory
in
mind.
Spark
is
very
memory
hungry.
The
first
thing
is
to
determine
if
your
data
is
partitioned
appropriately.
A
partition
in
this
context
is
merely
a
block
of
data.
A
If
possible,
partition
your
groups,
if
possible,
partition
your
data
before
spark
even
ingests
it.
If
this
is
not
practical
or
possible,
you
may
choose
to
repartition
the
data
immediately
following
the
load.
You
can
choose
to
repartition
to
increase
the
number
of
partitions
or
coalesce
to
reduce
the
number
of
partitions.
The
number
of
partitions
should
is
a
lower
bound,
be
at
least
two
times
the
number
of
cores
that
are
going
to
operate
on
the
data.
A
But,
having
said
that,
you'll
want
to
also
ensure
that
any
task
you
perform
takes
at
least
100
milliseconds
to
justify
the
distribution
across
the
cluster
and
then
the
next
thing-
and
this
is
the
same
thing
as
with
any
really
any
data
processing
pipeline-
you
want
to
filter
early
and
filter.
Often
so
do
things
like
apply
filters,
use
the
where
clause,
two
distinct,
those
sorts
of
things
early,
that
you
reduce
the
amount
of
data
as
you're
processing
and,
lastly,
make
use
of
pipelining.
A
So
here's
a
simple
example
using
a
query
and
subsequent
filtering
to
thin
the
data
set.
First,
we
select
only
the
fields
we
need
to
perform
the
analysis
in
this
case
age
and
email
address.
Next,
we
use
a
filter
to
get
the
over-the-hill
demographic.
Let's
assume
this
is
not
the
only
demographic
we're
going
to
process.
Otherwise
you
would
simply
include
this
filter
in
the
in
the
c
ql
statement.
We
will
then
apply
subsequent
operations
on
the
data
to
complete
the
analysis.
A
So
here
are
some
common
tasks
that
are
costly
and
require
a
shuffle.
So
these
are
the
kinds
of
things
you
want
to
be
careful
about
things
like
shorting
reduced
by
key
shug
and
group
by
key
because
they
require
shifting
data
across
the
cluster.
These
operations
require
the
data
to
be
compared
against
other
data
elements
which
is
expensive,
so
it's
important
to
learn
to
spark
a
pil
to
choose
the
best
combination
of
transformations
and
where
to
position
them
in
your
job.
So
that's
mostly
what
I
had
to
talk
about
today.
A
I
also
have
to
make
sure
I
give
you
a
shameless
book
plug
if
you're
interested
in
deploying
Cassandra
for
high
availability,
which
I'm
sure
we
all
are.
Then
this
is
here's
a
preview.
You
can
check
that
out
coming
out
this
month.
Otherwise,
I
appreciate
your
time
and
look
forward
to
any
questions
you
might
have.
A
So
your
if
I
understand
your
question
you're
saying
if
you
write,
if
you
try
to
run
a
job
that
requires
more
ram
than
you
have
available,
what
will
happen
so
it
kind
of
depends
in
most
cases,
what
we
have
seen
is
that
things
will
start
to
fail
and
it
will
report
back
that
it
is
unable
to
allocate
sufficient
resources
for
the
job.
So
there
there
are
some
things
that
it
can
spill
to
disk
on,
but
you
can't
really
rely
on
that
for
everything.
A
So
if
you're
familiar
with
doing
a
Hadoop
analysis,
the
sort
of
canonical
way
of
handling
this
with
cassandra
is
you,
as
you
use
data
center
replication.
So
you
have
an
analysis
data
center
that
may
be,
it
is
usually
virtual,
so
it's
located
co-located
with
an
existing
physical
data
center,
so
you
have
essentially
a
transactional
cluster,
a
transactional
data
center
and
then
an
analysis.
Data
center.
That's
getting
replicated
data
and
the
one
thing
I
would
say
too
is
be
careful
about
v
nodes,
I,
didn't
say
this,
and
the
talk
would
be
careful
about
V
nodes.
A
A
You're
talking
about
heterogeneous
meeting
like
spark
and
Hadoop
together
yep
we
actually
run.
We
still
have
some
existing
MapReduce
jobs,
and
we
do
that.
So
it's
you
know
you
have
to
manage
resources
a
little
bit
better.
There
are.
There
are
some
tools
like,
for
example,
cloud
eras
tool
purports
to
do
this,
but
I
have
found
it
not
to
be
very
good
at
doing
that,
so
you
really
kind
of,
if
you're
going
to
do
that,
you
have
to
kind
of
manage
it
yourself
right
now,
I'm
actually
very
interested.
A
A
Well,
one
of
them
is
what
I
showed
you,
the
the
histogram,
that's
obviously
a
very
rudimentary
use
case.
Our
primary
use
case
is
event
analysis
so
and
by
event,
you
know,
meaning
a
similar
sort
of
thing
to
the
event
talk
you
might
have
seen
this
morning.
If
you
went
to
that,
but
a
generic,
we
have
a
generic
event
API
that
does
it.
A
And
so
that's
essentially
are
our
primary
use
case,
and
then
we
do
various
sorts
of
analysis
on
those
events.
They
could
be.
You
know
a
user
scrolled
this
far
in
our
app
in
one
of
our
apps
and
clicked
on
this
content,
or
you
know
someone
reported
their
requested
weather
at
this
location.
You
know
all
sorts
of
things
everything
you
can
imagine.
We
capture
anything
else.
A
A
That
that
may
be
true
I
will
I
will
readily
confess
to
being
compared
to
been
using
cassandra
since
point
five,
so
I'm
a
unicorn
in
that
space.
But
I
am
a
I'm
relative
newbie
in
spark,
so
I
will
readily
admit
that,
and
I
will
tell
you
that
our
experience
has
been
that
the
that,
although
they
say
that
spilling
to
disk
sort
of
works,
well
that
in
our
experience
it
doesn't
work
very
well.
So
we're
looking
forward
to
that
working
better.
A
A
Yeah
group
by
group
by
is
is,
is
always
going
to
require
a
data
shuffle,
and
so
you
know,
I
mean
there's
so
many
variables
in
how
fast
you
can
distribute
shuffle
across
the
network
that
that's
there's
going
to
be.
A
lot
is
going
to
be
dependent
on.
You
know
variables
outside
of
spark
itself,
and
that's
why
you
know
I
recommend
that
you
do
those
sorts
of
things
sparingly
and,
as
after
you
have
filtered
as
much
as
you
possibly
can
makes
sense
anything
else.
A
Spark
and/or,
Scala
well,
I
will
say
this
in
terms
those
I'll
answer.
The
scholar
one
first
in
terms
of
Scala
there
is
a
the
the
programming
in
Scala
book
is
actually
really
really
good.
That's
how
I
learn
Scala
four
years
ago,
so
I
highly
recommend
that
you
don't
really
need
to
be
a
heavy
Scala
developer
to
get
started
in
spark
write
it
with
scholars.
In
fact,
spark
is
a
really
good
way.
A
I
would
say
to
kind
of
get
into
the
game,
because
there's
a
lot
of
really
esoteric
kind
of
features
in
scala,
and
I
would
the
only
bit
of
advice
I
would
give
you
in
learning
Scala
is
don't
don't
get
too
worried
about
what
the
zealots
say
on
Stack
Overflow
because
it
can
be.
You
know
you
can
get
some
really
intense
and
you
know
crazy
conversations
going
on
and
just
yeah
I
do
recommend
trying
to
write
it
in
a
right
scholar
in
a
functional
way,
but
at
the
end
of
the
day
it
is
a
functional
language.
A
So
you
know
you
can
wait
in
little
bits
at
a
time
if
you
want
to
as
far
as
learning
spark
the
actually
the
best
resource
that
I
found
so
far
is
the
spark
website.
You
know
it's
there's
not
it's
a
lot
of
experimentation
in
many
ways.
It's
where
Hadoop
was
years
ago,
where
we
were
all
sort
of
trying
to
figure
out
what
the
design
patterns
are.
So
that's
that's
what
I
would
say
anything
else.