►
Description
Speakers: DeWayne Filppi, Technical Account Manager at GigaSpaces
Slides: http://www.slideshare.net/planetcassandra/c-summit-2013-realtime-big-data-with-storm-cassandra-and-inmemory-computing-by-dewayne-filppi
This session will describe how to resolve the processing limitations by placing the streaming and data store interfaces in-memory as well, through an in-memory computing platform, and also how to resolve the complexity challenge by implementing a DevOps approach that abstracts all the underlying infrastructure and provides single-click management of all the application tiers and services, on any environment (private/public cloud, bare metal...). And the best news is that all this optimization can be implemented seamlessly, with no code change to your apps.
A
It's
a
major
player
in
the
in-memory
compute
space,
I'm
here
to
really
sort
of
rationalize
or
convince
people
that
combining
in-memory
compute
technology
with
cassandra
actually
makes
sense
for
certain
use
cases.
It's
you
know.
Obviously
it's
under
the
sort
of
the
banner
of
extremely
high
performance,
low
latency,
and
that's
that
means
different
things
to
different
people.
A
A
A
A
So
this
slide
is
meant
to
communicate
the
different
aspects
of
why
you
might
need
real-time
processing,
real-time
in
the
sense
of
now
real
time,
in
the
sense
of
real-time
events,
not
just
polling
a
table
or
running
pretty
well,
pretty
performant
queries
against
a
static
data
set.
But
how
can
I
be
notified
when
certain
things
happen?
How
can
I
do
real-time
aggregations,
real-time
counts
and
so
forth?
And,
of
course,
all
these
come
up
all
these
companies
use
in-memory
compute
in
order
to
enhance
their
performance.
A
So
I
think
we
can
break
down
big
data
analytics,
which
is
what
I'm
going
to
be
calling
the
the
aggregations,
the
counts
and
so
forth.
So
let's
just
call
them
analytics,
so
you
can
break
them
down
roughly
into
counting
correlating
in
research
and
these
each
of
these
actually
maps
to
a
different
level
of
real
time,
typically
in
going
left
to
right
in
a
decreasing
order
of
real
time,
for
example
it.
Let's
take
twitter
example.
A
It
doesn't
really
matter.
But
the
point
is
that
it's
not
it's
probably
not
terribly
computationally
expensive
and
it
needs
to
happen
instantaneously.
The
data
is
in
flight
typically
to
another
system,
to
cassandra
to
a
system.
That's
doing
long-term
storage.
Doing
you
know
vast
accumulation
of
data,
so
you
don't
have
a
lot
of
time
for
processing
correlating
what
devices
fail.
At
the
same
time,
what
features
gets
users
hooked?
These
are
higher
level
questions
so
now
you're,
rather
than
just
asking
what
you're
starting
to
ask.
A
Why
and
and
and
correlating
different
events,
probably
different
aggregations
against
each
other
across
time.
This
takes
more
time.
This
is
something
that's
typically
not
happening
at
the
scale
that
I'm
addressing
here
and
then
the
final
level
is
research,
and
this
would
equate
roughly
to
a
hadoop
job.
So
we're
talking
about
something
that
could
take.
You
know
many
minutes
to
to
complete.
A
A
A
So
again,
for
for
the
lowest
possible
latencies
you're
talking
about
event
driven
stream
processing.
By
definition,
you
don't
have
time
to
pull
a
static
data
source
or
even
a
static
data
source
living
in
memory
and
it's
extremely
high
resolution,
so
we're
counting
every
tweet
and
we'll
get
into
what
what
that
means.
As
we
go
forward
here,
ad
hoc
querying.
A
This
would
be
a
typical
application
of
a
cassandra
type,
a
big
data,
nosql
database,
doing
ad-hoc
querying
more
like
conventional
database
type
operations
and
then,
of
course,
the
long-running
batch
jobs,
extract,
transfer,
load
map,
reduce
and
low
resolution.
Now,
you're
kind
of
zooming
out
from
the
data
and
you're
finding
patterns
and
trends
and
so
forth.
A
A
So
the
data
the
data
is
partitioned
across
the
cluster.
This
is
not
in
the
same
sense
that
a
nosql
cluster
is
partitioned.
We're
not
talking
about
eventually
consistent
storage.
Here,
it's
more
akin
to
an
in-memory
database,
to
a
certain
extent
that
I
mean
you
could
view
it
as
an
in-memory
distributed
object-oriented
database.
A
A
Typically,
the
grids
in
a
grid
computing
system
like
ours
is
actively
managed
by
management
processes
and
agents
that
are
running
on
the
nodes,
with
the
containers
for
memory,
so
they're,
tolerant
of
failure,
the
the
system,
notices,
failures
of
components
restarts
them
as
needed
so
that
the
the
client
gets
a
continuous
view
of
a
what
amounts
to
a
large
memory
space.
Far
larger
than
a
single
node
can
hold
up
into
the
terabyte
ranges,
not
hundreds
or
thousands
of
terabytes
like
cassandra,
but
low
terabytes
and
the
other.
A
A
A
A
If
you're
doing
you
know
millions
of
millions
of
messages,
a
second
on
a
single
node,
then
you're
going
to
have
to
be
not
doing
much
in
your
in
your
aggregator.
But
you
could
do
you
know
much
more
in
depth,
processing
side
effects
are
calculated
and
since
they're
calculated
in
memory
in
the
grid,
they
they're
persisted
in
the
grid
and
if
you
recall,
the
grid
is
akin
to
an
in-memory.
You
know
it's
a
purse,
it's
an
in-memory
persistence,
so
the
data
lives
there.
It's
highly
available,
it's
reliable.
You
can
query
it
from
the
outside.
A
Okay,
let's
move
on,
so
I
guess
just
to
clarify
that
events
come
in
in
the
event,
if
we
take
say
a
click
stream
is
a
is
an
example
of
an
event,
something
perhaps
coming
from
a
weblog,
a
flumes
output,
something
like
this
flows
into
the
in
memory
grid.
Each
one
of
these
events
only
exists
in
order
to
aggregate
data.
The
aggregator
is
the
goal.
Once
the
event
has
been
processed,
the
data
is
evicted,
but
simultaneously
is
asynchronously
replicated
to
cassandra
for
doing
the
normal
cassandra
type
queries
you
might
want
to
do
that.
A
A
A
You
could
have
events
flowing
say
from
a
customer
for
an
example
and
have
have
your
database
of
customers
residing
in
the
grid
and
the
stream
flying
past
it
every
time
an
event
hits
it.
You
go
out,
run
a
query
on
that
on
that
customer
table
enrich
the
data
and
then
right
and
then
it
gets
written
automatically
asynchronously
to
cassandra.
In
the
background.
A
A
We
use
sql
for
the
read
side,
and
this
is
really
not
a
topic
for
this
particular
this
or
a
discussion
for
this
topic,
because
we're
talking
mainly
on
a
right
basis,
but
we
also
can
sit
in
front
of
cassandra
and
do
read-through
type.
Caching,
in
which
case
we
use
sql
in
order
to
pass
queries
through
to
the
underlying
cassandra
engine.
A
A
A
So
what
are
the
virtues
and
limitations
because
we're
going
to
get
to
storm
here?
It's
it's
nice,
because
it's
a
complete
stack
like
I
said
it's
got
the
it's
got
the
real-time
persistence,
the
the
super,
low,
latency
and
so
forth,
and
it
also
gives
you
transactions
and
all
this
good
stuff,
and
it
sits
on
top
of
cassandra,
it's
elastic
and
so
forth,
but
it
could
be
faster.
Okay,
high
availability
is
not
free.
A
I've
discussed
that
already
a
an
in-memory
data
grid
like
this
will
never
be
able
to
compete
performance
wise
with
storm
in
terms
of
raw
numbers.
As
far
as
in-memory
processing
there's
no
way
for
that
to
occur
because
storm
does
not
storm
does
not
bear
the
cost
of
persistence.
It
pushes
persistence
out
to
the
edges
essentially
outside
of
the
model.
So
so,
if
you
need
to
go
faster,
you
might
need
to
go
storm
for
for
the
event
processing
complex
event.
Flows
are
not
easy
to
assemble
or
understand
what
we
talk
about
here.
A
A
Storm
does
give
you
those
things:
it's
a
popular
open
source,
real
time
in
memory
streaming,
computation
platform.
It
has
a
distributed
runtime
and
an
intuitive
api.
So
it
runs
on
a
cluster
similar
to
a
in
memory
data
grid.
But
it's
quite
different.
It
doesn't
focus
on
storage,
it's
only
focusing
on
moving
data
through
as
fast
as
possible
and
transforming
it.
A
It's
also
scalable
and
fault-tolerant,
but
in
a
totally
different
way-
and
I
was
developed
a
back
type
open
source
by
twitter
and
I
recommend,
if
you're
interested
in
it,
definitely
go
check
out
stormproject.net,
because
it's
fascinating
project,
it's
got
the
concept
of
streams.
So
these
are.
These
are
abstractions
built
on
actual
streaming
model,
which
is
absent
from
a
generic
in-memory
data
data
grid
processing
system.
So
it's
got
concept
of
a
stream
which
you
define,
which
is
an
unbounded
sequel
to
sequence
of
tuples.
A
So
it's
a
simple
data
model
spouts,
which
are
essentially
the
sources
of
cues
and
then
bolts,
which
are
where
the
actual
work
is
done.
If
there's
any
kind
of
aggregations
being
produced,
filters
joins
aggregations
and
then
there's
topologies
and
all
the
topologies
are
a
container
for
these
things.
You
create
workflows
based
on
these
concepts,
putting
them
together
and
here's
an
example
of
a
topology
that's
been
put
together.
A
You
see
a
spout,
there's
a
there's
a
in
this
case.
It
says
zap
trident
spout
which
I'll
get
to,
but
that's
sort
of
selling
my
own
spout,
which
connects
to
xap
but
and
you
notice
how
it
connects
if
you're
familiar
with
cascading.
It
has
a
similar
type
api,
but
it
uses
a
builder
type
interface
in
order
to
connect
together
the
different
steps
in
a
really
a
stream
processing
workflow.
A
So
it's
not
just
getting
an
event
and
counting
it
or
something
it's
actually
routing
data
through
to
a
through
various
processing
steps
and
each
one
of
these
will
create
a
bolt
behind
the
scenes
and
it's
kind
of
hidden
from
you
and
you
can
give
a
parallelism
hints
to
tell
how
much
you
want
to
fan
out
processing
on
a
certain
level,
depending
on
the
computational
cost
of
it
and
so
forth.
And
then
you
just
submit
it.
A
It's
got
a
it
like,
I
said
earlier:
it
delegates
persistence
to
external
providers.
It
assumes
that
the
source
of
tuples
can
replay
them.
I
mean
for
the
most
part.
You
can't
have
unreliable
sources
of
tuples,
but
it
by
and
large
for
the
maximum
performance.
It
has
to
be
able
to
replay
tuples
and
cassandra.
A
If
we,
if
we
take
cassandra
as
an
example
of
a
persistence,
imagine
you're
doing
a
an
aggregation
and
before
I
would
talk
about
doing
an
aggregation
in
memory,
if
you
have
a
bolt
that
is,
is
aggregating
the
stream
as
it's
flowing
past,
it
has
to
write
into
some.
It
has
to
write
somewhere.
If
you
want
to
persist
it.
So
that's
going
to
go
to
a
data
store
to
cassandra.
A
A
So
storm-
and
this
is
just
to
point
out
some
of
the
differences
storm
is
optimistic
processing-
is
what
I'm
calling
it,
but
as
opposed
to
a
data
grid
which
is
really
pessimistic.
So
storm
assumes
that
tuples
in
flight
are
going
to
succeed
and
reach
their
end
state
and
their
end
state
is
defined
as
all
the
bolts
have
successfully
processed
that
tuple.
If
there's
any
kind
of
a
failure,
it
doesn't
restart
in
the
middle
or
something
it
doesn't
have
any.
It
doesn't
maintain
a
state.
A
In
that
sense,
it
will
command
the
spout
to
re-emit
the
tuples
so
to
get
high
performance
so
that
that
that
alone,
since
it's
not
attempting
to
to
persist,
you
know
continuously
that
that's
speeds
it
up
and
streamlines
it.
It
uses
batching
and
pipelining
for
performance,
so
a
the
highest
performing
spouts
will
emit
batches
of
tuples
together
and
multiple
batches
can
be
in
flight
simultaneously
through
the
system,
and
if
one
of
them
fails,
then
the
spout
has
to
be
able
to
replay
it,
which
means
the
spout
has
to,
of
course
persist.
A
A
A
A
Okay,
so
that's
really
fast!
It's
gonna
go
as
fast.
It's
gonna.
It's
certainly
gonna
go
as
fast
as
cassandra
can
go,
it'll
go
as
fast
as
you
want
it
to
go,
but
if
you
wanted
to
go
faster,
you
need
to
eliminate
the
non-memory
components.
So
when
you
start
thinking
of
ram
as
a
persistence
medium,
then
when
you
see
something
like
any
kind
of
a
architecture
that
actually
winds
up
writing
to
disk,
you
start
thinking.
A
We
synchronously
update
the
state
in
xap
here
rather
than
cassandra,
so,
instead
of
having
the
limiting
factor,
be
the
speed
of
which
cassandra
can
write
now,
the
limiting
factor
is
the
speed
at
which
xap
can
write,
which
we
have
already
discussed,
is
200
300
microseconds,
something
like
that.
The
and
then
zap
takes
care
or
the
grid
takes
care
of
updating
cassandra
asynchronously
behind
the
scenes
of
everything
that's
going
on,
including
by
the
way,
not
just
the
original
events,
but
anything
else
you'd
want
to
persist
in
cassandra
as
well.
A
How
are
we
doing?
I
think,
we're
getting
near
the
end
here,
so
a
sample,
here's
a
sample
architecture.
That's
got
everything
everything
under
the
sun
in
here
you
can
have
cassandra
serving
clients
up
here.
We
see
hector
sequel,
astonix
clients
here,
not
everything
needs
in-memory,
processing
right.
So
there's
no
reason,
there's
no
reason
to
pretend
that
everything
has
to
go
through
this
architecture.
So
there's
no
reason
why
everything
every
this
might
be
80
90
of
traffic
goes
through
this,
who
knows
events
and
queries
and
streams
can
go
into
xap.
A
Alternatively,
streams
and
queries
can
also
go
into
storm
because
I
didn't
mention
that
storm
also
will
serve
distributed.
Remote
procedure
calls
as
well,
but
it
doesn't
provide
persistence
in
memory
like
a
data
grid
does
so
the
streams
can
enter
into
xap
streams.
Those
streams
can
feed
storm
do
in
memory,
processing
the
results
of
state
and
other
events
that
you
want
to
happen
can
go
into
xaps
to
the
data
grid.
A
The
data
grid
can
then
actually
use
its
eventing
capability
to
notify
external
clients,
any
other
listeners
that
are
attached
to
the
to
whatever
is
occurring
in
memory.
These
queries
up
here
going
into
the
data
grid,
can
be
their
sql
based
sort
of
hibernate
like
queries
that
you
can
run
on
on
aggregators
and
objects
in
memory
and
then
asynchronously
along
this
path.
A
A
Some
components
in
order
to
facilitate
streaming
the
source
is
available,
the
the
spout
for
xap.
Is
there
it's
fully?
It's
a
transactional
spout
it's
batch.
It
can
replay
batches,
batch,
oriented
and
so
forth
and
a
more
background
on
the
effort.
I've
written
a
couple
blog
posts
and
I'm
part
three-
is
coming
soon,
I'm
a
little
bit
behind
on
my
blog
posts,
but
basically
how
this
whole
thing
was
built
and
put
together.
So
that's
it.