►
From YouTube: C* Summit 2013: Large Scale Data Ingestion, Processing and Analysis: Then, Now & Future
Description
Speakers: Ameet Chaubal, Technologist and Fausto Inestroza, Architect at Accenture
Slides: http://www.slideshare.net/planetcassandra/ameet-chaubal/
The presentation aims to highlight the challenges posed by large scale and near real-time data processing problems. In past, such problems were solved using conventional technologies, primarily a database and JMS queue. However these solutions had their limits and presented serious problems in terms of scale and redundancy. The new breed of products - a la Cassandra & Kafka, being innately distributed in their design, aim to tackle such challenges in a very elegant manner. The presentation will showcase some of the use cases of this genre from the industry and describe the solutions which have been increasing in their sophistication.
A
Mr.
presentation
in
30
minutes
allocated
to
us,
so
we're
going
to
be
talking
at
speed
of
Cassandra,
but
if
you
have
any
other
questions,
feel
free
to
come
by
the
experts
room
and
ask
anything
that
you
need.
So
my
name
is
Amy
chobo
and
with
me,
I
have
my
colleague
Fausto
and
we're
from
Accenture
I'm
from
the
Big
Data
group
within
Accenture
and
Foster
works
for
a
little
cooler
sub
group
with
an
eccentric
all
the
labs
where
they
do
rnd
stuff
on
new
technology.
A
So
that's
the
flow
through
which
we
are
going
to
walk
through.
First
of
all,
the
the
problem
which
was
in
for
a
financial
organization.
The
problem,
the
typical
problem
space
consisted
of
ingesting
high
volume
of
data
which
was
coming
from
an
internet
or
e-commerce
transaction
website.
The
objective
was
to
accept
the
data
store
it
in
some
sort
of
data
repository
perform
analysis
in
order
to
answer
questions
and
help.
A
A
So
how
this
is
a
very
typical
use
case?
How
have
people
been
solving
something
like
that?
So
you
have
your
standard,
some
sort
of
a
website
related
data
which
is
generating
your
transactions
or
messages
for
your
application
and
in
order
to
store
the
data
in
some
sort
of
a
database,
people
have
been
inserting
a
JMS
queuing
system
in
the
middle,
which
would
allow
you
to
decouple
the
front
user
facing
website
from
the
back
end
transient
data
storage
system,
and
that
just
allows
you
to
scale
your
infrastructure
in
a
more
balanced
fashion.
A
And
then,
eventually,
you
can
get
the
information
out
into
whatever
analytical
format.
You
want
some
details
about
our
specific
use
case,
so
we
were
dealing
with
about
five
thousand
messages
or
transactions
coming
in
per
second
and
the
transaction
size
around
1,
1
KB
for
each
message-
and
this
was
for
a
very
large
organization.
So,
throughout
the
day
the
messages
were
coming
in
so
we
might
get
into
200
250
300
million
messages
of
one
kilobyte
in
a
day's
time.
A
Obviously
we
had
the
the
queuing
system
and
then
the
data
storage
for
transient
storage
was
done
in
an
informants
traditional
database
and
eventually
the
data
was
extracted
into
a
massive
beta
War
House
hosted
using
terror
data
just
to
add
some
more
color
to
it.
So
what
was
the
problem
with
it?
The
the
main
problem
was
around.
A
The
transient
storage
obviously
was
over
a
period
of
time
not
able
to
keep
up
with
the
volume
that
we
were
talking
about
now
that
problem
percolated
down
stream
and
reverberate
'add
upstream,
as
well,
because
of
the
failure
of
the
system
to
keep
up
with
the
incoming
data
that
translated
into
loss
of
messages
in
the
queuing
system.
The
queueing
replication
was
an
issue.
A
The
other
thing
was
that
the
sort
sort
of
messages
that
were
captured,
which
were
supposed
to
give
us
the
ability
to
ask
questions
and
solve
customer
problems.
It
was
difficult
to
do
those
things
at
the
same
time
as
ingesting
the
data.
So,
keeping
up
with
the
read
and
writes
was
becoming
a
challenging
thing
as
well,
and
the
data
extraction
was
done
in
batches,
so
every
four
hours
we
would
extract
the
data
load
it
into
Terra
Terra
data
data
warehouse.
A
The
additional
thing
was
that
there
was
no
low
latency
type
of
application
that
was
available
because
the
system,
the
transient
storage
itself,
was
not
able
to
keep
up
with
its
main
functionality.
So
any
other
applications
such
as
low
latency
or
dashboard
sort
of
applications
were
pretty
far
out
in
in
future.
A
So,
just
to
recap,
the
the
problems
high
rate
of
ingestion
ability
to
keep
up
with
incoming
data,
no
low,
latency,
dashboard
sort
of
application,
and
also
we
didn't
have
any
way
of
doing
pattern.
Detection
on
the
incoming
data,
while
we
were
ingesting
it
in
order
to
make
some
sort
of
decisions
or
trigger
any
alerts.
A
So
what
what
do
we
do
is?
As
I
said
in
the
beginning,
the
first
thing
that
we
did
was
replace
the
transient
storage
with
Cassandra
because
of
all
its
features
in
terms
of
high
ingestion
capability,
so
rest
of
the
pieces
stay
pretty
much
the
same.
The
messages
were
ingested
put
on
the
JMS
queue
and
then
using
hector
klein
api's.
We
ingested
the
information
into
a
Cassandra
cluster.
A
So
what
what
is
the
role
of
data
model
in
in
all
this
this
process?
So
before
we
go
into
what
exactly
the
data
model
was
I
want
to
just
segue
into
what
are
some
of
the
things
that
are
missing
in
Cassandra
when
we
compare
it
with
a
traditional
or
DBMS
and
go
from
there?
So
I
don't
really
want
to
say
it.
These
are
shortcomings,
but
actually
there
are
opportunities
for
us
to
transform
the
way
we
have
been
dealing
with
the
databases
in
order
to
avail
of
new
features
that
are
available
in
systems
such
as
Cassandra.
A
So
it
might
take
a
slight
change
in
our
our
attitude
rather
than
saying
that
this
is
how
I've
been
doing
things
in
Tara
data.
Show
me
how
I
do
it
in
Cassandra.
Well,
we
would
slightly
change
the
question
and
say:
I'm
gonna
forget
how
I've
been
doing
things
with
Oracle
and
Tara
data
and
see
what
the
features
and
capabilities
in
Cassandra
are
and
then
tune
myself
and
my
business
and
my
way
of
doing
things
so
that
it
better
molds
itself
with
what
Cassandra
offers.
So,
obviously
there
are
no
joints
there.
A
No
ways
of
doing
aggregation,
queries
and
with
random
partitioner
were
not
able
to
organize
the
keys
in
in
a
certain
order.
So
if
we
look
at
them
as
opportunities,
that'll
give
us
the
impetus
to
structure
our
data
model
according
to
certain
features
that
are
available.
So
what
are
so
before
we
get
into
the
features
in
order
to
get
around
those
lack
of
features.
We
have
to
do
smart,
B
normalization
because
we
don't
have
joins.
A
We
have
to
use
client
or
application
along
with
the
database,
we're
all
used
to
doing
analytical
queries
in
Oracle
with
windowing
functions
and
lead
lag,
and
regression
and
what-have-you.
So
in
in
this,
when
you,
when
you
switch
gears-
and
you
are
in
Cassandra,
you're
gonna
have
to
leverage
the
application
so
together,
the
application
and
the
database
will
give
you
the
entire
solution.
So
that's
the
way
that
that's
the
lens
through
which
I
would
look
at
this
problem
space.
A
This
allows
us
to
change
the
way
we've
been
storing
the
data
in
a
vertical
way
in
a
relational
system
into
a
horizontal
fashion.
The
other
thing
is
that
all
the
data
that
is
stored
horizontally
is
stored
for
a
certain
key
is
stored
on
a
single
server.
That
also
gives
us
the
ability
to
access
it
very
very
quickly
in
order
to
perform
any
sort
of
calculations,
any
sort
of
analysis
on
all
the
records
pertaining
to
that
key.
A
The
other
thing
is
that
which
is
very
different
from
a
traditional
database.
The
name
of
a
column
is
of
great
importance
with
this
system,
which
is,
which
means
that
we
are
able
to
store
actual
important,
useful
information
embedded
as
part
of
the
name
of
the
column,
whereas
in
a
traditional
database,
all
we
could
say
was
social
security
or
first
name
or
last
name
equal
to
this
greater
than
this.
A
A
A
If
my
row
key
had
a
minute
value,
then
all
the
messages
coming
in
in
that
minute
are
going
to
go
horizontally
in
that
single
key,
and
if
you
have
a
thousand
node
cluster
or
whatever
100
node
cluster,
all
the
99
machines
would
be
sitting
idle,
not
doing
anything
except
the
one
machine
which
got
that
key
allocated
to
it
via
hash.
So
you
have
to
be
careful
how
you
construct
the
key,
so
in
this
case,
in
order
to
remove
such
hot
spotting
as
well
as
making
sure
that
everyone
gets
a
chance
to
play.
A
A
It
would
be.
It
would
be
wiser
to
structure
the
columns
in
such
a
way
that
the
granularity
of
these
columns
goes
on
increasing.
So
that
means
the
first
column
name.
Our
first
piece
that
you're
using
to
construct
your
composite
entity
contains
lot
of
rows.
Then
you
go
narrow
down
so
essentially
you're
drilling
down
from
the
first
one,
all
the
way
to
the
end,
which
would
allow
you
to
construct
where
slice
predicate
soar
actually
do
queries
against
this
data
in
a
more
meaningful
fashion.
A
B
So
let
me
talked
about
what
to
do
when
you
have
an
existing
system
that
you
need
to
bring
up
to
date.
What
happens
when
you
have
a
Greenfield
opportunity,
we're
trying
to
bring
create
a
new
product
from
an
emerging
technology
from
reading
technologies?
So
what
I'll
talking
about
today
is
the
eccentric
cloud
platform
and
what
the
essential
cloud
platform
is
is
a
cloud
brokerage
that
allows
us
to
provide
our
assets
to
our
clients
in
an
on-demand
elastic
manner.
On
top
of
that,
we've
built
Big
Data
capabilities
to
handle
different
classes
of
big
data
workloads.
B
The
first
iteration
of
this
was
batch
oriented,
so
it
was
essentially
doing
collaborative
filtering
as
a
service
to
provide
targeted
product
recommendations.
The
current
early
iteration
is
based
on
stream
processing
and
it's
what
I'm
going
to
talk
about
today
and
event,
detection
and
the
first
use
case
for
this
was
network
and
oolitic
s--.
B
So
why
network
analytics?
Well,
if
you
think
about
service
providers,
they're
kind
of
in
a
grim
precision
right
now
in
terms
of
network
loads?
Obviously
we
know
there's
been
a
proliferation
of
consumer
devices,
tablets,
smartphones,
set-top
boxes,
whatever
have
you
and
most
of
the
increase
in
that
data
has
been
in
video.
So
what
does
this?
Do?
It
puts
tremendous
load
on
the
network.
This
in
turn
causes
operational
costs
to
go
up.
It
makes
understanding
defects
within
the
network
more
difficult
in
an
fx
capacity
planning,
and
all
of
this
in
turn
makes
it
in
fluence.
B
As
the
end
user
experience
very
directly,
you
can
imagine
if
you
can't
understand
why
a
defect
is
happening.
You
have
no
slow
network
speeds.
You
have
buffering
events
if
your
video
is
being
slowly
trying
to
watch
Game
of
Thrones
on
your
iPad
and
you
can't
find
out
who's
getting
the
head
chopped
off
this
week.
You're
gonna
be
angry.
You're
gonna
move
to
the
service
provider
right,
so
it
affects
customer
churn.
So
really
it
affects
the
bottom
line
over
over
cuz
of
our
clients.
So
how
do
we
go
about
understanding?
B
So
in
order
to
do
that,
we're
ingesting
into
a
distributed
queue
in
this
to
finish
being
a
project,
Apache
Kafka,
we're
doing
pre-processing
and
storm
and
eventually
ends
up
in
the
Cassandra
story,
distributed
data
store,
as
a
me
mentioned,
because
the
data
stacks
allows
you
to
integrate
with
a
dupe
which
gives
us
access
to
the
entire
Hadoop
ecosystem,
which
allows
us
to
use
things
like
custom,
MapReduce
jobs,
hive
Pig
to
understand
what
the
correlations
are
and
what
is
actually
causing
those
suboptimal
speeds.
So
is
it
based
on
devices
that
people
are
using?
B
Is
it
based
on
the
browsers
that
people
are
using?
Is
it
based
on
location
or
all
users
from
a
certain
region
having
a
little
slow
speeds
once
we
know
that
I
want
to
understand
what
those
score
metrics
are
what
those
patterns
are.
We
can
then
implement
them
back
into
storm
so
that
next
time
you
have
those
messages.
Are
those
requests
coming
through?
You
can
understand,
what's
happening
and
rays,
events
based
on
that
in
the
simplest
use
case,
we're
writing
to
a
specific
column
family
for
further
analysis,
but
you
can
see
how
this
can
be.
B
Events
can
be
raised
to
external
systems,
other
systems
and
internally
to
clients
to
end-user
UI's,
and
finally,
we
visualize
both
from
the
from
the
datastore
and
from
the
real-time
event,
notifications.
You
think
things
like
d3
and
cross
filter,
as
you
can
see
that
the
pattern
itself
is
very
iterative.
You
know,
twitch
has
come
in
already
to
analyze
that
you
feed
that
back
into
the
system.
So
next
time
you
can
respond
to
that
accordingly,
so
I
can,
you
know,
have
a
whole
presentation
on
any
one
of
those
components.
B
Well,
first,
it's
looking
like
any
technical
system
interests
saying
what
do
I
need
from
this.
Well,
I!
Guess,
as
I
was
saying
earlier,
we're
really
looking
to
build
a
platform
that
can
handle
multiple
use
cases.
So
we
need
to
be
able
to
do
several
things:
do
processing
aggregations,
joint,
complex
computations
and
we
need
to
have
a
system
that
allows
us
our
framework
or
toolset
that
allows
us
to
build
for
all
those
and
gives
us
a
flexibility
to
build.
B
B
The
different
use
cases
we
get
from
having
low
level
primitives
that
it
allow
us
to
build
those
so
again
until
this
a
little
bit
further,
but
having
what?
Essentially
is
plain
old
Java
code,
that's
encapsulated
in
certain
classes
that
implement
certain
earth
vs.
allows
you
to
be
very
modular
in
how
you
build
these,
and
also
to
think
about
your
analytical
workflows
as
a
catalog
that
you're
composing
together
to
do
what
you
need
to
do.
B
The
flat
tolerance
is
gained
through
robust
failover
strategies
and
reliability
through
tunable
processing
guarantees.
So
you
understand
what
the
premium
is.
I
talked
about,
and
some
you
might
some
of
you
might
be
familiar
with
with
storm,
so
you
might
not
to
give
a
high-level
overview
the
concepts
the
topology
is
really
your
streaming
processing
application,
it's
analogous
to
a
MapReduce
job
except
imagine,
MapReduce
job
finishes.
Well,
this
lasts
forever
until
you
kill
it,
how
we
break
it
down
for
this
type
of
analysis,
it's
based
on
the
analysis.
B
Are
you
doing
so
suboptimal
network
speed
will
be
one
topology.
Geospatial
analysis
would
be
another.
You
can
do
those
simultaneously
but
again.
Well,
since
we're
trying
to
build
a
platform,
we
need
to
be
portable
across
different
clients
across
different
projects.
So
we
need
that
level
of
portability.
B
B
The
spout
is
what
takes
the
data
from
the
external
source
and
emits
it
into
the
topology.
So
in
our
case
it's
a
distributed
queue
in
Kafka,
but
it
could
be
api's.
It
could
be
other
databases.
Finally,
a
bolt
is
what's
actually
doing
the
processing
a
boat
takes
a
tuple.
There
are
some
processing
on
it
and
it
emits
it
to
the
next
one
again.
It
helps
to
break
down
everything
to
kind
of
slowest
level
of
processing.
So,
for
us
is
things
like
session
a
station,
speed,
calculations,
parsing
and
again.
B
It
helps
you
think
of
things
very
in
a
very
modular
way
to
be
able
to
think
of
these
processing
workflows
as
being
a
catalog
that
you
can
compose
together,
so
integration
with
Sandra,
why?
This
is
work
well
with
Cassandra?
Well,
if
you,
if
you
think
about
everything
that
just
talked
about
about
scalability,
about
having
high-throughput
parallelization,
all
of
that
complements
very
well
with.
B
What's
Cassandra
can
do
it
doesn't
help
us
if
it
doesn't
help
us
if
storm
can
process
thousands
upon
thousands
of
messages
a
second
if
the
database
can't
handle
those
writes
at
the
backend
right,
so
we
need
something
that
can
can
work
well
together
with
their.
What
we
did.
We
build
a
custom
bolt
because
we
were,
we
need
a
dynamic
columns
per
request
to
be
written.
I
would
urge
you
to
not
do
that
and
use
the
available
things
that
are
out
there.
B
There's
the
there's
a
great
community
around
storm
that
has
built
a
lot
of
a
lot
of
custom
classes,
especially
for
Cassandra,
so
I
would
go
out
and
look
for
those
and
finally
quickly
because
I'm
have
to
wrap
up
I'm
gonna
run
through
a
example
of
how
we
actually
do
this.
This
is
a
simple
topology,
showing
you
how
we
pull
from
the
kafka
spout
based
on
the
IP
we
group,
based
on
the
IP,
addresses
for
a
specific
user.
We
do
some
pre-processing
some
parsing
session
as
I.
Do
as
such.
B
You
create
the
session
understand
what
a
session
is.
This
is
arbitrary
right.
This
can
be
something
that
you
define
as
a
pattern
or
something
that
you
create
as
a
time
window
that
you
define
arbitrary,
but
the
nice
thing
here
is
that,
if
that
strategy
changes
you
can
plug
it
in
and
out
of
this
apology,
that's
the
the
power
that
it
gives
you.
We
calculate
the
speed
baton
per
the
session.
We
keep
a
running
tally
of
all
this.
B
So,
as
you
can
see
here
stream,
the
top
stream
is
doing
where
we
just
what
we
just
talked
about,
but
the
bottom
one
is
actually
funding
the
average
speed
per
the
location
and
the
location
across
all
users.
So
we
can
compare
that
user
speed
to
the
user
to
the
speed
of
all
the
other
users
within
that
region
and
compare
those.
B
And
I
think
I
am
out
of
time,
and
then
these
are
some
lessons
learned.
I
can
go
into
again
as
as
a
meet
mentioned,
come
talk
to
us
kind
of
at
a
time
here
but
come
talk
to
us
at
the
experts
lounge
and
we
happy
to
talk
to
you
any
either
mirror
meet,
and
my
team
is
here:
Sonali,
Mike
and
Neil.
Everybody
raise
your
hands
still
happy
to
answer
questions
so
with
that
and
thank
you.