►
From YouTube: Real-Time Analytics with Apache Cassandra at LoopLogic
Description
If you've ever wondered how to utilize Apache Cassandra for real-time analytics, then this is a meetup you won't want to miss!
We're excited to have Stephane Legay, CTO at LoopLogic, joining us to present on how LoopLogic uses Apache Cassandra for their real-time analytics use case. Stephane will be covering the LoopLogic/Cassandra architecture + how they process their data using a 2-pass system.
Go Daddy has graciously offered to host this event. Be sure to come straight from the office (or sofa) as food/beverage will be served.
Hope you can make it!
What You Will Learn in this Session
• Architectural Overview of Cassandra at LoopLogic (3,000 to 5,000 writes per second on a 2-node cluster)
• Processing Data Using a 2-Pass Process
- Event log messages get stored in Cassandra as well as Amazon SQS upon reception of the event.
- Log processor processes the SQS queue.
- Message updates hundreds of counters using Memcached counters.
- We then batch-write those hundreds of results in Cassandra and update indices.
About Stephane Legay
Stephane is the founder and CTO of the Phoenix based startup LoopLogic. LoopLogic provides users with tools to create, public and track who watches your videos. Before creating LoopLogic, Stephane was the CTO of OCEG. Stephane's skillsets include web application development, user interface design, C# programming, and project management in a wide variety of business applications.
Apache Cassandra at LoopLogic Case Study
http://www.planetcassandra.org/blog/post/apache-cassandra-at-looplogic-case-study
A
A
What
we,
what
we
do,
that's
kind
of
different
from
everybody
else
is
that
we
give
you
a
named
viewer
and
we
integrate
with
your
marketing
software.
So
if
you
use
something
like
salesforce,
constant
contact
or
mailchimp
to
actually
generate
leads
through
videos
and
then
give
you
kind
of
viewer
habits,
and
things
like
that,
so
that
you
can
actually
automate
marketing
based
on
video
usage,
when
we,
when
we
started
blue
project
at
first
analytics
in
terms
of
aggregate
analytics
globally,
were
kind
of
on
the
back
burner.
A
It
was
just
something
we
had
to
have
we're
not
too
worried
about
it,
and
we
started
with
some
php
package
that
was
just
doing
web
analytics.
There
was
no
clustering,
there
was
no
horizontal
scaling
built
in
or
anything
like
that.
The
performance
was
horrible.
It
was,
you
know,
processing
logs
every
night,
and
it
was
just
awful
so,
but
that
was
fine
at
first
and
then
we
ran
into
some
problems
too
much
volume,
too
much
data,
and
so
a
database
crash.
A
We
tried
to
scale
upward
problems,
and
so
we
had
to
go
back
to
the
drawing
board
and
kind
of
build
our
own
at
the
time.
Cassandra,
I
think,
was
that
when
we
started
messing
around
with
it,
it
was
0.6,
so
it
was
pretty
early
on.
We
started
working
with
the
guys
that
was
not.
Data
stacks
at
the
time
was
a
different
company.
That's
when
ellis
just
left
left.
Where
was
he
working
again,
the
cloud
people
cloud
files
and
stuff
anyway?
A
He
just
started
that
company
and
we
worked
with
those
guys
and
kind
of
went
into
cassandra,
because
what
we
liked
about
it
was
was
we
we
need.
We
needed
heavy
rights
right.
That
was,
that
was
main
key,
wanted
real-time
analytics
in
order
to
do
that,
if
you
don't
want
to
start,
you
know
the
idea
of
doing
big
data
type
of
thing
with
the
hadoop
cluster
and
all
that
stuff.
That
was
not
really
for
us.
We
didn't
want
to
have
like
timed
reports
and
timed
big
data
analytics
running
in
the
background.
A
A
So
let
me
there
you
go
so
this
was
our
use
case.
We
didn't
know
what
the
message
load
was
going
to
be.
I
mean
we're
a
startup.
We
went
from
you
know
a
few
channels,
one
day
to
suddenly
we
had
a
couple
microsoft
videos,
one
event
that
they
did
because
we
had
an
inside
track.
So
we
did
that
one
and
it
was
scott
guthrie.
At
microsoft
and
within
a
day
we
had
you
know
250
000
viewers.
A
You
don't
have
that
meeting
the
other
need.
We
had
because
of
the
type
of
analytics
we
run
is
that
whenever
we
send
a
message
to
the
to
the
cluster,
it
contains
some
session
information
so
instead
of
sending
one
event
per
increment,
so
let's
say
you
know
in
web
analytics.
That
would
be
a
page
view.
So
you
load
your
page.
A
You
got
an
increment
of
one
and
that's
the
only
message
and
once
the
message
process
is
gone,
we
actually
had
we
monitored
a
session
on
the
client
that
has
some
information
about
what
what
pieces
of
the
video
the
person
watched
where
they
were,
where
they
left
it,
whether
they
completed
all
that
stuff.
So
there
was,
there
was
a
persistent
state
that
needed
to
be
stored
and
also
used
with
all
of
the
delta
values
that
are
in
that
session
for
aggregate.
So
there
were
kind
of
two
pieces
to
that
equation.
A
A
We
processed
18
metrics
and
what
I
call
metrics
are
things
like
views
place
completions,
how
much
of
a
video
you
watched
how
much
bandwidth
was
consumed.
All
of
those
things
that
need
to
be
aggregated,
so
those
are
our
metrics.
The
dimensions
are
things
that
you're
going
to
be
filtering
on
and
ordering
your
data
from
your
data
on.
So
I'm
going
to
give
you
a
quick
example
of
what
that
looks
like,
because
I
think
it
is
going
to
make
it
easier.
A
Yeah,
so
this
is
an
example
of
video
analytics
page,
so
well
there
on
this
one
over
the
past
month,
we've
got,
you
know,
14
total
views,
and
then
you
can
so
these
are
all
all
of
our
metrics.
You
need
visitors,
first-time
visitors,
people
who
you
have
personal
video
and
then
these
are
our
dimensions.
So
the
dimensions
will
be
things
like
country,
cities
which
site
it
was
embedded
on,
and
so,
if
you
click
one
of
the
dimensions.
C
A
Thing
now,
the
other
thing
that
we
wanted
to
do
that
was
really
important.
Was
this
idea
for
drill
down?
So
if
you
go
to
something
like
google
analytics,
typically
you'll
see
your
analytics
dashboard
like
this
and
any
of
those
dimensions
here
you
should
be
able
to
click
inside
it
so
that
you
can
actually
view
the
subset
of
the
data
filtered
by
that
by
the
dimension.
A
So,
for
example,
if
I
go
into
right
here-
and
I
click
on
it
now-
all
of
my
data
is
filtered
by
city,
and
within
this
I
can
I
can
go
and
see
which
websites
where
people
from
the
city
watching
right
so
you've
got
kind
of
a
two
level
of
depth
so
that
you
can
filter
and
do
all
this
stuff,
and
this
is
pretty
much
real
time.
So
all
of
this
data
is
updated
within
about
15
seconds
unless
there's
a
there's,
a
log
jam
all
right.
So
let's
go
back
to
that.
A
All
right,
so
those
were
my
dimensions,
three
levels,
so
we
we
want
aggregates
at
the
channel
level.
So
that's
your
account
containing
a
few
of
your
videos,
the
video
level
and
then
within
a
video.
You
might
have
different
campaigns,
so
you
might
distribute
your
video
on
different
sites
and
have
kind
of
a
tracking
code
and
isolate
those
views
so
that
you
can
then
generate
analytics
on
this.
A
We
want
you
to
provide
segmentation
by
day
week,
month
year
and
all
time
since
the
video
existed
and
that's
what
I
was
showing
you
the
level
of
drill
down
and
then
another
key,
because
you're
thinking,
okay.
Well,
that's
easy.
I
just
created
a
bunch
of
fabulous
creative
keys,
push
that
into
cassandra
use
the
built-in
counters.
I
aggregate
all
that
and
I'm
just
kind
of
done
well,
okay,
first
off
how
many
of
you
actually
use
cassandra.
A
So
you've
got
two
all
right,
so
let
me
backtrack
a
little
bit.
I
just
assumed
that
everybody
here
was
familiar
with
it.
How
many
of
you
are
not
familiar
with
cassandra
at
all
and
you're
just
here,
as
I
just
want
to
check
it
out
all
right.
I've
investigated
you're
just
investigating
okay,
we
know
a
little
bit
about
it
all
right,
so
so
cassandra
has
a
feature
since
I
think
it
was
0.8
that
is
counters,
so
there's
built-in
counters
and
they're
eventually
consistent,
meaning
that
you
push
the
data.
A
It's
got
a
timestamp
it
aggregates
this
data
based
on
the
timestamp
and
then
eventually
it
will
be
consistent.
When
you
write
it,
you
can
make
sure
that
it's
consistent
across
the
entire
cluster,
so
that
if
you
read
it
back,
it's
going
to
be
accurate
or
if
you
don't
care,
you
can
just
say
just
get
the
data
process
it
later.
I
don't
care
if
it's
valid
right
now
and
cassandra
does
the
rest
of
getting
all
those
timestamps
and
just
creating
the
right
value
at
some
point
in
the
future
right.
So
this
is
great.
A
I
mean
it's
awesome,
it's
it's
amazing
technology.
The
problem
is
well.
Let's
say
that
now
you
want
to
actually
have
a
data
stream
of
aggregates
that
are
happening
in
real
time
right.
So
I'm
viewing
my
videos
and
you
want
to
have
a
trigger
right.
You
want
to
have
a
trigger
that
says.
Whenever
I've
got
100
people
watching
this
video,
I
want
to
get
notified
for
a
lot
of
different
reasons
like
for
us.
It
could
be
bandwidth
issue.
Oh,
this
user
was
free.
There
was
a
problem
in
the
software
they
just
used.
A
You
know
a
terabyte
of
data.
We
got
to
stop
this
so
or
you
can
have
you
know.
I
want
to
know
whenever
10
people
from
new
york
view
this
thing,
because
there's
something
going
on
new
york
and
I
want
to
start
a
campaign
there
right.
So
if
you
do
that,
and
you
do
all
of
your
aggregates-
you
know
in
cassandra-
you
have
no
visibility
of
that.
A
Unless
you
d,
you
do
reads
every
single
time
you
increment
and
that
process
would
be
slow.
You
need
to
write.
You
need
to
make
sure
it's
eventually
consistent
and
then
you
need
to
do
at
the
very
least,
a
quorum
read
to
make
sure
that
your
data
is
correct
and
then
process
whatever
you
need
to
do
so,
in
our
case,
we
didn't
want
to
do
the
well
first
off.
A
So
what
did
that
mean?
Well,
we
needed
to
have
a
queuing
system,
because,
even
though
cassandra
is
very
fast,
you
can't
assume
that
you're
gonna
have
suddenly
a
hundred
thousand
people
logging
into
your
system
and
you're
going
to
process
all
that
real
time.
Your
cluster
is
going
to
crash
and
everything
goes
down.
A
Those
were
all
time
by
year
by
day,
by
month
by
week
and
three
was
I
wanted
to
count
at
the
channel
level
and
aggregate
on
my
channel
level
at
the
video
level
on
the
campaign
level.
So
you
can
think.
Like
one
view,
one
person
clicking
play
is
going
to
generate
2
535
rights
right.
So
when
we
before
cassandra,
we
tried
to
run
our
algorithm
on
mysql
and
on
just
a
standard.
You
know
just
standard
server
right.
It
was
an
amazon
ec2
instance.
A
It
was
like
a
medium
cpu
and
we
stopped
at
about
100
rights
per
second
right
so
which
we
could
probably
have
optimized,
maybe
gone
as
you
know,
as
far
as
like
500,
but
it
was
just
not
nearly
enough,
so
we
so
with
cassandra
to
give
you
an
idea
right
now.
You
know
when
we
started.
We
had
just
two
notes
and
the
reason
we
did
two
notes.
We
wanted
some
redundancy,
so
they
actually
redundant
their
copy
of
each
other
right.
They
don't
so
think
of
like
if
it
was
a
raid
array.
A
It
would
be
what
is
that
rate
10
or
like
it's
just
a
mirror
of
each
other,
so
with
just
that
running
on
amazon
large
instances,
nothing
crazy,
nothing!
Fancy
right!
We
run
without
any
problem.
6
000
rights
per
second
without
one
problem
at
all
the
so
the
other
key
was
to
be
able
to
then
so
you
process
all
this
data.
A
So
you
get
you
get
your
metric
right,
you,
you
have
a
user
view
and
you're
going
to
have
you're
going
to
create
your
2535
aggregates,
and
then
you
need
to
write
them
well.
The
last
thing
you
want
to
do
is
write
sequentially,
because
not
only
is
it
kind
of
a,
I
mean
it's
more
processing,
but
the
other
problem
is
that
you're
going
through
the
pipe
to
communicate
with
your
database
server
for
every
single
right.
So
that's
not
great.
The
best
thing
to
do
would
be
to
be
able
to
do
one
big
batch
right.
A
D
A
You
to
do
that
right,
so
you
can
do
batch
writing
and
there's
no
limit
on
the
the
amount
of
data
that
you
passed
through
through
that
batch
right.
I
explained
to
you
that
some
of
the
messages
need
to
be
persisted
immediately
and
the
yeah
the
counter
values
process
at
the
app
level.
We
talked
about
that
and
the
problem,
with
eventually
consistency
for
calendars
all
right.
So
how
do
we
do
this?
A
So
what
we
set
up
was
a
system
that
that
used
apache
cassandra.
We
started
so
with
0.6
we're
running
1.0
right
now
and
we
use
a
couple
large
instances
how
many
of
you
are
familiar
with
aws
everybody?
Okay,
so
your
large
instance
is
like
I
don't
know.
I
think
it's
like
200
bucks
a
month.
Those
are
linux
instances
and
we.
A
We
have
a
couple
terabyte
ces
volumes
on
those,
so
you
can.
Actually.
I
was
talking
to
you
about
the
6000
rise
per
second,
the
number
one
bottleneck
is
going
to
be.
You
know
your
rights
on
the
on
the
drive
so
right
now
we're
we're
not
doing
any
kind
of
raid
at
the
on
the
large
instances,
they're
just
one
evs
volume
per
and
for
now
it's
it's
fine,
because
the
worst
case
scenario
is
we've
had
bottlenecks
where
it
took
a
few
hours
to
process
the
data,
but
nobody
was
really
bothered
with
that.
A
A
Okay,
so
sqs,
quick,
rundown!
It's
awesome
if
you
got
a
ton
of
messages
because
even
though
it's
it's
pretty
slow
in
latency,
when
you
write
and
read
your
message
compared
to
your
rabbitmq
or
something
like
that,
it
scales
virtually
infinitely.
So
as
long
as
you
work
with
that
latency
of
read,
write
and
you're
fine
with
that
you
can
scale
and
have
millions
of
messages
processed,
and
you
never
have
to
worry
about
it
and
it's
guaranteed
to
be
written
and
not
lost,
and
it's
just
great.
A
Think
it
was
about
50
milliseconds,
so
in
itself
it
you
know
it's
not
terrible,
but
compared
to
a
purely
in-memory.
You
know
node
that
would
be
sitting
next
to
your
application.
Node
is
slow.
You
know
and
we've
seen,
that
the
problem
is
that
it's
not
predictable
and
you
might
have
some
rights
that
might
be
up
to
250
500
milliseconds,
sometimes,
and
it
kind
of
depends
on
the
internal
connection,
because
you
don't
know
where
your
server
is,
where
the
sqs
servers
are,
and
there
can
be
a
ton
of
stuff
going
on.
So
it's
pretty
unpredictable.
A
A
And
we
didn't
want
to
run
our
own
servers
because
we're
already
maintaining
you
know
enough
servers,
so
sqs
was
good.
The
point
is
that
it's
text
based
expensive
to
serialize
and
do
all
this
stuff
so
and
also
in
how
many
messages-
and
it
can
start
costing
you
where
you
start
dealing
with
billions
of
messages.
I
mean
the
millions
of
messages
level
is
not
too
bad.
When
you
start
having
tens
hundreds
of
millions,
it
becomes
a
problem.
So
what
we
did
is
we
actually
have
a
two
pass
queueing.
A
So
when
you,
when
the
message
goes
into
the
app
server,
we
have
in
memory
buffer
that
stacks
up
to
10
20
messages,
and
it
calculates
the
size
of
the
serialized
version
of
it
to
make
sure
that
it
doesn't
go
above
the
amazon
size
limit
for
your
message,
which
is,
I
can't
remember,
like
how
many
characters,
but
it's
pretty
short.
It's
like
a
thousand
characters,
maybe
something
like
that,
and
so
we
stack
them
up
together
and
we
package
them
in
10
messages
at
a
time.
A
So,
in
your
in
your
sqs
message,
you
actually
have
10
serialized
json
serialized
messages
inside
it.
So
that's
the
way
we
did
it
and
that's
where
we
divided
our
cost
by
10.
the
danger
in
doing
that
is
that
you
know.
If
some
something
goes
wrong,
you
get
your
message.
You
get
10
of
them,
it's
impossible,
not
to
lock
that
message
at
the
message
level.
So
you
you
process.
Five
of
them.
You've
got
five
left
and
something
goes
wrong
well,
which
ones
were
processed
when
we're
not
now
you're
on
your
own.
A
So
it's
a
risk
right,
it's
a
risk
before
so
it
was.
It
was
worth
it.
Our
app
servers
were
not
net
in
c
sharp,
medium
instances,
just
standard
stuff,
nothing
fancy
and
so
for
what
was
interesting
was
for
the
distributed
counselors
in
the
lock
server,
because
we're
going
to
need
a
lock
server,
we're
going
to
start
doing
increments
on
our
own
right,
because
you
can
imagine
actually
I'm
going
to
switch
to
the
next
one.
A
Yeah,
I'm
going
to
talk
about
memcache
in
couple
slides,
I'm
going
to
run
through
this
one
really
fast.
So
the
way
it
works
is
your
your
client
just
is
watching
your
video
doing
stuff
and
seriously
we're
going
to
send
a
message.
The
app
server
is
going
to
do
two
things:
it's
going
to
send
directly
to
cassandra
the
data
storage
layer
just
to
log
off.
This
is
the
user
session.
This
is
the
the
current
data
right
update
these
values.
A
At
the
same
time,
it's
going
to
send
that
to
sqs
for
later
processing
for
the
actual
aggregates
right
that
way,
the
next
time,
the
user
reloads.
This
we're
obviously
going
to
read
the
data
for
the
session
directly
from
cassandra,
and
that
is
not
cued
right.
So
then,
where
it
gets
interesting
is
the
the
second
pass
where
so
we've
got
a
worker
cluster.
Here
of
you
know
a
few
ec2
instances
doing
all
the
work
of
aggregating,
calculating
doing
all
the
stuff
processing
messages.
A
Each
of
them
might
be
running
multiple
threads.
So
as
a
standard,
we
run
only
two
per
server
by
default.
There's
some
locking
mechanism
where
we
pause
the
threads.
When
there's
no
messages
and
all
that
stuff
and
then
and
then
boost
them
back
up
and
have
them
running
whenever
we
have
messaging
until
there's
no
messages
anymore,
so
at
any
point
in
time,
there's
actually
one
that
is
periodically
checking
and
as
soon
as
it
hits
a
message,
all
of
them
wake
up
and
pull
everything
that's
available.
A
We
also
have
a
ui
that
allows
us
to
increase
the
threat
count
per
server
whenever
we
want.
So
we
you
know.
If
we
have
a
lot
of
messaging,
we
might
have
10th
press
running
per
person,
but
you
can
see
how
you
know
if
you've
got
this
running
in
parallel
and
you've
got
an
increment,
you
know
of
you've
got
you
know
a
thousand
people
watching
your
video
and
they
are
watching
from
phoenix.
So
you
can
see
how
the
aggregate
on
phoenix
being
processed
by
you
know
potentially
20
30
40
threads.
A
At
the
same
time,
you
got
to
have
some
way
of
that
being
consistent,
right
and
so
you're
going
to
have
to
do
some
locking
there.
You
don't
have
to
do
some
locking
if
you
would
have
cassandra's
internal
calendars,
because
cassandra
will
do
that,
for
you
sandra
internally
will
do
we'll
get
all
the
messages
and
based
on
the
timestamps,
we'll
run
them
in
order
and
do
the
aggregate
and
then
be
done,
and
you
don't
have
to
worry
about
it.
Of
course
we
did
so
so
we
need
to
do.
A
We
need
to
have
a
distributed
lock
server
and
the
only
thing
it
does
is
that
we
give
it
a
key.
We
check
if
the
key
is
already
locked.
If
it's
locked,
then
we
just
don't
process
the
message
and
we
just
let
it
come
back
up
later
I
mean
we
keep
trying
getting
the
lock
on
that
key
for
a
little
while
a
few
seconds,
and
then,
if
we
can't
get
a
lock,
we
just
leave
the
message
around
for
further
processing
and
we
needed
a
distributed
counter.
A
So
we
just
needed
a
really
really
fast
way
of
counting
things
on
a
key
so
be
able
to
increment
a
value
based
on
a
key
in
a
really
fast
manner.
That
would
be
accurate
and
not
eventually
consistent.
That
would
be
actually
accurate
right
now
and
I
need
to
be
able
to
do
updates
on
that
key
multiple
thousands
of
times
per
second
right,
and
so
what
what
we
did
is
that
we
actually
used
memcached
for
both
of
these,
so
how
many
of
you
are
familiar
with
memcache,
okay,
so.
A
Well,
lock
the
value
read
the
value
increment,
the
value
store,
the
value
unlock,
the
value
you
can
just
say
increment,
and
it
will
increment
and
return
the
value
which
is
kind
of
nice.
The
funny
thing
was
we
built
that
whole
thing.
It
was
all
great
and
then
we
realized
that
some
numbers
didn't
work
and
when
we
ran
some
unit
tests
on
this
whole
thing,
we
realized
that
the
version
of
memcache
we
were
using
back
then
actually
was
not
atomic.
A
They
portrayed
it
as
being
atomic,
and
you
read
10
threads
doing
an
increment
on
the
same
key
and
doing
100
increments
each
and
you
ended
with
a
value
that
was
completely
unpredictable
between
993
and
999.,
and
so
that
was
a
problem,
but
they
they
fixed
it.
So
now
it's
okay!
It's
correct
passes
it's
all
great,
and
so
we
did.
We
do
all
the
increments
at
the
in
memphis.
So
how
does
it
work?
We've
got
our
sqsq
here.
A
A
What
I
need
to
do
now
is
loop
through
all
of
the
dimensions
that
are
included
and
create
keys
that
match
each
of
each
one
of
those
dimensions
for
each
dimension.
I
need
to
create
a
key
also
that
matches
a
time
period
day
week
month
year,
all
time
and
I
need
to
then
loop
through
each
dimension
through
the
dimensions
one
more
time
to
create
an
aggregate
for
people
in
phoenix
for
that
specific
website
right.
So
so
I
create
all
those
keys.
A
I
create
those
2545
keys
that
are
going
to
be
the
values
that
I'm
trying
to
increment
and
once
I
have
this,
this
list
of
keys,
I'm
going
to
call
them
cached
for
each
one
and
I'm
going
to
try
to
increment
them,
and
so
memcache
can
run.
So
I
mean
we,
we
run
tests.
You
know
I
can't
remember.
I
can't
even
remember
how
many
thousands
of
messages
we
could
run
per
second,
but
that
was
a
piece
okay,
so
you
do
your
first
pass,
but
memcache
being
memcached
and
it's
all
located
in
memory.
A
A
A
A
Memcache
is
going
to
come
back
with
either
the
value
that
you
set
to
increment
two.
So
if
I
say
increment
that
key
by
two
and
it
comes
back
with
two,
that
means
that
the
key
was
not
there
to
start
with.
So
for
all
of
those,
you
have
to
re-invalidate
all
those
keys,
so
you
create
a
different
list
of
of
keys
that
were
not
a
matching
main
cache
for
all
the
other
ones.
You're
set
you're
fine
and
you
create
a
timestamp
on
that
column.
In
cassandra.
That's
going
to
have
that
value.
A
You
create
a
timestamp
that
matches
the
exact
time
when
you
checked
when
you
checked
men
cached,
because
the
reason
why
you
do
that-
and
you
have
to
be
very
careful
with
your
clocks
across
your
cluster-
is
that
let's
say
you've
got
multiple
servers
that
are
incrementing
at
the
same
time
and
they
both
go
to
memcached
and
they
both
go
to
memcache
within
a
couple
seconds,
which
one
is
right
right,
because
both
of
them
are
going
to
write
to
cassandra.
Cassandra
will
eventually
get
those
things
together
and
pick
the
right.
A
So
what
do
we
do
with
the
values
that
were
not
found
in
memcache?
Well,
we
we
are
gonna
lock
all
of
those
keys,
so
for
all
of
those
keys,
we're
gonna
tell
the
entire
system.
Just
don't
I'm
working
on
these,
don't
touch
them,
wait
all
right,
and
so
I'm
going
to
the
log
server
and
I
say
for
all
those
keys,
just
stop.
You
know
put
a
lock
on
this.
I
don't
want
anybody
to
work
on
these.
If
some
of
those
keys
were
already
locked,
then
I
just
don't
work
with
them.
A
I
ignore
them
and
well
I
I
wait
until
those
are
available
or
just
put
the
message
back
in
the
message
queue
then.
What
do
I
need
to
do
I
need
to.
So
is
the
typical
double
locking
thing?
So
I
don't
know
if
you
know,
if
you
guys
have
worked
with
a
distributed
system
before
the
way
you
do
lock
on
the
value
that
you
want
to
update.
You
need
to
read
it.
A
If
you
figure
out
that
you
want
to
update
it,
you
have
to
lock
it,
you
have
to
read
it
again
and
then
you
can
update
it.
The
reason
is
that
between
that
time
I
know
it
doesn't
look
like
much.
It's
got
pulled
lines
of
code
right,
but
during
that
time,
when
you
write
it
and
then
you're
going
to
lock
it,
there
could
be
another
system
that
actually
updated.
A
So
you
always
want
to
do
another
read
of
your
data
right
after
you
lock,
so
we
do
another
pass
into
memcache
of
all
those
keys.
We
do
a
multi-get
to
make
sure
we
have
the
current
value
in
that.
If
anything
had
been
updated,
then
the
queue
was
now
a
match.
We
know
that
and
we
know
that,
since
it
was
locked,
it's
not
going
to
happen
again
so
we're
safe.
A
Now
we're
going
to
do
a
batch,
get
on
all
of
those
keys
in
cassandra.
We're
going
to
get
all
of
the
values
that
were
stored
and
we're
going
to
do
that
read
using
we
actually
use
a
quorum.
If
you
want
to
make
sure
that
you
want
to
do
a
read,
that
is
everything
in
the
system.
You
could
do
an
all
on
your
read
to
make
sure
that
you
cost
all
the
the
cluster,
but
supposedly
quorum
should
be
good
enough
to
have
an
accurate
value.
A
So
we
do
a
quorum
read
on
all
those
things.
Now
we
have
the
actual
data
that
came
from
cassandra.
We
can
increment
all
of
those.
We
can
set
them
back
in
memcache,
but
this
is
not
an
increment.
This
is
a
set,
but
this
key
here
is
the
current
value
and
I'm
gonna.
I'm
gonna
unlock
all
this,
this
thing
and
this
thing
and
let
everybody
else
run,
and
I
do
on
those
I
set
the
timestamp
to
the
exact
to
the
exact
value.
When
I
unlocked
each
of
those
keys.
Okay,
you
guys
following
me:
okay,.
A
So
yeah,
but
keep
in
mind
that
a
lot
of
those
are
reused
because
it's
an
aggregate
right
so,
even
though
you
might
have
a
key
that
might
be
a
combination
of
you
know,
people
who
view
that
site
on
mywebsite.com
from
france
that
key
is
going
to
be
reused
within
that
channel
for
future
people
who
come
from
that
website.
You
say
I'm
saying
so,
but
but
how
many
keys
are
we
storing?
A
A
Of
your
data
that's
actually
stored
in
cassandra,
it's
the
data,
that's
being
incremented.
After
a
while
when's
not
worked
on
that
video
is
not
being
used
or
whatever
is
gone
right.
It's
gone
from
memcache
just
in
cassandra.
What
I
can
tell
you
is
in
an
actual
gigabytes.
I
mean
right
now.
I
think
we've
got
a
100
200
gigabytes
of
data
and
those
are
only
in
cassandra
in
aggregates
the
memcache
cluster
is
actually
pretty
small.
We
only
work
with
eight
gigs
of
data
right
now.
A
D
A
Actually,
our
values
take
no
space
at
all.
The
entire
space
taken
by
sandra
is
only
in
the
keys,
because
the
keys
are
pretty
long.
I
expect
the
key
structure
in
the
next
slide,
so
we
unlock
our
values
and
then
I've
got
both
the
ones
that
were
correctly
aggregated,
the
ones
that
had
to
be
fixed
from
cassandra
unlocked
in
all
that
stuff,
and
I
do
a
batch
right
on
all
the
final
values
to
cassandra
and
I
can
go
on
and
go
on
with
my
life
and
process.
The
next
message.
C
A
C
And
locks
right,
I
was
just
thinking
that,
if
you,
if,
if
the
lock
and
the
values
associated
with
say
a
particular
key
or
part
of
the
key
like
the
prefix,
you
know
if
it
hashes
to
a
particular
m
cache
instance.
You
can
work
all
within
that
instance.
Then
your
failure
modes
and
time
synchronization
issues
become
less
severe.
So.
A
A
I
don't
know
how
they
call
that,
but
like
partitions
within
your
memcache
cluster,
and
so
we
use
that
so
we've
got
a
lock
partition
and
a
counter
partition
right
and
between
those
two
they're
not
going
to
override
each
other
right
and
for
hashing.
We
use
tiger
hashing
to
in
order
to
prevent
collisions
as
much
as
possible.
A
Tiger
tiger
hashing
is
just
a
hashing
algorithm.
I
didn't
write
it.
It
was
just
the
the
client
that
we
use
well
one
of
the
clans.
We
used
had
some
example
of
tiger
hashing,
and
then
we
used
that
to
actually
preprocess
all
of
the
keys
to
make
sure
that
they're
short
enough.
We
have
complete
control
over
them,
and
so
we
make
sure
that
they
they
will
fit
in
the
memcache
key
space,
and
so
we
can
actually
trick
it
by
keeping
some
elements.
So
so
memcache
itself
is
not
going
to
hash
the
keys
right.
A
So
it's
going
to
keep
the
value
and
we
we
actually
use
prefixes
and
things
like
that
that
are
outside
of
the
hash
in
order
to
make
sure
that
some
some
of
the
key
pieces
are
not
cannot
collide
because
they're
actually
different
on
a
few
characters,
but
we
we
have
not
have
any
collision
issues.
So
far,
does
that
answer
your
question?
A
A
It's
a
separate
cluster,
so
we've
got,
we've
got
other
ec2
instances,
it
actually
runs.
On
the
same.
I
don't
want
to
say
something
stupid.
We
used
to
have
them
on
the
cassandra
cluster
and
because
you
could
run
them
on
separate
servers
right
so,
first
off
they
don't
have
to
be
on
the
same
machines.
It
was
just
a
cost,
efficient
way
of
doing
things
and
we
we
actually
do
have
the
instances
on
the
workers
and
the
reason
why
we
do
that
is
twofold.
One
is
now.
A
A
A
A
D
A
There's
two
reasons:
I
don't
have
anything
against
them.
The
first
reason
is
really
dumb:
it
didn't
exist
when
we
wrote
it
so
there
was
that
but
and
then
we
we
thought
about
just
moving
away,
because
we
I'd
love
to
get
rid
of
this
right,
yeah,
just
rid
of
the
distributed
cameras
and
locks
and
all
that
stuff
and
just
switch
to
center,
but
because
we're
so
we've
got
an
event
model
around
those
processors
that
is
in
c
sharp
and
listening
to
those
events
and
those
counters
and
based
on
certain
rules
that
the
users
set.
A
We
can
use
those
triggers
as
notifications,
so
we
rely
on
the
app
actually
having
an
event
saying
you
know.
This
metric
has
been
updated
and
with
this
value-
and
you
know
in
the
same
process,
to
actually
trigger
those
events
and
put
a
new
message
in
a
different
queue.
If
you
do
them
cassandra,
you
don't
have
an
event.
You
know
you,
because
you
never
read
the
value
back.
A
A
A
Yeah
so
kind
of
to
finish
this
yeah.
I
wrote
this
this
afternoon,
but
I
think
that's,
it
would
be
actually
the
opinion,
but
for
us
to
just
ignore
bullet
point
number
one.
We
would
not
want
to
use
composite
columns
and
I'm
going
to
explain
to
you
why,
because
I
was
I
was
going
through
the
architecture
reading
the
code
and
trying
to
remember
why
I
wrote
this
this
way
and
then
I
thought
oh,
could
we
use
composite
columns?
Then
you
know
the
other
time
that
we
could
do
it.
A
That's
the
preferred
way,
and
I
thought
about
it-
it
wouldn't
work.
The
reason
is
so
the
way
we
structure
the
row
keys
are
you've
got
a
channel
id
video
id
campaign
id
period
being
what
I
call
period
is
day
week
month,
the
date
which
will
be
the
start
date
of
the
period,
the
dimension,
the
subdimension,
the
dimension
being
the
country
city.
A
That's
going
to
be
composer
for
channel
id,
nothing
for
the
video
id,
nothing
for
the
campaign,
ide
and
if
I
want
all
time,
nothing
for
the
period
and
nothing
for
the
date.
If
it's
all
time
and
then
for
dimension,
I
just
wanted
to
see
a
list
of
cities.
So
I
just
put
city
in
here
and
nothing
for
the
sub
dimension.
A
The
super
column
name
structure
show
below
the
row
key.
The
super
column
is
going
to
be
named
using
the
dimension
value
and
the
sub
dimension
value.
What
that
means
is
that,
if,
let's
say
I
wanted
to
run
a
filter
on
videos
that
were
viewed
on
my
wordpress
website
named
stefan
bluegay.com-
and
this
is
said-
you
mentioned
I'm
from
france-
I
want
to
see
who
viewed
it
from
our
home
country.
A
I
can
just
run
a
get
on
that
super
column
and
get
all
the
columns
within
it,
and
the
columns
are
going
to
be
my
metrics,
so
my
columns
are
going
to
be
bandwidth,
views
place,
completions
all
those
metrics
that
we
track
and
then
the
value
for
each
one
within
it
right.
So
that
allows
me
to
do
things
like
you
know.
If
I
want,
I
want
to
know
specifically
people
who
come
from,
I
want
to
know
all
the
countries
of
people
who
went
to
my
web,
who
viewed
my
video
through
stephanie.com.
A
I
could
run
a
get
slice
on
cassandra
on
the
super
column
right
by
saying
I
want
to
know
so.
It's
a
filter
by
by
website
by
embed
site
and
by
country,
and
I
want
to
run
slice
where
the
key
actually
starts
with
the
url
of
the
website.
So
stephanie.com
and
I
don't
pass
the
submission
value
because
I'm
not
looking
for
just
one
data.
I
want
to
know
that
entire
table
of
data
right
and
so
the
slice,
because
these
are
ordered
sequentially
based
on
a
on
just
string
values.
A
F
A
A
That
only
has
a
channel
id
one
key
that
has
a
channel
id
video
id
and
a
create
a
key
that
has
channel
id
video,
I'm
id
to
create
keys
for
each
of
those
figures,
with
different
periods
and
different
dimensions
and
different
sub
dimensions,
and
then
within
those
I'm
going
to
loop
through
all
of
the
possible
values
and
do
my
aggregates
and
do
that
batch
right
right
to
to
cassandra.
But
it's
important
for
me
to
have
a
structure
that
does
two
things
one
is.
A
I
need
to
be
able
to
find
that
value
so
based
on
the
type
of
report
I
know
so
the
key
with
that
is
that
you
have
to
know
what
reports
you're
going
to
be
pulling
up.
So
this
is
not
ad
hoc.
This
is
for
ad
hoc.
You
would
use
something
like
hadoop
and
do
big
data
queries
on
that,
and-
and
that's
great
I
mean
that
that's
awesome.
A
A
France,
well
the
way
you
do
that
is
you
actually
switch
them
so,
instead
of
website
being
here
and
country
being
here,
you
put
the
country
here
on
the
website
here,
and
the
country
is
going
to
be
the
first,
the
beginning
of
that
column.
Super
column
names.
Okay,
so
I
can
kind
of
query
my
data
in
a
really
easy
way
for
reports
that
requires
very,
very
little
weeds
to
actually
render
all
this
stuff,
but
still
allows
those
reports
to
be
dynamic
and
interactive.
B
A
What's
really
easy
to
do
using
this
is
to
just
introduce
it
in
the
list
right.
So
now
I
have
country
city
whatever
and
street
address,
and
in
your
code
you
just
add
it
to
that
list
of
things
you
care
about
and
in
in
the
code
itself
it
requires
no
change
and
you're
ready
to
go.
I
just
need
to
republish
my
code
right
because
I'm
looping
through
those
names-
and
I
know-
and
then
I
have
what
I
call
a
dimension
resolver,
I'm
not
running
my
laptop
here.
A
So
I
can't
go
in
the
code
to
show
you,
but
the
dimension
resolver
you
just
pass
it.
The
name
of
the
dimension
looks
at
the
message
and
say:
where
are
you
looking
for
the
country?
The
country?
Is
this
and
comes
back
right,
so
in
your
dimension,
resolver,
you
just
need
to
have
one
line,
saying
what
what
are
you?
Where
are
you
trying
to
handle?
That's
the
street
address
or
the
street
addressing
that
value
in
that
message.
A
I
can't
remember
what
I
said:
16
dimensions
or
something
right,
so
you
have
to
multiply
the
2645
increments
by
16
right,
because
now
you've
got
three
levels
of
dimensions
and,
to
be
honest,
after
we
saw
some
usage,
there
are
some
people
who
use
the
drill
down,
but
very
few.
It's
really
nice
for
demos.
People
love
it,
but
but
the
idea
of
having
a
second
drill
down
is
just
not
something
that
anybody
cares
about.
A
If
they
did,
that
would
require
quite
an
increase
in
our
capacity
standard
capacity
and
how
much
storage
we
would
have,
because
it
would
be
16
times
more
yeah,
not
worth
it
yeah.
So
I
I'm
going
to
go
back
into
why
I'm.
I
was
completely
wrong
about
the
composite
columns
in
this
scenario.
Here
nice
thing
about
this,
so
for
our
keys
we
use
our
keys,
are
randomized
right,
which
seems
to
be
the
best
practice
for
cassandra.
A
You
don't
want
them
sequenced,
because
that
makes
it
easier
using
the
the
randomization
of
the
keys
to
scale
your
cluster
up
and
down
and
not
have
an
issue
where
one
is
overloaded
and
the
other
one
is
empty
right.
So
not
having
your
key
sequence
is
the
best
thing
to
do
if
you
know
that
you're
going
to
scale
up
skinning
up
at
some
point,
the
the
problem
with
that
is
that
you
can't
do
a
slice
get
on
your
keys.
So
you
can't
say
I
want
my
keys.
A
That
start
with
this,
and
you
know
this
dimension
and
then
so
imagine
if
I
was
your
composite
keys
at
the
end
of
this,
I
would
actually
have
the
actual
dimension
value
and
the
subdimension
value
so
move
the
super
column
name.
Data
structure
backs
here
into
the
key
and
then
just
use
columns
and
just
use
a
standard
cassandra
column
family.
To
do
this
and
no
super
columns
right
the
primary.
If
I
can't
slice,
is
you
know
how
do
I?
A
How
do
I
get
that
slice
of
data
that
I
had
based
on
a
country
where
I
get
a
bunch
of
rows?
But
I
can't
so
the
way
we
would
do
that
would
be
to
actually
have
use
those
as
columns
and
then
all
of
those
will
point
at
the
value
to
another
row
in
a
different
table.
So
this
would
be
basically
an
index
and
the
increments
would
be
located
in
a
different
column
family.
A
So,
instead
of
using
super
columns,
you
would
have
to
use
two
column,
families
that
are
standard
one
with
the
composite
keys,
pointing
to
the
row
actually
containing
the
data
and
then
actually
go
to
the
data
with
another
multi-get
to
read
all
of
your
rows
and
when
we
started
messing
around
with
with
ordering
and
indices
that
we
wanted
to
write
on
our
own.
It
was
just
the
performance
was
not
good
enough,
so
we
stuck
with
this.
A
We
actually
had
pretty
good
performance
using
the
super
columns
and
using
slice
kits
because
now
now,
when
we
do
the
report,
we
actually
need
to
do
only
one
get.
We
don't
have
to
do
two
where
we
get
the
keys
first
and
then
we
get
the
data
so
now
you're
thinking
well,
yeah,
that's
great.
But
how
do
you
like?
If
I
you
know
if
I
showed
you
that
that
table
I
showed
you
earlier?
You
can
actually
sort
on
stuff,
so
you
can
sort
a
number
of
views,
completions
and
all
that
stuff.
A
As
you
probably
know,
cassandra
doing
that
cassandra
out
of
the
box
requires
you
know
putting
indices
on
your
on
on
your
columns
and
after
a
certain
amount
of
data.
It's
not
the
greatest
thing
to
do.
I
think
I
mean
last
time
I
checked
granted
recently.
I
haven't
really
been
up
to
date
on
cassandra.
I
don't
know
what's
going
on
right
now.
I
know
that
at
the
time
there
were
problems
with
second
readinesses
and
performance
when
you
started
dealing
with
tens,
hundreds
of
billions
of
rows
right.
A
Every
time
you
update
an
aggregate,
your
index
needs
to
be
updated,
and
it's
just
you
know-
and
I
don't
know
if
you've
tried,
maintaining
secondary
indices
and
cassandra,
but
it's
kind
of
a
pain
in
the
butt,
and
so
the
way
we
do
this
is
actually
on
the
reporting
engine
we'll
create
all
those
like
registered
memory.
So
we
just
get
all
the
data
we
need.
A
This
is
cut
at
a
low
level
enough
that
the
the
level
is
low
enough
that
we
can
get
all
the
data
we
need
and
then
process
all
those
calculations
in
memory
at
runtime
when
we
actually
run
the
report.
So,
for
example,
if
you
want
to
report
of
you
know
what
is
the
total
aggregate
of
people
who
played
something
from
france
between
the
15th
of
january
and
today,
I'm
actually
going
to
run
a
multi-get
on
all
of
those
keys
right.
A
So
I'm
going
to
have
all
those
aggregate
values
that
are
already
cut
for
me
by
day,
and
I
just
need
to
do
that.
On
the
you
know,
getting
30
keys
back,
create
the
aggregate
memory
and
just
give
it
to
you,
and
if
I,
if
I
want
my
thing
to
be
sorted,
so
I
want
all
of
those
values,
you
know
taking
the
example
of
a
list
of
countries
and
I
want
those
organized
by
completion.
A
I'm
just
going
to
get
the
entire
data
from
here,
the
entire
list
of
countries
with
sandra
and
sort
them
and
display
them.
So
and
that's
fine
for
us,
because
we
don't
have
any
dimension
where
the
actual
volume
of
data
the
possibilities
within
those
dimensions
is
huge,
we're
talking
in
the
hundreds,
maybe
in
the
thousands
right,
but
we
don't
have,
for
example,
millions
of
embed
websites,
there's
a
finite
number
of
countries,
finite
number
of
cities.
Finite
number
of
you
know
isps
that
are
going
to
be
presenting
your
data.
E
A
So
I
would
not
do
it
on
my
own
because
I
mean
I
remember
when,
when
those
guys
were
building
him,
it
was
a
major
pain.
But
I
mean
if
you
look
at
the
history
of
the
feature
and
the
amount
of
bugs
that
they
had
and
the
problems
with,
because
the
first
version
was
relying
on
a
vector
vector
clock
that
they
were
on
a
lock
server.
Because
of
you
know
the
problem
of
incrementing
data
and
all
that
stuff
and
they
finally.
A
A
A
So
we
run
that,
but
because
we
do
very
little
deletion,
we
actually
don't
delete
stuff
much
because
for
us
it
doesn't
cost
us
much
to
just
put
more
data
on
it,
and
so
we
don't
because
we
don't
do
much
deletion.
We
don't
have
to
do
the
compaction
very
often
because
that's
primarily
to
delete
to
get
rid
of
your
your
keys
in
order
to
save
space
space
is
cheap.
So
we
don't
really
bother
with
that
in
terms
of
the
nodes
being
in
maintenance
or
crashed
or
doing
compaction,
or
you
know
things
like
that.
A
We
we
just
decrease
the
number
of
when
we're
going
to
maintain
the
cluster
and
start
doing
work
on
it,
we're
going
to
decrease
the
number
of
threads
running
so
that
it's
not
so
heavy
on
the
on
the
cluster
and
then
we'll
increase
it
back.
That's
pretty
much
what
we're
doing
yeah!
I
read
a
paper
recently
about.
A
F
A
You
can
actually
have
supposedly
cassandra
perform
at
the
level
of
memcache
and
make
sure
that
a
lot
of
your
data
is
actually
already
there
in
memory,
in
which
case
retrieving
a
value
based
on
a
key
should
be
about
the
same
speed,
because
I
mean
the
latency
on
that
is
going
to
be
primarily
network
based.
You
know
the
actual
process
of
getting
that
from
memory,
whether
it's
c,
like
memcached
or
java,
like
cassandra,
is
not
going
to
matter
much.
D
A
So
you're
familiar
with
super
columns
and
columns
and
cassandra,
so
your
column
names
are
going
to
be
things
like
bandwidth
view
play
completion
right.
Those
are
the
those
are
the
pieces
of
the
message
that
I'm
going
to
be.
Those
are
the
kinds
of
events
that
are
happening
from
the
user.
Those
are
what
I
call
my
metrics
right.