►
Description
Speaker: John McCann, Senior Director of Applications Engineering at Comcast
SlideShare: http://www.slideshare.net/planetcassandra/nyc-tech-day-using-cassandra-for-dvr-scheduling-at-comcast
Comcast is developing a highly scalable cloud DVR scheduling system on top of Cassandra. The system is responsible for managing all DVR data and scheduling logic for devices on the X1 platform. This talk will cover the overall architecture of the scheduling system, data model, message queue and notification software that have been developed as part of this ambitious project. We'll take a deep dive into the details of our data model and review the implementation of Comcast's open-source, Cassandra-based clones of Amazon SQS and SNS.
A
A
So
the
idea
is:
we've
taken
everything
that
used
to
live
on
the
set-top
box
and
we've
moved
as
much
of
the
software
into
the
cloud
as
possible,
and
we
reinvented
the
user
experience
to
make
it
more
engaging
and
make
content
easier
to
discover
and
consume.
So
basically,
it's
called
the
x1
platform
for
xfinity
TV
and
it's
the
next
generation
xfinity
TV
experience.
It's
basically
a
brand
new
line
of
set-top
boxes
and
a
new
way
of
approaching
delivering
cable
software.
A
A
It
was
all
running
on
one
box
that
we
would
deliver
in
trucks
to
people's
houses
and
much
of
that
software
was
tightly
coupled
with
the
hardware
that
it
was
running
on.
So
it
made
it
really
difficult
for
us
to
update
software
frequently
try
new
experiences
with
certain
types
of
users,
and
what
made
matters
worse
was
that
the
hardware
that
we
gave
to
people
was
fragmented
in
terms
of
markets,
because
different
cable
equipment
was
available
in
different
market.
So
what
you'd
end
up
with?
Were
situations
like?
A
Oh,
we
have
this
great
new
guide
that
we
want
to
roll
out.
Oh,
but
it
doesn't
work
in
the
Atlanta
market
because
they
have
you
know,
cable
boxes
that
were
built
by
Motorola
instead
of
scientific
atlanta.
These
are
the
type
of
challenges
that
we
have
been
working
on,
overcoming
over
the
past
three
years.
So
what
we've
done
is
we?
Basically,
we
architected
the
software
that
runs
on
the
set-top
box
so
that
it's
entirely
decoupled
from
the
hardware.
It's
running
a
distribution
of
linux
that
we
have
put
together
on
the
device.
A
Drivers
are,
of
course,
given
to
us
by
the
hardware
manufacturers,
but
basically
everything
else
is
part
of
what
we're
calling
the
RDK.
It's
actually
something
we're
open
sourcing
to
select,
I,
guess,
cable
partners,
I,
believe
a
portion
of
it
is
already
open
sourced.
If
you,
if
you,
google,
Comcast
rtk,
but
it's
a
it's
really
exciting,
because
what
it
does
is
allows
us
to
run
a
thin
layer
of
software
on
on
the
set-top
box
and
then
move
every
everything
else
into
the
cloud.
And
basically,
what
we're
doing
is.
A
We've
got
a
platform
that
can
deliver
the
existing
cable
experience
in
terms
of
all
of
the
on-demand
content,
all
the
DVR
content
and
all
the
paper
mute
content
that
we
have
today,
except
with
the
brand
new
user
interface
and
running
on
brand
new
hardware
and
brand
new
software
and
we've
essentially,
basically
taking
out
taken
out
all
of
the
tight
coupling
between
the
devices
and
the
software
we've
also
decoupled.
All
of
our
services
that
are
running
in
the
cloud
so
that
you
know
individual
services
are
providing
specific
roles.
A
A
So,
let's
talk
a
little
bit
about
why
or
so
what
what
we're
doing
with
Cassandra
and
then
we're
gonna
talk
a
little
bit
about
why?
Why
we're
doing
it
so
we're
storing
customer
viewing
history?
This
lends
itself
well
to
Cassandra,
because
every
time
a
user
is
flipping
channels
and
they
form
a
channel
change.
We
keep
track
of
that.
This
isn't
necessarily
for
business
intelligence
purposes.
This
is
more
for
being
able
to
present
a
portion
of
the
user
interface
that
shows
the
user,
the
content
that
they
viewed
recently.
A
A
Netflix
is
a
little
bit
ahead
of
us
I
think
they
store
every
two
seconds
and
their
evidence
and
write
and
do
it
so,
but
we're
working
on
basically
scaling
our
system
so
that
we
can
store
resume
points
more
and
more
frequently.
We
store
we
store
device
and
account
preferences.
These
are
more
read.
Heavy
services,
but
they
also
lend
themselves
well
to
Cassandra,
because
we
like
the
like
the
fact
that
there's
no
structured
schema,
so
we
can
basically
treat
device
and
account
preferences
like
property
bags
like
key
value
stores.
A
If
we
have
to
add
something
new,
it's
you
know
really
trivial,
just
add
that,
as
a
new
column,
pay-per-view
purchases,
these
aren't
as
transactional
as
you
might
think,
in
terms
of
getting
charged
and
all
the
things
that
have
to
happen
on
your
accountant
and
and
and
on
your
bill.
That's
all
handled
by
the
cable
card.
A
A
We
like
the
fact
that
it's
multi
data
center,
the
thing
about
cable,
is
that
everyone
expects
it
to
be
working
all
the
time
and
if
it
doesn't
work,
the
first
thing
to
do
is
they
pick
up
the
phone
and
they
call
the
call
us
and
complain.
So
it's
really
important
that
we
can
be
resilient
to
all
sorts
of
outages,
including
full
on
data
center
outages,
which
have
happened
since
we've
started
with
with
this,
with
the
service
in
production.
A
A
We
like
the
fact
that
it's
that
it
works
well
on
for
four
use
cases
where
we're
writing
a
lot
of
data,
frequently
and
I
mentioned
a
few
a
little
bit
earlier,
but
DVR
status
updates.
So
if
you
imagine
the
Super
Bowl
or
the
Oscars
right
most
likely
we're
gonna
have
a
lot
of
users
recording
the
same
shows
it
was
popular
shows.
You
know
all
at
the
same
time
the
shows
going
to
start
and
the
box
is
going
to
send
a
message
into
our
DVR
system.
Saying
hey
I
started
this
recording.
A
We
need
to
persist
that
right.
We
need
to
persist
the
recording
status
for
all
of
these
devices.
So
you
know
if
we
have
millions
of
devices.
That
means
we
have
millions
of
rights
arriving
at
our
system
all
at
the
same
time,
we
need
to
be
able
to
sustain
those
rights.
They
don't
have
to
necessarily
make
their
way
down.
You
know
and
and
percolate
to
all
the
data
centers
immediately,
so
eventual
consistencies
an
okay
outcome,
but
it
is
important
that
we
don't
lose
data,
and
so
that's
that's
one
of
the
right
heavy
use
cases
that
we.
A
You
know
that
we
hope
to
be
able
to
you
know,
take
advantage
of
Cassandra
for
and
resume
points.
I
mentioned
that
as
well.
You
know
during
content,
playback,
you
know
in
primetime
a
lot
of
people
are
watching
watching.
Dvr
content
to
a
lot
of
resume
points
are
coming
in.
We
need
to
be
able,
to.
You
know,
have
a
system
that
can
scale
to
to
the
to
the
number
of
rights
that
we
you
know
when
all
the
users
are
watching
DVR
vaad.
At
the
same
time.
A
Another
thing
we
like
is
that
developers
instead
of
dbas
are
modeling
and
managing
the
data.
Maybe
this
is
more
of
a
organizational
thing
that
we
had
with
within
comcast,
but
one
of
the
challenges
we
had
the
mezuzah
divide
between
the
dbas
and
the
development
team,
and
we
think
that
cassandra
is
helping
to
alleviate
that,
because,
essentially,
the
developers
are
now
ultimately
responsible
for
the
data
modeling,
making
sure
that
the
data
is
structured
in
a
way
that's
performant
for
the
system.
A
You
know
it's
just
I
feel
like
it's
a
much
more
transparent
development
process
and
it
puts
more
responsibility
on
the
development
team,
but
it
also
removes
the
sort
of
notion
of
this
database
that
was
kind
of
we
were
using
Oracle
and
very
much
acted
like
a
black
box,
sometimes
where
it
felt
that
way
and
this
one's
a
little
fuzzy.
But
basically
the
idea
here
is
that
I
don't
know
this
may
be
a
personal
opinion,
but
I
feel,
like
the
open-source
nature
of
Cassandra
kind
of
encourages
in-house
expertise.
Think
about
a
big
company.
A
Is
you
get
a
lot
of
consultants
and
Oracle?
When
is
just
the
type
of
technology
that
would
lend
itself
well
where,
if
there's
a
problem,
hey
calling
the
consultants
and
they'll
come
in
and
fix
it
well,
they'd
come
in,
and
you
know
help
you
out
for
for
a
few
months
and
then
they're
gone,
and
we
just
really
didn't.
A
Have
the
in-house
expertise,
in
my
opinion,
especially
with
on
the
development
side,
to
really
be
able
to
maintain
a
quality
of
service
that
we
that
we
really
need
so
I'm
going
to
be
talking
specifically
today
about
the
work
we've
done
for
the
DVR
on
the
excellent
platform,
the
whole
user
experience
has
been
reinvented,
the
DVR
in
particular,
and
it's
been
it's
been
sort
of
redesigned
not
only
from
a
user
experience
standpoint,
but
also
from
a
technology
standpoint,
architectural
II.
We
basically
have
a
system,
that's
really
not
that
surprising
or
complicated.
A
We
have
a
bunch
API
servers.
They
just
expose
rest
pole
interfaces
for
interacting
with
the
DVR
data.
You
know
they
provide
data
to
the
user
interface
and
they
provide
recording
instructions
to
the
set-top
box.
We
have
a
bunch
of
worker
services
that
are
responsible
for
calculating
recording
angels.
So
really,
in
this
case,
the
input
is
all
of
the
stuff
that
the
user
wants
to
record.
These
could
be
things
like.
Oh
I,
just
want
to
record
this
show.
Just
you
know
one
off
show
because
I
hit
the
record
button.
A
These
could
be
series
recordings
I
like
to
show
you
know.
Whatever
show
you
want
to
record,
you
know
Tribble
out
my
wife
likes
True,
Blood,
youporn,
true
blood,
and
you
get
all
those
episodes.
You
can
record
personalities.
So
you
know
if
you're
a
Tom
Cruise
fan
you
you.
Can
you
record
that
and
basically
the
workers
need
to
figure
out
when
all
of
the
contents
of
on
that
has
that
particular
actor
or
actress,
and
then
sports
teams
as
well.
A
So
you
put
in
your
favorite
sports
teams
and
the
DVR
system
essentially
goes
out
and
figures
out
all
the
games
that
that
team
is
playing
in
and
schedules
as
recording.
So
that's
kind
of
a
pretty
heavy
duty,
type
type
of
task,
and
so
we've
sort
of
segmented
that
work
off
to
talk
to
these
worker
processes
that
basically
have
a
queue
and
their
whole
job
is
to
drain
the
queue
and
read
in
a
set
of
recording
instructions
which
has
been
aggregated
over
time
by
user
setting
preferences
and
setting
recordings
and
changing
their
recording
options.
A
And
then
it
just
outputs
a
recording
schedule
actually
go
back
for
one
second,
a
couple
other
things,
so
we
have
at
the
moment
we
have
two
data
centers.
We
intend
to
expand
to
more
as
we
roll
the
product
out
to
other
markets
and
in
our
in
our
user
base
grows.
We
have
obviously
Cassandra
running
a
ring
in
both
data
centers
and
we're
using
and
we're
using
a
network,
topology
network
topology
strategy
to
replicate
that
data
across
data
centers
right
now
our
rings
are
relatively
small.
Part
of
the
reason
for
that
is.
A
We
have
some
pretty
hefty
hardware,
we're
using
SSDs
I'll
get
into
the
hardware
stuff
later
right.
Now
we
have
four
node
rings
in
two
data
centers.
Our
intention
is
to
probably
expand,
we'll,
probably
double
in
size,
I,
anticipate
that
we'll
probably
doubled
in
size.
28
no
drinks
once
we
you
know
start
to
put
some
real
load
on
the
system
are
used
with,
is
initially
going
to
be
pretty
limited,
but
we're
expecting
a
pretty
substantial
growth
in
users
cutting
over
to
this
platform
by
the
end
of
the
year,
we're
using
network
topology
strategy
for
our
replication.
A
Basically,
we
write
to
one
data
center.
It
gets
gets
replicated
over
to
the
other
data
center.
Currently,
replication
factor
is
set
to
34,
pretty
much
all
of
the
column,
families
that
are
used
for
DVR,
so
basically
three
copies
of
the
data
in
each
data
center
in
local
quorum
for
most
operations.
So,
basically,
when
the
data
is
consistent
on
two
of
the
three
nodes
that
it's
going
to
be
eventually
consistent
on
the
operation
succeeds.
A
So
we
also
have
a
portion
of
the
system,
that's
used
for
queuing
and
met
and
notifications,
and
this
is
called
the
comcast
message-
bus
it's
developed
independently
from
the
DVR
system.
It's
going
to
be
used
throughout
all
of
the
systems
within
x1,
but
it's
actually
pretty
exciting
technology
that
we've
put
together
on
top
of
cassandra
that
I
want
to
talk
a
little
bit
about.
It
is
open
source,
so
you
can
download
the
source
from
github.
But
the
idea
is
it's
a
it's
a
horizontally
scalable,
queuing
and
notification
system.
A
So,
if
you
have
in
our
case
with
DVR,
we
have
a
queue
of
work
that
needs
to
be
drained
by
the
these
worker
processes.
We
have
an
implementation,
that's
completely
compatible
with
Amazon,
SQS
and
SNS,
so
the
queuing
system
is
compatible
with
SNS.
Sorry
at
with
SQS,
and
so
any
integration
work
that
you've
done
to
test
things
out
and
into
any
initial
prototyping
that
we've
done,
you
know
against
the
amazon
services
will
automatically
work
in
our
production
instances
of
Comcast
message:
bus.
A
The
notification
system
is
used
to
deliver
notifications
to
other
systems
that
are
interested
in
knowing
that
DVR
data
has
changed.
So
the
user
interface
is
a
good
example
of
that
lots
of
times.
User
interface
will
have
and
we'll
get
into
this
more
later,
but
the
user
interface
is
going
to
have
somebody
cached
representations
of
the
data
both
sort
of
on
screen
in
the
form
of
user
UI
elements
that
have
been
rendered
as
well
as
serialized
in
in
a
memcache
cluster.
The
notifications
are
important
to
allow
us
to
invalidate
that
data.
A
You
know
when
appropriate
and
then
notifications
are
also
used
to
deliver
messages
to
the
set-top
box
to
tell
it
that
a
recording
schedule
has
changed.
So
this
is
important
when,
if
you
go
when
you
said
a
series
recording,
though
the
worker
system
is
going
to
go
through
and
recalculate,
your
recording
schedule
is
going
to
have
to
deliver
that
schedule
back
to
the
set-top
box
and
the
set-top
box
is
going
to
have
to
use.
You
know
the
updated,
recording
schedule
and
sort
of
change
all
of
the
things
that
it
was
reporting
based
on
the
new
schedule.
A
So
we
persist
all
of
the
messages
in
in
Cassandra,
but
Redis
is
actually
used
as
a
caching
layer.
So
the
reason
we're
using
Redis
here
is
well
a
couple
of
reasons.
One
on
the
nature
of
this
type
of
service
is
that
it's
read
heavy,
so
we
need
to
have
very
good,
read
performance.
The
second
thing
is:
Redis
provides
a
lot
of
really
sort
of
clever
and
cute
little
data
structures
that
make
your
life
as
a
developer.
A
A
lot
easier,
particularly,
I
believe,
the
one
that
we
like
in
particular
here
is
is,
is
that
is
the
notion
of
a
list
structure
and
and
these
sort
of
type
of
high
level
structures
don't
really
exist
in
Cassandra,
and
our
attempts
to
implement
them
didn't
really
perform
as
well
as
we
would
have
liked.
So
what,
as
it
turns
out,
redness
is
kind
of
perfect
for
that,
and
the
system
can
operate.
Fine
without
redness
or
red.
Us
again
is
just
a
caching
layer.
A
If
all
of
our
Redis
nodes
go
down,
we
can
still
continue
to
operate
on
top
of
Cassandra,
but
but
but
Redis
is
there
and
it
actually
works
pretty
well
as
a
caching
layer
in
this
system,
like
I,
said,
is
completely
open
source
in
apache
license.
You
know,
comcast
isn't
necessarily
known
for
open
sourcing
saw
for
open
sourcing,
their
software,
but
we're
we're
changing
our
tune
of
it,
and
so
this
was
kind
of
the
first
real
project
that
we
underwent
that
we
have
fully
open
source.
A
Ok,
so
I'm
going
to
do
a
use
case
explore
for
basically
setting
a
series
recording,
so
the
user
goes
in
through
any
number
of
discovery.
Paths
to
find
a
particular
series
of
a
TV
show
that
they
want
to
record
and
they
set
that
recording.
So
it
all
starts
with
the
user.
Interacting
with
the
set-top
box,
you
know
with
a
remote.
The
set-top
box
has
a
one
terabyte
hard
drive
built
into
it.
A
The
set-top
box
is
going
to
be
responsible
for
actually
performing
the
recordings
and
storing
the
video
data
on
this
on
the
hard
drive,
and
it
needs
to
be
given
instructions
about
what
to
record
so
that
the
box
has
for
integrated
quam
tuners.
So
it
can
other.
Only
three
are
enabled
right
now
for
performance
reasons,
but
so
you
can
basically
watching
One
Show
and
reporting
two
things
at
a
time,
but
the
boxing
instructions
about
what
to
do.
It's,
a
dumb
box.
A
All
of
the
logic,
actually
lives
lives
in
the
cloud
so
on
anything
that
is
drawing
on
screen
comes
from
the
cloud.
It's
like
a
remoted
view,
I
system,
so
it's
got
a
persistent
TCP
connection
with
the
UI
server.
So
the
UI
server
sending
a
bunch
of
messages
down
all
the
time
to
tell
it
what
to
draw
when
the
user
pushes
a
button
on
the
remote.
The
set-top
box
sends
a
message
up
to
the
UI
server
and
you
know
tells
it
what
button
was
pressed.
A
You
I
server
responds
to
it
by
sending
a
bunch
more
commands
down
to
change.
What's
on
screen
the
closest
thing
that
we
have
found
out
there,
that
is
like
this
is
X
Windows
remoting,
but
it's
a
pretty
interesting
sort
of
lightweight
you
ID
system,
so
there's
a
very
thin
client
on
the
box,
and
this
allows
us
to
have
a
single
implementation
of
user
experience
that
can
work
on
a
variety
of
different
devices
with
different
capabilities,
because
the
UI
is
so
lightweight
from
a
set
from
the
set-top
box
side.
A
A
It's
also
going
to
be
responsible
for
fielding
requests
from
the
user
to
record
something
new
delete,
recordings,
change,
recording
properties
and
so
forth,
and
then
the
US
server
has
a
its
own
cache
of
the
data,
primarily
that's
just
there,
so
that
we
can
scale,
and
so
it's
not
that
we
don't
trust
Cassandra's,
read
performance.
It's
it
allows
us
to
reduce
the
number
of
HTTP
requests
going
over
the
network.
A
The
fact
of
the
matter
is:
there's
some
overhead
associated
with
HTTP
there's
some
overhead
associated
with
JSON
serialization
d
serialization,
and
all
these
things
you
know
we
want
to
try
to
reduce
as
much
of
that
overhead
as
possible.
So
we
do
use
a
mem
cache
to
cache
recordings,
and
things
like
that.
A
A
Worker
Q
is
implemented
in
comcast
message:
bus
as
I,
as
I
mentioned.
Basically
a
workers
just
going
to
pick
up
that
that
operation
off
the
queue
so
actually
skipped
a
step.
So
at
this
point
here
the
API
server
has
written
to
Cassandra
the
instruction
that
the
user
has
has
has
taken.
So
in
this
case,
if
they've
recorded
a
new
show
that
recording
instruction
is
captured,
cassandra
and
then
the
work
is
is,
is
is
pushed
on
to
the
work
queue.
So
the
work
is
popped
off
the
queue
the
worker
picks
it
up.
A
The
worker
now
reads
all
of
the
recording
instructions
from
Cassandra
and
basically
will
get
into
the
data
model,
but
it's
basically
just
a
sequence
of
instructions
that
were
performed
in
order.
It
was
recorded
two
series,
don't
record
the
series
record,
this
team
don't
record
this
team
change
the
recording
priority
of
this
team,
and
these
are
just
a
set
of
instructions
that
are
input
to
to
the
to
the
worker
process.
That's
going
to
basically
be
recalculating
the
recording
schedule,
recording
schedules,
we
calculated
it's
written
to
Cassandra
and
then
a
notification
is
sent
to
the
set-top
box.
A
This
tells
the
set-top
box
that
there
is
a
new
set
of
recordings,
a
new
recording
schedule
to
to
execute
so
now
the
set-top
box.
It
has
to
go
fetch
that
recording
schedule.
We
don't
want
to
send
the
whole
thing
in
the
payload
of
the
notification,
so
we
basically
just
say:
hey
something
changed,
and
so
now
the
set-top
box
is
going
to
go
out
and
fetch
the
the
updated
schedule
and
and
start
executing
in
the
schedule.
Just
looks
something
like
record.
You
know
this
show
on
this
channel,
starting
at
this
time
and
ending
at
that
time.
A
It's
just
really
nothing
more
than
a
list
of
of
instructions
that
are
prioritized.
That
say,
you
know,
start
time,
end
time
and
then
ultimately
channel
number,
and
then
the
set-top
box
knows
how
to
execute
that
recording
schedule
and
as
it's
doing
it,
it's
going
to
it's
going
to
send
HTTP
request
to
the
API
server
to
notify
it
of
reporting
status
updates.
So
you
know
when
the
recording
starts
when
it
completes,
if
it
fails
all
that
information
is
sent
to
the
API
server.
A
What
might
have
happened
when
it
was
down,
and
so
as
all
this
is
happening.
Well,
as
this
as
the
schedule
recalculation
has
been
delivered
to
the
set
top
box,
the
worker
process
is
going
to
notify
the
UI
server
in
the
form
of
a
notification
that
says
hey
for
this.
For
this
set-top
box,
we
have
a
new
recording
schedule.
A
The
scheduled
recordings
are
the
set
of
hoardings
that
are
in
progress
or
what
have
you
so
that's,
basically
how
the
how
the
system
works
in
a
nutshell,
we're
going
to
talk
a
little
bit
about
how
we've
modeled
the
data
specifically
there's,
there's
three
there's
three
primary
data
points
here
and
the
models
are
actually
a
lot
easier
than
then.
You
might
think
that
the
first
one
is
the
recording
instructions.
So
basically,
this
is
essentially
the
output
from
the
from
the
worker
processes.
Basically,
it's
I'm.
Sorry.
This
is
the
input
to
the
worker
processes.
A
This
is
basically
all
the
stuff
that
the
user
has
said.
They
want
to
record
its
things
like
TV
shows
individual
airings
of
episodes,
personalities
and
and
sports
teams,
and
things
like
that,
and
basically
it's
just
stored
sort
of
in
the
order
that
it
came
in
and
we
basically
kind
of
go
through
it
and
we
can
sort
of
recreate.
You
know
from
that.
We
can
we,
you
know
we
create
what
the
set-top
box
should
be
recording
right
now,
so
we
have
a
composite
roky
of
account
ID
device
ID.
A
So
this
lets
us
read
all
of
the
recording
instructions
in
a
single
operation
and
then,
basically,
we
have
column
names
of
time
uuid,
which
ultimately
allows
us
to
maintain
all
of
the
all
the
instructions
in
the
order
in
which
they
arrived.
And
then
the
the
value
of
the
columns
is
basically
just
the
instruction.
So
it's
just
a
payload
that
says
you
know
it's
just
some
object
that
says:
okay,
you
know
what
is
the
type
of
instruction?
Is
it
like
a
one-time
recording?
Is
it
deleting
a
series?
A
The
sort
of
next
column
family
is
essentially
the
recording
schedule.
This
is
the
output
of
the
recording
schedule.
Calculation.
Basically,
what
it
is
is
a
list
of
the
things
that
the
set-top
box
needs
to
record.
It's
basically
the
schedule,
that's
delivered
to
the
set
top
box
and
its
current
right,
so
it'll
be
basically
if
I
am
a
set-top
box
and
I
need
to
start
executing
recordings
right
now.
A
This
is
this
is
this
is
the
data
and
it
obviously
gets
updated
pretty
frequently
as
data
changes
as
recording
is
complete,
so
we
need
to
always
sort
of
keep
it
up
to
date.
Again
we
use
composite
roki
kind
of
device
ID,
so
we
can
sort
of
get
everything
on
once
and
then
each
property
of
the
recording
schedule
is
stored
as
a
column.
A
So
basically,
we'll
use
a
will
be
using
composite
column
names,
so
be
something
like
recording,
ID
slashed
and
then
the
property
name
for
the
column
name
and
then
for
the
value
it'll,
be
whatever
the
value
of
that
property.
Is
you
know
the
recording
status
or
the
recording,
ID
or
whatever
it
is,
and
this
allows
us
to
retrieve
properties
of
a
specific
recording
using
a
slice
query.
So
it's
pretty
inexpensive
operation
to
retrieve
all
the
properties
of
a
scheduled
reporting
update,
a
property
of
a
scheduled,
recording
or
retrieve
the
entire
recording
schedule.
A
This
model
allows
for
all
those
use
cases
in
a
pretty
performant
way,
and
likewise,
the
completed
recording
schema
is
exactly
the
same.
I
mean
it's
basically,
you
know
just
these
are
recordings
that
are
done
there
on
the
box
they're
completed
and
we
store
them
exactly
the
same
way
with
the
the
with
the
composite
columns.
A
In
terms
of
what
we're
using
for
hardware,
so
we
were
originally
running
on
virtualized
hardware
and
shared
storage.
We
strongly
suggest
that
you
don't
do
that.
We
are,
we
actually
I,
think
we
were
down
to
just
one
system
in
production
right
now,
still
running
on
shared
storage,
pretty
much
everything
else
is
on
commodity
hardware.
A
So
what
we
have
is
22
like
magnetic
you
know,
standard
spinning,
needle
hard
drives
for
the
operating
system
and
that's
going
to
be
raid
0,
sorry,
raid
1,
and
then
we
have
basically
solid-state
drives
for
the
Cassandra
data,
and
so
right
now
we
have
two
in
there
and
we
have
room
for
up
to
four.
So
what
we
found
was
the
onboard
controller
wasn't
really
fast
enough
to
keep
up
with
the
hard
drives.
So
we
basically
had
a
separate
lsi
controller
put
in
that
drives
just
the
samsung
solid-state
drives.
A
A
It's
a
little
bit
more
about
solid-state
drives
and
why
we
went
to
it.
So
you
know
that
they
are
expensive
right.
So
just
adding
those
drives
increase
the
cost
per
node
by
fifty
percent
on
that
sounds
pretty
substantial,
but
really
the
benefit
paid
off,
because
we
get
about
four
hundred
percent,
a
capacity
increase
per
node.
So
you
know
we
can
run
smaller
Cassandra
rings
and
we
can
end
up.
A
The
other
thing
about
the
solid
state
drives
that
we
really
liked
is
when
you
look
at
sort
of
the
some
of
the
performance
analysis
and
the
load
test,
we
did
lots
of
times
what
you'll
see
is
lots
of
95th
sand,
99s
kind
of
all
over
the
place
in
in
your
load
test
results.
One
thing
we
found
was
just
moving.
A
The
solid-state
draw
has
made
those
99
and
95th
percentile
lines
look
really
flat
and
it
really
changed
a
lot
about
how
we
look
at
95th
and
99
s,
because
now
things
are
a
lot
more
leveled
and
less.
You
know
all
over
the
place.
So
so
we're
pretty
happy
with
that.
So
we
learned
a
lot
of
lessons
moving
to
Cassandra.
We
really
didn't
have
much
knowledge
in-house
about
Cassandra
at
all.
When
we
started
with
this,
so
you
know,
and
and
and
honestly
the
learning
curve
was
relatively
steep.
A
A
That
would
that
we
came
across
on
so
this
is,
I
don't
know
if
this
is
necessarily
particular
to
Hector,
but
one
of
the
challenges
that
we
found
sort
of
initially
was
when
we
were
configuring,
our
the
max
number
of
connections
in
Hector's
connection
pool
first
off
whatever
you
is.
So
if
you
have
a
bunch
of
API
servers
that
max
active
value
applies
to
how
many
maximum
connections
can
be
open
to
each
node
in
the
cluster.
A
So
if
you've
got
an
eight
node
Cassandra
ring,
you
know
whatever
max
active
value
you've
set
on
your
on
your
API
server.
You
got
to
multiply
that
by
eight,
and
then
you
multiply
that
by
how
many
API
serves
you
have
and
that's
how
many
connections
are
currently
open
to
Cassandra.
Well,
why
this
matters
is
that
we
found
that
in
a
particularly
during
under
high
load,
we've
found
that
a
lot
of
open
connections
to
Cassandra
will
cause
Hector
to
think
that
the
certain
nodes
are
down.
A
So
what
we
ended
up
with
was
we
initially
configured,
and
this
was
a
huge
mistake.
We
initially
configured
our
API
service
to
have
a
max
active
value
of
150
and
we're
testing
against
an
eight
node
cluster
and
we
were
like.
Why
is
Hector
reporting
all
hosts
turn
down
we're
seeing
host
coming
down
and
up
like
all
the
time
in
the
in
the
logs
that
Hector
was
producing.
It
was
also
needed
because
there
were
too
many
connections
open.
So
it
sounds
like
an
obvious
one,
but
it's
really
not.
We
kind
of
just
learned
it
through
experience.
A
So
it's
not
thinking
the
manual
about
that.
You
know
what
I
mean,
so
we
we
reduced
it
I.
Think
right
now
we
run
a
value
of
20
so
that
that's
20
I,
think
that's
even
kind
of
high
for
probably
potentially
20
connections
per
node
I
think
we
need
to
actually
bring
it
down
a
little
further,
but
that's
a
that's
a
good
one!
I
don't
know
if,
what's
the
other
one
st
annex
would
have
the
same
problem,
but
we
certainly
were
burned
by
that
row.
A
Cash
got
us
in
a
lot
of
trouble,
rokosz
another
one
of
those
things
like
yeah
turn
it
on.
Why
wouldn't
we
want
a
row?
Cash
right,
we're
like
why
is
it
off
by
default?
You
know,
of
course,
we
want
to
row
cash.
No,
you
don't
always
want
it,
because
you're
going
to
suffer
performance
penalties
in
certain
circumstances.
A
A
So
what
we?
What
we
found
was
just
disabling
the
RO
cash
really
did
help
our
performance
a
lot.
Now,
that's
not
to
say
you
always
want
to
disable
it.
I
would
certainly
suggest
it.
If
you
are
reading
an
entire
row,
you
know
the
entire
row
it
wants
and
your
row
is
sort
of
bounded
terms
of
how
wide
it
can
get.
A
A
Cassandra
can't
compact
that
data.
So
what
happened
was
the
data
that
we
deleted
got
tombstones,
but
it
never
actually
got
compacted
so
certainly
on
a
minor.
The
the
minor
compaction
would
essentially
do
nothing
with
these
with
it
with
the
data
that,
even
when
we
ran
a
major
compaction,
if
there
were
pending
right
to
that
row,
the
major
compaction
would
skip
the
row
all
together.
So
what
we
ended
up
with
is
a
data
set
that
should
have
been
just
a
couple
of
gigabytes
ballooning
into
like
60-70
gigabytes
and
it
made
compaction
slow.
A
It
made
repairs
slow.
The
ops
guys
were
really
mad
at
us.
It
was
really
unpleasant,
and
so
it's
actually
we're
using
we're
on
one
dot
one
in
in
some.
In
most
of
our
rings,
we've
been
told
that
the
issue
is
better
in
12,
I,
guess,
I'll
believe
it.
When
I
see
it
I,
don't
I
didn't
actually
dig
into
the
specific
commit
that
the
supposed
to
have
addressed
the
issue,
but
it's
supposedly
better
than
one,
not
two,
but
the
whole
point
is:
if
you're
going
to
be
like
you
know,
deleting
a
row.
A
You
have
to
do
one
column
at
a
time.
If
you're
going
to
be
writing
to
that
row
immediately
again
so
row
deletions
are
fine.
If
you
don't
plan
to
write
to
that
row
again
any
time
soon,
but
they're,
not
okay.
If
you're
going
to
be
doing
immediate
rights
of
new
columns
to
that
row
really
burned
us,
we've
been
like
scheduling,
major
compaction
'he's.
You
know
basically
every
night
to
try
to
mitigate
the
problem.
A
Ultimately,
the
the
way
we're
getting
off
of
this
is:
we
have
a
brand
new
column
family
and
we're
migrant
and
obviously,
we've
repaired.
We've
fixed
the
the
code
to
not
delete
the
row
but
delete
individual
columns.
We've
got
a
whole
new
column,
family
and
sort
of
we're
doing
an
on-demand
migration.
Where,
when
a
user
attempts
to
read
this
data,
we
look
to
see
if
it's
in
the
new
column
family,
it's.
If
it's
not
there,
we
fetch
it
from
the
old
column
family.
A
A
Don't
don't
run
Cassandra
and
shared
storage
if
you
can
avoid
it
like
I
said,
we've
done
a
bunch
of
performance
analysis
and
everything
points
to
the
DL
360s
being
vastly
superior
in
terms
of
in
terms
of
a
hardware
setup
for
running
Cassandra,
and
that's
all
that's
basically,
all
I
have
I
will
open
up
to
questions.
Sure.
B
D
A
We've
been
we
started
about
a
year
ago.
Well,
we
started
about
a
year
and
a
half
ago
doing
some
sort
of
analysis
of
what's
out
there.
You
know
we
looked
at
things
like
HBase,
a
couple
other
options
and
most
of
the
team
was
leaning
towards
Cassandra.
We
started
coding
on
it
probably
about
a
year
ago.
A
The
first
surface
that
we
did
was
what
we
call
the
personalization
service
of
restoring
like
resume
points
in
last
hist,
the
viewing
history
we
had
about
three
implementations
of
that
before
we
really
gotta
write
just
from
a
data
modeling
perspective
and
just
getting
the
code
to
work
and
that
took
probably
three
or
four
months
and
a
smaller
organization
I
would
expect
things
would
go
much
more
quickly.
We're
pretty
sort
of
careful
about
well,
not
necessary
careful,
but
we
bet
our
process
out
a
lot,
but
I
would
say
that
it
was
it
was
it
was.
A
A
A
You
know
we
have
some
replication
factor
associated
with
that
data,
so
we
can
lose
any
number
of
worker
nodes,
any
number
of
Cassandra
nodes
without
actually
suffering
data
loss
of
the
items
in
the
queue
Oh
Redis
is
just
for
a
caching
layer
so
like
so
so
we
we
use
Redis
for
basically
to
improve
the
Reed
performance
of
the
work
queue.
It's
it's
a
it's
a
it's
a
cache
optimum,
it's
an
optimization
just
on
the
read
side
and
if
the
Fred
exposed
ow,
never
ething
still
works
I'll,
be
in
the
great
degraded
state.
It
still
works.