►
From YouTube: How Hulu Scales Services to Support 400 Million Plays: C* , Redis, and SSD-Based Hardware
Description
Check out the full Cassandra Week post here: http://www.hakkalabs.co/articles/hulu-scales-cassandra-redis-ssd
Hulu must scale web services to meet the demands of an ever growing number of users. During this talk, Matt Jurik discusses the initial use case for Cassandra at Hulu: the video progress tracking service known as HUGETOP. While Cassandra provides a fantastic platform on which to build scalable applications, there are some dark corners of which to be cautious. We will provide a walk-through of hugetop and some design decisions that went into the hugetop keyspace, our hardware choices, and our experiences operating Cassandra in a high-traffic environment.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
B
What
is
so,
I
don't
know
if
we're
pretty
well
known
at
this
point
more
so
than
a
couple
years
ago,
but
we
are
an
online
streaming
company,
so
we
provide
a
lot
of
premium
content
like
television
episodes
and
whatnot
and
movies
available
online.
So
we
have
a
lot
of
licensing
deals
with
our
content
partners
fox
abc
nbc.
So
that
way
you
can.
A
B
And
a
little
bit
more
about
the
tech
team.
Actually,
as
I
mentioned,
we
have
a
lot
of
services
that
we
develop,
we're,
try
and
embrace
the
service,
oriented
architecture
methodology,
and
we
also
follow
the
unix
philosophy.
So,
if
you're
familiar
with
it,
it's
pretty
much
that
we
try
and
keep
our
services
encapsulated
in
terms
of
functionality.
B
Don't
do
too
much
at
one
time,
just
kind
of
focus
on
a
core
problem
and
solve
it
above
and
going
along
with
that.
We
have
a
lot
of
small
teams
that
are
focusing
on
these
different
problems
and
making
sure
that
we
can
actually
get
a
really
good
solution.
That
scales.
Well,
has
you
know
good
performance
and
can
handle
the
traffic
and
going
and
going
along
with
that?
B
Also
we're
really
focused
on
using
the
right
tool
for
the
job
and,
as
we'll
see
soon,
we
found
that
cassandra
fit
our
needs
pretty
well
for
some
of
these
services-
and
you
know,
there's
no
barrier
in
terms
of
you-
know:
business
bureaucracy
or
red
tape
for
us
to
use
cassandra.
It's
it's
what
works
built
for
us
and
they
carry
us
through
a
lot
of
different
services.
We
can
use
what
actually
solves
our
solves
our
problems
and
finally,
we
do
a
lot
of
cross-team
development.
B
You
know
we're
all
developing
services
and
there's
a
lot
of
common
functionality
and
there's
also
some
devops
related
devices.
So
if
you
go
to
hulu.com
and
you're
logged
in
as
you're
browsing
the
pages,
you
can
see
progress
indicators
showing.
Oh
hey,
you
watched
this
episode
or
you
know
you're
only
halfway
through
another.
B
This
is
all
provided
via
the
huge
shop
service
and
the
other
fun
feature
of
cubeshop
is
a
session
carrier.
So,
as
you're
watching
you
know
a
show
on
a
laptop
or
something,
maybe
you
know
it's
time
to
head
out
time
to
head
to
work.
So
as
you
leave
the
house,
you
know
you
take
your
phone
as
soon
as
you
get
on
the
bus,
you
take
the
phone
out
and
go
to
the
hulu
app.
B
Part
of
the
process
is
q.
Shop
is
really
related
more
to
the
real-time
requests
that
come
in
and
we
leave
the
analytics
work
to
a
entirely
separate
pipeline,
which
is
built
up
and
offline.
So,
ultimately,
what
this
means
is
we're
not
going
to
be
running
mapreduce
jobs
or
using
a
spark
on
top
of
cassandra.
Cassandra
is
purely
for
low
latency,
request
serving
and
stuff
that
goes
along
with
that.
B
B
B
C
A
B
Busier
ones
at
hulu,
because
there's
a
lot
of
people
constantly
watching
videos
we're
getting
updates
on
where
people
are
in
videos,
we're
also
trying
to
service
requests
for,
like
the
various
trades
that
you
saw
with
the
progress
indicators.
So
there's
a
lot
of
a
lot
of
concurrency
in
here,
and
we
use
geometry
with
python
to
ensure
that
we
can
handle
extremely
high
load
now
in
terms
of
the
actual
data
stores.
B
The
two
data
data
stores
we
use
are
redis
and
originally
was
mysql,
and
the
idea
here
is
the
redis
shards.
Are
there
as
pretty
much
casual?
Now
we
do,
we
did
have
persistence
enabled,
but
you
know
we
have
a
very
large
cluster
to
handle
a
lot
of
that
reload
and
yeah.
It's
it
handles
coordinated
traffic
and
then,
on
the
other
side,
we
have
a
python
application
that
is
talking
to
the
mycl
servers
and
the
actual
mysql
cluster
consisted
of
just
four
really
beefy.
B
B
B
B
B
And
the
reason
why
we
switched
to
crabby
was
because
the
python
drivers
at
the
time
were
not
very
good
for
cql
and
at
the
point
where
we
actually
started
working
on
this,
it
was
a
toss-up
between
thrift
and
cql,
but
going
forward,
we
really.
We
realized
that
cql
really
had
had
a
better
future
than
the
threat
and
wanted
to
embrace
that.
And
so
we
went
ahead
and
created
a
very
thin
api.
Just
with
the
java
the
datastax
java
driver.
B
Yeah
very,
very
good
question.
One
is
that
we
were
running
out
of
space
with
the
mysql
servers,
so
we
wanted
to
have
a
little
bit
of
extra
headroom
at
that
point
and
the
other
thing
is
replication.
So
with
mice
devel,
we
only
had
you
know
a
master
and
then
slaves.
There
wasn't
three
copies
of
the
data,
but
with
cassandra
now
we're
going
to
hold
actually
three
copies
on
disk,
so
we
need
more
actual
hardware.
For
that.
B
B
B
You
know
we
looked
at
hbase
and
a
couple
others
and
there
were
drawbacks
to
all
of
we
initially
did
this
hbase
wasn't
highly
available,
our
hbase
relied
on
hdfs,
which
was
not
highly
available
due
to
the
single
name
issue.
That's
since
fixed,
but
yeah
too
late,
so
yeah,
I
just
didn't,
have
the
performance
we
were
looking
for.
B
One
of
the
big
things
that
we
really
liked
about
the
santa
was
that
it
was
highly
available
and
it's
not
just
for
the
cases
where
servers
fail.
It's
really
really
nice
to
have
a
database
where
we
can,
you
know,
take
a
server
offline,
upgrade
it
tweak
some
gc
settings
or
make
changes
without
having
to
totally
take
a
master
or
slave
offline
and
going
along
with
the
entire
idea
of
cassandra.
You
know
no
node
is
special
they're
they're,
all
pretty
much
the
same.
So
you
know
it's
not
like.
B
B
B
B
B
B
B
B
Yes,
so
the
we
realized
that
we
really
didn't
want
to
rely
on
right
as
a
pure
data
store
sure
it
has
persistence
and
can
handle
quite
a
bit
of
load,
but
we
were
really
happy
with
the
persistence
models
there.
You
know
there's
rdb
and
pendulum
files
or
whatever,
but
they
didn't
have
the
same
actual
durability
that
cassandra
could
offer.
So
we
figured-
let's
not
even
use
rest
for
that
persistence
instead,
we'll
use
cassandra
purely
for
the
system.
Persistence.
B
B
C
B
A
B
A
B
B
B
There
were
physical,
no
vms,
didn't
want
the
overhead
of
having
a
zen,
hypervisor
or
something
sitting
in
the
middle.
We
wanted
to
actually
have
the
jvm
run
directly
on
the
hardware
and
just
six
hard
drives
nothing
special
there.
Three
two
gigs
of
ram
we're
using
level
of
compaction
and
just
treating
those
six
drives
as
a
jbod.
There
was
no
rain,
no
software
right
now,
hard
growth
rate,
it's
very
simple
and
we
ended
up
using
level
compaction
there,
because
we
could
get
very
good
distribution
of
data
across
all
six
drives.
B
We
didn't
have
to
worry
about
really
large
ss
tables,
building
up
from
size,
tier
compaction
or
something
we
had
very
little
ones
almost
yeah
very
evenly
across
all
six,
and
this
worked
pretty
well
for
the
rights.
We
had
done
a
couple
imports
to
test
performance
and
we
were
very
happy
with
the
how
the
whites
were
looking.
But
then
we
actually
went
to
query
the
data
and
not
so
happy,
but
I
don't
know
what's
happening.
There
is
there's.
B
So
take
two
was
to
switch
to
ssds,
and
the
approach
here
is
that
I
also
went
with
the
physical
boxes,
but
the
catch
is.
We
also
realized
that
we
had
to
disable
c
states.
So,
if
you're
familiar
with
c-states,
it's
pretty
much
what
your
laptop
does,
but
for
servers.
What
your
laptop
does
is
when
it's
sitting
here
it
down-clocks
the
cpu,
so
really
you're
only
running
at
about
60
of
the
power
available.
We
want
it
to
be
at
100.
B
If
there's
any
spare
spare
time
available,
we
wanted
to
be
at
full
power
and
kind
of
an
interesting
story.
There
is
being
on
the
core
team.
We're
super
interested
in
minimizing
the
latency,
because
all
of
our
services
are
on
the
critical
path
and
with
that
we
kind
of
have
a
little
competition
drawing
within
our
team
of
who
could
make
their
service
the
fastest.
B
So
you
know
we
improved
things
a
lot.
We
started
to
look
at
our
graphs
and,
as
we
started
to
reach
peak
hours,
you
know
the
rps.
The
request
per
second
would
start
to
go,
but
then
latency
would
actually
decrease
during
peak
hours
or,
like
you
know,
let's,
let's
add
more
traffic
I
mean
like
you
know
we
can
get
lower
latency
just
by
throwing
more
rps
at
it.
It's
a
big
win.
What
was
actually
going
on.
There
was
during
peak
hours,
the
cpu
started
to
bump
up
their
frequencies
and
as
soon.
B
Yeah
three
and
five
consideration
there
is
that
we
want
to
tolerate
failure
pretty
well,
it's
way
easier
to
rebuild
that
radio
rig
with
a
single
failed
driving
than
it
is
to
let
the
camera
have
to
shuffle
data
around
and
you
know,
run
a
repair.
It's
just
if
the
rate
controller
can
just
copy
the
data
from
other
local
drives.
B
For
sure
we
didn't
end
up
going
with
that
because,
as
we
start
to
calculate
the
replication,
we
have
three
replicas
already
times
another
day.
Another
data
set,
so
we
have
six
represents.
If
we
went
with
the
grade
10
all
the
time,
then
we
end
up
with
12
copies
of
data
on
disk,
which
is
pretty
excessive.
B
B
Yes,
if
you
yeah,
I
don't
have
a
graph
here,
but
you
know
we
can
watch
the
file
system
buffer
usage
over
time
and
you
can
see
it
increase
very
sharply.
As
you
know,
right
after
the
restart
as
the
ssp
start
pulling
in
more
data,
it
gets
held
in
ram.
So
that
way,
any
subsequent
requests
for
that
data
can
just
be
serviced
out
of
ram
other
than
hitting
disk
again.
B
B
B
Yes,
the
key
space
design,
there's
a
two
different
queries.
Two
different
sets
of
queries
that
we
want
to
handle.
One
is
those
queries
that
I
was
talking
about
before.
Hey
tell
me
the
last
position
for
user
x,
video
live
very,
very
straightforward.
Just
have
a
simple
views
table
where
the
user
id
is
the
partition
and
the
video
id
is
the
clustering
key.
B
We
just
we
thought
about
trying
to
use
a
blob
to
hold
all
that
data,
but
the
issue
with
trying
to
hold
it
all
as
a
blob
is
the
modified
write
cycle
like
that.
You
may
have
just
heard
from
the
previous
presentation,
and
so
with
this.
If
you
actually
split
up
the
individual
attributes,
then
you
can
update
them
individually,
not
have
to
worry
about
any
using
modified
lights,
and
the
other
case
is
for
a
daily
log
that
we
had
to
provide
to
another
service
so
part
of
our
recommendation
team
of
all
the
views.
B
So
that
way,
they
can
use
this
to
train
their
models
and
figure
out
the
recommendations
to
use
for
the
next
day
to
incorporate
it
into
any
other
of
their
real-time
services
as
well
and
with
mysql.
What
we
would
do
is
every
single
day
we
would
run
a
select
star
from
view
videos
and
they
would
have
to
scan
through
the
entire
data
set
on
all
of
the
mysql
servers,
and
we
had.
C
A
B
B
So
the
way
that
we
approached
it
this
time
was
that
every
single
day
we
say,
okay,
there's
going
to
be
120
rows
designated
for
holding
the
views
that
come
in,
and
so
with
these
views
or
with
all
those
rows.
What
we
do
is
we
just
have
some
very
simple
starting
scheme
where
we
write
pretty
evenly
to
all
three
of
them
or
all
128,
just
as
we
will
start
to
mysql
starters,
and
this
way
rather
than
say,
okay,
we
just
have
one
row
that
holds
the
entire
day's
worth
of
views.
A
A
B
B
B
So
when
we
read
so
you
know,
we
replicate
data
to
the
cassandra
cluster
and
then
we
have
to
backfill
the
data
from
mysql
and
cassandra,
and
if
we
are
very
careful
with
the
way
we
specify
the
column
timestamps,
we
ensure
that
all
the
new
data
that
we
started
writing
to
cassandra
doesn't
get
overwritten
by
the
old
data
that
comes
into
cassandra
from
mysql
big
thing.
There
was
time
very
helpful,
yes,
operations
operationally
cassandra
is
very
smooth
too
much
issues
use
a
internal
tool
for
automating
your
repairs.
B
B
A
B
B
Access
to
all
the
histograms
for
the
latencies
of
every
single
panel
family
on
every
single
node,
as
well
as
data
sizes
of
the
columns
families
number
of
connected
users
so
forth.
It's
extremely
helpful
to
have
this
there
when
you're,
trying
to
troubleshoot
issues,
because
you
can
pinpoint
the
exact
cause
and
dig
into
it
very
very
deeply
because
of
the
sheer
number
of
metrics
that
it
exposes,
but
the
big
problem
there
is.
It
exposes
so
many
that
it
actually
overwhelms
gravity
and
yeah.
We
to
give
you
an
idea
of
how
much
data
there
is.
B
Originally,
we
were
sending
metrics
from
cassandra
to
grab
like
internal
metrics
from
cassandra
into
graphite
once
every
10
seconds,
and
we
were
sending
roughly
10
megabytes
per
second
of
metrics,
just
a
graduate.
So
the
volume
of
data
from
the
metrics
alone
was
a
sizeable
fraction
of
what
some
of
the
nodes
were
actually
dealing
with
for
the
queries,
the
not
very
interesting,
I
guess
the
you
can
pretty
easily
tell
if
something's
going
wrong.
B
C
C
B
That
have
seen
some
corruption
as
well,
so
it's
not
100
bulletproof
and
when
we
do
see
it,
though
there's
two
tools
that
we
go
to,
one
is
no
tool
scrub
and
the
other
is
ss
table
scrub.
The
first
one
no
tool
scrub
pretty
much.
What
happens
is
you
can
connect
to
a
live
node
and
that
live
node?
Then
itself
executes
a
snapshot
and
then
starts
cleaning
up
the
ss
tables
and
with
ss
table
scrub
you
do
it
offline,
so
the
node
is
actually
going
to
be
down
when
you
do
that.
B
The
reason
why
I
say
that
things
are
really
really
bad.
You
use
as
a
table.
Scrub
is
a
couple
weeks
ago
we
ran
into
an
issue
where
cassandra
wrote
out
a
bad
ss
table
and
as
soon
as
it
tried
to
read
it,
it
was
trying
to
allocate
some
very
large
bike,
bumpers
and
in
doing
so
is
causing
javascript
each
face
out
of
turning
errors,
and
as
soon
as
it
happened,
cassandra
would
crash.
So
every
time
he
tried
to
restart
his
sampling
crash
again,
we
had
to.
B
C
B
B
B
B
And
so
initially
you
have
no
extra
space,
because
really
you
just
the
snapchat
is
a
directory
of
hard
lines.
But
when
notification
starts
copying
the
new
files,
then
suddenly
you
have
two
copies
of
the
data,
and
because
of
this
you
want
to
make
sure
that
you're,
always
under
fifty
percent
disintegration.
B
B
The
performance
of
the
cassandra
is
really
really
good,
but
during
our
import,
the
initial
import
of
micelle
data
and
cassandra,
we
were
running
into
some
issues
with
the
gc
and
we
ended
up
figuring
out
some
settings
that
allowed
us
to
get
g1gc
to
outperform
a
concurrent
market
suite
and
from
the
graph.
You
can
see
that
the
green
line
up
top
is
g1
gc
and
the
bottom
one
is
cms
and
the
graph
sorry
it's
small
to
read
the
graph
is
us
doing
a
small
import
of
the
entire
data.
B
We
said:
okay,
let's
see
how
it
performs,
let's
try
and
improve
it,
and
we
ended
up
running
it
for
about
eight
hours,
and
with
this
you
know,
we
can
see
that
g1dc
is
consistently
higher
than
cms
and
we
were
pretty.
We
were
surprised
with
how
well
performed
actually
because
cms
is
pretty
much
the
go-to
garbage
collector
within
java.
So
it
was
a
good
win
for
us.