►
Description
A brief intro to how Barracuda Networks uses Cassandra and the ways in which they are replacing their MySQL infrastructure, with Cassandra. This presentation will include the lessons they've learned along the way during this migration.
Speaker: Michael Kjellman, Software Engineer at Barracuda Networks
Michael Kjellman is a Software Engineer, from San Francisco, working at Barracuda Networks. Michael works across multiple products, technologies, and languages. He primarily works on Barracuda's spam infrastructure and web filter classification data.
A
Welcome
everyone
to
this
edition
of
our
cassandra
cassandra
community
webinar
series
I
am
delighted
to
have
with
us
today.
Michael
Kelman
from
Barracuda
Networks
Michael
just
gave
this
presentation
last
week
at
the
Bay
Area
Cassandra
summit
Michael
was
up
on
the
big
stage
in
front
of
hundreds
of
people.
So
you
know
that
was
a
high-pressure
event
I'm
hoping
today.
This
will
be
a
lot
easier
for
you,
Michael
just
kick
back
and
relax
just
a
couple
of
housekeeping
items.
So
as
always,
we
will
take
questions
at
the
end.
A
Please
use
the
Q&A
panel
inside
of
WebEx
to
ask
your
questions.
I
will
read
them
to
Michael
at
the
end
and
we'll
get
through
as
many
as
we
can
so
without
further
ado,
I'd
like
to
hand
it
over
to
Michael,
to
talk
to
us
all
about
their
migration
from
MySQL
to
Cassandra
and
things
that
in
hindsight
he
wish
he
had
known.
Welcome
Michael
thanks.
B
Christian
thanks
everyone
for
spending
the
next
hour
of
your
life
with
me.
I
am
an
engineer
at
barracuda
networks
and
sort
of
a
little
bit
about
what
we
are.
We
are
a
network
security
company.
We
make
spam
firewalls
web
filters,
backup
appliances
etc.
We
were
an
appliance
company,
but
we
are
moving
into
the
cloud
space
and
with
that
we
sort
of
you
know,
had
a
big
data
problem.
B
Hopefully
this
week
can
finish
up
binary
support
for
the
native
protocol
for
cql
and
I
have
been
at
the
company
for
six
years
so
because
of
that
I
sort
of
touch
everything
all
over
the
place
in
the
company.
So,
a
little
bit
about
our
Cassandra
cluster,
we
have
been
in
production
for
two
years
with
Cassandra
and
our
cluster
has
grown
significantly
and
also
our
usage
has
grown.
So
we
started
around
08
and
currently
we
are
on
125.
It
gets
now
a
little
closer
to
126,
with
some
custom
patches
that
we
have
as
well.
B
We
are
24
nose
in
two
data,
centers
12
in
each
we
use
to
two
terabyte
spinning
disks,
no
rate
at
all
I.
Let
Cassandra
take
care
of
the
raid
one
small
SSD.
We
used
for
column,
family
64
years
RAM,
and
then
we
use
on
puppet
and
cobbler
cobbler
as
a
red
hat
projects
that
lets
you
go
from
bare
metal
all
the
way
through,
sir,
to
exclude
and
sorta
manage
that
process
in
puppet
is
used
to
manage
sort
of
config,
etc,
and
we
target
load
about
600
gigs
I'll
get
into
that
a
little
further
down.
B
So
what
is
real
time?
This
is
sort
of
an
interesting
thing
for
us
in
spam.
Real
time
is,
is
you
know
pretty
much
a
negative
right?
You
you
want
to
get
it
before.
Anyone
seen
it
because
of
the
first
element
of
a
brand
new
campaign
comes
out
now:
that's
re
gun
through
to
someone's
inbox
and
just
as
importantly,
being
fast
on
on
false
positives,
false
positives
that
are,
how
do
we
get
faster
and
so
Mike
sequel
became
a
huge
hindrance
to
what
our
you
know.
B
Real-Time
behavior
was
going
to
be
and
we
weren't
real
time
enough.
So
here
is
a
lovely
piece
of
of
diabetes
spam
and
this
got
through,
unfortunately,
in
our
old
code
to
about
300
in
boxes,
and
this
is
what
we
call
advertising
them.
So
the
primary
goal
is
to
get
people
to
go
to
this
free
diabetes
thing
and
give
them
their
information
etc.
And
our
problem
here
is
that
there's
quite
a
bit
of
different
angles.
We
have
like
12
to
15
different
layers
that
we
try
to
spice
them.
B
So
in
our
particular
case
my
sequel
was
was
causing
a
huge
delay
and
how
fast
we
could
classify
all
these
different
layers
and
so
I
sort
of
to
summarize
before
I
go
into
the
details
of
the
rewrite
you
know.
Clearly,
cassandra
has
helped
us
here.
We
have
decreased
our
latency
for
our
API.
This
number
actually
I
got
I
get
below
1.5
milliseconds
today,
so
that's
pretty
cool
the
key
numbers.
Here
too,
we
can
now
store.
33
million
objects
that
we
track
at
all
times
in
our
database
versus
in
my
people.
B
We
could
only
keep
about
4
million
total
and
we
started
hitting
performance
problems
around
2
million,
and
obviously
you
know
the
number
of
rows
you
can
keep
them.
My
sequel
is
going
to
depend
on
your
workload
because
we
have
a
very
high
rate
of
insertions
and
changes.
It
caused
the
the
you
know,
you
know,
and
in
my
item
to
sort
of
eat
it
a
little
bit
and
another
really
important.
That
is
that,
because
of
Cassandra,
we
can
now
serve
all
33
million
or,
however
many
records
we
have
to
all
of
our
customers.
B
So
with
my
sequel,
we
had
only
were
only
able
to
serve
about
three
hundred
and
fourteen
thousand
and
then
on
this
one's
key
to
the
number
of
times.
We
have
to
see
something
before
we
actually
decide.
Okay,
hey!
This
may
be
something
we
want
to
take
a
look
at
so
when
it's
Cassandra
we
do
it
on
the
first
request.
So
the
first
time
we
see
anything
in
the
field
that
comes
to
us,
we
start
investigating
and
having
our
automated
classification
systems
go
to
work.
In
my
sequel,
we
may
have
had
a
threshold
that
said.
B
15
different
units
in
the
fields
would
have
to
report
that
back
to
a
cetera,
pretty
much
doing
everything
we
could.
You
know
to
start
to
reduce
the
load
on
my
sequel,
we're
now
multi
data
center,
we're
in
two
data
centers
versus
one-
that's
obviously
a
huge
benefit
to
our
customers
and
then
here's
the
real-time
numbers.
We
went
from
an
average
of
about
eight
minutes
of
classifying
a
domain
down
to
three
seconds,
and
so
three
seconds
actually
means
that
we
pretty
much
have
stopped
all
of
this
particular
types
of
spam
from
getting
through
to
our
customers.
B
So
this
article,
how
to
survive
a
ground-up
rewrite
without
losing
your
sanity
was
tweeted
by
Jonathan
Ellis.
It
would
be
a
patch
chair
of
Cassandra,
and
it
was
an
article
that
we
were
about
a
month
from
finishing
our
rewrite
and
I
wish.
I
had
read
it
or
that
had
been
written
when
I
started
my
rewrite
but
highly
recommend
anyone.
You
know
who's
thinking
about
doing
this,
to
take
a
look
at
this
article
and
pretty
much
what
the
summation
of
the
article
joel
is.
B
One
of
the
original
who's
me
original
engineers
at
microsoft,
and
he
also
co-founded
back
overflow,
and
so
he
told
a
tale
about
the
current
company
he
works
at
and
they
had
to
rewrite
going
on.
At
the
same
time,
one
was
business
driven
where
sales
had
actually
been
told
that
they
couldn't
sell
to
customers
who
had
a
number
of
transactions
per
second
higher,
because
it
was
going
to
bring
down
the
system
for
all
their
previous
customers
and
the
other
one
engineering
themself
they
decided
hey.
B
You
know
this
coches
is
an
optimal
anymore
and
they
were
a
dotnet
microsoft
house
using
ms
equal
and
pretty
much.
What
happened
was
the
one
that
wasn't
business,
driven
the
one
that
just
sort
of
engineering
said
hey.
This
could
be
a
lot
better.
This
is
sloppy
that
one
failed,
because
all
the
features
that
everyone
said
they
didn't
want
well
now
when
they
went
lives,
those
features
needed
to
be
implemented.
You
know
yesterday
and
then
the
one
that
that
succeeded.
B
Thinking
there
are
performances
that
are
and
so
past
engineering
decisions
are
now
preventing
implementation
of
your
new
business
requirements.
So
people
would
come
to
me
and
they
had.
You
know
this
is
a
particular
type
of
spam.
How
do
we
catch
it?
Well,
I
wasn't
able
to
do
anything
necessarily
or
you
know,
just
not
do
a
hundred
percent
of
the
job
right,
maybe
I
would
catch
it
after
three
to
four
hundred
messages
had
come
through
all
of
our
customers,
but
I
want
to
capture
the
first
one.
B
I
don't
want
to
catch
it
on
three
or
four
hundred
and
and
also
increasingly
in
my
particular
use
case.
We
found
that
threats
are
becoming
smarter
and
more
targeted,
and
this
is
pretty
important,
because
now
we
can't
just
wait
for
15
to
20
units
in
the
field
to
report
back
to
us,
every
customer
is
just
as
important
and
just
because
they
have
one
targeted
campaign
and
no
one
else
does
it's
not
fair.
You
know
that
we
don't
give
them
an
equal
priority
in
our
in
our
cloud
systems
and
so
legacy
systems.
B
Obviously,
everyone
has
code
that
themselves.
If
they
haven't
ridden
I
would
beg
to.
You
know:
I've
looked
at
my
own
code
even
just
a
week
later
and
said
what
the
heck
was
I
thinking,
I'm
sure
you
know
many
people
can
associate
with
that,
and
you
start
putting
all
these
layers
of
duct
tape
around
your
database.
So
my
sequel
can't
guilt.
So
then
the
first
thing
you
do
is
you
go
okay,
well
master
masters
too
risky,
so
I'm
gonna
put
in
place.
B
If
you
put
in
all
these
reads
lips
well,
now
you've
got
slave
labs
all
over
the
place
and
now
you've
got
you
know
the
bin
log.
Isn't
there
and
you've
got
to
do
these
crazy
gymnastics
all
over
the
place
to
get
that
to
work
and
then
you've
got
latency
between
your
day.
Enslave
replication
just
stops
and
then
one
data
center,
your
application,
isn't
working,
etc
and
I'm
sure
anyone
then
he
went
master
master
has
had
problems
with
IDs
cetera
and
that's
just
a
whole.
B
You
know
another
problem
and
and
on
our
my
sequel
system
right
now
we
run
about
a
minimum
of
one
second
slave
lag,
even
on
the
weekends,
sometimes
as
as
high
as
30
seconds,
and
so
my
sequel
performance
in
trying
to
scale
it
just
wasn't
going
to
work.
We
didn't
have
a
real
good
way
to
shard
our
data,
and
so
what
we
would
do
is
throw
hard
we're
at
it.
So
we
we
add
a
16
disk
raid,
60
box,
with
256
gigs
of
ram
and
16
cores,
etc.
B
B
You
can't
just
you
know,
keep
adding
read
slaves
because
you
haven't
fixed
the
underlying
problem,
and
so
you
have
to
hit
the
reset
button
and
in
our
case
this
was
moving
to
a
different
set
of
tools
with
the
main
one
being
cassandra,
and
so
our
requirements
were
to
plan
for
continuous
failure,
and
I
would
argue
that,
even
five
years
ago
the
thought
of
failure
and
your
persistence
tier
continuously
was
unthinkable.
You
know,
so
you
throw
tons
of
raid
cards
of
things
you
throw
redundant.
B
B
You
also
want
it
to
be
easily
scalable,
you
need
to
add
nodes,
you
need
to
remove
nodes
and
there
shouldn't
be
a
single
point
of
failure
that
you
can
think
of
I
like
to
put
a
smiley
face.
Next
to
that,
because
clearly,
there's
always
the
things
you
didn't,
think
of
that
bring
down
the
entire
stack
and
then
also
many
smaller
boxes,
verse,
one
monolithic
boxes,
I
talked
we're
using
really
really
cheap
boxes,
one
of
the
benefits
of
being
an
appliance
company
that
we
actually
make
our
cloud
solutions.
B
Now
that
we
use
internally-
and
so
you
know,
our
boxes
are
about
as
bare
bones
as
possible
and
I
think
this
is
a
trend
in
the
industry,
and
so
it
allows
us
to
essentially
get
n
number
of
nodes
for
that,
one
that
we
used
to
have
and
so
back
to
sort
of
the
rewrite.
You
know
you
have
to
get
technical
buy-in
from
all
the
parties
right.
B
B
It's
a
great
database,
but
there
comes
a
point
in
time
where
those
people
also
need
to
understand
that
there's
real
business
implications
here,
the
longer
you
wait,
because
if
you've
already
started
to
hit
capacity
problems
on
my
sequel,
you
know
this
rewrite
is
going
to
take
time
and
so
the
longer
you
wait,
the
more
you
defer
the
problem.
You
kick
the
can
down
the
road.
You
know
the
hard
is
going
to
be
to
get
that
rewrite
in
on
time
before
your
customers
really
start
noticing
significant
problems
in
your
application.
B
So
what
I
would
say
you
should
get
a
real
big.
My
sequel
advocate
in
your
company
to
sort
of
argue
against
you
in
a
public
forum
and
have
that
argument
and
sort
of
have
them
talk
about
all
the
different
ways
that
they
would
also
maybe
in
a
different
team,
try
to
scale
my
soup.
Well,
have
you
done
this?
Have
you
done
that
if
you
don't
have
a
good
answer
for
what
they're
saying
go,
try
to
find
it
and
then
come
back
to
them
and
say
Cassandra
still
the
right
solution?
B
So
it's
the
business
requirement
in
our
case
were
hey.
You
can't
go
and
reimplement
this
for
a
year
and
come
back,
and
so
we
were
pretty
much
told
within
three
months.
They
needed
certain
deliverables
and
it
actually
created
a
bit
of
an
engineering
problem
because
underlying
it
was,
you
know,
needing
to
set
up
all
the
Cassandra
infrastructure
and
all
of
our
shared
common
projects
that
other
tools
would
use,
and
so
it
also
required
us
to
operate
the
systems
in
hybrid.
B
So
right
now
we
have
both
I'm
systems
running
in
production
and
hopefully
next
week
we
will
move
a
hundred
percent
of
our
customers
over
we're
about
a
quarter
right
now
in
the
new
code
and
three
quarters
on
the
on
the
old
code.
So
this
is
about
the
time
in
the
presentation.
I
show
you
this
really
beautiful
graph,
showing
how
great
cassandra
is
and
how
perfect
everything
is
going
to
be
and
how
you
can
just
drop
it
in
place
and
boom.
Everything
is
going
to
be
perfect.
B
What
I
should
clarify
here
is
that
this
was
taken
on
a
Saturday
when
we
have
very
minimal
email
traffic,
and
this
is
what
things
could
look
like.
So
cassandra
is
not
the
direct
my
vehicle
replacement
and
it's
not
magic
bullet
to
solve
everything.
There
is
no
tool
in
software
engineering
that
is
a
magic
bullet
and
so
v1
here
you
can
see.
This
was
me
forgetting
that
I
turned
off
a
key
cash
on
a
particular
column,
family
and
so
I
fixed
that
and
you
can
notice
things
stabilized
quite
a
bit
v2.
B
It
turns
out
that
I
had
not
realized
that
there
was
a
database.
Excuse
me,
a
cross
data
center
load,
balancing
policy
in
the
new
datastax
driver.
That's
why
I
fixed
that
and
much
to
my
bosses.
Dismay
push
the
third
major
code
change
out
within
a
few
hours
and
then
v3
here
you
can
see
things
flattened
out
and
now
we're
down
to
that
2.41
average
I
was
talking
about
I,
don't
know
what
happened
around
eight
o'clock
there
so
clearly,
there's
still
a
few
little
things
that
that
could
be
improved
in
Sao
Paulo.
B
You
know
migrating
it's
painful.
It's
really
painful
have
I
mentioned
it's
painful,
there's
tons
of
rewriting
there's
tons
of
regressions
that
are
going
to
creep
in
you
know,
and
you
know
so
why
do
it
and
I
would
argue
that
cassandra
is
the
best
option
for
your
persistence
tier
at
the
moment
and
I
look
down
the
line
at
other
products
that
are
being
ridden
or
other
trends
in
the
industry.
B
There's
nothing
really
to
compete
right
now
with
Cassandra,
and
so,
if
you
have
a
business
reason
which
I'm
sure
many
people
do,
don't
let
your
database
hold
you
back.
Don't
don't
let
my
sequel
prevent
you
from
implement
things
correctly.
You
know,
don't
don't
say,
hey
yeah.
We
can
do
this
and
implement
this.
This
half
half
you
know
a
solution
because
it's
not
going
to
be
great
and
you're
not
going
to
be
really
proud
of
it,
and
so
personally,
I
like
to
be
proud
of
what
I
produce
I
don't
want
anyone
getting
spam
and
their
inbox.
B
So
the
lessons
we
learned,
the
good
things
that
we
did.
We
really
really
thought
about
our
data
model
up
front
and
my
boss
and
I.
We
joke,
because
we
had
the
conversation
for
two
hours,
just
whiteboarding
it
over
and
over
again
about
three
times,
and
then
we
would
let
it
we
come
back
and
we
go
wait.
What
what
have
we
decided
to
do
here
and
we
weren't
taking
notes?
B
We
were
just
like
boarding
it,
someone
go
in
and
raised
it
and
if
we
go
do
it
again
and
then
we
realized
hey
the
last
time
we
weren't
right,
and
so
we
spent
a
ton
of
time
up
front
doing
that,
and
so
you
know,
the
old
adage
measure
twice
cut
once
is
really
important
and
also
what
we
did
to
is
our
all
the
components
that
we
use
as
we
implemented.
We
started
realizing
that
what
we
thought
we
need
wasn't
actually
what
we
needed.
B
Yet
we
didn't
really
have
to
change
our
underlying
architecture,
and
this
is
really
important.
We
figured
out
how
modular
it
was
and
I
think
going
forward
to
it's
going
to
allow
us
to
not
need
to
necessarily
rewrite
the
entire
stack,
which
is
which
is
pretty
important.
You
know
breaking
things
apart
versus
having
everything
rely
on
one
monolithic
object
is
it's
pretty
important
and
so
the
bad
lessons
we
learned
consider
migration
and
deliver
requirements
from
the
very
beginning.
B
What
I
thought
I
could
not
usually
do
was
to
cut
over
our
you
know,
hundred
thousand
customers
on
the
same
day,
and
we
have
many
many
different
deliverables
that
we
produce
out
of
cassandra.
The
thought
that
you
can
switch
over
all
of
them
at
once
is
not
going
to
work
and
you're,
probably
going
to
have
a
pretty
bad
week.
If
you
do
that,
so
how
are
you
going
to
migrate
customers?
How
are
you
going
to
not
have
them
notice
regressions?
How
are
you
going
to
get
real
live
traffic
onto
you
know
your
parallel
system.
B
B
Adjust
expectations,
I
didn't
expect
to
be
relying
on
legacy
systems
for
so
long.
In
fact,
I
stood
up
on
stage
at
the
summit
and
said
that
we
this
week
would
be
live
a
hundred
percent.
Will
product
management
wanted
to
wait
another
week
to
get
some
more
feedback
and
so
I'm
still
relying
on
those
legacy
systems
for
quite
a
few
customers?
So
you
know
you
the
problems
that
you
had
before
aren't
just
going
to
magically
go
away
as
much
as
I
and
wish
they
did
so.
B
You
know
you're
going
to
have
to
probably
think
about
how
are
you
going
to
keep
the
duct
tape
together
for
long
enough
and
I?
Don't
think
we
did
that?
Well
enough.
Also
thinking
data
is
really
important.
So
in
our
particular
use
case
we
were
thinking
data
from
about
15
different
sources
and
pulling
them
all
into
Cassandra
and
what
happened
was
I.
B
I
synced
some
data
and
the
epoch
that
we
had
for
a
time
stamp
was
Bureau
where
it
shouldn't
have
been
well
naively
and
my
think
script
I
said
that,
where
the
you
know
the
initial
sync
where
it
was
greater
than
0
instead
of
greater
than
or
equal,
and
so
we
missed
a
bunch
of
classifications
than
our
sink
and
what
ended
up
happening
was
I
blocked,
German,
google
samsung
com
on
the
day
of
the
galaxy
for
release
ups
com.
Luckily
I
left
fedex
alone
for
all
of
our
customers
and
it.
A
B
A
pretty
painful
day
and
obviously
merging
data
from
you
know,
many
different
sources
is
you're
going
to
introduce
problems.
It's
not
an
easy
task,
and
so
you
know
take
your
you're
sinking
of
your
data
more
seriously
than
your
production
code.
You
know,
don't
just
write
a
one
off
script
that
you
think
you're
going
to
be
able
to
run
the
day
that
you're
going
to
do
your
migration
cut
over
everyone,
because
you
know
it
takes
time
to
move
all
that
data
to
so
you
know
I'll
get
a
little
bit
more
into
sinking.
B
So
one
start
with
the
queries,
so
I
think
there's
a
a
bit
of
a
you
know:
stigma
with
quote
no
sequel
and
I
sort
of
hate
that
term
no
sequel,
because
I
think
it
means
no.
No
thinking
to
some
people.
When
you
do
you're
my
sequel,
you
know
table
and
schema.
You
spend
a
lot
of
time.
Thinking
about
your
primary
foreign
key
relationships
and
how
are
you
going
to
lay
things
out
and
how
are
you
going
to
you
know
reduce
I/o,
etc?
B
Just
with
Cassandra,
you
know:
I,
oh
and
the
laws
of
physics
still
play,
and
so
you
really
need
to
think
about,
especially
with
tons
of
data.
How
are
you
going
to
lay
it
out?
Do
you
want
to
use
counters?
You
know
remember
right
now,
in
the
current
counters
implementation,
it's
sort
of
like
a
best
guesstimate.
If
you
it's
pretty
accurate,
but
if
you
start
getting
tons
and
tons
and
tons
of
counters,
you
may
be
a
few
off
and
then
composites.
How
can
you
lay
out
your
schema
with
composites
and
now
it's
dql.
B
Obviously
that's
all
under
the
hood
to
make
sure
that
you
can
retrieve
data
efficiently.
So
you
need
to
optimize
for
your
use
case
if
you're
doing
time,
series
data
you're
really
doing
rights
right.
So
you
know
how
are
you
going
to
do,
roll
ups,
etc?
There's
many
articles
online
and
don't
be
afraid
of
right.
So
Cassandra
really
excels
it
right.
You
can
throw
millions
of
rights
of
this
thing
and
storage
is
super
cheap.
B
Nowadays
you
can
get
256
gig
SSDs
for
like
a
hundred
bucks,
and
so
you
know
if
you
need
to
read
some
data
out
right
at
three
different
times
into
three
different
column
families.
So
then
you
can
pull
it
instead
of
trying
to
think
of
these
complex.
You
know
fo
giants
you're
going
to
do
you
know
in
a
tea,
normalized
world
and
so
I'll
go
into
tombstones
in
the
queue.
But
you
know
you
really
need
to
think
about.
B
If
you
have
a
really
high
delete,
SS
tables
and
Cassandra
are
immutable,
so
once
they're
written
in
their
appended
to
they
don't
change,
and
so,
when
a
delete
happens,
a
tombstone
that
the
little
flag
is
appended
to
the
end.
Saying:
hey
look,
you
know
everyone's
distributed
system.
This
is
not
valid
anymore,
it's
deleted,
so
that
data
doesn't
just
magically
go
away
and
you
do
a
compaction.
And
so,
if
you
don't
remove
that
data
and
the
compaction,
it
still
has
to
go
through
and
iterate
through.
B
All
of
that-
and
so
you
really
need
to
think
about
tombstones
counters
composites
and
how
you're
going
to
lay
out
to
think
differently
regarding
reads
I.
Think
because
my
sequel
performance
is
so
horrible.
We
tend
to
think
like
I
need
to
get
all
my
data
at
once
and
pre
populate
my
cash
well
I'm
going
to
try
to
pull
out
all
four
million
records,
and
you
know
load
this
crazy.
You
know
table
and
and
and
create
a
Tokyo
cabinet
file
etc
from
it
who
knows
with
Cassandra,
and
that's
probably
the
wrong
and
the
wrong
approach.
B
So
instead
you
know
how
are
you
going
to
do
it,
and
so
there
are
cases
where
you
do
need
to
iterate
over
every
row
and
do
a
read.
So
in
these
cases
we
use
Hadoop
and
pig,
there's
plugins,
just
natively
and
Cassandra,
that
that
will
work
with
Hadoop
there's
examples
if
you
just
get
the
the
sorcerer
or
our
binary
distributions
from
the
Apache
Cassandra
website
and
then
in
our
case
we
had
a
use
case
where
we
needed.
Real-Time
ordered
results
on
about
15
different
columns
at
all
times
instantly
in
real
time.
B
So
times
like,
when
did
this
last
element
hit
across
32
million
rows
and
so
that
served
to
be
pretty
difficult
because
in
a
distributed
world
now
you
need
to
go
get
reads
from
all
these
different
nodes
and
mugging
together
and
so
doing.
Ordered
results
is
really
hard.
So
we
use
elastic
search
here
for
that
one
particular
use
case
sinking
back
to
that
really
take
it
more
seriously
than
production
code
and
so
design
your
sink
to
be
continuous.
So
we
sink
all
the
time
and
a
mistake
that
I
naively
made
to
was
oh
hey.
B
I
can
just
you
know,
iterate
from
the
last
time,
I
think
the
current
time
and
move
all
of
it.
Well,
what
I
forgot
is
that
there's
some
data
that
really
really
needs
to
get
to
the
new
coded
faster
there.
Some
is
sort
of
important
and
then
there's
the
stuff
that
I
really
don't
care
if
it
gets
there
within
24
hours,
and
so
you
really
need
to
think
how
can
I
split
up
my
my
sink
into
different
buckets,
etc,
and
that's
probably
one
of
my.
B
My
biggest
mistakes
was
not
thinking
about
priority
and
not
taking
the
sink
code
super
seriously
and
doing
a
regression,
testing
etc.
For
don't
use,
cassandra
is
the
queue,
so
previous
development
in
our
system
actually
is
using
my
sequel
in
some
ways
as
a
queue,
and
so
we
have
a
single
thing
that
goes
and
pulled
a
particular
table
and
sees
hey
there
any
changes
and
it
grabs
all
of
them,
and
then
it
deletes
all
of
them.
That
was
probably
not
the
best
solution.
B
Then,
and
cassandra
is
just
as
bad
as
that,
even
though
it's
really
tempting
so
here's
a
great
blog
article
you
can
read
on
you
know
the
technical
reasons
why
you
shouldn't
use
kassandra's
too
cute.
It
goes
back
to
tombstone,
and
so
instead
we
use
Casca,
which
came
out
of
linkedin.
It's
now
an
Apache
project.
It's
awesome,
08
is
a
really
really
great
release:
multi
publisher,
multi
consumer
and
it's
written
to
disk
to
sew
things
go
down,
it
can
replay
trans
transaction.
B
You
really
don't
want
to
use,
at
least
with
the
hotspot
oracle
JVM
any
more
than
eight
gigs,
because
you'll
start
dealing
with
garbage
collection
and
so
GC
pause,
essentially
to
stop
the
world
garbage
collection
where
the
whole
JVM
just
pauses,
everything
trying
to
get
rid
of
and
walk,
though
the
whole
stack.
And
so
you
can't,
even
if
you
have
64
gigs
of
ram,
you
can't
just
go:
throw
64
gigs
on
the
Java
heat,
and
so
you
need
to
plan
your
capacity
in
mind
of
this.
B
So
what
I
recommend
is
get
a
piece
of
hardware
you're
happy
with
use
the
stress
tool
which
comes
with
Cassandra
benchmark
that
one
node
and
then
figure
out
with
your
replication
factor.
How
many
of
these
nodes
do
you
need
and
how
many
data
centers
and
then
you
know,
do
that?
Io
is
still
a
concern.
So
just
because
it's
you
know
Cassandra,
and
just
because
it's
new,
and
just
because
it's
awesome
doesn't
mean
that
I,
oh,
isn't
a
problem.
B
You
know
if
you
have
a
7200
rpm
disk
you're
going
to
have
just
as
many
problems
and
and
so
that
sort
of
brings
me
to
my
sequel.
Hardware
is
not
Cassandra
hardware
right,
so
using
lots
of
cheap
disks
and
lots
of
cheap
on
boxes
to
replace
one
giant
expensive
box.
It
is
definitely
the
way
to
go,
because,
if
you're
going
to
assume
in
your
design
that
you're
going
to
be
continuously
failing,
you
know
if
you
have
one
or
two
or
three
boxes
and
you
load
them
up
to
1.2
terabytes
worth
of
data
per
node.
B
It's
going
to
really
hurt
when
you
lose
that
note
right,
but
if
you
lose
one
and
you
only
have
200
to
600
gigs
on
it,
and
you
know
it's
easy
to
bring
it
up,
you
don't
need
to
copy
1.2
copying
1.2
terabytes
of
data.
You
know
it's
pretty
hard,
it's
rough,
so
you
know
keep
that
in
mind.
Don't
forget
what
you've
known
for
your
entire
life
in
terms
of
I/o
and
so
love
your
inner
op
self
so
distributed
system,
they
move
complexity
to
operations
and
so
a
little
story.
B
This
is
the
wrong
solution.
To
think
of
you
know,
automate
and
test
in
your
dev
production
and
then
staged
it,
because
what
you've
done
is
by
adding
all
these
layers
of
nodes,
etc.
You've
added
a
ton
of
complexity
to
your
applications,
and
so,
if
you
don't
really
think
about,
how
are
you
going
to?
You
know
automate
this
and
add
a
node
right
when
a
node
dies,
if
it's
not
super
easy
to
bring
it
back.
You've
just
created
more
problems
to
yourself,
so
we
use
puppet.
You
can
use
chef.
B
If
you
want
I'd,
recommend,
silligans
puppet
or
a
puppet,
you
could
create
a
tarball
and
arson
get
around
and
have
it
expands
whatever
works
for
you,
but
automate.
Automation
is
really
important
when
you
start
talking
about
distributed
systems,
and
so
there's
also
an
the
code
written
by
Sicilian
from
the
Cassandra
team
called
CCM
and
what
it
allows
you
to
do
is
create
a
local
dev
cluster
on,
like
you
know
your
macbook
pro
or
whatever,
and
so
the
coolest
thing
here
is
my
coworker.
B
We
actually
wrote
a
script,
it
creates
a
brand
new
CCM
cluster,
it
populates
the
schema.
It
goes
to
our
production.
Cluster
and
grabs
live
data
that
just
hit
and
then
populates
it
into
your
dev
cluster,
and
so
the
cool
thing
is
here.
You
know
it's
literally
one
run
of
the
command
line.
We
can
all
get
our
own
dev
nodes
with
real
live
data.
Obviously,
you
don't
have
the
load
on
it,
but
you
can
use
stress
tool
for
that
and
then
you
can
operate
Hadoop
bondage.
B
You
can
operate
everything
you
can
test
your
new
code
and
it's
really
nice.
So
I'd
recommend
you
know,
learning
CCM
at
the
beginning,
rather
than
later,
seven,
some
maintenance
required.
So
as
I
mentioned,
it
is
a
java
app,
and
so
you
need
to
learn
some
of
the
java
tools
that
are
in
the
and
the
jdk
they're,
really
not
that
hard.
I
know
that
there's
a
little
bit
of
like
this.
Oh
man,
there's
all
this
stuff.
I
don't
know
believe
me,
I
was
there
and
I
think
everyone's
been
there.
So
J
console
is
really
cool.
B
It
will
let
you
see
all
the
different
mDM's
management
ways.
You
can
touch
cassandra
internally
while
it's
running
and
you
can
also
see
performance
of
the
JVM
etc,
and
so
you
can
look
at
j
console
it's
just
in
the
in
the
bin.
Folder
of
the
jdk
you
download
on
the
Oracle
site,
there's
also
J
stack
if
you
have
problems
etc.
So
you
know
you
don't
need
to
become
a
job
at
expert
and
I.
B
Don't
want
to
convey
that,
but
getting
a
little
bit
of
an
underlying
understanding
about
how
do
I
monitor
it,
because,
as
a
Linux
person,
you
know
the
inclination-
maybe
oh
I'm,
just
going
to
estrace
the
pit
and
take
a
look
at
what's
going
on
where's,
my
I,
oh,
what's
the
problem?
If
us
trace
them,
the
jvm
you're
just
going
to
see
a
bunch
of
few
texas
and
it's
useless,
so
you
really
need
to
get
inside
java
to
understand
what's
going
on,
and
you
need
to
do
that.
B
B
So
it
spreads
the
load
out
and
if
we
do
it
frequently
enough,
let's
ranges
are
out
of
sync
meaning
less
data
needs
to
be
streams,
which
means
less,
I,
oh
so,
and
then
one
thing
that
we
also
use
is
google
ax,
which
exposes
jmx,
which
is
on
the
java
management
over
HTTP.
So
we
just
use
perl,
but
you
can
use
anything.
You
want
to
automate
this
stuff,
so
you
know
rolling,
restarts
upgrades.
You
know
what
our
performance,
metrics,
etc.
We
pull
all
that
out.
That's
free!
B
So
we've
got
two
product
lines:
a
hundred
percent
on
Cassandra
and
then
spam
firewall,
which
is
our
biggest
next
week.
We
have
a
firm
date,
we're
switching
everyone
off,
and
you
know
we
achieved
the
real-time
response.
So
the
most
important
thing
here
is
that
our
product
management
teams
or
sales
teams
they're
excited
to
go
now
sell
spam,
because
we
don't
have
to
really
hide
behind
metrics
anymore.
We're
really
proud
of
our
numbers.
B
You
know
a
rate
that
are
all
this
trip
turns
it's
just
not
friendly,
and
so
we
have
some
older
engineers
who
may
not
be
as
excited
to
learn
newer
technologies,
but
they
want
the
power
of
Cassandra,
and
so
everyone
understands
select
star
from
food
and
so
I,
really
like
the
fact
that
you
know,
if
there's
people
who
are
thinking
about
the
data
model
etc
and
how,
to
you
know
properly
use
Cassandra
in
their
in
their
use
case.
You've
now
made
it
really
accessible
to
everyone
to
also
get
it
that
data
as
well
and
calf.
B
It's
a
pretty
cool
way.
If
you
really
really
really
need
you,
no
more
transactional
stuff,
it
makes
it
even
better
drop
in
my
sequel
replacement,
that's
also
coming
into
data,
and
so
finally,
just
to
wrap
it
up.
The
Cassandra
community
is
freaking
awesome,
so
you
know
there's
actually
this
there's
other
solutions.
You
should
definitely
go
out
and
do
your
homework.
So
there's
react
which
is
Erlang
based
HBase,
which
is
really
just
a
key
value
pair
a
solution,
but
it
scales.
B
Well,
I
would
argue
Cassandra
scales
just
as
well
and
then,
if
you
want
to
spend
millions
of
dollars,
you
can
go
with
Oracle.
But
you
know
why
would
you
when
you
have
a
great
solution,
the
scales
that
has
an
awesome
community
where
there's
great
commercial
options-
and
you
know,
most
importantly,
the
people
who
were
working
on
this
project?
They
love
this
thing,
it's
their
life
and
so
their
goal
is
to
make
this
thing
work
for
you
and
for
your
project.
So
you
know,
IRC
is
there
for
you:
I
didn't
really
use
IRC
beforehand.
B
So
don't
be
that
afraid
of
it.
Just
google
free
nodes
and
you'll
get
the
URL.
You
can
connect
to
hash
tagged,
Cassandra
and
you
know
there's
people
there
who
just
want
to
help
you
when
you
hit
a
snag,
when
you
don't
know
what
to
do
and
there's
also
the
mailing
list
user
at
Cassandra,
Apache,
org
I'm,
which
has
tons
and
data.
A
B
A
You
can
see
any
previous
webinars
that
we've
done
at
the
links
that
we're
displaying
their
they're
also
all
available
on
Planet
Cassandra
and
then
also,
if
you're
interested
in
learning
more
about
Cassandra
and
would
like
to
take
training.
These
are
our
upcoming
training
dates
with
the
URL
there
as
well.
So
Michael
fasten
your
seat
belt.
Are
you
ready
ready?
A
B
B
What
I
would
say,
though,
is
that
most
of
the
people
who
are
looking
for
a
replacement
for
my
sequel
will
not
be
happy
with
HBase.
So
you
know
any
good
engineer
is
going
to
go.
Do
their
homework
and
you
know
clearly
there's
a
lot
of
good
use
cases
for
each
base,
but
excuse
me,
especially
with
cql
now
I
think
that
Cassandra
is
a
significantly
better
option
than
HBase
and
performance
numbers
as
well
too.
You'll
see
that
Cassandra
actually
in
most
benchmarks,
significantly
outperforms
as
you
increase
the
size
of
your
cluster
HBase.
A
B
Internally,
we've
used
puppet
now
for
three
or
four
years
and
I
think
the
project
has
matured
personally
I,
just
don't
like
chef
and
and
to
be
fair
to
I,
don't
like
Ruby,
so
I'm
not
advocating
puppets,
because
it's
in
Ruby
I'm
just
advocating
because
I
think
it's
the
best
option.
I
think
there's
a
much
more.
B
A
support
for
puppet
I've,
also
heard
of
many
horror
stories
in
production
with
chef,
I've
heard
pretty
much
no
horror
stories
with
puppets
and
so
I
guess:
I
haven't
personally
run
chefs
in
production,
but
I
have
friends
who
have
run
chef
in
production
in
there
now
all
using
puppets.
So,
but
you
know,
there's
great
built-in
tools
in
AWS
for
chefs
to
definitely
do
that.
The
key
that
I
would
stress
is
you
know,
make
sure
when
you
pick
your
automation
framework,
it
should
work
for
you.
It
shouldn't
work
against
you.
B
So
if
you're
constantly
fighting
it,
you
know
it's
going
to
make
things
worse
and
I.
Think
then
you
decide.
Oh
well,
I'm
not
going
to
automate
anything
anymore
because
I
I
hit
that,
and
so
you
know
whatever
works
best,
for
you,
I
think,
is
the
best
solution.
But
right
now,
I
think
puppet
is
the
best
option.
A
A
So
what
is
the
release
timeline
for
20
and
when
does
the
SAE
that
stage
stacks
enterprise
plan
a
20
based
release,
so
my
understanding
is
that
to
dot
0
will
be
available
sometime
in
july.
So
let's
say
before
the
end
of
july:
it
will
go
to
a
boat
and
then
get
released,
and
I
know
from
a
data
stack
enterprise
perspective.
We
are
aiming
to
have
to
dot
0
in
the
stacks
enterprise
in
our
October
release
time.
A
B
So
we
it's
a
little
bit
of
a
hack
job,
I'll
be
a
hundred
percent
honest.
We
have
a
casket
q
or
essentially
any
time
as
any
modification
on
an
element
week.
You
it
and
then
we
have
a
process
that
goes
through.
You
know,
if
you
do
those
requests
with
within
a
one
or
two
second
period
and
then
repopulate
them
in
bulk
every
30
seconds
back
into
elasticsearch.
So
there's
a
maximum
of
about
a
30
second
lag
in
our
elastic
search
index
triggers
are
coming
into
that
oh
and
they're
pretty
flexible.
B
So
what
I
will
play
around
with
and
then
I'm
a
little
concerned
about
performance
of
putting
a
trigger
for
every
single
update?
Obviously
the
you
know:
it's
probably
gonna
just
kill
our
performance.
It
won't
be
a
real
option,
but
I
made
you
know
you
may
be
able
to
hook
in
a
trigger
and
then
update
elasticsearch
there.
B
You
know
I
think
in
practice,
though
it's
even
though
it's
quote
a
half
job,
it's
not
really
hockey.
You
know.
Ninety-Nine
percent
of
our
data
is,
and
sink
and
I
have
a
sanity
script.
I
run.
You
know
to
make
sure
that
everything
is
in
sync,
after
that,
it's
not
perfect,
but
it's
pretty
good,
and
you
know
our
main
use
case
is
to
get
those
sorted
orders.
So
we
still
rely
on
kassandra's
our
actual
answer
I'm,
when
we,
when
we
render
the
actual
element.
B
So
pearl
casa
has
cql
support
and
that
works
over
thrift
currently
and,
as
I
said.
Hopefully,
hopefully,
fingers
crossed
I'll
get
some
time
to
finish
up
the
native
protocol,
and
so
you
can
just
do
execute,
execute
statements
in
perl,
like
you
normally
would,
with
the
you
know,
python
driver,
etc,
and
then
you
get
back
a
result
set
and
you
can
iterate
over
them.
A
Okay,
great
thomas
asks
could
reddit
have
been
considered
as
an
alternative
when
migrating
from
MySQL
Thomas
I
don't
know
too
much
about
Redis,
but
we
have
on
planet
Cassandra
a
five-minute
interview
with
Instagram
that
we
did
recently
where
they
talk
about
migrating
from
Redis
to
Cassandra,
but
Michael
just
interested.
If
you
have
any
perspectives,
I
yeah.
B
A
B
We
have
a
intern
actually
working
on
a
project
with
Redis
and
he
almost
brought
down
one
of
our
infrastructures
with
it
inherently
I
understand
the
problems
they're
trying
to
solve
with
Redis,
but
there's
a
reason
that
memcache
is
awesome.
Right
I
mean
it
is
just
a
key-value
in-memory
cache
and
this
hybrid
solution
that
reddit
has
got
going
with
now.
G
do,
and
you
know,
replicate
the
stuff.
You
know
treat
your
caches
of
cash
and
treat
your
persistence
layers
and
persistence
layer.
This
hybrid
solution
of
trying
to
keep
the
stuff
is
just
not
great.
B
You
know,
and
Cassandra
you've
got
a
cache
that
you
can
use.
If
that's
not
enough
for
you,
if
you've
got
some
really
really
heavily
cashable
items,
throw
memcache
in
front
of
it
as
well.
You
know
I,
not
bashing,
reddits,
but
I've.
Yet
to
see
someone,
you
know
really
come
out
there
and
say
hey
look.
This
thing
is
frickin
awesome.
In
fact,
people
seem
to
be
migrating
away
from
it.
So.
A
B
So,
as
I
mentioned
the
very
beginning
of
the
talk,
we
have
24
nodes
right
now
and
we
have
12
in
each
data
center
two
data
centers
for
all
of
our
important
data.
We
use
a
replication
factor
of
three
and
we
have
some
binary
objects
that
we
use,
that
we
store
in
Cassandra
and
for
those
we
use
a
replication
factor
to
the
data.
Also
too,
we
lose
it
wouldn't
be
the
end
of
the
world,
so
I
don't
really
mind
having
a
replication
factor
too,
but
also
storage.
A
B
We
had
15
different
sources
and
there
was
this
love
and
affinity
for
you
know
putting
stuff
in
these
flat
files,
and
what
happens
is
if
you
got
flat
files
all
over
the
place
on
these
boxes
that
aren't
backed
up
with
critical
business
data.
I
mean
stuff
that
makes
your
company
money
is
now
on
these
non
backed
up
flat
files
so
also
to
from
a
performance
standpoint.
You
really
need
to
be
loading
this.
You
know
multi
Meg's
file
in
and
out
of
memory
and
trying
to
parse
it
with
new
lines,
etc.
B
You
know
that's
the
old,
my
sequel
thinking
is
doing
that
stuff.
You
know
the
the
biggest
challenge
you
know
other
than
getting
away
from
my
sequel
is
getting
my
company
to
get
off
of
the
flat
file
mentality.
You
know
put
it
somewhere
where
it's
going
to
be
backed
up.
It's
going
to
be,
you
know,
compressed
it's
going
to
be
stored.
It's
going
to
be
replicated
it's
going
to
be
managed
by
an
Operations
team.
There
is
0
reason
why
you
got
to
keep
all
these
flat
files
around
anymore,
great
quivering.
B
A
B
Yeah
so
we
use
the
SSD
to
pin
so
in
11
and
above
you
can
actually
sit
on
symlink
or
you
can
pin
column
families.
If
you
look
at
the
structure
of
the
of
the
data
directory,
you
can,
you
can
pin
a
particular
column,
families
folder
to
the
SSD,
and
so
that
may
be
a
use
case
for
very
read,
sensitive
latency.
You
can
move
just
one
over.
B
A
B
So
I
did
personally
most
of
the
rewrite
myself.
My
colleague
shetal
you
know
was
really
really
important,
but
she
actually
was
hired
on.
It
had
never
used
Cassandra
when
we
hired
her
on
and
so
there's
a
bit
of
a
ramp-up
period.
But
I'd
say
you
know
we
wouldn't
have
got
here
without
the
help
so
primary
and
then
my
other
call
a
Colin.
He
really
helped.
He
wrote
all
the
cql
support
into
pro
casa,
which
we
really
wanted
to.
B
B
Facebook
has
done
in
my
sequel
to
scale,
there's
no
question
about
it,
but
I
also
talked
to
someone
at
Facebook
and
they
have
six
layers
of
caches,
sometimes
in
between.
You
know
when
you
click
something
and
when
it
gets
to
my
sequel,
I've
logged
into
Facebook
before-
and
it
says
due
to
your
particular
database
being
under
maintenance,
you
won't
be
able
to
log
into
Facebook.
You
know
I,
don't
personally
think
in
this
day
and
age
you
should
be
able
to
say,
hey
we're
going
to
take
that
database
top
line,
and
you
just
can't
login.
B
You
know,
even
if
it's
affecting
point
zero,
zero,
zero.
One
percent
of
your
millions
of
users
is,
you
know
it's
still
kind
of
a
bummer
and
you
can't
get
on
when
you
want
to
use
it.
So
I
think
obviously
there's
ways
you
can
scale
it.
But
you
know
we
reached
the
point
where
my
sequel
was
no
longer
the
solution
for
us.
B
If
you
have
not
reached
the
point
where
my
sequel
is
the
solution
for
you,
you
probably
don't
have
a
business
reason
to
do
the
rewrite
and
you
probably
shouldn't
do
it
because
you're
going
to
introduce
regressions
and
it
will
take
your
focus
away
from
the
real
application
logic.
You
should
be
writing,
but
in
our
case
we
couldn't
scale
my
sequel
anymore,
even
with
all
the
internal
knowledge
of
my
sequel.
A
B
A
B
B
So,
on
the
purl
side
we
use
pro
casa
and
on
the
Java
tab,
we
were
using
the
data
stacks
Java
driver
which
is
awesome,
and
so
we
are
pretty
much
very
soon
hopefully,
and
a
month
will
be
a
hundred
percent
of
the
native
protocol
that
was
introduced
in
1-2
and
so
after
apps
tnx
was
a
great
client
for
netflix.
You
know
it's
really
awesome,
but
we
are
personally
or
as
a
company
we're
moving
on
from
thrift.
So
if
you
still
have
legacy
code,
I
would
and
have
used
at
the
Onyx.
I
didn't
really
like
Hector.
A
A
B
We're
currently
so
we're
about
six
terabytes
right
now
of
data
stored
in
Cassandra.
Our
numbers
there,
with
the
33
million,
are
a
little
low
because
of
the
fact
that
we
couldn't
keep
stuff
on
my
sequel.
We
were
actually
going
into
leading
data
that
we
didn't
think
was
pertinent
anymore
and
so
I
expect
our
data
to
balloon.
Now
that
we
don't
have
to
artificially
delete
data
so
I,
you
know
I
the
current
number
66
ER
of
it.
Okay,.
A
B
A
B
A
B
Yeah,
you
know
if
you
I,
think
the
key
here
is:
you
need
to
make
sure
that
your
ops
are
going
to
be
ready
for
it.
So
if
you
get
a
three
node
cluster
and
you
lose
one
node
and
you
have
a
replication
factor
of
three,
you
will
have
transactions
failing.
So
yes,
it's
what
Cassandra
still
for
what
it's
great
for
but,
as
I
said,
it's
not
a
magic
bullet
right.
B
A
B
Since
there
I
don't
have
like
a
link
offhand
right
now,
but
there's
a
row
cash
and
there
is
a
key
cash
and
Cassandra
and
in
12
they're,
both
off
heap
and
they
were
moved.
It
used
to
be
je
na,
but
now
to
Java
miss
gun
safes.
So
the
benefit
here
is
that
you're
not
limited
by
the
Java,
keep
when
you're
using
your
cash.
Do
you
can
set
that
and
optimize.