►
From YouTube: Cassandra Day 2014: From Proof of Concept to Production
Description
This talk will cover how to load test your Cassandra cluster for your applications schema and other best practices to gain confidence in your Cassandra deployment before you run in production.
Jake Luciani is an Apache Cassandra developer at DataStax, as well as a committer on Apache Cassandra and Apache Thrift. His previous employer was Blue Mountain Capital in NYC, building a next generation market data database on Cassandra.
A
Hi
I'm
Jay
luciani.
I
work
at
data
sex
on
the
core
cassandra
team
I've
been
working
on
Cassandra,
since
that
Kim
came
out
basically
and
I
thought
I
would
give
a
talk
on
all
the
things
that
you
need
to
do
between
getting
something
up
and
running
in
Cassandra
like
a
little
application
and
then
going
to
actually
production
eyes
it
and
all
the
things
that
you
know
don't
normally
get
talked
about.
But
it's
kind
of
like
the
job
of
us
as
operators
of
a
cassandra
cluster.
A
So
this
is
how
we
build
software
right.
We
just
we
build.
We
build
some
software
production
eyes
it
and
we
prop
it.
But
what
happens
in
between
nothing?
We
just
roll
it
out
right,
like
that's.
How
well
I've
been
in
places
where
this
happens?
They're
like
proof
of
concept,
that's
production
done
roll
it
out
that
doesn't
always
go
well,
but
in
reality
you
know
we
have
all
these
things.
We
do
right.
We
test
we
performance
operations,
monitoring
the
key
metric
here
is
the
thing
I
think
step.
A
Two
is
really
we're
focused
on
preparation
of
like
and
we're
basically
trying
to
prepare
for.
Like
our
worst
case
scenarios,
this
book
is
actually
a
good
book
on,
unlike
all
the
horrible
things
that
went
wrong,
what
when
they
develop
nuclear
weapons-
and
you
know
they
had
to
come
up
with
like
all
these
different
plans
of
action
like
what
happens
in
this
scenario.
A
In
that
scenario,
and
that's
kind
of
like
what
you
do
in
in
in
a
production
scenario,
you're,
basically
trying
to
plan
for,
like
all
the
nightmare
things
that
could
happen
like,
oh,
you
know,
we
lost
a
machine,
we
lost
a
disk.
You
know
we
basically
lost
our
our
backups.
You
know
we,
we
had
a
huge
spike
in
traffic
and
you
know
we
need
to
add
more
nodes.
A
There's
like
a
bazillion,
different
thing
that
could
go
wrong
and
you
know
our
job
is
to
kind
of
come
up
with
like
what
are
the
bad
things
that
can
happen
in
this
application.
What's
acceptable,
what's
not
not
acceptable
and
what
can
I
plan
for
so
that
when
it
does
happen,
I
don't
like
completely
have
a
panic
attack,
because
I've
been
in
like
a
production
environments
with
you
know
when
things
go
wrong
and
you
know
I'll,
just
I
I'm
not
I'm
not
made
I'm
not
cut
from
that
cloth.
A
I
can't
I
can't
do
that
stuff,
it's
very
stressful.
So
so,
but
before
we
begin
like
some
of
the
basic
things,
you
know
a
lot
of
the
marketing
materials
and
stuff
like
that
will
be
like.
Oh,
you
don't
really
need
to
know
like
how
to
use
the
software
to
use
it
like
you
have
to
know
some
low
level.
You
know
you
have
to
be
able
to
log
into
the
unix
box.
You
have
to
know
how
to
like
use
the
basic
commands
you
have
to
know
like
how
to
check
how
look
loaded
your
disks
are.
A
A
A
So
it's
it's
very
easy
to
like
do
something
on
all
the
boxes
like
tail
all
the
logs
or
like
look
at
op
across
all
of
them.
So
those
are
kind
of
things
that
I
I
like
to
use
okay
anyway.
That
was
just
sort
of
a
background
slide.
So
in
terms
of
like
what
you
do
after
you've
built
your
proof
of
concept
right.
So
so
so
you
you've
you've
modeled
your
data
in
in
Cassandra
you've.
You
didn't
normalize.
Based
on
your
queries.
A
You
know
what
your
queries
are,
and
this
is
something
I
recently
came
back
to
data
sex
after
working
on
a
adduction
cluster
for
a
couple
years,
and
this
is
a
tool
that
I
wish
I
had
so
I
it
and
it's
basically
it's
basically
our
stress
tool
but
but
for
but
for
your
specific
schema
right
so
because
you
know
all
the
benchmarks
that
that
pepper,
that
Patrick
showed
in
the
last
last
slide
that
sound
like
a
very
like
simple.
You
know
clean
schema
with,
like
you
know,
there's
there's
nothing
really
complicated
going
on.
A
So
of
course
it's
like
super
fast,
but
you
know
in
your
scheme
mobile
that
you're
using
sets
and
using
matching
doing
all
this
different
types
of
compaction.
You
want
a
load
test,
like
your
queries
against
your
schema,
so
this
allows
you
to
do
that.
It's
not
actually
into
one
yet,
but
it's
it's
under
review.
It'll
go
in
close,
but
you
can
actually
download
it
now
and
it
works
against
a
20
cluster.
A
So
it's
just
a
stress
tool
and
the
way
and
the
reason
why
you
want
to
do
this-
is
because
you
want
to
push
your
cluster
2222
to
the
limit.
If
you
don't
know
when
your
cluster
breaks,
then
you
really
don't
know
like
what's
what's
going
to
happen
right,
so
this
is
sort
of
the
first
step
of
production.
Izing,
something
is
figuring
out.
You
know
when
does
it
fall
over
and
how
does
it
fall
over
and
like
what
can
I
do
to
make
that
not
happen,
and
when
can
I
know
that?
A
Okay,
it's
time
to
go
and
buy,
buy
more
hardware,
so
the
way
it
works
is
there's
a
yellow
file
and
I'll
show
you
in
a
second
and
then
you
basically
just
right
right
with
that
command
line
and
there's
a
bunch
of
options
that
you
can
set
like,
consist
his'n,
sea
level
and
number
of
threads
normal
stuff
and
then
for
reads.
You
can
basically
have
multiple
queries
that
you
you
you
can
run
against
that
that
data
and
what
it
does
is
it
it.
A
A
A
A
Yeah,
so
the
way
this
works
is
usually
defined.
Your
key
space
you
put
in
your
CTL
key
space
definition.
So
real
cluster,
yes
Mike,
one
okay.
So
if
you
need
to
put
in
your
you
know
like
real
key
space
information
and
run
it
on
on
your
real
posture,
you
can
do
that
now
very
easily.
You
can
also
you
know,
create
your
your
table.
A
A
Distribution
of
data,
so
for
that
name
column
I
basically
want
it
to
be.
You
know,
so
you
see
you
want
something
that
represents
your
your
particular
data
set.
You
know
like
the
names
will
be.
You
know,
between
length
of
one-and-twenty,
it'll
kind
of
have
like
a
uniform
pattern
or
or
gas
en
or
it's
fixed.
So
you
can
use
all
these
different
distributions
and
then
down
here
is
where
you
set
your
queries
right.
So
I
can
just
show
like
okay,
if
I
just
want
to
select
something
by
a
particular
name.
A
That's
one
of
my
queries
and
I
limit
it
to
100
the
second
one.
I
do
a
range
query
and
you
just
put
in
bind
variables
and-
and
the
stress
will
now
like
it
based
on
this
seed
down
here
that
last
thing
you
don't
really
need
to
touch
it,
but
it
basically
guarantees
that
that
that
that
the
data
you
right,
we'll
we'll
be
discoverable
by
by
the
readers,
because
it's
all
random
data,
so.
A
A
A
A
You
should
probably
still
do
that
because
you
want
to
know
how
your
application
handles
under
load,
but
this
is
a
good
first
step
just
to
get
comfortable
around
the
Cassandra
side,
so
you
can
see
where
the
peaks
are
so
yeah
if
I
let
this
run
I
be
nice
to
let
it
run
because
at
the
end
that
gives
you
a
nice
little
summary
which
you
can
chart
you
can
plot
out.
So
it
shows
you
what
your
tail
lit,
latency,
czar
and
stuff
like
that.
A
A
A
A
A
Usually,
a
lot
of
you
know
Doc's
that
need
to
be
read
for
this
stuff,
but
I
think
this
is
some
good
general
guidelines
to
keep
in
mind
for
your
hardware.
You
know,
keep
in
mind,
Cassandra's
not
really
well
suited
for
anything
like
more
than
a
terabyte
per
node.
So
if
you're
planning
on
running
like
a
20
terabytes
per
node,
you're
you're
going
to
be
in
a
world
of
hurt,
at
least
in
the
short
at
least
until
we
fix
it.
A
So
that's
one
of
the
things
that
we're
trying
to
focus
on
is
like
dense
nodes,
because
it's
something
that
a
lot
of
people
you
know
ask
for
or
expect.
So
that's
something
that
we're
focused
on
now
this
this
is.
This
is
only
in
the
case
of
a
pure
Cassandra
cluster.
If
you,
if
you're
using
you,
know,
spark
or
DSC
with
with
with
with
with
a
CFS
those
nodes
can
be
more
dense
because
they
they
can
actually
handle
larger
nodes,
a
larger
disk
usage,
but
in
terms
of
your
pure
key
spaces
and
column
families.
A
Those
need
to
stay
less
than
one
terabyte.
So
and
ideally-
and
this
is
kind
of
a
controversial
thing,
since
most
people
have
a
very
fixed
set
of
hardware-
they
get
like
the
to
you
boxes
with
the
8
drives
in
the
front,
and
so
therefore
they
can
scale
up
to
like
you
know:
10
terabytes,
20
terabytes,
but
you
know,
as
I
said
in
the
first
slide,
it
doesn't
really
work.
A
So
the
thing
that
you
want
to
keep
in
mind
is
like
if
you
can
get
like
small
boxes,
a
lot
of
small
boxes
like
like
a
blade
chassis
with
separate
disk
and
power
and
network.
That's
actually
the
best
use
case
vote
for
for
Cassandra.
It's
kind
of
a
hard
sell
to
management,
I
think
because
they're
like
oh,
this
is
a
database.
It
needs
to
be
on
like
database
class
hardware,
but
it
really
doesn't.
A
It
actually
does
much
better
on
this
hardware,
and
the
operations
are
much
much
faster
because
if
you
need
to
repair
something,
if
you're
using
all
of
your
cpu's
on
a
relatively
small
data,
set
like
that
that
that
that
that
that
completes
much
faster,
a
larger
box
with
64
cores-
and
you
know,
200
gigs
of
ram
or
any
of
that
stuff.
And
if
you
do
have
those
larger
boxes,
you
can
run
vmware
with
you
know
a
disparate
vm,
perhaps
or
you
could
try
to
get
the
LXE
containers
working.
A
A
Cassandra
on
it,
but
I've
never
actually
done
that
so
I
don't
know
how
well
that
works.
The
the
ec2,
the
I
two
instances
are
kind
of
made
for
this,
this
kind
of
workload
and
obviously,
if
you
can
get
SSDs
like
that's,
that's
the
best
case,
your
commodity
SSD.
Now,
I
think,
is
about
like
30
like
a
dollar
per
gig.
Alright,
so
you
can
get
like
800
gig
SSD
for
like
800
bucks.
It
really
is
in
that
bad.
A
So
in
terms
of
the
UNIX
level
stuff,
there's
probably
a
lot
of
things,
I've
missed
and
if
you
have
any
things
chime
in,
but
you
know
turn
off
swap
you
should
have
enough
RAM
on
the
box.
You
should
have
like
a
32,
gig
or
64
gigs,
and
if
it
swaps,
that's
you're,
basically
hosed
so
so,
just
just
turn
it
off
turn
off
cpu
speed.
This
is
a
one
thing
that
a
lot
of
people
get
get
caught
up
on.
A
If,
if
it's
your
own
hardware,
you
know
the
default
distributions
turn
this
on,
and
it
makes
sure
your
your
your
tail
latency
is
very
sporadic
when
that
turns
on
and
off.
You
should
to
the
deadline,
colonel
schedule
that
that
that
controls,
the
the
disk,
the
disk
I/o,
the
socket
buffer
sizes.
Those
are
things
if
you
read
like
a
sea
10k
article.
A
He
recommends,
like
all
the
those
options
that
you
should
set
installed:
Numa
control,
which,
if
you
turn
that
on
then
Cassandra
itself
will
start
up
with
with
the
right
Numa
settings
for
itself,
which
basically
is
interleave
all
because
we
don't
really
have
a
new
numa
process.
Yet
at
this
point
raise
your
limits.
I
didn't
finish
this
one,
the
no
file
and
the
there's
a
couple,
other
big
ones
that
you
want
to
raise
and
stress
your
disks.
Just
just
like
we
stressed
our
our
Cassandra
cluster.
A
What's
your
worst
case,
throughput
is,
if
you're
completely
discount
the
deployments.
This
is
sort
of
like
dev
ops
ii,
stuff,
like
you,
should
be
using
like
chef
or
puppet
or
ansible,
or
something
where
you
can
reliably
reproduce
your
your
deployments.
This
is
nice
because
it
makes
it
easy
to
like
roll
out
and
roll
roll
back,
and
you
know
it's
sort
of
a
general
thing.
You
should
release
your
own
artifacts
to
a
central
location
from
from
your
build
servers
and
then
and
then
you're,
you
know
chef
Orpik
and
can
pull
it
from
from
there.
A
You
know
you
shouldn't
be
rolling
your
own
tar
balls
and
then
you
know
ft
peeing
them
over
and
then
unzipping
them
and
restarting,
like
that's
so
bad
habit
to
be
in.
If
I'm
sure
everyone
knows
this
stuff,
and
then
you
should
do
this
for
Cassandra
as
well.
You
know
opscenter
gives
you
the
ability
to
roll
out
new
nodes,
but
it
doesn't
give
you
the
ability
to
upgrade
them.
So
I
think
in
general
is
you're
better
off
just
doing
your
own
package.
A
There
are
like
chef
packages
and
and
puppet
packages,
so
it
makes
it
easier
to
to
upgrade
things
because
your
configs
are
consistent
across
nodes
and
all
that
stuff.
You
have
to
worry
about
it
in
terms
of
monitoring,
you
need
to
stretch
their
system
and
where
it
breaks
down
and
use
that
to
basically
inform
your
you're
learning.
A
You
need
to
know
your
SLA
is
like
this,
something
you
you
should
ask
your
stakeholders
up
front
like
how
fast
is
this
need
to
be
like?
What's
our
minimum
like
what's
our
maximum
latency,
that
that
we
can,
we
can
have
and
not
be
considered
down,
and
you
should
you
should
set
those
per
per
layer,
so
you
say,
like
my
you
know,
our
application
layer
has
to
respond
within
like
two
two
milliseconds
internally.
A
You
know
Cassandra
needs
to
come
back
within
two
milliseconds,
so
total
worst-case
four
milliseconds
that
kind
of
stuff,
so
ops
center
is
nice.
For
you
know
all
things
Cassandra
do
this,
but
you
also
need
to
monitor
your
own
system.
So
if
you
have
your
own
system
and
you
want
to
integrate
Cassandra,
if
you
go
to
this
blog
post,
you
know
the
metrics
layer
in
Cassandra.
A
All
the
stuff
that
goes
into
opscenter
can
also
be
pushed
out
to
you
know,
ganglia
or
a
group,
graphite
or
Riemann
and
specifically
to
Cassandra
the
things
that
I
would
monitor
are
like
pending
compaction.
You
want
to
know
like
so
once
you're
once
you're.
If
you're
writing
too
much,
and
you
just
can't
keep
up,
you
have
a
few
things
that
you
can
do.
You
can
raise
your
concurrent
compaction
to
use
more
cores
and
and
burn
through
that
faster.
You
can
also
tell
your
writers
to
slow
down.
A
You
know
you
can
throttle
on
the
application
side.
There's
a
lot
of
things
that
you
can
do,
because
once
your
compaction
start
to
rise,
then
it
starts
to
slow
down
your
your
reads
right
because
it
has
to
read
more
more
things
to
find
the
data
you
also
should
keep
track
of
your
exception
counts.
Obviously,
if
that,
if
that
triggers,
then
there's
a
there's,
a
error
in
the
log
files
that
you
need
to
go
find,
and
you
should
check
your
disk
space,
because
if
you
start
running
a
disk,
you're
really
going
to
be
in
trouble.
A
So
Cassandra
ops,
like
you,
should
know
the
basics
of
Cassandra
outside
of
like
ops
center
any
of
these
things,
you
should
read
the
docs.
You
should
know
how
to
do.
You
know
a
bootstrap.
You
should
know
how
to
repair
your
own
data.
You
should
know
that
if
you
lose
a
node,
you
can
rebuild
it.
You
should
know
like.
If
your
data
files
get
a
bit
rot,
then
you
can
run
scrub
on
them.
Stuff.
Like
that
you
should
you
should
test
these
things
out.
A
You
should
break
a
SS
table
and
and
try
to
try,
try,
try
to
fix
it,
and
if
you
do
find
an
issue,
you
should
raise
a
jira
ticket.
This
helps
us
identify
things
out
in
the
wild,
because
you
know
working
in
a
closed
at
dev
team
on
sort
of
theoretical
problems.
We
we
we
don't
see
what
you
guys
see.
So
you
have
to
help
us
there
by
you
know
if
you
guys
find
a
problem,
just
open
up
a
jira
ticket.
A
Another
thing
that
you
know
is
useful
to
build
into
your
application
layer
is
achieved,
your
own
consistency.
You
are
shark
I
like
that
cover,
but
when,
when
things
go
wrong
in
your
cluster,
you
want
your
application
to
stay
up,
and
it
doesn't
mean
that
you
should
always
write
and
read
at
at
consistency
level
one
because
if
your
system
can't
deal
with
eventual
consistency
and
you
need
to
run
a
quorum
horam,
but
if
something
goes
wrong,
you
can
deal
with.
You
know
a
couple
hours
of
you
know
of
in
consistency.
A
This
is
like
a
tricky
subject
right
because
you're,
you
technically
don't
really
need
backups,
but
you
should
do
them
anyway,
because
if
someone
accidentally
like
logged
in
and
deleted
all
the
data,
you
need
them
or
if
you
write
some
bad
code
and
it
goes
in
and
corrupts
all
your
rights
a
bunch
of
garbage
to
to
to
the
wrong
table
or
you
login
from
dev
that
kind
of
thing,
so
the
best
thing
to
do
is
just
keep
a
on
going.
You
know
the
last
hour
of
all
the
data
keeping
a.
A
Snapshot
and
the
snapshot
basically
is
a
really
lightweight
way
of
making
a
hard
link
of
all
the
SS
tables.
So
so,
if
each
hour,
you
just
create
a
fresh
link
and
then
if
something
goes
wrong,
you
based
I
have
an
hour
to
to
go
back.
He
can
go
back
in
time.
One
hour
if
you
had
to
the
problem
with
with
these
local
snapshots.
Is
it's
not
necessarily
free,
especially
with
a
lot
of
compaction
going
on,
because
it
keeps
the
like
old
files
around?
So
if
it
so,
if
it
rewrites
a.
A
A
The
problem
with
doing
a
traditional
full
backup
is
in
Cassandra,
there's,
there's
no
master
and
slave.
So
basically
all
replicas,
you
know,
have
all
copies
of
the
data
and
you
kind
of
need
all
of
them,
because
if
you're
not
running
at
a
high
consistency,
you
don't
know
which
data
is
on
which
nodes
necessarily.
So
you
kind
of
need
to
keep
all
the
data
around.
So
the
only
thing
that
really
makes
sense
is
to
do
a
full
machine
backup,
and
it's
really
painful,
because
it's
if
you're
running
at
like
one
terabyte
per
node,
it's
it.
A
It
takes
a
really
long
time.
So,
in
the
end
of
the
day,
it's
like,
if
you
keep,
if
you
do
like
this
once
per
week,
and
then
you
keep
you
know
the
the
new,
the
new
SS
tables,
which
have
been
flushed
for
the
past.
You
know
a
few
days
and
you
know
so
you
can.
You
can
come
up
with
with
strategies
to
do
this,
but
it's
kind
of
on
a
case-by-case
business,
because
some
people-
don't
don't
even
do
this
and
you'd,
be
surprised,
there's
a
lot
of
people
that
don't
even
do
backups
at
a
minimum.
A
A
A
Cassandra
moves
pretty
fast
and
I
know
it's
hard
to
upgrade
your
boxes,
but
I
think
with
within
the
same
major
version
and
make
sense
to
keep
track
of
the
point
releases
and
just
as
a
general
rule,
it's
always
snapshot
your
data
before
you
do
an
upgrade,
so
you
can
roll
back,
which
leads
to
the
this
whole
idea
of
a
canary
node
if
you're
rolling
out
a
new
version
of
your
application
or
you're
rolling
out
new
version
of
Cassandra.
Just
do
it
on
like
a
single,
node
and
you'd.
A
Be
surprised
like
oh
there's,
this
new
configuration
file
that
I
didn't
you
know
tweak
and
therefore
the
node
won't
start.
So
if
you,
if
you
kick
off
some
process,
that
does
a
rolling
restart
across
all
your
nodes
and
and
they
all
go
down-
that's
happened
to
me
before
too,
and
it's
always
very
stressful,
because
then
you
have
to
go
and
panic
and
try
to
fix
it
all
as
quickly
as
possible.
A
So
if
you
just
do
it
on
a
single
node
and
just
watch
it
for
a
few
minutes
and
make
sure
everything's
working
the
way
you
want
and
then
just
continue
on,
this
is
something
just
a
common
technique.
People
do
pre
prod.
This
is
also
something
that's
kind
of
tricky
in
cassandra.
Is
you
know
you
can't
have
like
a
full
copy
of
your
production
in
some
like
dev,
environment
or
test
environment?
Everyone
wants
it,
but
it's
very
hard
to
justify,
because
it's
just
like
all
this
expensive
hardware-
that's
just
sitting
there.
A
You
would
want
to
set
up
your
data
to
be
where,
if
there's
a
particular
data
point
which
is
causing
trouble,
you
can
pull
over
just
that
one
point
into
your
test:
cluster
on
the
Cassandra
level.
Stuff
like
these
are.
This
goes
back
to
the
stressing
you
stress
to
figure
out.
You
know
how
can
I
tweaked
my
cluster
to
make
it
faster?
A
If
you
know,
but
by
default,
the
read
and
write
pools
are
like
32
in
a
32
threads
and
if
you're
running
on
a
64
core
box
or
something
then
you
know
you
should
raise
those
your
internal
compression.
If
you're
running
a
multi,
multiple
data
centers,
you
know
turn
that
on
don't
turn
on
internally,
it's
just
not
really
necessary.
A
You
should
also
lower
your
requests
timeouts.
This
helps
with
with
your
tail
Layton
sees.
You
know
like
your
nine-year
99
99
percentile
will
just
be
timeouts,
it'll,
just
be
awful
timeouts
or
your
99,
depending
on
your
your
system.
So
if
you
lower
that
from
10
seconds
22
seconds,
it
kind
of
helps
those
people
who
happen
to
hit
those
those
those
slow
things
right,
so
it
so
it
can
retry
faster.
A
You
should
set
your
concurrent
factors,
I
found
to
like
a
one
quarter
of
your
cores
that
helps
in
terms
of
getting
more
getting
a
striking
a
good
balance
between
serving
live
requests
and
keep
keeping
up
with
with
compaction
and
into
one
as
as
as
Patrick
bedroom.
We
have
these
off
heap
mem
tables,
which
helps
with
with
your
heap.
A
Another
thing
which
I'll
mention
that's
a
really
good
idea
is,
is
turn
on
authentication,
even
if
you
don't
use
it
just
set
some
like
hard-coded
password,
but
for
but
for
prod,
and
this
saves
you
from
accidentally
like
writing
to
the
wrong
database,
because
that
that
happens
a
lot.
So
if,
if
you
just
turn
on
some
prod
password,
that's
like
hard-coded
in
the
prod
deployment
script.
That
at
least
covers
you
from
accidentally
logging
into
the
wrong
machine
and
doing
something
and
that's
all
I
had
and
oh
wow
did
I
hit.
My
I
hit
my
time.