►
Description
Speaker: Matt Kennedy, Solution Architect: Big Data at Fusion.io
Slides: http://www.slideshare.net/planetcassandra/c-summit-eu-2013-cassandra-on-flash-performance-efficiency-lessons-learned
Flash Memory technology, deployed as server-side PCIe or solid state disks (SSDs), is emerging as a critical tool for performance and efficiency in data centers of all scales. This presentation will discuss how the use of Flash impacts Cassandra deployments in terms of configuration, DRAM requirements and performance expectations. Ideas on leveraging C*'s cutting-edge data-center awareness to blend flash and disk storage nodes for cost and workload efficiency will also be shared. Flash media itself will be examined from a physical perspective to understand endurance issues. Data on write amplification under bulk-load and operational workload conditions will be presented to explain the impact to Flash of C*'s Log Structured Merge Tree architecture and the associated compactions. Finally, we will examine strategies to make Cassandra more Flash-aware using both conventional techniques as well as emerging Non-volatile memory (NVM) programming capabilities. Lessons learned from real-world customer deployments will be shared to complete this presentation.
A
This
is
cassandra
on
flash.
The
the
subtitle
of
this
talk
is,
you
know,
performance
and
efficiency.
Lessons
learned,
and
I
want
to
kind
of
talk
about
those
two
terms
in
relation
to
each
other,
a
lot
of
people
think
of
flash
memory
as
just
a
way
to
make
something
fast,
but
it's
really
not
all
about
that.
A
The
efficiency
side,
in
my
mind,
is
actually
a
little
more
important
because
it
lets
us
start
to
think
about
building
cassandra
clusters
in
a
different
way
that
gives
us
potentially
a
way
to
run
our
operations
using
fewer
resources.
It's
about
doing
less
with
more
so
if
we
really
want
to
figure
out
how
we're
addressing
cassandra
and
efficiency,
it's
important
to
look
at
what
efficiency
really
means.
A
The
state
or
quality
of
being
efficient
that
doesn't
help
us
much
so
this
brings
us
to
efficient,
especially
of
a
system
of
especially
of
a
system
that
should
be
ore,
machine,
achieving
maximum
productivity
with
minimum
wasted
effort
or
expense
all
right.
So
now
we
know
that
we're
looking
at
kind
of
a
minimization
problem,
that's
better
all
right!
A
How
many
of
you
are
like
in
a
lunch
coma
right
now,
because
I
I
can
kind
of
amp
up
the
energy
level
if
that
will
help
a
little
bit,
because
I
I'm
honestly
feeling
a
bit
of
a
sugar
crash
right
now.
So
when
we,
when
we
talk
about
this
efficiency,
one
of
the
things
that's
important
to
look
at
are
the
characteristics
of
the
storage
media
that
we're
using.
A
A
The
equation
doesn't
come
out
in
favor
of
flash,
but
for
a
database
system,
particularly
an
oltp
type
database
system
like
cassandra,
that's
really
not
the
metric
that
we
necessarily
need
to
be
looking
at
okay.
So
if
we're
talking
about
cassandra
and
flash,
maybe
it
helps
to
understand
a
little
bit
more
about
what
flash
is.
A
So
this
is
a
pictorial
representation
of
a
flash
memory
cell.
It's
been
around
since
1980,
invented
by
dr
fujio
masuoka
at
toshiba,
and
it
is
really
the
first
mainstream
persistent
memory.
So
what
do
we
mean
by
persistent
memory?
It's
a
it's,
a
solid
state
storage
cell
that
does
not
lose
its
information
when
power
is
taken
away
from
it,
so
you
can
turn
it
off.
Put
it
somewhere
turn
it
back
on
your
data's
still
there
I
I
do
want
to
kind
of
accelerate
this
talk
a
little
bit
normally.
A
I
cover
a
little
bit
more
about
the
operations
of
nand
flash
memory,
but
we'll
see
how
much
time
we
have
left
at
the
end.
We
might
come
back
at
this,
but
some
jargon
that
is
good
to
know
about
as
you're
looking
at
flash
memory
technologies,
you
hear
things
like
slc
mlc,
even
tlc.
Now
these
refer
to
essentially
the
density
of
the
cell.
So
slc
is
a
single
level
cell
or
single
layer
cell,
which
means
each
cell
holds
one
bit
of
information.
A
Multi-Layer
cell
refers
to
flash
cells
that
hold
two
bits
of
information,
and
tlc
refers
to
three
bits
of
information.
Basically
and
then
the
other
piece
of
jargon
that
is
good
to
be
aware
of
is
this
term
ftl,
which
is
the
flash
translation
layer
and
the
flash
translation
layer
is
really
it's
logic.
A
That
is
there
to
assist
with
some
of
the
quirks
of
flash
as
a
storage
media,
and
if
we
have
a
little
bit
more
time,
we
can
come
back
to
some
of
those
quirks,
but
I
don't
want
to
get
into
too
much
before
talking
about
you
know
the
main
point
of
this
talk,
so
flash
is
now
available
to
us,
because
we
have
all
these
consumer
electronics
devices
that
really
needed
it.
As
a
storage
media.
A
I
don't
know
if
any
of
you
remember
the
battle
days
of
you
know
personal
electronics,
but
it
used
to
be
that
when
you
had
a
palm
pilot,
you
had
to
keep
it
charged
or
it
would
forget
what
it,
what
you
told
it
and
you'd
run
out
of
battery,
because
you
were
up
late
playing
bejeweled
and
your
battery
would
die
and
you
had
to
go
back
and
hot
sync
and
you
lost
your
days
worth
of
stuff
and
it
wasn't
good,
but
now
digital
cameras,
iphones
thumb
drives.
This
is
my
favorite
graphic.
A
A
So,
to
give
you
an
idea
of
how
that
manifests
in
terms
of
system
performance
between
the
two
architectures,
this
is
kind
of
what
it
looks
like
in
terms
of
the
number
of
hops
required
to
get
data
from
your
storage
into
your
dram.
If
you've
got
say,
ssds
behind
a
raid
card
right-
and
this
is
what
you
can
do.
If
you
have
a
technology
that
can
basically
dma
directly
into
the
dram.
So
this
results
in
essentially
an
advantage
in
latency.
A
We
see
that
in
real
world
applications
response
time
very
low
latency
at
low
q.
Depth
is
important
because
it
allows
the
machine
to
be
efficient,
even
when
we're
not
in
like
a
benchmarking
scenario,
where
we're
throwing
a
whole
bunch
of
I
o
at
the
machine
and
allowing
the
operating
system
to
you
know
cue
things
up
and
kind
of
do
batch
and
parallel
operations.
A
A
It's
not
just
online
fast
reads
and
writes
in
a
simple
kind
of
key
value
store
people
are
using
it
for
complex,
analytic
jobs,
and
so
what
does
that
do
to
hotness
and
coldness
of
data?
We
kind
of
think
of
hot
data
as
the
data
that
was
mostly
recently
written
and
most
recently
being
read
right,
but
if
you've
got
an
analytic
job,
what
does
that
do
to
the
nature
of
what's
hot
and
what's
cold
right?
A
A
But
then
we
hit
32
gigabyte,
dimms
and
the
price
goes
through
the
roof
through
most
of
the
suppliers.
So
but
that's
okay!
This
is
scale
out
software
right.
So
what
do
we
do
when
we
find
ourselves?
In
this
scenario,
we
just
add
a
node:
it's
actually
cheaper
to
get
another
node
with
16
gigabyte
dimms
than
it
is
to
put
a
whole
bunch
of
32
gigabyte
dimms
into
a
node.
You've
already
got
so
we
do
that.
A
A
A
A
But
cassandra
is
a
system
where
people
do
grow
into
the
thousands
of
nodes
and
while
that's
really
really
cool,
we
want
to
make
sure
that
we're
doing
that
efficiently.
We
don't
want
to
waste
resources,
we
don't
want
to
waste
power.
Power
is
expensive
and
it
also
you
know,
there's
this
carbon
footprint
side
effect
of
that.
So
here
we're
seeing.
A
A
So
how
do
we
use
all
flash
storage?
We
just
put
it
in
and
mount
varlive
cassandra
at.
You
know
where
you're,
where
you've
got
the
flash
set
up,
and
essentially
the
only
extra
tuning
that
could
be
suggested
is
in
many
cases
you
can
actually
turn
compaction
throttling
off
which,
as
we'll
see
later,
can
result
in
some
nice,
even
performance,
and
you
don't
have
to
worry
about.
A
You
know
catching
up
in
terms
of
compaction
which
is
kind
of
nice.
This
is
kind
of
the
most.
I
think
interesting
of
these
options.
I
want
to
give
a
shout
out
to
marcus
erickson
at
spotify,
who
developed
this
capability
to
essentially
it's
kind
of
simple.
Actually,
what
it
does
is.
It
gives
us
consistent
directory
placement
for
column
families.
A
So
now,
if
we
scratch
our
heads
and
think
about
that,
for
a
second
say,
okay.
Well,
if
I
have
a
consistent
path
that
my
column
family
is
always
going
to
be
at
well,
then
I
just
need
to
make
the
mount
point
for
my
flash
drive,
the
column
family
directory.
Now,
I've
pinned
a
column
family
to
a
flash
drive
so
and
you
can
go
up
and
you
can
say
all
right
I'll
do
the
same
thing
with
key
spaces,
and
now
I've
pinned
a
key
space
to
a
column
family.
A
So
that's
really
kind
of
nice.
We
can
have
an
active
data
set
in
a
given
key
space
or
column
family
on
flash,
and
I
can
have
other
column,
families
and
key
spaces
that
live
on
disk
that
maybe
don't
have
these
particular
performance
requirements
that
we're
using
the
flash
stuff
for
so
this
is
good.
This
is
a
nice
way
to
have
a
balanced
system,
this
one's
a
little
more
complex,
but
we've
I've
seen
in
other
talks
today.
A
People
talking
about
you
know
the
the
orange
telecom
talk
mentioned,
how
they
use
a
separate
hadoop
data
center
that
is
spinning
disk
based,
even
though
they
have
a
flash
based,
active
active.
You
know
two,
two
other
data
centers
set
up
for
for
active
use,
but
they've
got
this
kind
of
batch
data
center.
That
is
loaded
with
spinning
disk,
so
we
can
kind
of
take
that
idea
and
if
we're
willing
to
make
some
sacrifices
in
our
applications,
we
can
get
even
more
efficient
with
our
use
of
flash
and.
A
One
of
the
reasons
this
comes
up
is
that
I
talk
to
people
that
don't
like
storing
replicas
on
flash
that
just
kind
of
conceptually
bugs
them
they're
saying
flash
is
more
expensive
than,
and
I
don't
want
to
have
copies
of
things
on
flash.
I
want
the
copies
of
things
on
disk,
so
you
can
kind
of
achieve
that.
You
could
either
like
write
your
own
snitch
to
kind
of
do
this
and
figure
out
where
your
reads
should
go,
or
you
can
just
say
all
right.
A
A
That
is
a
combination
of
many
block
devices.
You've
got
this
cache
layer
sitting
there
in
front
of
your
other
spinning
media,
so
there's
a
couple
of
open
source
ones,
and
then
we
also
have
a
commercially
available
one.
The
open
source,
flash
cache
is
probably
the
best
known
as
developed
by
facebook,
and
it
is
a
a
right
through
cache
bc.
Cat
or
be
cash
is
also
a
right
through
cache.
So
this
is
terminology.
Caching
terminology
that
may
not
be
familiar
to
everybody.
A
Okay,
I
have
to
apologize.
I
reverse
the
terminology
in
my
slides.
That's
how
you
know
clear
these
caching
terms
are
so
flash
cache
and
be
cache.
Can
work
in
write
back
mode
and
write-back
mode
allows
you
to
primarily
write
to
the
cache
and
then
at
some
later
point
persist.
Those
writes
off
to
the
spinning
disk.
A
So
this
allows
you
to
kind
of
accelerate
rights
as
your
as
you're
also
cashing
reads
in
cassandra:
it's
not
that
necessary,
because
rights
are
not
in
the
critical
path.
So
to
speak,
and,
to
be
honest,
it's
a
lot
safer
to
just
deal
with
caches
in
write
through
mode
right
back.
Caching
is
a
relatively
safe
thing
to
do
on
a
flash
device,
because
flash
is
persistent,
but
in
general,
it's
more
complex
to
implement
a
right
back.
Cache
that
isn't
going
to
screw
up
your
data,
so
right
through
caches,
tend
to
be
safer.
A
A
A
They
built
a
workload
generator
that
allowed
them
to
do
cache
friendly
workloads.
So
how
do
you
do
that?
You
essentially
use
a
long
tail
distribution
for
your
key
generator,
which
means
that
you're
going
to
have
some
keys
that
are
frequently
generated
by
the
workload
generator
and
then
a
long
tail
of
keys
that
are
infrequently
generated
by
the
workload
generators,
so
this
essentially
emulates
a
hot
data
set,
which
is
very
handy
for
testing
software
that
was
essentially
built
around
caching
right.
A
A
It
is
roughly
a
five
to
one
dram
to
storage
ratio,
so
this
is
a
10
hour
run
of
a
50.
50
read:
write
uniform
distribution.
Now,
while
we
do
have
kind
of
a
wide
band
here
of
the
the
performance
over
time,
it
stays
relatively
high.
You
know
our
low
spikes
only
go
down
to
you
know:
44
000
operations
per
second
over
five
nodes
and
what
you're
seeing
here
is
compaction
doing
a
pretty
good
job
of
keeping
up,
even
while
there's
a
very
heavy
workload
going
on.
A
If
you
ever
do
ycsb
benchmarks,
please
always
kind
of
include
these
95th
and
99th
percentile
numbers
that
ycsb
so
graciously
spits
out
for
you,
because
they
really
do
help
illustrate
the
the
not
so
good
case,
part
of
the
scenario
right
so
you're
always
going
to
have
some
requests
that
are
slower
than
others.
So
here,
even
10
hours
of
heavy
read
write
workload.
Our
99th
percentile
for
the
read
latency
is
under
50
milliseconds.
A
A
A
So
we're
looking
at
a
combination
of
a
couple
things
here.
But
if
we
look
at
the
75
thread
workload,
we
still
have
fairly
high
throughput
we're
at
a
significant
percentage
of
the
200
and
300
thread
workload.
But
we
have
much
lower
latency,
even
on
the
99th
percentile
of
the
read
workload
here
we
are
at
five
milliseconds.
A
So
obviously,
since
this
is
a
95.5
workload,
we're
doing
far
less
rights
and
there's
less
compaction
going
on.
So
we
can
have
commensurately
faster
reads,
so,
even
at
75
threads
we're
pushing
you
know
over
11
000,
readops
per
second
per
node
and
maintaining
sub
5
millisecond
latency,
even
at
the
99th
percentile,
which
is
really
really
good.
A
A
So
one
of
the
best
write-ups
on
cassandra,
consolidation
of
all
time
is
by
adrienne
calcroft
of
netflix,
and
this
is
when
amazon
ec2
actually
came
out
with
flash
enabled
nodes
and
to
to
kind
of
summarize
this.
What
they
were
able
to
do
was
take
a
total
of
84
nodes.
I
think
that
were
a
combination
of
cassandra
nodes
using
the
the
high
memory
instances
that
ec2
had
plus
memcache
d
instances
that
they
were
using
for
caching
in
front
of
cassandra.
A
So
I
highly
recommend
finding
this
blog
post
and
reading
about
it,
because
that
really
is
kind
of
really
gets
you
thinking
about
how
how
efficient
it
is
to
follow
this
model
of
scaling
out
for
dram,
so
that
our
ever
expanding
working
set
fits
into
memory.
Maybe
the
better
thing
to
do
is
just
to
assume
that
the
entire
thing
is
working
set
so
to
illustrate
what
this
looks
like
physically.
A
A
We're
talking
about
you
know
the
ability
to
remove
racks
from
cassandra
clusters,
which
is,
I
think,
pretty
cool.
I
mean
we.
I
think
we
should
all
as
system
architects
and
everything
try
to
be
a
little
bit
more
cognizant
of
the
amount
of
power
we're
consuming,
because
that's
a
lot-
and
this
is
simply
a
different
way-
to
build
out
scale-out
architectures
that
you
know
had
cassandra
been
designed
in
an
era
where
flash
was
this
economical.
A
A
A
A
Range
from
cheap
ssds
are
now.
If
you're
buying
hundreds
of
them,
you
know
70
cents,
a
gig
fusion.
I
o
the
the
list
price
for
fusion.
I
o
for
scale
out
deployments,
is
like
three
dollars
and
ninety
some
cents
a
gig.
But
if
you're
looking
at
a
deployment
where
you're
talking
about
you
know
more
than
10
cards,
we
will
do
everything
possible
to
get
you
the
best
price
and
it
will
be
competitive
with
you
know:
conventional
ssd.
A
Sure
very
good
question,
so
flash
does
wear
out
every
time
you
write
to
it.
You
damage
the
media
a
little
bit
and
that's
where
ftls
come
into
place.
They
can
help
and
they
have
all
gotten
much
better
over
the
last
several
years.
So
we
don't
see
things
like
hot
spots
in
ssd's
anymore,
where
you're
rewriting
one
particular
spot
over
and
over
and
over
again
and
causing
a
failure
by
virtue
of
the
virtue
of
the
fact
that
you've
worn
out
one
spot
on
the
drive,
so
none
of
them
really
suffer
from
that
anymore.
A
We've
all
got
good.
Wear
leveling
technology
in
our
ftls,
but
certainly
endurance
is
one
of
the
fusion.
I
o
advantages.
We
have
probably
the
best
flash
endurance
in
the
industry,
but,
as
you
point
out,
replacement
is
a
a
factor
in
how
you're
building
clusters
so
with
cassandra
using
jbod
mode.
It's
actually
pretty
convenient
with
the
the
front
swap
front,
swap
drives
to
just
be
able
to
take
them
out
and
replace
them.
So
here
when
we
deploy
fusion,
I
o
in
these
kinds
of
nodes.
A
A
There
are
those
kind
of
half
use
systems
where
you've
got.
You
know
a
single
socket
or
maybe
a
dual
socket
with
pci
slots
that
you
can
pull
out
the
whole
half
use
server
and
service
it
that
way.
So
we
we
do
make
sure
that
when
we
do
these
deployments
we've
at
least
talked
about
those
kinds
of
issues,
but
you're
also
going
to
be
replacing
pcie-based
flash
a
lot
less
frequently
than
conventional
ssd
type.
Flash.
A
I
want
to
say
pcie
fast
pcie
as
a
bus
is
way
faster
than
even
our
drives,
so
what
like
one
drive
will
get
a
gigabyte
and
a
half
per
second
of
bandwidth
off
or
throughput
off
the
drive,
whereas
the
entire
sata
bus,
I
think,
can
be
rated
for
six
gigs.
You
really
have
to
use
an
awful
lot
of
drives
to
get
up
to
that.
You
know
it's
really
for
cassandra,
it's
more.
A
The
latency
numbers
that
are
of
interest
and
what
you
typically
see
in
a
conventional
ssd
is
latency
numbers
in
the
range
of
you
know.
A
small
number
of
hundreds
of
milliseconds,
where,
as
this
is
or
pcie
flash
in
general,
is
tens
of
milliseconds
just
because
it
there's
that
direct
access
to
the
cpu
and
dram
complex.