►
From YouTube: This Week In Cassandra: Tales from the Field 4/15/2016
Description
Link to blog referenced in video: http://bit.ly/1MwM8c4
A
Alright
we're
back
with
another
this
week
in
Cassandra
this
week
we
have
with
us
Ryan's
fila
from
datastax.
He
has
seen
about
500
grape
clusters
and
maybe
500
broken
clusters,
not
really
sure
if
that
ratio
is
correct,
but
he's
seen
a
lot
of
clusters
in
his
life
he's
going
to
be
talking
with
us
today
about
some
common
anti-patterns
and
things
that
people
just
constantly
seem
to
do
wrong,
which
is
a
really
good
time.
Ryan.
Thank
you
for
joining
us
nice
to
have
you
thanks
for
having
me
preciate
it.
A
We
also
have
Luke,
Tillman
and
Sheree
hi
I
also
day
to
stack.
So
it's
it's.
A
data
stack
screw
today,
not
a
bad
time.
You
know
we're
people
too,
so
we
get
to
talk
so
first
big
news
in
the
Cassandra
community
I
got
my
first
haircut
of
the
year.
Just
want
to
I
mean
there's
lots.
I
mean
there
were
lots
of
rumors
leading
up
to
it.
It's
confirmed
the
hair
has
been
cut,
just
everybody
can
relax
now,
but
also
we
have
the
Cassandra
3.5
release,
which
is
just
gone
out.
A
B
Actually,
dow
really
for
the
the
cuidado
branch,
we
have
only
four
bug
fixes,
but
most
of
them
are
critical.
In
fact,
two
of
them
are
critical
bug
fixes
for
sassy.
One
of
them
is
about
the
sassy
going
om
if
you
have
to
brick
SS
tables
and
the
second
one
is
like
a
really
really
nasty
index
corruption.
So,
if
you
are
planning
to
you
Stassi,
please
use
three
dot:
five
and
not
tweet
at
four.
B
B
The
test,
what
the
idea
was
if
I
want
to
full
scan,
not
really
full
scan
but,
like
a
fetch,
a
third
of
my
table
out
of
Cassandra
into
spark
for
analytics
right.
It
is
an
analytic
scenario,
so
I
want
to
use
a
see
to
filter
down
a
little
bit
of
my
data,
and
this
is
how
it
works.
So
I'm,
just
fetching
out
of
the
data.
Ok,.
B
B
A
B
A
Very
cool
I'm,
so
35
that
you
know
we
talked
about
this
before
the
odd.
If
you
can
tell
what
kind
of
release
it
is
by
the
number
at
the
end,
if
it's
an
odd
number,
it's
a
bug-fix
release,
35
is
just
bug
fixes.
There
were
no
new
features,
so
if
you're
sort
of
like
bleeding
edge-
but
you
don't
want
to
be
like
as
bleeding
edge,
is
very
high,
maybe
you
should
think
about
rolling
out
Audrey
leases.
A
Maybe
if
you
don't
care,
maybe
you're
more,
like
Luke
and
you're
like
I'll,
just
put
anything
into
production,
I,
don't
even
care!
If
it's!
If
it's
been
tested
by
anybody,
just
go
for
it.
You
know
you're
more
of
like
a
free
spirit.
I
then
you
could
put
three
four
or
perhaps
36
when
it
comes
out
right.
A
A
D
Maybe
there's
a
race
condition.
Maybe
a
millisecond,
but
I
always
want
to
get
the
right
answer,
and
so,
where
this,
this
sort
of
thought
process
at
least
I
find
people
are
surprised
by
is,
if
you
have
a
transient
failure
and
let's
say
that
you
fail
a
right
to
your
local
data
center.
Maybe
one
of
your
replicas
or
two
of
your
replicas
were
down
you're,
going
to
fail
over
to
that
remote
data
center.
But
now
you
have
this
extra
latency.
D
A
D
This
is
where
a
lot
of
people
to
do
this,
actually
don't
like
the
result.
In
the
end,
they
don't
like
the
extra
latency
that
they
get
when
they
fail
over
and
two
it
doesn't
cover
the
common
modes
that
occur.
So
you
end
up
in
this
sort
of
anti
Goldilocks
world
where
you
don't.
You
are
too
slow
to
satisfy
your
SaaS
and
you're
too
inconsistent
to
satisfy
your
implied
contract,
and
so
it's
it's.
D
D
Because
your
app
servers
we're
talking
to
in
number
of
Cassandra
nodes
and
so
now
you're
taking
now
you're
going
over
particularly
a
constrained
pipe
and
sending
all
that
data
over
and
if
you've
thought
about
that
and
you've
size.
For
that,
there's
no
problem
with
it
like
if
you
like,
I,
have
customers
that
have
over
10
gig
links
between
data
centers.
You
know
they're
actually
close
to
each
other,
they're,
actually
just
different
failure
zones,
and
so
they
have
the
latency
where
they
can
go.
Okay.
D
All
these
app
servers
are
going
to
talk
to
this
other
data
center
and
that's
fine
and
that's
that's-
that's
completely
acceptable,
but
most
people
that
do
this
haven't
thought
through
that,
and
it's
one
of
these
running
with
scissors
type
processes.
Just
because
you
can
do
it
doesn't
mean
it's
a
good
idea,
yeah
and-
and
so
you
definitely
really
want
to
be
careful,
you
want
to
think
about
alright.
A
So
that
you
hit
it,
you
hit
an
interesting
point
a
little
bit
earlier.
There
was
you
talked
about
having
to
tend
geek
link,
I.
Think
one
of
the
interesting
things
that
a
lot
of
people
find
using
is
kind
of
the
difference
between
bandwidth
and
latency.
Right
people,
frequently
like
you,
could
have
you
know
a
10,000
gigabit
connection,
but
the
distance
between
your
data
centers
you're,
just
fundamentally
limited
by
the
speed
of
light
there
right.
D
A
D
A
D
C
Just
going
to
say
that
I
think
you
know
he
kind
of
points
out
in
this
article
too.
If
you
know,
if
you're
listening
to
this
and
you
go
and
you
check
out
the
article
on
planet
Cassandra,
he
does
kind
of
point
out
the
different
ways
that
you
can
fail
over.
So
as
opposed
to
doing
this
sort
of
round
robin
you
know
this
or
this
DC
aware
kind
of
failover
policy
in
in
the
driver.
He
does
talk
about
failing
over.
You
know,
basically
the
entire
data
center
and
and
uses
netflix.
C
I
mean
we
love
to
use
netflix
as
the
example
of
you
know,
but
yeah
he
totally
uses
netflix
and
talks
about
failing
over
at
the
app
app
level,
and
I
think
the
point
that
brian
made
is
is
important
because
a
lot
of
times
the
reason
you
know
that
people
are
doing
you
know
multi
data
center.
Cassandra
in
the
first
place
is
because
they
want
either
disaster
recovery.
You
know
kind
of
scenario,
or
they
want
to
get
the
data
closer
for
better
Layton
sees
you
know
better
customer
experience,
essentially
right.
C
They
want
their
their
clients
that
are
in
Europe
to
get
you
know
like
low
latency,
good
good
experience
with
their
with
their
site,
and
so
you
know
this
idea
of
having,
like
you,
know
your
European
data
center
potentially
spewing.
All
these
queries
over
to
your
you
know
us
East
data
center
you're,
essentially
going
to
make
the
latency
worse
and
kind
of
defeat.
The
purpose
you
know
of
why
you
did
it
why
you
went
multi
DC
in
the
first
place,
so
interesting
read,
definitely
recommend
checking
out
the
article
yeah
right.
A
On
I,
so
this
is
this
is
kind
of
an
interesting
segue
for
us
one
of
the
many
really
cool
parts
by
having
you
on
Ryan
is
we
get
to
talk
about
the
like
other
ways
in
which
people
have
tried
to
do
things,
and
you
know
either
succeeded
or
I
think
you've
seen
a
lot
of
their
failure.
I
mean
you've
been
basically
I
mean
you
handled
solution,
your
solution,
architect,
for
a
long
time,
now,
you're
doing
a
lot
of
support
and
really
kind
of
helping
people
through
problems
that
they
see.
A
A
D
D
A
D
One
is
not
as
common
as
people
doing
quorum
or
each
quorum
more
often
than
they
probably
intend
to
for
their
use
case
mm-hmm.
So
that's
really
common
and
and
generally
misunderstandings
in
sea
levels
and
one
common
very
common
manifestation.
That
is
the
downgrading
consistency,
retry
policy,
where
people
will
have
a
very
aggressive
initial
retry
policy
and
then,
of
course,
it'll
fell.
It'll
fall
back
all
the
way
to
one
which
never.
D
Yeah
and
I
and
actually
I've
had
some
really
good
debates
with
really
smart
people
defending
that
and
and
I
respect
it
and
I.
Think
if
you
understand
the
nuance
and
I
don't
want
to
tell
people
not
to
run
with
scissors,
you
know
if
you
know
what
you're
doing
and
you're
okay
with
the
one
and
you're
whatever,
but
it's
it's
sort
of
a
transient
failure
of
ditz,
lower
consistency,
pauses
lower
policy,
and
so
you
wouldn't
know
about
it.
A
I
there
is,
there
is
a
case
for
it
and
if
you
have
a
system
where,
like
I
work
for
a
company
where
we
were
doing,
we
were
doing
some
analytics
basically
right
and
we
would
like
we,
we
tried
our
best
to
have
the
information,
be
really
really
accurate.
But
there
were
times
where,
like
you
know,
maybe
there
was
a
problem
so
so
we
would
rather
show
our
page
to
someone
with
a
result
and
a
thing
that
says
like
warning
like
this
may
be
totally
correct,
but
here's
something
anyway
and
so.
B
A
Ya,
that's
the
case
not
just
like
not
just
like,
oh
whatever,
like
know,
if,
if
you're
comfortable
with
being
like
yeah
I
downgraded
to
one
and
liked
I,
don't
need
to
tell
anybody
like
if
you
feel
like
you,
don't
actually
need
to
tell
people,
then
you
probably
could
have
used
one
of
the
first
place
like
I,
completely
agree
with
you
yeah,
but
that
that's
the
nuance
right.
It's
like
if
it
meant,
if.
D
No
no
I
mean
you
have
to
like
just
be
honest
with
yourself
about
what
a
consistency
level
means
at
some
point
and
right.
So,
if
you're
going
to
use
something
like
any
database
that
accepts
multiple
rights,
whether
that's
in
a
global
data
center
sense
or
na,
because
you
know
this
applies
to
whether
using
my
sequel
or
Oracle
and
you're
using
multi
dinosaur
application.
You
have
multiple
sources
of
truth
as
soon
as
you
have
that
you
have
to
be
aware
of
that
inconsistency.
D
A
D
A
D
In
fact
they
would,
they
should
have
been
using
lightweight
transactions
actually
and
their
exact.
That's
a
use
case.
We
should
locking
mechanisms
another
case
you
should
use
lightweight
transactions
and
not
like
a
downgrading
consistency.
Retry
policy
so
but
I
mean
this
stuff
is
coming
because
it
sounds
good.
It
sounds
like
oh
I
want
to
be
really
available.
Is
why
I
picked
Cassander
in
the
first
place,
it's
very
seductive,
yep
and
and
and
and
to
see
I've
had
multiple
people
say.
A
D
A
Yeah
thatthat's
I
mean
that
the
delete
okay,
the
deletes,
are
an
interesting
one,
because
obviously
you
can
have
Sami
data
right
like
we
need
to
get
those
tombstones
in
place
before
Greece
Jeezy
gray
seconds
by
live,
but
there
are
there
are
tons
if
there
are
absolutely
cases
where
you
need
to
do
repair,
even
if
you've
never
issued
a
delete.
I've,
you
have.
You
worked
with
people
who
have
just
been
like
what
whatever.
D
D
Yeah,
so
I
actually
did
a
blog
post
about
it.
Called
Cassandra
repair
should
be
called
mandatory
maintenance
and
instead
of
like
no
tool,
mandatory
maintenance
or
something
like
that
or
you
know,
like
a
warning-
should
go
in
the
logs,
you
haven't
run
repair
in
XY
and
z.
It's
it's!
It's
just
a
very
thing.
D
I
think
a
lot
of
is
a
lot
of
people
come
from
a
background
where
you
don't
have
to
run
this
sort
of
operation
on
the
database
and
I
I
can
see
the
argument
for
because
of
the
nature
of
where
cassandra
is
such
a
mind.
Shift
I
can
see
the
argument
for
maybe
it's
something
we
should
be
handling
on
our
own.
Maybe
it's
part
of
a
server
process.
Yeah.
B
D
You
know
where
people
opportune
compaction
or
they
make
too
much
of
the
node
a
compaction
engine
that
it
has
nothing
left
to
do
anything
else.
Like
my
all
time,
favorite
then
throttle
the
compaction
which
people
will
use
to
get
out
of
a
problem,
but
a
lot
of
customers
run
that
as
their
normal
default
compaction,
throughput
setting,
isn't
Ronald
and
then
level
then
they'll
submit
a
ticket
for
why
do
I
have
bouncing
latency
okay,
so
this
is
really
common.
D
But
again
we
you
have
to
remember
the
historical
like,
like
sometimes
I
sound,
like
I'm,
being
like
hard
on
people,
but
this
is
all
news:
is
all
complicated,
we'd
have
30
years
of
third
normal
form,
training
and
different?
You
know
strategies
with
databases
that
now
we're
telling
people
the
opposite
thing
up,
and
you
know
they
haven't,
haven't
think
about
compaction
in
the
same
way
that
we
now
just
you
know
have
to
for
them,
and
tuning
is
not
ideal.
We
need
some
more
auto
tuning
in
Cassandra
I.
D
Think
to
make
this
something
that
they
don't
have
to
get
wrong,
but
yeah,
so
compaction
is
really
tough
for
people
to
tune
correctly.
I
see
people
crank
current
compactors,
really
high,
that's
very
common
they'll
have
spindles,
and
forty
eight
cores
and
they'll
have
concurrent
compactors
at
like
48
and
and
they're
7200rpm
hard
drives
will
be
dying
and
spindles
going
everywhere.
Yeah.
A
Well,
well,
even
even
you
know,
even
in
the
Cassandra
world,
or
just
the
tech
world
in
general,
like
enough,
has
changed
the
last
five
like
five
years
like,
let's
not,
let's
forget
about
the
last
30
years.
The
advice
that
you
may
have
gotten
five
years
ago
or
three
years
ago
about
like
let's
st.
JVM
tuning,
may
be
completely
different
right
now,
for
instance
like
if,
like
the
all
the
advice
that
everybody
came
out
two
years
ago
about
tuning
the
JVM,
for
let's
say
right,
heavy
workloads
is
like
horrible.
A
It
a
problem
because
I
you
need
a
really
big
new
gen
size.
If
you
want
to
do
read
heavy
workloads,
because
you
have
a
lot
of
these
short-lived
objects
and
if
your
new
gen
is
too
small,
then
they
get
promoted
prematurely
I,
don't
it
with
a
lot
of
you
end
up
with
a
lot
of
full
gc's,
because
you're
just
constantly
like
reading
a
ton
of
data
off
disc
and
pushing
it
at
your
old
jam
and
then
you
full
GC.
A
C
C
A
So
it's
hard
to
keep
up
with
it,
and
and
it's
like
and
you've
got
this
distributed
system
and
like
it's,
you
know,
maybe
you
have
it
spread
across
500
servers
and
it's
just
like
how
does
this
thing
behave
like
it's
just
hard
to
understand,
I
think
sometimes
so
yeah.
I
agree,
there's
a
lot
of
areas
that
we
can.
We
can
do
to
improve
these
things
like
there's
a
lot
of
auto
throttling
things
that
can
happen.
A
Yes,
one
hundred
percent
on
that
kind
of,
like
kind
of
like
how,
like
you
just
brought
up
to
you,
want
g1,
will
auto
size,
the
number
of
new
generation
regions,
old,
gen,
survivor
or
like
it
will
take
care
of
everything
for
you
eventually.
I
think
that
you
know
you'd
probably
want
to
see
that
out.
Your
database
too
yep.
A
The
last
thing
I
wanted
to
touch
on
with
you
was
storage,
because
people
this
is
this
even
like
even
I-
hit
this
and
I'm
not
dealing
with
people
in
the
field
as
much
as
you
are
I'm
dealing
with
people
that
are
new
to
the
Cassandra
world,
Luke
I,
like
you
and
I,
have
gotten
this
question
million
times.
We
hi
you've
probably
seen
this
a
bunch
of
times.
I
mean
you
in
the
beginning.
You
were
like
dude,
say
San
around
me.
A
D
Oh
boy,
so
recently
I
had
a
ticket
where
somebody
had
four
nodes
per
vm
and
they
were
all
sharing
the
same
one.
That
was
pretty
bad
and
it's
like
a
72,
its
engine,
rpm
spindles
and
raid
five
configuration.
So
that's
that's
pretty
bad.
The
worst
was
I
had
a
six
node
cluster,
all
sharing
the
same
lon
and
that
one
was
falling
over.
A
B
I
didn't
see
really
sad
people
using
Sun
because
I
stop
them
before
they
use
it.
But
I
have
like
a
very
interesting
question
like
oh
by
the
way
we
want
to
do
some
performance
benchmark,
but
it's
really
really.
The
resort
are
very
really
bad,
so
I
asked
them.
So
how
did
you
do
your
performance?
Oh,
we
pop
up
like
us
type
virtual
machines,
and
then
we
start
injecting
data
and
then
just
fail.
Well,
it's
5dr
machine
sharing,
the
same
disk.
Sorry,
there
is
no
magic
all
the
way
to
same
issues.
That's.
C
Funny
I,
like
people
benchmarking
on
sands,
like
I,
think
one
thing
that
that
people
don't
know
is-
and
this
is
true
benchmarking
Cassandra
in
general-
is
they'll.
You
know
they'll
run
a
benchmark
that
runs
for
five
or
ten
minutes
or
half
an
hour
or
something
like
that,
and
data
never
ever
actually
gets
flushed
to
disk
in
the
first
place,
or
they
don't
run
it
long
enough
where
compaction
actually
kicks
in
so
like
they
kind
of
get
this
false
sense
of
security
that
you
know.
Oh
my
sands.
C
A
D
A
D
I
guess
just
really
before
you
get
before
anybody
gets
started
with
Cassandra,
make
sure
that
you
think
about
things
as
how
they
have
to
work
not
as
magic
or
not
as
Cassandra's
just
handling
it
for
you
magically.
This
is
a
distributed
database
and
if
you
don't
understand
how
something
works,
ask
definitely
ask
it
on
the
IRC.
There's
some
really
help
for
people
there
yep.