►
From YouTube: C* Summit 2013: In Case Of Emergency, Break Glass
Description
Speaker: Aaron Morton, Apache Cassandra Committer
Slides: http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass
The design of Apache Cassandra allows applications to provide constant uptime. Peer-to-Peer technology ensures there are no single points of failure, and the Consistency guarantees allow applications to function correctly while some nodes are down. There is also a wealth of information provided by the JMX API and the system log. All of this means that when things go wrong you have the time, information and platform to resolve them without downtime. This presentation will cover some of the common, and not so common, performance issues, failures and management tasks observed in running clusters. I'll discuss how to gather information and how to act on it. Operators, Developers and Managers will all benefit from this exposition of Cassandra in the wild.
A
How
to
change
things
in
Cassandra?
Maybe
that
you
want
to
change
some
queries
that
are
running
slow.
You
want
to
make
them
more
faster,
maybe
that
you
started
with
a
simple
single
data
center
installation
and
you
want
to
go
to
a
multi
data
center
set
up
before
we
get
into
that
a
little
bit
about
myself,
I'm,
freelance
Cassandra
consultant
I
live
in
wellington
in
new
zealand
and
I'm
a
committer
on
the
Apache
Cassandra
project.
A
So
we're
going
to
look
at
the
platform
we're
going
to
look
at
how
you
work
with
the
green
in
Cassandra
to
maintain
availability
and
performance.
We'll
look
at
some
of
the
tools.
You've
got
to
help,
diagnose
and
fix
problems,
we'll
look
at
some
real
life
problems
that
we've
seen
out
there
and
will
step
through
a
common
maintenance
task
just
growing
to
a
multi
data
center
cluster
platform.
Ways.
It's
pretty
easy
to
draw
a
diagram
like
this
draw
a
circle,
put
some
dots
on
it
and
say
that's
our
cluster.
A
In
production,
though
we
have
these
clients
that
connect
to
the
nodes
in
the
cluster
ask
those
nodes
to
do
work
for
us.
Those
nodes
have
to
communicate
with
other
nodes
to
achieve
our
result,
and
even
more
in
production
is
a
very
dynamic
relationship
between
clients
and
nodes.
We
could
have
our
triangle
clients
here
that
just
out
of
luck
all
talking
both
talking
to
the
same
node,
it's
doing
a
right.
That
node
is
not
a
replica
for
this
row,
and
so
it's
communicating
with
three
other
nodes
in
the
cluster.
A
We
could
have
our
square
clients
here
using
a
balancing
client
like
asti,
onyx
or
the
native
binary
protocol
and
they're
all
want
to
read
the
same
row.
So
it's
chosen
one
of
the
replicas
for
that
row
and
the
reed
is
happening
with
a
low
probability
of
reg
repair.
So
we
only
have
to
talk
to
one
other
node,
so
a
job
as
operators
and
developers
and
architects
is
to
make
sure
that
clients
can
always
talk
to
note.
A
Luckily,
for
us,
some
clever
people
wrote
down
the
constraints
that
we
work
in
when
we're
working
with
a
system
like
Cassandra.
We
have
to
care
about
consistency,
availability
and
partitions
consistency.
Wise
we've
got
a
nice
choice.
Here
we
can
have
strong
consistency.
We're
all
reads
return
the
same
result.
We
achieve
that
when
the
number
of
replicas
that
are
involved
in
successfully
reading
a
value
and
successfully
writing
the
value
is
greater
than
the
replication
factor
and
the
easiest
way
we
achieve
that
is
by
using
corn
for
reads
and
writes.
A
Lots
of
systems,
though,
can
work
with
eventual
consistency
where
the
number
involved
is
less
than
the
replication
factor.
There's
always
a
cost
involved
in
consistency,
we're
waiting
for
other
nodes
to
acknowledge
that
they
completed
the
task
and
that
cost
can
be
a
little
bit
higher
when
you're
in
a
multi
data
center
situation,
if
you're
using
corn
or
each
quorum
with
multi
data
center
you're
introducing
cross
data
center
latency
into
the
request
path
for
your
clients.
A
When
we
talk
about
availability,
we've
got
a
really
simple
heuristic.
Here
we
just
need
to
maintain
consistency,
level
up
nodes
for
each
token
range
in
our
cluster.
If
we
do
that,
we'll
always
be
over
the
service,
the
requests
at
the
required
consistency
level.
We
look
at
a
nine
node
cluster.
We
could
lose
three
nodes
that
are
evenly
distributed
and
maintain
one
hundred
percent
availability
for
doing
reads
and
writes
of
Korra
I
call
this
the
best
case.
Failure
scenario
range,
a
has
replicas
one
two
and
three,
and
we
lose
replica
one.
A
We
still
got
replicas
two
and
three:
we
can
maintain
a
quorum,
we
can
still
do
reads
and
writes,
and
that
holds
for
all
of
our
token
ranges
out
there
in
the
cluster.
So
that's
the
best
case.
A
worst-case
failure
scenario
looks
like
this:
we
lose
two
nodes:
they
are
adjacent
to
each
other
Rangers.
A
and
B
have
now
lost
two
of
their
replicas,
so
then
no
longer
available
for
quorum
operations.
A
Finally
partitions,
when
you
shut
down
the
node,
it
doesn't
create
a
petition
in
Cassandra.
We
only
care
about
partitions
when
nodes
are
actually
servicing.
Client
requests.
Petitions
happen
when
the
network
fails
add
a
little
bit
more
to
that
petition
fails
when
the
network
fails
or
the
networking
in
Cassandra
fails,
and
we
end
up
with
a
situation
like
this.
Our
client
connects
to
a
node
asked
it
to
do.
A
write.
Node
wants
to
talk
to
three
other
nodes,
but
it
can
only
get
to
one
of
them.
A
At
that
point
there
we
may
not
be
able
to
a
participant.
We
may
not
be
able
to
complete
the
request
at
the
required
consistency
level
and
suddenly
were
unavailable
for
that
request.
That's
looking
at
the
top
view.
What
happens
when
we
get
to
a
node?
The
storage
engine
we
have
is
optimized
for
writes
the
storage
engine
in
your
relational
database
is
optimized
for
reads
that
said,
our
right
path
is
pretty
similar.
A
First
thing:
we
do
with
your
mutation
when
we're
processing
it
is
we
append
it
to
our
right
ahead
log
that
gives
us
durability,
and
we
take
the
columns
that
you
provide
it
to
us
in
the
mutation
and
we
merge
them
into
the
men
table
using
the
timestamps
to
reconcile
any
collisions
that
men
table
is
always
in
memory.
The
clue
is
in
the
name
and
updating
it
is
a
lock
free
operation.
So
this
is
a
very
fast
right
path.
A
Once
we've
done
that,
we
acknowledge
back
to
the
client
and
we're
done
later
on,
we
might
detect
that
memory
users
getting
high
and
we
flush
this
meant
able
to
disk
to
create
a
new
SS
table.
This
is
one
continue.
This
is
just
continuous
rights
and
we
might
be
creating
faster
in
the
tens
or
the
hundreds
of
Meg's,
depending
on
configuration,
OSS
tables.
Of
course,
there
are
multiple
files
on
this.
A
We
have
the
data
DB
component,
which
has
some
metadata
your
roki
and
then
column
name
equals
value
and
the
timestamp
and
then
column
name
equals
value
in
the
timestamp
and
then
the
next
row
and
the
next
row.
We
have
the
index
DB
component,
which
has
the
roti
and
the
offset
for
that
row
key
in
the
data
DB
component,
and
we
have
the
filter
component,
which
contains
the
bloom
filter
which
will
look
at
shortly.
A
So
after
a
while
after
a
while,
our
storage
engine
has
been
running
around
making
all
these
new
files
and
having
a
great
time-
and
we
end
up
with
a
situation
like
this-
we've
got
a
column
family.
It's
got
5,
SS
tables,
we've
got
one
row,
that's
fragmented
over
three
of
those
SS
tables.
So
now
the
read
part
has
a
lot
of
extra
work
it
needs
to
do
and
in
general
the
read
path
has
to
go
to
HSS
table.
Read
all
the
columns
merge
the
results
back
together.
Of
course,
it's
a
lot
more
performant
than
that.
A
What
we
do
is
to
start
with.
Is
we
look
at
the
bloom
filter?
This
is
kept
in
memory
and
then
will
tell
us
if
the
roki
we're
looking
for
definitely
does
not
or
may
exist
in
this
SS
table
with
a
certain
probability.
If
that
says,
to
keep
looking,
we
then
go
to
some
samples.
We
have
in
memory
of
that
index,
DB
component
by
default,
we
sample
every
128
keys
and
we
look
for
the
key.
That's
prior
to
the
one
that
we're
trying
to
read
and
with
that
that
gives
us
an
offset
into
the
index.
A
Db
component,
we
drop
down
to
disk
seat.
To
that
position,
read
the
head
find
the
Roky
we
want
to
read
that
gives
us
the
offset
into
the
data
DB
component,
which
we
jump
over
to
and
we
start
reading
your
columns
might
read
all
the
columns
there
or
specific
pages
of
columns
if,
depending
on
your
query,
so
when
we
put
this
together
and
we
want
to
read
some
columns,
it
looks
like
this
for
SS
table
1,
we
go
through
the
bloom
filter.
A
We
look
at
the
index
samples
that
gives
us
an
offset
into
the
index
DB
component,
so
we
drop
down
to
disk,
we
seek
their.
They.
We
find
the
offset
into
the
day
the
DB
compose
that
we
go
to
that
file
and
we
do
a
read
there
same
story
here
on
this
is
table
2
s.s
table
3.
We
go
to
the
bloom
filter
and
it
says:
stop
looking.
There
is
no
chance
euro
teens
going
to
exist,
so
we
don't
have
to
go
to
disk
SS
table
for
a
good,
read
SS
table
5.
A
Here
is
our
degenerate
case.
We
go
to
the
bloom
filter
and
it
says,
keep
looking
and
then
we
look
at
the
index
samples
and
we
find
the
prior
key.
So
we
drop
down
to
disk
into
the
for
the
index
DB
file
and
we
seek
and
we
scan
to
try
and
read
the
key
that
we're
interested
in
and
we
can't
find
it.
This
is,
what's
called
a
bloom
filter,
false
positive
we've
wasted
some
disk
I/o
here,
but
that's
okay,
because
we
save
disk
I/o
in
other
places.
A
Everyone
wants
things
to
go
faster,
so
the
first
level
of
tation
we
have
is
the
key
case.
This
sits
after
our
bloom
filter.
So
we
look
at
SS
table
1.
Now
we
go
to
the
bloom
filter.
It
says,
keep
looking,
we
go
to
the
key
cash
and
we
get
a
hit,
and
the
key
case
has
the
row
key
and
its
position
in
the
day
to
DB
component
for
one
particular
SS
table.
So
we
skip
going
to
the
index
DB
file
and
we
save
some
I/o
same
on
SS
table
2,
3
and
4
work.
A
When
we
get
to
SS
table
4,
we
go
to
the
key
cash
and
we
get
the
key,
miss
that's
okay
and
may
not.
It
may
be
a
new
file
and
SS
table.
5
is
still
our
degenerate
case.
The
bloom
filter
says,
keep
looking,
we
get
the
key
cache
miss
and
we
end
up
going
to
disk
of
wasting
that
disk
I/o
to
make
things
go
even
faster.
There's
a
row
case.
This
guy
sits
on
top
of
everything.
Takes
you
the
default
one
take
to
your
OC,
realises
it
and
stores
it
off
heap.
A
A
A
So
it's
the
platform,
understand
consistency
and
availability.
We
know
when
we've
got
the
partition
and
we
have
an
idea
about
how
to
get
the
best
out
of
the
storage
engine,
the
tools
we
have
and
it
comes
time
to
administer
this
and
the
first
one
is
a
knowledge
of
what
Cassandra
does
internally
I
did
a
talk
earlier
today
on
internals.
The
code
base
is
very
approachable,
there's
great
information
out
there
on
the
user
list,
and
things
like
this
about
how
to
understand
what
cassandra
is
doing.
A
I'd
urge
you
to
treat
cassandra
as
a
system
that
you
can
understand
and
not
a
black
box
like
a
proprietary
database,
most
important
thing
that
I
have
a
look
at
our
the
logs.
You
can
configure
that
using
log4j
server
properties.
You
can
update
that
file
and
the
configurator
will
look
for
changes
until
you
can
change
the
logging
running
system.
A
You
can
also
update
it
using
the
storage
service
mb
often
debug
level
logging
in
the
production
system
is
quite
excessive,
so
you
can
update
the
logging
and
set
it
for
just
one
class,
in
this
case
the
Cassandra
server
when
it
comes
time
to
reading
the
logs.
We
care
about
the
time
stamps,
but
we
also
really
care
about
the
thread
ID
in
the
first
line
here:
the
metered
flusher,
which
is
the
class
that
watches
memory,
usage
and
flushes
column
families
to
desk
once
with
once
they've
used
enough
is
running
on
the
optional
tasks
thread.
A
This
is
just
thread
one
on
the
thread,
pool
called
tional
tasks,
it's
doing
its
business
and
then
it
calls
column
family
store
when
it
wants
to
flush.
Slap
me
to
disk
column.
Family
store
can
get
called
to
flush
things
to
disk.
If
you
use
node
tool
and
you
do
a
flush
or
you
do
a
snapshot,
it
could
be
called
if
we
have
to
flush
things
to
disk
because
the
commit
logs
rolled
over
and
we
want
to
flush
the
disk.
So
there's
a
couple
of
different
ways.
A
This
get
to
call
knowing
that
it's
been
called
on
the
optional
task.
One
thread
lets
us
work
this
out
and
say:
okay,
metered,
flush,
Iran,
and
because
of
that,
the
column
family
is
now
being
flushed
to
disk
on
the
last
line.
Here,
we're
actually
writing
to
disk
we've
jumped
over
to
another
thread,
a
thread
in
the
flush
rider
pool.
This
is
we're
actually
putting
stuff
on
disk
next
level
of
logging.
A
That's
handy
to
know
about
is
the
GC
logs,
if
you're
ever
having
garbage
collection
problems,
you
should
enable
this
on
one
or
two
of
your
machines.
Give
you
lots
of
insight
into
what's
going
on.
The
sorts
of
things
you'll
see
can
be
a
little
confusing
if
you
want
to
learn
more
I
urge
you
to
go
and
buy
a
book
called
Java
performance
by
Charlie
hunt.
It
came
out
late
last
year
and
has
lots
of
great
detail
on
this.
This
is
an
example
where
pärnu
is
running
par
news.
A
The
garbage
collector
on
the
new
heap
when
it
runs
it
actually
pauses
all
the
application
threads.
So
we
don't
like
it
running
for
too
long
here
we
can
see.
It's
kicked
in.
This
is
with
some
non
default
settings.
They'll
talk
about
later,
there's
a
thing
that
it
will
log
called
the
tenure
and
distribution.
This
tells
me
how
many
objects
have
survived,
how
many
rounds
of
pärnu
again.
These
were
non
default
settings.
A
I'll
talk
about
them
later
and
when
in
power
new
finishes,
it
tells
me
okay,
the
eden
space
where
objects
are
initialized
is
down
to
zero.
We've
got
a
bit
more
space
in
the
survivor
space,
but
I've
now
I
can
easily
see
when
partner
was
running.
How
long
it
goes
for
the
output
that
you
see
in
the
Cassandra
logs
from
the
GC
inspector
garbage
collection,
events
that
are
measured
to
take
over
200
milliseconds.
So
it
may
not
see
that
things
that
you're
getting
a
lot
of
garbage
collection
takes
190
milliseconds.
A
You
won't
see
those
in
the
compact
in
the
standard
Cassandra
logs.
Do
you
want
to
know
more
enable
the
GT
logging
no
tool?
Is
our
go-to
guy
on
the
command
line
for
getting
stuff
done?
No
till
info
is
a
pretty
simple
overview
of
what's
going
on,
but
it
also
includes
their
cash
information
for
the
key
in
the
rotation
that
recent
hit
rate
that
we
see
up
there
anytime,
you
see
recent
metrics
on
anywhere
in
Cassandra.
A
They
are
the
metrics
from
the
last
time
that
thing
was
red
from
so
if
you've
got
ops,
center
or
graphite
or
something
running,
and
it's
also
collecting
information
on
your
key
cash
or
something
like
that.
Every
time
that
every
time
that
poles
it
resets
the
recent
things,
so
you
might
see
numbers
like
zero
for
your
recent
hit
rate
and
get
a
bit
confused
if
you're
using
ops
center
or
something
that's
the
answer,
you
can
always
calculate
the
overall
hit
rate
using
the
requests
and
the
heads
that
are
outputted
here.
A
Nowtell
ring
is
my
old
friend.
Now
it's
been
superseded
by
status
for
a
certain
for
most
things,
but
here's
an
example
of
it
in
a
multi
DC
cluster.
It
can
get
a
little
bit
confused
because
it
doesn't
understand
these
token
assignments
until
we
give
it
a
key
space
once
we
give
it
the
key
space,
it's
able
to
use
the
replication
strategy
and
tell
us
what
we
wanted
to
see,
which
is:
we've
got
a
nice
effective
ownership
spread
across
all
these
nodes.
A
Motul
status
has
pretty
much
the
same
information
with
the
addition
of
the
host
ID
and
the
tokens
column.
Tokens
are
the
number
of
tokens
that
this
node
has
host.
Id
is
the
way
that
we
now
identify
uniquely
identify
nodes
in
the
cluster
we
used
to
do
that
by
token.
Of
course,
if
the
nodes
got
256
tokens,
we
can't
identify
that
anymore.
A
Cf
stats
is
a
great
kind
of
static
view
of
what
is,
in
your
column,
family.
The
sorts
of
numbers
that
I
look
for
here
are
the
space
use
total.
This
is
the
total
bytes
on
disk
that
we
have
for
this.
Guy
number
of
keys
is
an
easy
way
to
see
if
you've
just
got
a
lot
of
data
in
there.
The
read
and
write
latency
for
the
last
request,
so
they're,
not
particularly
useful
balloon
filter,
is
look
at
we'll.
Look
at
that
guy
soon
and
the
compacted
row
minimum
maximum
mean
you
see.
A
This
is
a
Colin
family
with
a
300
me
grow,
which
means
we
probably
need
to
do
some
work
on
the
data
model,
see
if
histograms
is
a
great
tool
that
I
use
all
the
time
that
can
be
a
bit
hard
to
read.
The
offset
column
is
just
a
number.
It
doesn't
have
a
meaning
until
you
read
it
with
another
column
when
you're
reading
it
with
the
SS
tables.
A
The
offset
is
the
number
of
SS
tables
that
were
actually
read
from
completing
a
request,
so
we've
got
35,000
529
requests
that
touched
three
SS
tables
the
right
latency
and
the
read
latency
are
the
latencies
for
just
the
right
in
the
read
thread
executing
the
command
doesn't
include
any
wait
time
and
the
offset
there
is
in
microseconds,
so
we
had
1150
requests.
Take
10
microseconds
to
complete
four
row
sides.
It's
the
bite
size
of
a
row
fragment
in
a
particular
SS
table
is
not
the
entire
row
over
the
entire
or
overall
SS
tables
and
column.
A
Count
likewise,
is
a
count
of
columns
for
a
fragment
of
a
row
in
one
that's
a
stable,
not
the
entire
count
of
columns
for
the
row
over
all
SS
tables.
I
mentioned
that
CF
histograms
latency
is
only
for
the
very
small
task
of
the
right
and
the
rethread.
If
you
want
to
get
an
idea
of
overall
request
latency,
you
use
no
tool
proxy
histograms
again.
The
offset
here
is
in
microseconds
and
the
count
of
the
number
of
requests
from
the
point
of
view
of
the
coordinator
that
took
that
long.
A
So
this
includes
all
of
the
latency
for
communicating
with
other
nodes
waiting
around
in
queues
for
things
to
get
done
stuff
like
that
jmx,
you
really
need
to
be
familiar
with
jmx.
You
need
to
understand
what
it
does.
If
you
look
at
the
mbean
interfaces
in
the
code,
this
is
where
the
jmx
mbeans
are
defined.
They
all
have
really
documentation
on
them.
You
can
go
and
have
a
look
and
understand
what
these
do:
access
jmx
by
jconsole.
That
can
be
a
bit
tricky
in
production.
Depending
on
how
things
are
set
up.
A
You
can
use
tools
like
mx4,
j
or
a
couple
of
others
that
will
give
you
HTTP
access
to
it.
This
is
one
of
my
favorites.
It's
a
command
line
tool
called
jmx
turn
I
put
this
on
the
box.
It's
one
jar
I
can
connect
to
it,
and
I
can
interrogate
jmx
in
the
face
and
I
use
this
all
the
time
to
pull
stuff
out.
Lastly,
if
things
are
going
badly,
you
might
want
to
dump
the
heap.
You
can
do
that
using
gmaps
dump
it
out
to
a
file
be
warned.
A
This
could
get
quite
large
if
you've
got
an
eight
gig
heap
you're
going
to
make
a
file
that
you
know
could
be
up
to
eight
gigs
inside
and
then
you
want
to
put
it
into
something
like
your
kit
or
map
the
memory
analyzer
tool.
In
this
example
here
we're
looking
at
bloom
filters
gone
wild.
You
can
see
in
the
top
list.
A
This
is
ordered
by
the
size
of
the
allocation
size
and
there's
about
four
gigs
of
long
arrays
in
this
heap
when
I
select
on
that
it
gives
me
a
back
trace
and
see
these
long
arrays
were
used
by
open
bit
set,
which
are
used
by
the
bloom
filter
which
are
used
by
the
SS
table
reader.
So
this
is
an
example
of
what
you
would
see
if
you
were
using
pre,
Cassandra
1.2,
and
you
had
loads
and
loads
of
data.
A
On
your
note,
so
it's
the
platform,
quick
overview
of
the
sort
of
tools
that
you
need
to
work
with
it.
What
are
the
problems?
Did
we
see?
Well,
we
used
to
see
this
a
little
bit.
I
haven't
seen
it
on
for
a
long
time
in
anything
that
I
recalled
production,
but
here's
an
example
of
the
user
list.
In
the
last
few
months,
I
think
compaction
was
running
and
there
was
an
error.
Writing
the
new
compacted
file,
something's
corrupt
in
one
of
the
assess
tables
that
was
being
compacted.
A
We
know
these
SS
tables
are
immutable.
So
what
could
it
be?
It
could
be
some
disk
I/o
problems.
It
could
be
a
bug
that
maybe
it
was
bug
in
an
older
version.
This
file
has
been
hanging
around
for
a
long
time
could
be
that
you
change
the
key
validator
and
that's
reordered
things,
there's
an
easy
fix
for
this.
There's
a
tool
called
no
tool
scrub.
It's
a
bit
like
compaction.
It
just
rewrites
your
SS
table
into
a
new
one.
We're
done.
We
can
move
on
dropped
messages.
Everyone
gets
this
sometimes
drop
messages.
A
Look
like
this
in
the
logs.
They
look
like
this
at
the
bottom
of
no
tool,
TP
stats
and
they
are
a
symptom
of
another
problem,
not
a
problem
themselves.
You
could
have
excessive
garbage
collection
going
on
we'll
look
at
that
shortly.
You
could
have
overloaded
I.
Oh,
you
should
be
monitoring
your
I/o,
so
you
understand
how
many
ops
you're
using
have
an
idea,
perhaps
of
how
many
you've
got
it
could
be
an
overloaded
know
that
could
just
be
poorly
SPECT.
A
It
could
be
on
an
m1
large
and
you
really
should
be
on
an
m1
extra
large.
You
could
be
doing
wide
reads
for
lots
of
data
that
take
a
long
time.
You
could
be
doing
large
batches
for
reads
and
writes
each
row
you
ask
for
in
the
read,
becomes
a
number
of
tasks
in
the
thread
pool.
So
if
you
ask
for
a
thousand
rows
in
a
read
using
RF
three,
you
will
get
about
three
thousand
tasks
put
into
the
thread
pulls
on
the
replicas.
A
High
read
latency
is
something
we
have
to
deal
with,
often
and
there's
a
number
of
reasons
for
it.
Let's
step
through
an
example,
use
node
info
have
a
look.
We've
got
a
key
cash.
We've
got
a
pretty
good
key
cache
hit
rate
of
ninety
percent.
So
that's
great.
We
working
as
we
expect
go
the
CF
stats.
What
can
we
see?
Number
of
keys
on
this
note
is
over
a
billion.
That's
quite
a
lot
of
data
here.
The
read
count
is
about
267,000
bloom
filter,
false
positives.
A
We
saw
those
before
128
odd
thousand,
but
the
false
ratio
is
zero,
because
this
is
the
recent
false
ratio
and
the
monitoring
tool
on
this
cluster
had
just
checked.
It
was
a
quiet
time.
It
was
day
for
me
so
night
for
you
guys
over
here
and
false
ratio
was
zero.
Let's
dig
a
little
deeper
see
if
histograms
have
a
look
at
that
SS
tables.
A
If
I
CSS
table
/
read
above
about
four
I
use
that
as
a
flag
to
go
and
dig
into
the
data
model,
because
that
means
where
we
spread
over
a
couple
of
tears
in
the
sized
head
model,
but
when
we
get
up
to
this
level,
it's
ridiculous
there's
something
wrong
here.
Obviously,
if
we
look
at
the
column
count,
we've
got
about
20
columns
per
row
per
row,
fragment
so
kind
of
narrow
rose.
We
look
at
the
row
size
or
about
a
kilobyte
in
size,
so
we've
got
narrow
roads
really
high
SS
table
count.
A
Bloomfield
a
false
positive
rate
looks
a
bit
odd,
but
we've
got
a
high
number
of
them
jump
over
into
jmx
turn.
Connect
on
to
a
node,
get
the
management
beam
for
the
column.
Family
called
bloom
filter,
false
ratio.
This
gives
me
the
false
ratio
from
the
time
the
service
started,
the
one
that
says
recent
gives
me
the
reason.
A
So
you
can
see
fifty
six
percent,
which
is
a
ridiculously
high
number,
we're
going
to
lot
to
wrong
I'm
going
to
lots
of
SS
tables
when
we
don't
want
to
go
back
and
confirm
this
in
our
CF,
stat
270
or
thousand
reads
130
odd
thousand
false
positive.
That's
about
fifty
percent!
The
calls
on
this
was
this.
A
Node
had
over
a
billion
rows
and
a
bunch
of
nodes
in
this
cluster
did
we
had
to
take
some
aggressive
steps
to
reduce
memory,
and
so
we
set
the
bloom
filter
FP
chance,
which
is
a
colon
family
setting
to
0.1
the
default
for
size.
Tiered
compaction
strategy
is
point
01
the
and
the
default
for
level.
Compaction
is
point
one,
the
higher
the
setting,
the
more
likely
you
are
to
get
false
positives,
but
the
smaller
the
bloom
filters
are.
So
when
the
fixes
we
did
on
this
was
we
changed.
A
Our
read
queries
to
select
by
column
name
and
that
limits
the
number
of
SS
tables.
It
goes
to
I've
talked
about
this
last
year
in
query
performance,
talk,
we're
able
to
short-circuit
the
query
if
you
say
find
me
foo
bar
baz,
when
we
find
the
most
recent
copies
of
three
bar
bears,
we
can
stop
looking.
We
don't
have
to
go
and
look
at
all
the
recessed
tables
long-term.
You
should
move
to
my
cassandra.
A
One
point
to
the
bloom
filters
are
off
heat,
and
that
saves
us
a
lot
of
time,
so
we
could
have
been
running
with
a
point.
01
for
the
bloom
filter,
false
positive
rate
and
taking
the
bloom
filters
off
the
jvm
heap
would
have
alleviate
a
lot
of
garbage
collection
pressure.
Gc
problems
might
see
this
in
the
logs.
The
heap
is
full
we're
going
to
flush.
The
disk
concurrent
mark,
sweet
it
sweep
is
the
collector
that
runs
on
the
older
heap
that
doesn't
pause.
A
A
This
is
the
view
of
a
node
that's
going
through
from,
and
this
is
a
view
from
one
node
looking
at
another,
that's
going
through
a
lot
of
garbage
collection
as
its
pausing
is
not
keeping
up
with
its
gossip
tasks,
so
it
gets
marked
as
dead,
and
then
it
comes
out
upon
you.
It
does
me
the
gossip
gets
martyrs
up
then
goes
back
into
parnu
gets
marked
as
dead
again,
so
it
flaps
up
and
down
at
this
point
here,
you're
pretty
much
dead
in
the
water
I
saw
this
quote
recently
very
recently.
A
Garbage
collection
is
the
result
of
workload
and
configuration.
The
workload
stuff
is
the
important
things.
Looking
your
day,
the
model
for
wide
rows,
large
right,
unbounded,
multi,
reads
and
writes.
This
can
happen
sometime
if
you're
reacting
to
something
the
user
does,
and
you
don't
really
have
a
bound
on
that
and
the
users
doing
things
you
didn't
expect
and
suddenly
you're
reading
a
thousand
rows
when
through
testing,
you
only
ever
followed
one
certain
path,
and
you
read
a
couple
of
rows
if
you've
got
very
wide
rows
that
might
correlate
to
compaction.
A
This
is
some
aggressive
tuning
to
slow
down
compaction.
Concurrent
compactors
is
the
number
of
SS
tables.
We
try
the
compact
at
the
same
time
defaults
to
the
number
of
threads.
We
can
reduce
that
to
two
makes
a
bit
slower,
compaction,
throughput
megabytes
per
second
defaults
at
sixteen
and
slow
that
down
and
in
memory
compaction
limit
means.
If
the
rows
are
bigger
than
this,
we
don't
try
to
materialize
the
whole
row
in
memory
again.
This
is
all
aggressive
tuning,
monitor
their
compaction.
Think
about
reversing
it
once
you've
resolve
the
underlying
issue.
A
To
get
some
insight
on.
What's
going
on
and
again,
this
is
to
buy
time
to
resolve
the
underlying
issue.
I
might
run
something
like
this
on
the
heap
1200
Meg
new
size,
a
decrease,
the
survival
ratio
to
for
this
means
I've
got
an
eight
hundred
mega
eatin
heap.
This
is
where
objects
are
created
and
to
200
Meg
survivors
increase
the
max
tendering
threshold
to
for
this
slows
down
the
rate
of
tendering
for
objects
from
the
new
heat
to
the
tenured
heap,
I,
look
in
the
logs
and
I'm
looking
for
patterns
like
this
age.
A
This
was
these
nodes
have
very
wide
rows
and
1.3
billion
rows
on
them.
So
there
was
over
three
gigs
of
bloom
filters
because
it
was
using
the
older
default
for
the
bloom
filter
FP
chance,
as
well
as
like
pre
one
point,
0
point:
0,
0,
0
744,
so
bloom
filters
were
huge.
We
increase
the
bloom
filter,
FP
chance,
2.1
again
that
really
aggressive
setting
we're
taking
a
hit
here
on
some
latency
from
one
of
the
column,
families
and
went
to
the
default.
A
The
modern
default
of
point
0
1
for
the
others
that
really
reduced
our
bloom
filter,
use,
I,
increase
the
index
interval
from
128
to
512
and
the
same
time
increase
the
key
cash
and
then
started
running
with
these
settings,
which
just
designed
to
slow
down
tenure
of
objects.
Again,
this
is
designed
to
buy
time
to
try
to
resolve
the
underlying
she's.
A
Finally,
I
mean
the
look
at
the
partition
that
I
saw
a
while
ago,
but
it's
the
most
interesting
problem,
I,
probably
ever
dealt
with
I
was
going
through
a
cluster
and
I
was
doing
a
rolling
restart
to
change
some
config
restarted
no.23
its
up.
We
can
see
that
the
gossip
is
active,
I
always
check
before
I
restart
anything,
but
my
expectation
of
which
nodes
are
up
and
down
is
true,
so
go
to
another
node
and
I.
A
Look
at
the
note,
I'll
ring
and
it
says
no
23
is
down
I
p60
10
I
go
back
to
New
23.
It
thinks
that
everyone
else
is
up.
So
we've
got
a
problem
here.
No
23
thinks
everyone's
up.
Everyone
else
thinks
no
23
is
down.
So
this
was
a
running
production
system
in
the
run-up
to
Christmas.
Getting
a
little
bit
nervous.
Now,
no
23
could
serve
requests
for
keys
in
its
primary
token
range
at
local,
quorum,
quorum
and
all
consistency.
It
thought
the
entire
ring
was
running
every
other
node.
A
When
I
tried
to
read
a
row
that
was
from
the
token
rains
it
was
primary
for
23
could
do
local,
corn
and
corn
that
couldn't
do
all
so
they
swore
that
part
of
this
system
was
down.
The
next
thing
to
do
was
relax.
The
application
is
up
we're
selling
stuff,
it's
the
run-up
to
Christmas,
but
the
system
works
time
to
dig
a
little
deeper.
A
Gossip
is
the
thing
that
nodes
used
to
communicate,
to
tell
them
whether
or
not
they're
up
and
down
so
no
tool
has
a
command
to
see
the
internals
of
what
we're.
Storing
gossip
info
tells
me
that
the
status
of
no
23
is
normal,
dig
a
little
deeper
go
to
the
logging
enable
trace
logging
on
the
gossiper.
This
guy
runs
all
the
time.
A
A
So,
let's
get
a
bit
more
of
the
logging
turn
on
more
logging
on
no
20
here,
I
can
actually
see
in
the
first
that's
highlighted
no
23,
IP
6010,
actually
delivering
gossip
to
this
node
and
the
next
line
it
outputs
and
says
this
is
all
of
these.
Are
the
generation
and
the
version
numbers
that
we
have
for
all
of
these
IPS,
and
the
next
thing
I'd
expect
to
see
is
a
call
to
the
failure
detector
on
no
20
that
says:
hey
I,
just
heard
from
load
6010,
no
23.
A
You
should
mark
that
guys
up
because
we
just
heard
from
it
directly,
but
that
didn't
happen,
and
the
problem
was
that
gossip
has
a
two
phase
approach
to
its
things.
It
has
a
generation
number
which
is
initialized
when
the
server
bootstraps
two
seconds
past
the
epoch.
If
you
were
ever
seen,
gossip
a
generation
number
that's
higher
than
yours,
then
you
agree
that
everything
that
other
guy
has
to
say
about
the
node
is
true
and
more
relevant
than
what
you've
got.
There's
also
a
version
number,
but
that's
secondary
to
the
generation
number.
A
A
Looking
in
the
system
tables
using
the
command
line
interface,
you
can
see
the
generation
number
stored
there
and
the
time
stamp
on
this
guy
was
also
at
2025.
So
every
time
the
server
restarted
and
it
incremented
its
generation
number.
It
tried
to
do
a
right
with
the
current
timestamp
Cassandra
ignored
that,
because
that
timestamp
is
less
than
the
one
that
had
in
disk
and
we
couldn't
increase
the
generation
number
because
we
couldn't
increase
the
generation
number
after
added
restarted.
A
Its
version
number,
which
is
the
the
next
level
down
from
the
generation
number,
was
ignored
all
the
time.
None
of
the
other
servers
paid
any
attention
to
it
because
it
thought
it
was
an
older
version
of
that
server
running.
The
fix
here
was
to
do
a
delete
with
a
high
timestamp
flush
that
disk
run
a
compaction
to
get
the
tombstone
purged
and
then
restart
and
everything
was
back
to
normal
and
that's
an
example
of
a
partition
where
we
had
this
network
failure
of
a
bit
higher
up
in
the
stash.
A
The
application
was
able
to
keep
up
when
we
were
able
to
find
time
to
calmly,
go
and
fix
it.
Lastly,
maintenance
I
think
one
of
the
biggest
maintenance
tasks
that
you'll
ever
do
is
expand
to
multi
data
center.
If
you
haven't
done
it
already,
hopefully,
you'll
get
to
do
that
soon.
There's
a
number
of
steps
involved.
First,
we're
going
to
look
at
the
snitch.
The
simple
snitch
puts
all
of
your
nodes
into
rac1
and
data
center
one.
Obviously,
if
we
want
multiple
data
centers,
that's
not
really
acceptable.
We've
got
a
bunch
of
options.
A
Property
files,
niche
uses,
one
property
file
lists
all
the
IPS
and
their
data
center
and
the
rack
that
same
file
has
to
be
distributed
to
all
nodes.
We've
got
the
rack
inferring
snitch
that
uses
the
second
octet
of
the
IP
for
address
as
the
data
center
name
and
the
third
octet.
As
the
rack
theme
and
we've
got
these
things,
I
call
the
gossip,
be
snitches
easy
to
snitch
easy
to
multi
region
and
gossiping
property
file
snitch
when
the
easy
to
snitch
needs
to
answer
the
question
of
what
is
my
data
center
and
rack?
A
It
uses
the
easy
to
reflector,
gets
the
AWS
region
and
availability
zone
and
uses
those
as
DC
and
rack
when
it
has
to
answer
the
question:
why
there's
a
decent
rack
of
a
remote
node,
it
looks
in
the
gossip
state
for
a
remote
node
and
expects
to
find
that
it's
put
its
data
center
and
rack
in
there.
So
you
can't
update
to
the
AC
to
snitch
in
a
rolling
restart,
because
the
other
nose
won't
have
updated
their
gossip
state,
but
the
gossiping
property
file.
Snitch
is
a
bit
different.
A
It
uses
a
config
file
that
lists
my
and
only
my
data
center
and
rack
when
it
needs
to
answer
the
question
about
someone
else's
data
center
and
racket
looks
in
the
gossip,
but
if
it
can't
find
it
there,
it
falls
back
to
the
same
property
file
as
the
property
file
snitch.
So
you
can
upgrade
to
this
guy
in
the
rolling
restart
and
we're
going
to
use
that
today.
A
word
of
warning.
First,
if
you
change
the
snitch
and
you
will
know,
changes
its
data
center
or
its
rack,
your
data
goes
missing
still
on
disk.
A
Okay,
so
we're
going
to
move
to
this
guy,
go
to
the
existing
nodes,
go
to
the
cassandra
topology
properties,
and
here
I'm,
assuming
you're,
coming
up
from
the
simple
snitch
list,
all
the
existing
nodes
and
put
their
existing
data
center
and
rack
in
there.
I
also
set
the
default
at
the
bottom
of
this
file
for
any
node
not
listed
to
be
the
new
data
center,
because
I
want
any
node.
That's
accidentally
started
when
we're
going
through
this
process
to
be
isolated.
A
I
don't
want
to
accidentally
join
my
old
data
center
and
start
causing
me
trouble
now
go
to
the
Cassandra
racked
DC
properties.
This
is
the
one
used
by
the
gossiping
property
file
snitch
on
the
existing
nodes
and
they
update
their
data
center
and
the
rack
not
really
technically
necessary,
but
I'd
like
to
see
the
same
config
on
every
node.
A
Finally,
we
do
a
rolling
restart
on
the
existing
machines
after
we
update
their
yanil
file
to
the
gossiping
property
file,
snitch
and
now
we're
done-
and
this
is
one
of
these
great
maintenance
tasks
that
leaves
you
exactly
where
you
started.
Hopefully
all
of
our
nodes
are
sitting
there.
They
haven't
changed
data
center
or
rack,
but
we've
made
a
change
under
the
hood
next
thing.
A
A
The
token
range
that
contains
the
Roky
that
we're
interested
in
select
that
one
and
then
keeps
counting
until
it
gets
to
the
replication
factor
and
says
done
so
we
end
up
with
a
pretty
simple
picture
like
this:
foo
is
going
to
get
stored
and
in
this
token
range,
the
first
blue
node
is
the
primary
replica
for
it
and
the
other
nodes
are
they're,
ordered
by
their
token
ranges.
So
they
get
selected.
The
network
topology
strategy
is
more
complicated
in
orders.
A
The
token
ranges,
but
within
the
scope
of
a
data
center
it
finds
the
token
range
that
contains
the
rookie
that
we're
interested
in
and
select
that
and
then
moves
on
to
find
the
first
unselected
token
range
again
in
the
oil
in
their
order.
That's
from
a
different
rack
than
the
one
we've
already
started
at
and
selects
that,
and
it
follows
this
process.
If
it
hasn't
picked
up
enough
token
Rangers,
then
it
goes
loops
around
and
start
selecting
a
second
token
range
from
the
racks
that
we
have.
So
you
end
up
with
a
picture
like
this.
A
Here's
food
in
the
token
range,
the
first
node
in
each
rack,
has
got
a
replica
of
it.
If
you've
only
got
one
Rako,
you
end
up
with
a
picture
like
this,
which
is
the
same
picture
that
we
have
for
our
simple
strategy
and
that's
what
we
want.
We
don't
want
to
magically.
Have
the
replication
strategy
now
say
this
node
that
was
on
this
row.
That
was
on
those
one
two
and
three
is
now
unload
14
and
seven
that
would
be
bad
out.
Datarow
suddenly
disappeared.
A
So
again,
if
you're
changing
the
replication
strategy,
be
careful
if
you're
got
existing
multiple
racks
in
there
to
do
it
either
using
the
command
line,
CLI
or
cql,
you
can
go
and
update
the
keyspace
configuration
here,
I'm
putting
it
in
with
network
topology
strategy
and
I'm
setting
the
RF
23
for
the
existing
data
center.
Assuming
that's
what
we
had
and
I
put
the
new
data
center
in
with
an
r
f
of
0.
A
Finally,
now
we'll
put
on
the
new
nodes,
we're
going
to
disable
the
auto
bootstrap
feature
this
used
to
be
a
setting
in
yama.
It's
not
there
because
we
wanted
on
all
the
time,
but
you
can
put
it
in
and
say
that
the
false
we'll
look
at
that
in
a
second
remember
we're
using
the
gossiping
property
file
snitch.
That's
not
the
default
remember
to
set
the
seeds
so
that
the
nodes
in
the
new
data
center
know
how
to
find
the
nodes
in
the
old
data
center
and
the
convention
we
use
is
three
seeds
from
each
DC.
A
A
Remember
we've
also
got
to
set
the
Cassandra
racked
DC
properties.
This
time
these
are
the
new
nodes
they
get
set
to
be
in
the
new
data
center
and
their
rack.
We
can
ignore
the
Cassandra
topology
properties.
We
don't
need
that
now
when
we
turn
these
new
guys
on,
they
will
do
nothing,
and
that
is
exactly
what
we
want
them
to
do.
They
will
join
the
ring,
they
will
have
zero
data
and
there
won't
be
getting
any
traffic.
A
You
shouldn't
see
anything
coming
through
in
TP
stats
for
right
and
reads:
that's
exactly
what
we
want
we're
going
to
update
the
replication
factor
next
same
process
we
use
before.
We
turn
that
on
two
three
and
after
we've
done
this,
your
clients
will
be
talking
two
coordinators
in
the
old
DC.
Your
old
DC
will
be
sending
the
right
to
coordinate
two
nodes
in
the
old
DC
and
sending
it
over
two
nodes
in
the
new
DC.
Your
client
will
have
to
wait
for
that
response.
A
A
A
A
So,
in
summary,
we
talked
about
platform
and
tools.
We've
looked
at
some
problems.
The
first
thing
you
need
to
do
when
something
goes
wrong
is
relax.
Try
the
understand
is
this
affecting
production,
or
is
this
just
Cassandra
failing
in
a
way
that
means
that
our
app
can
still
run
think
about
the
platform
and
what
it's
trying
to
achieve
at
the
high
level,
with
all
of
the
nodes
at
the
low
level,
with
the
storage
engine,
use
those
tools
to
understand
and
try
and
diagnose
and
say,
okay,
what
level
of
requests
can
we
serve?
A
Can
we
are
we
working
at
corn?
If
not
is,
are
we
not
working
at
quorum
for
the
entire
ring
or
just
for
a
couple
of
token
ranges?
You
might
be
able
to
do
that
just
by
looking
at
the
ring
up
with
all
the
status
output
and
always
maintain
availability.
So
to
do
that,
I
am
ultra
careful,
of
course,
I
always
work
on
other
people's
databases,
so
that
makes
me
double
careful,
but
I
don't
go
out
and
make
a
change
on
every
node
and
then
see
the
impact.
It's
a
great
idea.
A
A
A
So
the
question
is
what
they
were:
users,
a
dashboard
to
see.
What's
going
on,
for
monitoring
from
what
I
do
as
a
consultant,
I
generally
use
what
people
have
in
place.
Ops
center
is
a
great
tool.
It
gives
you
a
good
overview,
it's
easy
to
set
up
and
get
going
if
you've
got
an
existing
system,
you've
got
graphite
or
something
else.
A
It's
a
really
good
idea
to
roll
that
into
your
existing
system,
so
that
you
get
all
of
the
OS
level
metrics
that
you
might
already
be
monitoring
on
that
box,
put
together
with
the
job
with
the
Cassandra
ones
and
you're
going
to
be
interrogating
jmx.
If
you
get
out
on
the
net,
you'll
find
some
examples
of
that
out.
There
yeah.
A
C
A
You
accidentally
change
your
nose,
DC
or
and
ring
reverse
it.
The
data
is
not
lost.
The
data
is
never
deleted
off
this.
There
is
two
two
phases
here:
finding
out
what
know
to
look
at
and
then
actually
reading
the
data
off
the
node
and
we've.
If
we
have
that
mistake,
we've
mucked
up
the
first
bit
and
we're
just
looking
at
the
other
nodes.
If
we
can
roll
back
we'll
start
to
look
and
the
data
will
still
be
on
desk.
D
A
It
necessary
to
take
notes
out
of
the
ring
for
major
compaction.
No,
but
that's
qualified
on
the
reason
that
you're
doing
it.
If
it's
because
you've
got
a
node
that
is
really
sick,
it
might
make
sense,
but
compaction
and
repair
or
all
online
operations.
I
know
sometimes
I
don't
feel
like
they
might
take
a
large
tax
on
their
node,
but
they're
meant
to
be
online
great.