►
From YouTube: Netflix: Active-Active Behind the Scenes
Description
Speaker: Roopa Tangirala, Senior Cloud Data Architect at Netflix
High availability is an important requirement for any online business and trying to architect around failures and expecting infrastructure to fail, and even then be highly available, is the key to success. One such effort here at Netflix was the Active-Active implementation where we provided region resiliency. This presentation will discuss the brief overview of the active-active implementation and how it leveraged Cassandra’s architecture in the backend to achieve its goal. It will cover our journey through A-A from Cassandra’s perspective, the data validation we did to prove the backend would work without impacting customer experience. The various problems we faced, like long repair times and gc_grace settings, plus lessons learned and what would we do differently next time around, will also be discussed.
A
A
Thank
you
all
for
taking
time
and
coming
to
the
summit
today,
and
especially
to
this
talk
for
those
who
don't
know
me
I'm
Roopa.
Thank
you
allah.
I
am
the
engineering
manager
at
netflix.
We
have
a
wonderful
flower
database,
engineering
team
and
we
provide
cassandra
as
a
service.
Our
team
actually
primarily
manages
all
cassandra
clusters
upgrades
them.
Does
data
modeling
capacity
planning,
monitoring
anything
related
to
cassandra?
We
try
to
serve
as
a
service
without
further
ado.
A
Let's
go
in
the
topic
of
my
presentation
is
active,
active
Cassandra
behind
the
scenes,
few
slides
about
Netflix
right,
you
all
no-dvd
Netflix
started
as
a
DVD
by
mail
company
and
within
few
years
we
actually
have
expanded
into
online
streaming
and
are
pretty
much
on
all
the
devices
name.
It
you
name
it
we
are
on
it.
Be
gaming
console
be
HDTV,
beat
blu-ray
player,
tablets,
phones.
We
are
pretty
much
on
all
the
devices
and
try
to
provide
very
rich
content
right.
We
have
very
thousands
of
titles
in
the
library,
along
with
that.
A
We
try
to
have
our
own
originals
actually,
which
add
flavor
and
lot
of
peas
are
like
house
of
cards.
Orange.
Is
the
new
black
are
pretty
popular
titles
and
people
like
to
do
binge-watching
right
and
it
becomes
very
critical
as
a
service,
we
need
to
be
always
up
and
running
so
high
availability
is
very
critical
for
us,
and
that
is
what
we
will
be
covering.
How
did
we
achieve
that?
A
So,
let's
start
active
actor
right.
What
is
active
active?
What
do
we
mean
by
that
just
a
simple
definition
and
also
called
dual
actor?
It
is
a
phrase
to
describe
network
of
independent
processing
nodes
connected
to
a
replicated
database
right
in
case
of
failure,
a
failed
node
that
request
either
go
to
the
existing
nodes
or
our
load
balanced
across
the
remaining
nodes.
So
this
is
a
very
high
level
definition
why?
A
Why
do
people
want
to
go
active
utter
right,
so
as
companies
which
start
primarily
as
a
small
biz
and
start
in
their
own
data
center,
they
actually
have.
You
know
the
solving
Enterprise
IT
solutions.
They
are
architected
around
having
their
own
data
center,
their
own
IT
ops,
but
as
they
scale
and
grow,
it
becomes
very
critical
that
they
need
to
go
to
web
scale,
solutions
right
and
for
web
scale.
Solutions
because
it
is
very
critical
for
their
application,
as
well
as
their
infrastructure,
to
be
rapidly
scaling
and
even
then
be
highly
available.
A
So
high
availability
is
really
very
important
for
any
online
business
to
succeed.
So
one
such
effort
at
Netflix
was
an
active,
active
implementation,
and
this
talk,
this
talk
will
actually
walk.
You
through
active
active,
are
active,
active
implementation
from
Cassandra's
perspective.
So
what
what
was
the
process
we
used
for
doing
active
active?
What
were
the
validations
we
have
to
do
and
what
are
the
takeaways?
What
will
we
be
doing
different
next
time?
A
So,
let's
look
at
a
few
failure
scenarios
right.
There
are
a
lot
of
ways
in
which
things
can
fail,
even
in
cloud
or
in
data
center.
So
does
an
instance
fail?
Yes,
it
can.
It
can
be
a
bad
hardware
or
it
can
have
some
latent
issues
or
it
can
also
be
related
to
some
push
on
that
instance
right
and
how
do
we
test
bad
instances
we
actually
tested
through
chaos?
Monkey
does
a
zone
fail
zone
in
Amazon?
Is
a
data
center
right?
A
A
Does
a
region
fail
region
failures
are
really
very,
very
rare
right,
but
they
do
happen.
We
have
been
hit
Netflix
actually,
as
a
company.
We
push
the
limit
of
my
software
and
hardware
as
well,
and
we
have
seen
failures
in
the
past,
which
may
be
you
know
in
the
data
center.
When
we
were
running
in
Oracle,
we
would
see
a
lot
of
failures
which
which
are
one
in
a
million
chance
of
happening,
but
we
will
get
hit
right,
so
region
failures
are
similarly
like
that
we
were
hit.
A
I
will
cover
the
history
behind
it,
but
region
failures
also
happen,
and
it
is
very
specific
to
mostly
region-wide
configuration
issues
right
and
how
do
we
test
that
we
actually
tested
using
chaos?
Calm,
so
so
cross
case.
You
know,
eventually
everything
will
fail
right.
So
what
defines
a
service
is
how
they
architect
around
failures
and
even
then
try
to
be
highly
I,
try
to
be
highly
available.
So
it's
very
important
that
you
have
to
keep
your
services
up
and
running
by
embracing
isolation
and
redundancy
right.
So
what
do
we
mean
by
isolation?
A
Isolation
is
basically
a
sharding
and
partitioning
the
regions
in
such
way.
Such
that
changes
in
one
region
do
not
impact
the
others
right.
Regional
outages
in
one
region
do
not
have
any
effect
on
the
other
region
and
network
partition.
Events
net,
like
network
partitioning
between
the
regions,
do
not
affect
the
functionality
or
operations.
A
What
do
we
mean
by
redundancy?
Redundancy
is
basically
making
more
operating
much
everything
right.
You
have
zones
and
you
have
regions,
so
you
distribute
your
services
and
your
back-end
in
such
a
way
that
they
are
spanning
multiple
regions
and
multiple
zones
so
that
you
get
more
availability.
So
a
brief
history
of
why
we
had
to
look
into
active,
active
or
why?
A
What
what
was
the
triggering
event
which
actually
made
us
go
active
actor
so
back
in
Christmas
Eve
in
the
year,
2012
Netflix
suffered
a
multi
our
outage,
and
it
was
actually
a
regional
outage
related
to
a
maintenance
process
where
data
was
inadvertently
run
against
production
and
it
took
a
while
I
took
out
our
front
load,
balancers
elastic,
load,
balancers
and
it
it
was
actually
a
very
bad
experience
because
customers
were
not
able
to
stream.
So
we
and
as
a
service,
we
are
Chile
strive
for
being
highly
available.
A
We
want
to
be
different
in
that
ways
and
it
became
very
critical.
So
we
went
back
to
the
drawing
board
and
you
know
thought
through
the
process.
How
can
we
use
Cassandra's
architecture
and
make
it
work
for
us
and
create
a
solution
which
will
actually
make
us
very
highly
available
and
agile
right?
So,
if
you
look
at
this
architecture
in
there
are
a
lot
of
components.
I
know
it
can
be
confusing,
but
I
want
you
to
focus
on
the
lowest
layer
where
I
am
showing
the
multi-regional
Cassandra
cluster,
which
is
spanning
the
two
regions.
A
One
is
us
East
one
and
the
other
is
us
best
to
and
we
have
a
replicated
crash
in
front
and
then
you
see
all
the
mid-tier
services
and
the
different
apps
and
the
front
ends
which
are
actually
taking
in
your
request,
so
in
a
very
stable
state.
Based
on
your
location,
your
request
is
geo,
routed
to
either
us
east
or
us
west
right.
A
So
this
is
a
good
state,
happy
state
and
when
there
is
a
failure
or
say
we
had
the
same
elb
issue
again,
all
the
traffic
will
be
routed
to
the
region,
which
is
healthy,
and
this
is
pretty
seamless
right
and
suddenly
you
have.
You
know
if,
within
few
seconds,
your
request
go
to
the
region
which
is
highly
available
and
then
there
is
no
disruption
to
the
service
or
the
customer
experience.
A
So
this
this
is
the
basic
theme
like
what
was
the
history,
and
why
did
we
go
with
active
actor
and
now
I
will
go
over
the
process?
Actually
what?
What
are
the
different
steps
from
Cassandra's
perspective,
which
we
had
to
make
sure
that
active
active
would
work?
So,
let's
go
with
the
process.
The
very
first
process
is
actually
identifying
all
your
clusters,
which
need
to
be
extended
so
think
about
it
like
as
a
company
we
are
have.
We
are.
A
We
have
to
maintain
90
clusters
and
not
each
and
every
application
needs
to
be
in
you
know
in
not
each
and
every
application
needs
to
be
used
to
service
a
particular
request.
There
are
a
lot
of
applications
like
logging
or
some
batching
applications
which
don't
need
to
be
active
active,
so
you
have
to
work
with
your
application
owners
to
identify
which
data
needs
to
be
replicated,
and
this
seems
a
very
simple
task,
but
it
takes
a
lot
of
cycles.
So
I
said,
let's
call
that
out.
A
Then
we,
as
we
know,
as
you
know,
we
are
running
in
Amazon
environment.
We
for
an
island
cluster.
You
would
be
using
if
you
are
Alan
cluster,
you
would
be
using
ec2
snitch
ec2
snitch
is
basically
using
private
eye
peas
for
its
communication
between
the
zones.
So
when
you
have
to
convert
it
into
multi-regional
cluster,
you
have
to
actually
change
the
snitch
to
use
ec2
multi-region
snitch
and
which
will
use
public
IPS
to
talk
across
the
regions
now,
the
think
of
it
in
amazon,
iraq
as
a
zone
and
a
region
as
a
data
center.
A
That's
how
its
configured
right.
So
you
once
you
do
this.
This
is
a
yeah
Mel
change.
It
would
need
a
restart,
but
this
change
is
very,
very
important.
The
it
doesn't
mean
that
in
ec2
multi-region
snitch,
the
private
eye-
peas
are
not
used
at
all
within
a
region,
which
is
the
lesson
address.
Cassandra
will
still
use
the
ssl
port
and
use
the
private
eye
peas,
but
it
is
only
for
the
profs
regional
traffic.
It
is
using
the
public
eye
peas.
A
This
is
another
configuration
change
which
you
need
to
do,
which
is
actually
we
have
exposed
this
in
premium
open
source.
It
has
pre
a
multi-region
enable
to
true,
basically
what
it
will
do.
Is
it
it
actually
makes
sure
that
you
are
able
to
communicate
cross
across
the
regions?
So
initially
what?
If
you
are
using
the
ec2
snitch,
you
will
be
using
the
storage
port
and
using
private
eye
peas
right
so
enabling
this
property?
A
Actually
it
creates
the
required
firewall
holes
so
that
your
cluster
roads
are
able
to
talk
and
look
at
other
nodes
in
the
other
region
using
public
IPS,
so
basically
converting
that
port
from
storage
pod
to
ssl
port.
This
is
specific
to
Amazon.
Here
you
had
a
question.
Priyam
is
a
actually
sidecar
which
manages
Cassandra
it's
open
source
that
is
developed
by
our
developers
and
Netflix.
So
basically
it
will
manage
Cassandra,
it
can
start
Cassandra
I
can
stop
it.
A
Does
the
backups
restores
so
it's
basically
added
functionality,
which
is
not
there
in
Cassandra
out
of
the
box,
then
you
actually
will
do
the
rolling
restart,
because
you
have
done
one
yamel
setting
and
the
other
is
a
security
group
setting.
Then
you
would
have
to
do
a
rolling
restart
for
your
snitch
changes
to
come
into
place.
A
The
next
step
is
actually
spinning
up
clust,
spinning
spinning
up
the
nodes
in
the
new
region.
As
you
see
like
here,
we
have
highlighted
that
say.
For
example,
you
have
a
three
node
Cassandra
cluster
and
your
application
is
in
u.s.
East.
This
is
the
light
cluster
of
taking
all
the
live
traffic
right
and
you'd
made
the
snitch
changes.
You
did
a
rolling
restart
for
this
us
east
side,
and
now
you
spin
up
nodes
in
the
u.s.
vests
too.
Now
what
this
becomes
is
a
one
global
ring
right,
the
right
side
nodes.
A
Don't
have
the
data,
yet
they
just
have.
They
are
just
part
of
the
ring,
but
without
any
the
data.
So
this
is
the
you
have
to
actually
spin
up
those
nodes
once
those
nodes
are
up
and
part
of
a
ring.
What
you
do
is
you
actually
update
the
keyspace?
If
your
cluster
has
five
key
spaces
or
ten
key
spaces,
you
have
to
update
each
and
every
of
them
individually.
A
The
reason
is,
like
you
have
to
add
the
new
region
which
you
are
trying
to
add
to
the
global
ring
in
the
gee
option,
so
that
your
kids,
now
after
this
point
when
after
you
do
this
update
and
schemas,
are
in
agreement,
whatever
rights
you
do
in
the
u.s.
East
region
will
automatically
be
replicated
in
to
us
verse,
two,
so
the
rights
that
this
actually
enables
your
right
to
go.
You
know
it
is
you're
the
new
region
which
has
been
added.
It's
taking
all
the
live
right
traffic,
but
I
did
not
show
any
application
there.
A
So
we
are
not
ready.
Yet.
We
cannot
just
spin
up
apps
in
the
new
region
because
the
historical
data
is
missing
and
there
are
other
things
which
need
to
be
done
will
go
through
so
after
this
update.
What
you
need
to
do
is
basically
to
get
all
the
history
back
right.
You
can
run
the
node
tool
rebuilt
command
on
the
new
region.
In
our
case
it
was
the
u.s.
west
to
region.
A
A
We
did
not
see
any
performance
on
the
live
cluster
impacted
because
of
the
rebuild
because
he's
just
streaming
the
data
it's
similar
to
bootstrap,
but
it's
actually
much
quicker
before
this
rebuild
was
there
like
we
rebuild,
was
introduced,
I
think
in
one
I
am
not
sure
exactly
which
version,
but
we
had.
We
had
to
extend
prior
to
one
as
well
and
in
those
scenarios
we
were
actually
using
repair
and
think
about
it.
Like
the
new
cluster
new
region.
A
Does
not
have
any
data
set
and
the
repair
was
actually
taking
very
very
long
because
it
has
to
populate
all
the
data.
So
this
really
helped
no
tool
rebuild
is
a
great
tool
also
whenever
we
see
that
sometimes
cassandra
does
not
come
up
with
data
due
to
whatever
reason
we
do
use
no
to
rebuilt
to
basically
repopulate
the
data
set.
It
gives
us
a
you
know
faster
edge
to
have
the
data
seeded,
and
then
we
can
do
the
repair.
A
The
next
process
is
running
the
node
tool.
Repair,
no
tool
repair
is
very
very
you
have
actually
I.
Think
most
of
you
who
are
using
Cassandra
very
seriously
and
have
production
installations
would
understand
like
this
repair
is
really
really
it
takes
time,
but
it
is
very
necessary
right
because
eventually,
Cassandra's
is
an
eventually
consistent
system
and
repair
is
the
there
are
repair
and
other
knobs
as
well
to
make
the
data
consistent,
but
you
should
definitely
run
repairs.
A
So
this
was
the
whole
process
of
making
your
one
live
cluster
and
extending
taking
it
and
extending
into
the
new
region
where
you
want
to
set
a
factor
vector
right
so
with
we.
We
knew
you
knew
the
process,
we
said.
Ok,
we
are
ok
now,
but
then
you
know
our
developers
and
all
our
application
owners.
They
had
a
very
important
question.
They
said:
ok,
we
are
think
we
are
actually
tasking
Cassandra
for
the
replication
and
we
trust
Cassandra's
replication.
But
how
long
does
it
take
for
a
right
to
replicate
to
the
other
region?
Why?
A
Because
we
are
using
it
for
active
active.
That
is
very
critical.
We
don't
want
a
lot
of
delay
and
in
what
scenarios
do
delay
happen?
If
we
identify
them,
we
can
work
around
it
right.
So
I'll
go
over
the
validations
which
we
did
to
make
sure
Cassandra
as
a
service
and
Cassandra.
The
technology
for
active
active
would
work
and,
if
he'll
be
able
to
answer
any
questions.
So
for
this,
what
we
did
was
we
actually
did
a
global
benchmark.
Cassandra
global.
A
We
took
a
very
heavy
right,
intensive
application
and
we
restored
it
from
backup
followed
the
same
procedure
which
I
explained
to
you
in
the
previous
slides
the
whole
process
of
extending
a
u.s.
East
cluster
into
u.s.
vests
too.
So
it
was
18
terabyte,
backup
data.
We
actually
back
a
we
restored
it
from
s3,
extended
the
cluster
into
u.s.
vests
too,
and
we
just
kept
running
load
right,
so
the
validation.
What
we
wanted
to
do
was
we
did
1
million
rights
in
one
region
and
read
those
1,000,000
back
and
we
it's
just
way
after
500
millisecond.
A
We
waited,
but
it's
not
needed
like
it's
just
to
prove
that
whatever
reads
you
do
and
we
did
it
with
cl
one.
That
is
the
main
difference
here,
because
we
wanted
to
see
if,
even
if
we
do
not
use
quorum
or
higher
consistency,
if
we
just
use
CL,
1
and
just
write
in
one
region,
does
all
the
data
go
through
or
like
how
many
times
do
the
like?
How
many
records
do
not
make
it
through
the
other
region?
What
percentage
of
income
syston
see
do
we
have
because
we
don't
have
those
tools
exposed?
A
Yet
it's
very
important
question
people
keep
asking
how
much
entropy
do
I
have
in
the
system
when
should
I
be
running
repair
when
should
I
should
not
be
running
repair?
So
since
we
didn't
have
all
the
tools
in
place,
we
actually
ran
this
1
million
benchmark
and
we,
this
is
thinking
all
nodes
are
up.
There
was
no
network
partition,
no
issues
at
all
with
any
of
the
nodes.
A
We
actually
were
able
to
read
back
all
the
1
million
without
any,
without
with
no
data
loss,
so
that
actually
helped
us
validate
that
okay,
Cassandra
replication
does
work
and
it
can
be
used
for
activator.
The
other
thing
we
wanted
to
prove
was
how
does
Cassandra
perform
when
there
is
a
thundering
herd?
So
when
you
have
two
regions
in
place
and
one
region
is
taking
to
both,
the
regions
are
taking
fifty
fifty
percent
of
traffic
and
suddenly
there's
a
there's,
an
issue
with
the
region.
A
You
have
to
fail
over
to
the
other
region
right,
so
suddenly
you're
the
region
which
is
active
and
healthy.
It
receives
double
the
traffic.
So
we
want
you
to
see
what
kind
of
problems
we
have
and
what
are
the
effects
on
Layton
sees
on
the
application
side
when
we
actually
move
the
traffic,
so
we
did
test
for
thundering.
A
Her
too
I'll
show
you
some
graphs
in
the
future,
slides
which
will
actually
show
you
the
things
we
were
measuring,
but
this
is
some
one
of
the
important
validation
we
need
to
do
whenever
we
want
to
make
sure
you
know
you
have
enough
capacity
and
cluster
can
handle
test
for
retries.
So
if
you're,
one
region
is
not
performing
well
and
it
is
slow
or
its
latent,
you,
your
application
is
bound
to
make
lot
of
retry
calls
and
so
move.
A
Even
one
is
thundering
herd
and
over
that
you
would
also
have
retries
in
place
when
the
traffic
is
moved
to
the
newer
region.
So
this
we
also
wanted
to
make
sure
that
when
we
are
doing
all
these
retries,
what
are
the
effect
on
Layton
sees.
How
is
the
application
behaving
so
for
all
these
tests?
What
are
the
metrics
we
were
using?
We
were
basically
focusing
on
ninety-fifth
and
ninety-ninth
read
latency,
and
this
is
both
at
the
cassandra
coordinator
level
as
well
as
at
the
client
level,
because
it
is
very
important
that
these
both
should
match.
A
If
there
is
a
mismatch,
that
means
either
you
have
some
bottleneck
in
your
app
or
there's
something
going
on
at
the
Cassandra
side
right.
So
this
is
one
thing
which
is
critical
drop
metrics
on
Cassandra,
so
if
Cassandra
is
under
stress
or
it
is,
it
doesn't
have
enough
capacity,
you
will
see
that
you
know
it
starts.
Dropping
the
coordinator
will
start
dropping
metrics,
be
it
mutations
beat
read
reads
there
are
so
many
metrics
there,
so
it's
we
actually
in
in
our
team.
We
monitor
this
matrix
and
that
actually
shows
us.
A
The
health
of
the
cluster
look
for
exceptions.
We
looked
actually
are
there
any
exceptions
growing
when
we
are
moving
this
traffic
over
when
we
are
saying
that
doing
the
Thundering
Herd
testing?
Are
there
any
exceptions
on
Cassandra?
How
is
Cassandra
behaving?
How
is
the
heap
like
is
hiep
a
regular
C
sought
pattern,
or
is
it
actually
going
up
causing
Layton
sees
and
other
kinds
of
problems?
So
we
did
want
to
capture
that
what
are
the
thread
spending?
So
this
is
another
indicator.
If
a
system
is
actually
having
difficulty
coping
up,
you
will
see
the
thread.
A
Spending
increasing
on
a
node,
so
we
have
like
I
think
if
you
saw
Chris
ptosis
talk,
we
have
Atlas
as
a
monitoring
system
and
we
have
ways
to
you
know
pinpoint
a
particular
node
which
actually
is
a
troublesome,
node
or
is
causing
issues.
So
during
all
these
tests,
we
were
measuring
these
metrics
to
see
the
health
of
Cassandra
a
just
a
quick
slide
on
the
configuration
for
the
test.
The
back
Cassandra's
configuration
was
24,
node,
SSD
cluster.
A
So
that
in
the
three
replicas
in
each
region
right
so
as
we
extended
this
region,
it
became
six
replicas
total.
So
so
that's
the
other
thing
as
and
when
you
add
more
regions-
and
you
add
more
replicas-
you
keep
adding
more
data
set,
so
your
cluster
gross,
but
then
so
the
next
slide.
This
one
actually
showing
you
the
normal
load,
so
that
particular
production
cluster
we
had
in
prod
it
was
taking
around
12
k,
reads
per
second
and
it
was
taking
around
1000
through
the
batch
he
write
operation,
so
it
was
going
up
to
1.2.
A
So
since
we
wanted
to
test
for
thundering
herd,
we
did
like
we
started
at
25
to
25
k,
reads
per
second
and
around
700
writes
per
second,
and
you
see
that
sudden
bump,
because
we
did
want
it
to
be
sudden.
We
didn't
want
it
to
be
gradual,
because
now
we
know
gradually
caches
fill
up
at
the
latest,
the
performance
improves.
But
what?
If
there
is
a
sudden
increase?
How
is
the
performance?
So
you
see
those
that
thundering
herd
effect
there,
and
these
are
the
latencies.
A
These
latencies
don't
mean
anything
for
us
for
this
particular
application
before
it
was
15
milliseconds
and
then
during
the
Thundering
Herd.
When
we
increase
the
read
ops,
it
became
245
milliseconds,
which
is
very
good
for
her
95th
and
ninety-ninth
went
from
40
to
80,
but
the
point
I'm
making
here
is
it's.
This
is
very
specific
to
the
application
which
we
were
using
and
the
load
we
were
generating
the
payload
it
was
having
so
it
these
numbers
can
change
for
your
app
right.
So
these
each
application
will
have
different
SLA.
A
It
will
have
different
sensitivity
to
latency.
So
you
have
to
make
an
effort
to
know
your
clusters
and
see
what
is
what
is
their
ceiling
right.
So
this
is
another
thing
right
we
did
want
for
this
particular
cluster.
We
wanted
to
know
how
much
maximum.
When
can
we
drive
the
read
ops
and
write-offs
without
impacting
the
latency,
for
example,
these
guys
they
had
SLA
off
below
99
to
be
below
100
millisecond
and
if
it
is
more
than
that,
they
didn't
want
like.
That
means.
There's
some
problem
right
for
them.
A
They
have
the
time
outset
and
all
according
to
that.
So
it
is
very
important
that
you
determine
the
ceiling
for
your
clusters
and
each
cluster
will
be
different
depending
upon
the
application
test
for
Network
partition.
So
this
was
also
important
when
you
have
two
regions,
and
you
are
saying
that
we
will
be
using
it
for
active
active.
There
will
be
instances
when
one
region
is
not
able
to
talk
to
other
right.
A
So
what
we
did
for
this
particular
case
was
we
did
it
in
tests
and
we
updated
the
IP
tables
so
that
the
nodes
are
not
able
to
talk
on
the
SSL
storage
port
across
region
and
we
could
actually
mimic
a
region
outage.
So
in
one
region
we
saw
that
all
the
other
nodes
in
the
other
regions
were
marked
down,
so
this
was
only
seeing
the
local
ring.
We
actually
I'll
show
you
in
the
takeaways,
but
we
did
find
out
lot
of
cases
where
people
were
using
a
quorum
and
that
would
fail.
A
So
what
are
the
takeaways?
What
did
we
learn
from
it?
The
very
first
takeaway
is
this
right.
Repairs,
after
extension,
are
very,
very
painful
so
and
since
you
are
doubling
the
data
set
and
you
are
adding
a
new
region
which
is
never
repaired
before
it
takes
a
long
time
and
we
have
few
clusters,
which
are
you
know,
hundreds
of
nodes,
and
so
that
adds
up.
So
it's
very
important
that
you
you
understand
that
any
time
you
are
adding
a
new
region,
you
have
enough
time
for
repairs
and
remember
in
the
process.
A
I
was
showing
you
that,
after
repairing
actually
there's
a
slide,
which
is
missing
because
after
repairing
is
the
place
where
you
can
enable
the
applications
in
the
new
region
so
that
they
can
start
taking
traffic
right.
So
it's
only
after
repair
that
you
should
have
your
applications
ready
to
access
the
new
region.
So
your
whole
migration
time
can
very
much
depend
upon
the
size
of
the
cluster,
the
time
it
takes
to
repair.
So
it's
not
just
like
you
have
to
plan
for
this
activity
time
to
rip
yeah.
A
No
rip
you
do
rebuild
for
the
historical
data,
but
rebel
will
not
does
not
mean
that
you
should
not
repair.
You
definitely
have
to
repair
after
rebuild
as
well
right,
so
repair
is
the
is
the
final
thing
in
the
whole
whole
operation,
because
rebuild
will
not
make
it
very
consistent
across
all
the
nodes.
Rebuild
is
actually
copying
data.
It's
like
similar
to
bootstrap,
where
it
is
getting
data,
but
it
matches
the
nearest
hash
and
gets
the
data
from
that
particular
token
right.
A
So
it's
it's
much,
so
you
do
definitely
have
to
repair,
even
after
rebuild
so
time
to
repair
depends
upon
the
number
of
regions.
We
have
few
clusters
where
we
are
extending
four
regions,
so
it
does
depend
it
does
actually
repairs
do
ya
know
we
did
not
have
any
notes
enable
here
we
yeah
we
have
not
tested
me
knows
yet.
We
do
not
know
yeah.
We
still
have
to
prove
that
it
would
work
for
us
personally
like
we
have
like.
A
We
have
test
environment
where
we
have
to
refresh
from
prod
to
test
every
week,
and
we
have
not
figured
out
a
story
there,
because
it's
very
difficult
to
do.
Those
backups
and
restores
so
so
a
time
to
repair
also
depends
on
the
replicas.
So
we,
since
we
are
running
in
Amazon
environment
and
we
have
three
zones,
we
use
three
replicas
so
that
you
know
we
can
take
care
of
the
zone
outages.
Even
if
one's
own
completely
goes
down,
we
still
are
available
right.
So
that
is
why
we
actually
have
three
replicas
in
each
region.
A
So,
for
example,
if
we
have
four
regions
you
will
have
12
replicas,
so
it
your
repair
time
actually
would
increase.
It
also
depends
upon
the
data
size,
of
course,
right.
The
bigger
the
data
set
they're
more
time.
It
will
take
more
than
the
data
size.
It
is
the
entropy
right.
If
you
are
changing
your
data,
a
lot
time
to
repair
would
increase
yeah.
A
Yes,
we
do
yeah,
we
do,
but
the
thing
is
like
each
use
case
is
so
different
and
even
if
we
say
it's
better,
not
to
delete,
of
course,
because
I
will
show
you
there
is
the
problems
to
delete
as
well,
but
applications
do
have
to
use
they
that
they
Cassandra
delete,
works
perfectly
well,
because
compaction
will
clear
off
the
tombstones
and
like
as
a
service.
It
is.
It
is
designed
for
that
right.
So
we
we
try
to
stay
away
from
any
standardizing.
We
actually
serve
full
freedom
and
responsibility.
A
We
do
tell
them
the
basic
guidelines,
but
it
is
up
to
the
developers
actually
right
so
that
ways
we
are,
we
do
not
want
to
be
the
bottleneck.
Basically,
our
team
will
tell
the
guidelines
will
help
them
work
through
the
whole
cluster
and
the
design
at
all,
but
we
will
not
impose
any
restriction
saying
that
hey
guys,
don't
do
this
or
do
this,
so
this
is
very
important
takeaway.
You
should
adjust
your
GC
grace
before
after
I
said.
After
extension,
it
should
be
before
extension.
A
That
is
the
error
in
this
slide.
So
GC
grace
is
actually
a
column
family
setting.
It
is
defined
in
seconds
and
default.
It
is
ten
days,
so
you
should
definitely,
since
after
region
is
added
repairs
take
more
time.
You
should
definitely
tweak
your
GC
grace
so
that
it,
the
number
of
days
fit
the
amount
of
time
it
will
take
you
to
repair
the
cluster.
A
This
is
another
important
factor
like
beware
of
deleted
columns
right
so
at
one.
So
if
you
have
an
application,
which
is
tombs
like
deleting
columns
at
a
very
fast
rate,
you
should
be
careful
of
increasing
the
GC
grace,
because
otherwise
you
will
have
a
row
which
is
actually
having
lot
of
tombstone
data
which,
since
you
have
increased
the
GC
grace
that
they
will
not
go
away
and
your
ro
gets
so
big
and
you
can
run
into
heap
issues.
We
had
issues
where
we
extended.
A
The
GC
grace
didn't
realize
that
the
app
was
actually
underneath
deleting
so
many
times,
and
it
caused
all
kinds
of
heap
issues,
because
that
all
those
tombstones
have
to
be
loaded
up
when
the
request
is
being
served,
though
the
coordinator
will
not
send
you
show
you
the
deleted
data,
but
the
the
replicas
actually
passed
that
data
to
the
coordinator,
so
that
adds
to
the
heap
issues.
Quite
a
lot
have
a
run
book.
This
is
very
important
because
you
definitely
need
to
make
sure
whatever
processes
you
have.
A
They
ideally
should
be
automated
and
you
there
is
no
other
way.
Actually,
you
can
test
all
the
scenarios.
You
definitely
need
to
automate
all
the
failure
scenarios
and
we
have
all
the
monkeys,
as
I
was
showing
you
before
the
simian
army,
where
we
tests
are
failures,
but
sometimes
fail.
They
also
don't
work
right,
so
you
do
definitely
need
run
book
and
need
to
maintain
that
run
book
as
and
when
your
services
are
being
added.
A
A
A
Oh
yeah,
that
ways
you
are
saying,
so
we
actually
we
don't
never
add
nodes,
1
node
at
a
time
like
what
we
do
is
we
try
to
double
the
cluster
because
of
the
way
our
tokens
are
arranged.
We
don't
just
add
two
or
three
nodes.
We
actually
double
the
cluster,
but
at
the
same
time
we
can
bring
one
note
at
a
time
we
are
not
using
vinos,
yes,
I,
consistency
level.
This
is
important
in
multi-regional,
cluster
quorum
is
not
equal
to
local,
quorum
and
quorum.
A
As
I
was
saying,
one
of
our
one
of
the
applications
we
found
out
was
using
quorum
calls
and
it
failed
miserably
right.
So
it's
very
critical
that
you
identify
all
your
applications
and
make
sure
you
are
not
using
quorum
for
consistency.
We
also
had
one
very
like
it
was
a
little
bit
of
surprise
to
us.
So
what
happened
is
as
soon
as
we
had
one
region
taking
live
traffic
and
reads
and
writes,
are
happening
properly,
no
problems.
A
So
as
soon
as
we
added
the
new
nodes
in
the
other
region
and
rebuild
the
cluster,
we
started
seeing
the
99th
percentile
latency
in
the
main
region.
Spiking
up
right
and
we
didn't
understand
what
you're
saying
hey,
that
was
only
happening
for
cl1
calls.
So
that
was
a
surprise,
because
we
didn't
understand
why
we'll
see
l1
go
cross
region,
but
there
is
an
edge
case
with
snitch
can
decide
and
actually
throw
some
calls
across
and
that's
why
your
99
gets
stipend.
A
So
for
this
Jason
Brown,
who
is
a
committer
and
he
was
at
Netflix,
he
actually
created
local
one.
He
passed
Cassandra
in
6202
and
we
recommend
local
one
for
read
so
that
your
reads
for
even
when
you
are
using
CL
1,
you
should
use
local
one
so
that
they
are
within
the
region
and
they
are
not
going
cross
and
local
quorum
for
rights.
A
A
You
should
definitely
use
one
right,
CL
one
or
like
local
one
in
this
case,
because
that
ways
your
reads
will
be
much
faster,
but
if
you
are
a
very
sensitive
application
and
you
definitely
want
consistent
data,
you
should
use
local
quorum
for
reads
and
local
koro
for
writes
that
ways
you
are
data
within
that
region
is
consistent
and
it
gets
repaired
at
the
when
you
are
making
all
those
reads
calls
Cree
create
chaos.
If
this
is
very
important
right,
because
you
set
up
some
application
or
this
infrastructure,
you
need
to
make
sure,
does
it
work?
A
Will
it
work
when
we
have
real
failures
and
we
have
all
these
simian
army
tools
which
actually
be
used
to
create
chaos
and
make
sure
whatever
we
decide
it
worked
last,
but
not
the
least
benchmark.
It
is
time
consuming,
but
it
is
worth
it
and
we
learned
a
lot
of
lessons
doing
the
benchmarks
and
you
know
doing
all
those
validations
which
actually
helped
us
move
to
this
active,
active
and
use
cassandra
at
the
back
end.
We
do
we
are
considering
as
christos.
If
you
have
seen
christmases
talk
in
the
morning,
we
are.