►
From YouTube: PagerDuty: Span the WAN Yes you can!
Description
Speaker: Paul Rechsteiner, Principal Engineer
Most Cassandra usages take advantage of its exceptional performance and ability to handle massive data sets. At PagerDuty, we use Cassandra for entirely different reasons: to reliably manage mutable application states and to maintain durability requirements even in the face of full data center outages. We achieve this by deploying Cassandra clusters with hosts in multiple WAN-separated data centers, configured with per-data center replica placement requirements, and with significant application-level support to use Cassandra as a consistent datastore. Accumulating several years of experience with this approach, we've learned to accommodate the impact of WAN network latency on Cassandra queries, how to horizontally scale while maintaining our placement invariants, why asymmetric load is experienced by nodes in different data centers, and more. This talk will go over our workload and design goals, detail the resultant Cassandra system design, and explain a number of our unintuitive operational learnings about this novel Cassandra usage paradigm.
A
A
Today
we're
going
to
talk
about
spanning
the
LAN
and,
yes,
you
can
pay
to
duty
uses
Cassandra
for
an
awful
lot
of
our
production
infrastructure
and
we
use
it
in
a
relatively
novel
way
that
most
other
Cassandra
deployments
don't
use
we're
going
to
talk
about
how
we've
actually
deployed
Cassandra
in
a
way
that
gives
us
extreme
durability,
even
in
the
face
of
data
center
outages
and
complete
data
center
edges,
not
just
an
availability
zone
here
and
there
so
span
the
LAN.
Why
are
we
doing
it?
A
Give
you
a
little
bit
of
background
about
Pedro
duty
and
also
about
some
of
the
problems
that
we
face
so
first,
what
does
pagerduty
do?
Pedro
duty
is
an
incident
management
solution
that
also
does
alerting
and
notifying
so
in
the
bad
old
days.
It
used
to
be
that
there
is
no
jos
and
whatever
tools,
and
they
would
basically
just
spew
alerts
and
your
entire
on-call
team
would
be
getting
alerts
and
would
be
painful.
Pedro
duty
is
basically
an
intermediary
agent,
where
we
accept
all
of
the
alerts
and
notifications
that
are
coming
from
your
monitoring
tools.
A
Until
somebody
is
awake
to
fix
the
problem,
this
is
obviously
a
pretty
important
job,
because,
if
you're
having
a
problem,
you
won't
know
about
it.
Unless
we
tell
you
about
it,
so
if
Pedro
Duty
is
down,
then
our
customers
can
be
down
and
they
don't
know
about
it.
So
we
take
reliability
very
seriously.
It's
not
like
a
consumer
internet
app
where
you
know
it
be.
You
know
bad
if
we
were
out,
but
it's
not
fatal.
A
Customers
will
just
be
a
little
bit
irked
with
a
consumer
application.
If
you
look
at
the
structure
that
we
have
it's
relatively
straightforward,
we
have
monitoring
systems
that
our
customers
they
send.
The
events
into
pagerduty
pagerduty
collects
the
events
processes
them.
We
convert
them
into
incidents
as
appropriate
or
update
incidents.
In
response
those
events,
we
have
an
on-call
workflow,
which
is
escalation.
Policies
on
call
rotations
scheduling.
All
of
that,
after
determining
who
needs
to
be
alerted
we'll
actually
get
the
alert
set.
A
A
Now
that's
all
well
and
good,
but
if
the
trunk
in
the
middle
breaks
down
your
kind
of
up
the
creek
and
it
doesn't
matter
how
much
redundancy
you
have
with
your
providers
or
anything
so
some
history,
pedra
duty
was
originally
built
as
a
monolithic
ruby-on-rails
stack
with
my
sequel,
as
the
data
store
Patriots
been
around
for
about
six
years
now-
and
you
know
this
is
a
very
effective
way
to
build
a
product
get
it
out.
There
get
people
using
it.
We
were
hosted
entirely
in
u.s.
A
A
There
were
a
number
of
the
failures
that
actually
affected
all
the
availability
zones
in
East,
one
so
I
know
that's
not
supposed
to
happen
at
its
promise
that
it
won't
happen,
but
in
practice
it
does
happen
and
while
it's
not
a
very
frequent
occurrence,
it's
something
that
deeply
affects
a
product
such
as
Pedro
duty
and
it's
something
that
we
decided.
We
really
can't
leave
as
basically
just
a
shrug
our
shoulders
and
sorry
customers.
That's
there's
nothing!
We
can
do
about
it.
A
We
have
to
do
better
than
that,
and
so
we
decided
that
we
had
to
evolve
our
system
design
so
that
we'd
be
resilient
to
failures.
Just
like
this,
so
we
have
some
design
goals
for
what
we're
going
to
do
to
deal
with
this.
We
wanted
to
have
continuity
during
a
data
center
drop,
that's
an
availability
zone
or
an
entire
region.
A
We
wanted
no
operator
intervention
to
make
that
happen.
We
can't
lose
data
if
we're
receiving
events
from
customers.
We
are
guaranteeing
that
we
are
going
to
send
a
notification
and
write
and
wake
the
right
person
up.
It's
not
okay
to
say,
oh
well,
you
know
we
had
a
couple
minutes
of
downtime
as
we
did
a
flip
and
any
events
that
we
gather
during
this
couple
of
minutes.
I'm.
Sorry,
we
just
didn't
notify
anyone.
We
that
was
not
an
acceptable
solution.
A
If
we
were
going
to
go
down
that
path,
so
we
want
to
make
sure
that
there
was
no
data
loss
whatsoever.
We
also
can't
really
delay
data.
So
it's
not
okay
to
say
that
oh
well,
the
DC
went
down
for
half
an
hour
and
what
we'll
do
is
we'll
wait
till
it
comes
back.
Hopefully
they
can
repair
a
fiber
cut
in
half
an
hour
but
I'm
skeptical.
But
if
it
came
back
in
half
an
hour,
we
can't
just
send
out
d
alerts.
A
Then
half
an
hour
is
a
lot
of
extra
response
time
for
a
critical
problem
for
one
of
our
customers,
and
so
that's
not
acceptable.
So
there
really
is
a
shelf
life
for
this
data
as
well,
and
we
have
to
make
sure
that
it
not
only
is
it
durable,
but
it's
durable
and
accessible
right
away,
even
if
a
data
center
goes
down.
A
Lastly,
we
want
to
make
sure
that
we
could
always
send
out
timely
notifications,
and
it's
really
important
to
note
that
this
is
where
it's
a
bit
different
than
a
lot
of
uses
of
Cassandra.
The
time
frame
that
we've
got
to
be
successful,
for
this
is
actually
measured
in
the
tens
of
seconds.
It's
not
like
a
sub
second
kind
of
operation,
so
we're
not
trying
to
create
the
highest
performance
type
of
system
that
you
can
imagine
what
we're
instead
trying
to
do
is
create
a
system
that
will
work
all
the
time.
A
We've
got
a
fair
bit
of
time
to
play
with
and
as
long
as
we
can
make
it
reliable
we're
going
to
have
a
really
good
system.
So
what
did
we
decide
to
do?
First
of
all,
we
knew
that
it
had
to
be
masterless,
so
it
was
going
to
be
a
pure
based
and
clustered
data
store
that
we're
going
to
base
this
on.
We
also
can't
tolerate
staleness,
so
we
actually
wanted
synchronous
when
replication,
we
also
are
going
to
be
managing
state
in
Cassandra.
This
isn't
just
data,
it's
not
just
kind
of
like
information.
A
This
is
actually
the
state
of
the
events
and
of
the
notifications
that
are
being
processed,
and
so
we
really.
We
really
need
consistent
reads
if
we're
going
to
be
doing
that.
So
if
we
write
an
update
to
the
state
of
something,
we
want
to
make
sure
that
as
we're
about
to
process
it
next,
we
find
exactly
that
state
and
what
was
written
out
of
this,
we
opted
to
use
Cassandra.
A
A
Cassandra
has
massive
throughput
people
send
thousands
of
operations
per
second
per
node.
Probably
what
we
see
recently,
hundreds
of
thousands
of
operations
per
second
are
being
achieved,
and,
honestly
we
don't
care
we're
not
really
using
that
much
throughput.
It's
not
like
we're
doing
big
data,
we're
not
doing
massive
amounts
of
data,
that's
boring
into
it!
It's
a
relatively
modest
workload
that
we're
applying
to
the
system,
and
so
we
don't
really
need
the
mass
of
throughput.
The
Cassandra
supports.
A
We
also
don't
have
lots
of
data,
we're
actually
just
tracking
the
in-flight
operational
data
once
the
event
comes
in
and
we
processed
it
and
we've
sent
a
notification
we're
done
so
we
really
just
have
a
working
set
that
we're
trying
to
manage
in
Cassandra
rather
than
an
a
big
accumulation
of
data.
So
over
time
we're
not
really
caring
about
horizontal
scalability.
Either
I
mean
yes,
it's
going
to
grow
over
time
as
our
customer
base
grows,
but
it's
not
going
to
be
the
massive
growth
of
customer
base.
A
Plus
accumulation
plus
larger
data
sets
so
really
our
Cassandra
cluster
sizes
are
relatively
modest.
A
lot
of
applications
tolerate
eventual
consistency,
as
I
mentioned,
because
we're
actually
managing
state
eventual
consistency
gets
very
awkward
and
something
that's
very
hard
to
work
around
at
the
application
layer.
B
A
There's
an
awful
lot
of
Cassandra
deployments
that
rely
on
or
take
advantage
of,
a
high
right
to
read
ratio.
An
awful
lot
of
data
goes
in
and
some
data
comes
out,
but
it's
really
optimized
for
high
rights
and
that's
actually
part
of
the
fundamental
design
of
Cassandra
as
well,
that
it
is
fundamentally
geared
towards
a
high
write
throughput
and
we
don't
really
care
about
that.
And
lastly,
cassandra
is
often
used
in
situations
where
you
do
want
and
need
that
high-performance
individual
operations,
particularly
for
rights.
A
A
So,
let's
get
into
a
little
bit
of
how
we
actually
use
Cassandra
with
and
made
a
LAN
spanning
system
design.
So
a
very
simple
view
of
how
we
build
our
services
at
pagerduty.
We
have
a
clustered
application,
which
is
all
over
the
business
logic
that
is
a
Custer
of
services
of
hosts
each
one
of
them
can
do
exactly
the
same
job
as
any
other
one.
So
it's
also
a
masterless
application
cluster
we
use
to
underpinnings
for
it.
We
use
a
shared
cross
data
center.
A
Datastore
Cassandra
will
get
into
details
about
that
in
just
a
minute
and
we
also
have
distributed
coordination,
which
we
use,
zookeeper
for
which
is
also
cross
data
center.
Brief
digression
to
quorum
consistency
systems.
So
when
you
talk
about
Cassandra
or
zookeeper,
well,
looky
brunos
much
but
Cassandra
react
dynamo
and
similar.
These
qwarm
consistency
systems
take
each
piece
of
data
and
they
make
sure
that
they're
going
to
be
replicated
to
multiple
nodes
in
the
cluster.
A
This
is
you
to
call
the
replication
factor
or
n.
You
also
usually
have
a
consistency
factor
for
your
rights,
which
says
how
many
of
the
end
replicas
need
to
acknowledge
the
right
before
you
say
it's
successful,
and
this
is
combines
with
the
read
consistency
which
is
saying
when
you
want
to
read
a
value.
How
many
nodes
do
you
need
to
get
a
result?
Back
from
from
the
replicas
for
that
piece
of
data
before
you
consider
the
read
successful.
A
A
So
how
do
we
use
this?
Well,
we
actually
set
up
our
Cassandra
cluster
for
one
of
our
services
with
replication
factor
of
five.
We
do
have
three
data
centers
and
then
we
take
advantage
of
Cassandras
DC
aware
placement
strategy.
This
is
a
really
core
part
of
this
design,
and
what
we
do
here
is
cassandra
lets
you
say
how
many
of
the
replicas
for
each
piece
of
data
do
you
want
to
appear
in
each
of
the
data
centers.
A
So
in
our
setup
we
actually
say
that
we
want
two
replicas
in
dca,
two
replicas
in
DC
be
and
one
replicas
in
DCC
then,
and
so
that
actually
means
that
if
you,
if
you
lose
a
data
center,
you're,
not
actually
losing
all
the
replicas
so
you're
ensuring
that
not
all
replicas
are
clustered
in
one
spot.
They
are
spread
out
internally
in
Cassandra,
it's
almost
as
if
you're
creating
a
separate
ring
in
each
data
center
and
you're
setting
the
replication
factor
within
each
one
of
those
rings.
A
We
then
use
the
right
consistency
of
three
which
you
could
also
call
quorum
and
a
rican
systen
sea
of
three.
When
these
combined,
you
end
up
having
the
funny
situation
that
every
single
right
is
going
to
hit
more
than
one
data
center,
because
there's
not
two
don't
have
three
replicas
in
any
single
data
center.
It
also
means
that
every
read
is
going
to
hit
at
least
two
data
centers,
because
you
can't
get
three
nodes
to
contact
of
the
replica
set
to
get
your
values.
A
So
in
the
data
layer,
just
a
summary
of
what
we've
got,
it's
safe
against
data
center
failure,
since
if
one
goes
away,
the
data
always
going
to
exist
in
at
least
one
other
data
center.
At
all
times,
we
do
get
consistent
reads
of
acknowledged,
writes.
It
is
expensive,
though,
to
do
multi
data
center
reads
and
writes,
and
we'll
get
into
that.
In
a
few
minutes
talking
about
how
this
works
in
production,
we
are
managing
state
and
there
is
some
complication
here
because
of
the
data
layer.
A
We
know
how
to
get
consistent
reads
of
individual
data
items,
but
if
you're
looking
at
application
level,
consistency
frequently
there's
more
than
just
individual
data
items
involved.
So
to
do
that
we
actually
need
some
properties
or
you
would
like
to
have
some
properties
that
are
closer
to
transactions.
Cassandra
does
have
lightweight
contractions,
but
they're
limited
in
that
they
only
affect
one
column,
family
or
table,
and
if
you
wanted
to
have
long-running
transactions
where
your
actor
interleaving
some
operations
with
some
logic
in
application,
there's
no
real
way
to
support
that
with
Cassandra.
A
So
we
actually
had
to
enforce
that
by
building
transactions
at
the
application
liber
layer
to
put
on
top
of
this.
So
just
a
relatively
straightforward
run-through
of
this.
So
we
have
transactions
which
are
going
to
be
a
sequence
of
cassandra
operations
and
some
business
logic.
We
end
up
needing
to
implement
them
as
being
idempotent
because
they
could
be
interrupted
halfway
through
and
then
you're
going
to
have
to
redrive
them.
Unfortunately,
there's
no
real
way
to
back
them
out.
A
We
also
need
to
enforce
transaction
ordering,
and
we
do
that
epic
application
level,
so
we
use
zookeeper
as
a
way
to
support
that.
We
use
some
relative
course
walking
where
we
can
take
a
distributed
lock
on
an
entity
and
then
use
that
to
access
the
data
in
Cassandra
thats
related
to
that
entity.
We
also
need
to
make
sure
that
if
a
transaction
is
interrupted,
there
could
be
some
stale
data
or
some
inconsistent
data-
that's
left
in
Cassandra,
and
we
can't
leave
that
there
so
before
another
transaction
can
start.
A
A
So
let's
get
into
what
actually
happens
in
production,
so
we
talked
about
multiple
data
centers
and
that's
great.
If
you're,
actually
just
in
multiple
availability
zones
in
the
same
region,
you're
Layton
sees
are
actually
pretty
nice.
It's
really
designed
that
way,
but
if
you're
using
actually
independent
data
centers
like
multiple
regions
or
even
multiple
data
center
providers,
you're
going
to
end
up
with
non-trivial
Layton
sees
between
the
data
centers.
In
our
case,
we
also
look
at
the
network
diversity,
in
that
we
don't
want
to
have
a
single
point
of
failure
upstream
of
the
data
centers.
A
That
could
in
fact
give
us
a
multi
data
center
failure,
which
is
something
that
we
don't
really
want
to
contemplate.
So
the
result
of
that
is
that
we
get
you
know
non-trivial
and
relatively
uneven
network
latencies.
You
can
see
up
on
the
diagram
right
now
that
you
know
we
have
also
a
bit
of
asymmetry
here
as
well,
so
we
ended
up
with
one
data
center
d,
CA
and
ECB,
which
are
approximately
twenty
four
milliseconds
round-trip
time.
A
We
also
have
DCC,
which
is
also
24
milliseconds
from
dca,
but
it's
only
three
milliseconds
from
DC
be
so
some
transit
is
quite
quick
and
some
transit
is
a
little
bit
longer
and
you
know
well,
it's
all
well
and
good
to
label
diagram
like
this.
It
doesn't
really
give
you
the
sense
of
how
painful
this
is
until
you
visualize
it
correctly.
A
So
what
you
really
want
to
do
is
this
is
what
the
data
center
layout
is
and
then
you
can
start
getting
a
much
better,
intuitive
feel
for
how
things
actually
degrade
or
how
things
will
actually
perform.
So,
let's
look
at
how
actual
cassandra
reads
and
writes
work.
So
there's
a
client
which
is
your
application.
It's
going
to
connect
to
one
of
the
nodes
in
X
and
ur
cluster
and
that'll
be
the
coordinator
for
the
operation
in
order
to
complete
that
operation.
A
That's
saying
how
many
of
the
replicas
need
to
complete
the
operation
before
you
can
respond
back
to
the
client,
so
in
the
case
of
Rights
you're,
still
sending
it
out
to
all
five
of
the
replicas,
but
you
don't
have
to
wait
till
three
of
them
come
back
and
then
you
say,
mission
accomplished,
client,
you're
good
to
go.
It's
it's
been
written
completely
now.
A
The
funny
thing
is
how
this
actually
pans
out
with
the
right
consistency,
level
and
also
the
multiple
data
centers.
So
if
you
have
a
read,
that's
being
coordinated
from
a
node
in
DC,
be
or
in
DC,
see
the
three
closest
nodes
that
will
be
respond,
but
three
closest
replicas
are
all
within
three
milliseconds,
so
it's
actually
a
pretty
snappy
response.
On
the
other
hand,
if
you're
actually
coordinating
an
operation
from
cast
one
or
cast
two
over
here,
it's
got
two
replicas
that
are
very
nearby,
but
it's
going
to
have
to
hear
back
from
one
replica.
A
That's
all
the
way
over
here
and
that's
going
to
take
at
least
24
milliseconds.
So
not
only
do
you
have
these,
you
know
very
asymmetrical
Network
round
trips,
but
it
also
has
a
very
direct
impact
on
the
performance
and,
if
you're
distributing
your
requests
evenly
to
the
different
coordinators
in
the
different
data,
centers
you're,
going
to
end
up
with,
in
this
case,
like
an
eight
full
difference
in
performance
of
the
Cassandra
operations
that
you're
actually
performing
now,
we
found
very
surprisingly
another
Layton's
effect
on
the
system,
and
it's
not
just
on
the
performance
other
requests.
A
This
was
kind
of
unintuitive
we'd,
seen
a
lot
of
interesting
things
in
production,
where
some
of
our
nodes
were
experiencing
much
higher,
like
elevated
CPU,
and
we
had
initially
attributed
to.
It
was
actually
all
happening
in.
I
believe
west
one
and
we
had
kind
of
you
know
our
inner
gossiping
was
like.
Oh
well,
that's
actually
the
old
amazon
hardware.
It's
crap
here
west
to
actually
has
better
instances.
I
mean
yes,
there's
socially
the
same
sizing
and
everything,
but
there's
slightly
better
instances.
A
Maybe
maybe
zen
is
more
up
to
date
or
something,
and
so
we
attributed
it
to
that
when
we
dug
into
it
later
on.
We
found
that
that
actually
wasn't
the
case
and
it
actually
was
Cassandra
and
how
it
actually
performs
the
operations
that
was
changing
the
load
on
our
different
nodes
and
in
particular,
we
found
that
the
per
node
read
volume
was
substantially
different
in
the
cluster
and
when
I
say
the
per
node
read
volume,
not
that's
not
who's,
coordinating
it.
A
It's
the
individual
operations
that
are
being
performed
on
the
replicas
themselves,
so
you'd
expect
to
see
a
very,
even
load
across
all
of
the
nodes
in
the
cluster
and
instead,
what
we
saw
as
I
mentioned
is
a
very
skewed
load
where
some
nodes
were
very
light
and
some
nodes
were
quite
a
lot
more
heavy
looked
at
visually.
This
is
actually
what
we
saw
when
we
finally
investigated
in
our
dashboards,
and
we
saw
that-
and
you
can
see.
A
A
Every
time
you
do
a
right,
regardless
of
which
data
center
it's
originating,
which
coordinator
it's
originating
on
or
anything
and
that's
cool,
but
with
reads:
there's
a
little
optimization
in
Cassandra,
which
says
the
replication
factor
is
5.
That's
fine,
but
the
rican
systen
sea
level
is
only
three
we're
going
to
choose
three
replicas
and
read
from
just
three
replicas
and
get
the
responses.
And
then
we
respond
back
to
the
client
and
it's
not
going
to
bother
putting
extra
requests
out
to
those
other
replicas.
That's
fine!
A
So
how
does
it
actually
choose
which
replicas
to
reach
out
to
well
Cassandra
actually
chooses
the
fastest
and
that's
good,
but
in
our
case,
because
we
have
such
an
asymmetric
network
latency
that
ends
up
dominating
the
speed
of
all
requests
and
so
you're
almost
always
going
to
have
your
reads
directed
to
the
three
network
nearest
replicas
that
hold
the
data.
That's
all
well
and
good,
but
still,
why
do
we
get
uneven
load?
And
why
is
it
so
obviously
skewed?
Let's
consider
the
situation
when
Cassandra
3
is
the
coordinator
for
a
request.
A
The
nearest
nodes
that
it's
going
to
find
or
the
newest
replicas
that
it's
going
to
find
are
all
in
data
center
be
and
data
center
see.
So
it's
always
going
to
hit
three
four
and
five,
and
it
would
be
absolutely
idiotic
for
it
to
try
to
read
any
value
from
Cassandra
one
or
Cassandra
to
theirs
too
far
away,
it'll
never
be
in
the
top
three
fastest.
So
don't
even
try.
A
The
same
thing
happens
if
Cassandra,
four
or
Cassandra
five
is
used
to
coordinate
a
request,
a
read
request:
it's
always
going
to
go
and
hit
those
three
nodes.
There's
never
an
opportunity
or
good
reason
to
go
further
now,
on
the
other
hand,
what
happens
if
Cassandra
one
is
coordinating
a
request?
It's
going
to
take
the
three
nearest
nodes
so
to
the
nearest
nodes
or
cast
one
and
cast
to.
Where
is
the
next
nearest?
A
Well,
it's
pretty
much
going
to
be
a
random
choice
between
34
and
five,
depending
on
maybe
the
actually
operation
performance
or
maybe
varying
network
latency
that
snooze
things
on
on
the
day.
That's
actually
happening.
So
that
means
that,
for
anything,
that's
coordinated
over
here,
there'll
be
a
little
bit
of
load
over
here,
and
some
of
the
load
will
also
be
applied
over
on
the
right
hand,
side
two
nodes:
three,
four
and
five.
A
If
we
go
when
we
do
a
very
mathy
kind
of
analysis
of
how
the
requests
end
up
tallying,
you
can
see
that
if
we
applied
for
the
same
data
item,
if
we
did
five
requests
one
to
each
of
the
coordinator
nodes
you'd
expect.
If
we
apply
to
cast
one,
then
we
had
actually
send
one
operation
notes
cast
11
to
cast.
You
end
a
third
of
an
operation
randomly
to
cast
four
five
and
three.
A
The
same
thing
happens
with
cast
to
11
and
a
third
and
third,
a
third,
because
it's
kind
of
going
to
choose
randomly
between
them.
On
the
other
hand,
with
cast
three
four
or
five
coordinating
you're,
always
going
to
hit
three
four
and
five
with
a
request.
If
you
tally
up
all
the
load,
you
find
that
on
average,
for
these
five
requests,
one
for
each
coordinator
that
cast
one
and
two
will
each
service
to
read
operations
and
cast
three
four
and
five
will
each
be
servicing
3.66
operations
so
factor
of
about
1.8.
A
This
was
like
a
really
fascinating
find
for
us
that
completely
explained
why
the
load
was
actually
higher.
White.
Sorry
why
the
CPU
was
higher,
also
why
the
read
rate
was
higher,
but
the
rights
really
didn't
matter
and
they
were
even
and
you
could
see
it
being
incredibly
evenly
distributed
between
all
the
nodes.
This
also
had
an
outsized
effect
on
Pedro
duty
in
particular,
because
we
actually
have
a
very
high
read
to
write
ratio
in
our
usage
of
Cassandra.
A
So
in
most
clusters,
you
probably
wouldn't
actually
end
up
seeing
cpu
skew
because
of
this,
but
as
pagerduty
because
of
our
workload,
this
has
an
inordinate
effect
on
what
we
see
going
back
to
the
actual
metrics
that
we
had
before
they
don't
line
up
exactly
with
the
theory.
We
are
saying
that
we
should
see
about
a
1.8
factor
here,
we're
seeing
in
the
actual
measured
metrics
something
around
about
a
1.5
factor.
A
That's
probably
because
we
actually
do
have
exactly
uniform
distribution
of
the
requests
amongst
the
coordinators,
and
so
that
skews
the
results
a
little
bit.
But
it
was
still
very
satisfying
to
find
this
correlation
as
a
consequence
of
this
asymmetric
network
layout
that
we
have
so
there's
also
a
few
interesting
implications
about
scaling
out,
because
now,
with
asymmetrical
per
node
read
volumes.
It
also
means
that
the
amount
of
CPU
and
the
amount
of
disk
I/o
that
you
need
in
each
data
center
is
different
because
the
nodes
are
going
to
be
experiencing
different
loads.
A
A
This
is
kind
of
an
interesting
effect
to
see,
but
don't
fall
into
the
trap
of
saying:
oh
well,
we'll
just
provision
according
to
that,
because
it's
not
the
worst
case,
and
it's
not
the
failure
case
that
you
need
to
make
sure
that
you're
going
to
be
able
to
deal
with
and
so
in
the
end
you're
pretty
much
going
to
end
up
with
same
size
nodes
across
the
board.
Anyway,.
A
So
this
does
happen.
We
actually
had
this
in
May
I
think
was
the
31st
or
maybe
the
30th
one
of
our
data
centers
dropped
off
the
internet.
All
of
our
hosts
in
that
data
center
were
one
hundred
percent
unreachable
for
about
five
hours
as
one
of
our
ops
engineers
tweeted
out
by
by
light
purple
one
a
page
of
these
data
centers
still
having
a
major
outage
and
we're
still
going
strong.
This
is
a
graph
showing
activity
at
our
application
layer,
and
you
can
see
that
the
light
purple
drops
out.
A
That
was
actually
the
application
host
that
was
in
that
data
center.
We
had
absolutely
zero
customer
impact.
We
actually
had
zero
pages
for
anyone
other
than
in
the
operations
team
that
was
dealing
with
infrastructure
that
was
down
everything
just
continued
to
hum
along
no
data
was
lost.
It's
also
important
to
notice
that
in
this
case,
any
events
that
had
been
received
into
that
data
center
that
went
down
they
were
not
stranded
there
either
because
they
had
actually
been
stored
to
another
data
center.
A
A
We
also
had
another
interesting
application
of
this
architecture
in
August,
so
we
actually
decided
not
exclusively
because
of
that
outage
in
our
DCC
provider,
but
for
a
few
other
factors
that
we
want
to
move
to
a
different
data
center.
So
we
actually
we're
going
to
move
our
entire
fleet
from
data
center
c
to
a
brand
new
data
center
with
a
different
provider.
We
were
cassandra
clusters.
We
actually
could
just
drop
a
node,
add
a
new
node
in
the
other
data
center.
Everything
would
repair
and
we're
done.
A
There
is
no
application
level
migration
needed
at
all
and
there
was
zero
customer
impact.
This
was
all
done
cluster
by
cluster
moving
all
the
nodes
over
in
a
period
of
a
few
weeks,
which
is
pretty
fun
and
when
you
think
about
any
kind
of
system
architecture
that
isn't
actually
doing
replication
like
this,
it
would
be
will
say
harder
to
do
a
migration
like
that,
especially
with
no
downtime
and
no
customer
impact
whatsoever.
A
So
it's
a
really
cool
way
that
you
can
get
benefits
from
it,
not
just
in
terms
of
reliability
but
in
time
I
guess
it
is
in
reliability,
but
this
is
really
taking
down
the
data
center
yourself
and
replacing
it
with
a
new
data
center.
We
also
actually
had
an
interesting
kind
of
side
effect
that
we
saw
just
last
week,
so
there
was
a
fiber
cut
that
occurred
between
data
center.
A
and
data
center
be
because
of
the
way
the
data
centers
are
connected.
A
All
transit
from
A
to
B
was
going
through
this
a
single
path
and
it
got
routed
to
something
that
was
really
horrendous
that
we
actually
didn't
have
control
over
because
we're
at
the
cloud
level.
So
the
latency
went
actually
from
24
milliseconds
all
the
way
up
to
200
milliseconds,
that's
pretty
horrible
and
it
actually
wasn't
pleasant,
because
that
meant
that
we
now
had
more
links
that
were
quite
latent.
Now
it
didn't
actually
hurt
the
system
very
much.
We
did
experience
some
slowdown
in
processing,
but
everything
continued
to
take
along.
A
You
can
see
how
the
yellow
actually
leaps
in
here-
and
this
was
actually
at
the
application
layer,
because
we
actually
found
that
more
load
was
going
to
some
of
our
application
hosts.
That
was
in
one
of
the
data
centers,
and
we
felt
that
it
was
a
little
bit
heavy
loaded.
So
we
just
added
some
more
hosts.
It
wasn't
at
the
Cassandra
level.
That
was
fine,
but
we
did
make
the
change
at
the
application
layer
what
was
also
kind
of
cute.
A
If
you
see
this
extra
bit
at
the
right-hand
side
there,
that
was
actually
the
East
one
outage
that
happened
over
the
weekend.
We
also
have
this
and
if
it
was
actually
slid
a
little
bit
to
the
left,
it
would
be
a
really
interesting
scenario
which
we've
had
in
the
past,
which
is
when
one
of
our
cloud
providers
goes
down.
Not
only
is
it
affecting
us
and
to
create
degrading
the
pagerduty
system,
but
it's
probably
degrading
a
huge
number
of
our
customer
systems.
A
So
not
only
does
pay
dirty
go
into
a
degraded
state
in
terms
of
Cassandra
clusters
processing,
but
we
experienced
two
three
four
times
the
regular
amount
of
volume,
and
so
when
it
comes
time
to
actually
plan
for
how
much
capacity,
we
need
it's
a
very
careful
thing,
because
not
only
you
really
do
have
to
combine,
and
you
expect
that
you're
going
to
have
your
biggest
peaks
when
you're
in
your
worst
degradation
mode,
which
is
usually
not
the
planning
that
everyone
wants
to
do,
it's
very
inconvenient.
It
turns
out
to
have
to
plan
that
way.
A
So,
let's
head
back
to
where
we
started,
what
have
we
learned
so
Wayne
spanning
synchronous,
replication?
It
actually
is
a
thing
you
can
do
it
with
Cassandra,
it's
not
a
very
common
way
of
using
Cassandra,
but
you
can
build
some
pretty
cool
things
if
you
stretch
it
this
way,
data
layer,
consistent
reads
are
actually
practical,
with
Cassandra,
given
the
right
workload
and
application.
In
our
case,
the
extra
latency
is
not
a
killer.
A
We
did
have
to
create
a
fair
bit
of
scaffolding
at
the
application
layer
and
use
a
fair
bit
of
unfortunate
kind
of
static
analysis
to
be
able
to
come
up
with
the
right
things
and
to
build
sequin
things
sequence,
things
correctly,
but
it
does
work
and
you
can
make
it
work
when
you
put
this
into
production,
the
network
topology
actually
has
a
pretty
interesting
impact,
not
only
on
performance
and
latency,
but
also
on
per
node
load
and
how
you
plan
for
failure
scenarios
and
how
you
actually
provision
the
system.
Lastly,
kind
of
is
a
summary.
A
A
A
C
A
So
the
question
was:
what
are
we
doing
to
manage
the
connectivity
between
the
data
centers?
We
have
a
semi-custom,
ipsec
infrastructure
setup
that
we
tunnel
all
over
traffic
through.
It's
not
precisely
a
VPN,
it's
more
point
to
point
form
from
host
to
host,
and
we
also
use
that
to
to
only
establish
the
tunnels
where
traffic
is
actually
allowed.
So
it's
also
kind
of
access
control
between
the
hosts
is
that
does
that
answer
your
question
all
right.
D
A
D
A
So
we've
thought
about
that.
The
the
network
diversity
is
a
really
big
crimp
in
any
of
the
plans
that
we
have.
We
are
looking
for
basically
like
a
cloud
provider
where
we
can,
we
can
get
you
know
basically
instances
and
a
certain
amount
of
manageability
there
aren't
that
many
that
fit
our
criteria
and
are
within
a
reasonable
distance.
A
That's
something
that
we
honestly
I
I,
don't
recall
that
we've
had
any
conversation
about
that
internally
and
we
did
actually
also
migrate
away
from
the
eastern
part
of
the
u.s.,
mainly
because
East
one
wasn't
wasn't
as
reliable
as
we
wanted
it
to
be.
I
know:
that's
improved
a
little
bit
over
the
last
couple
of
years,
but
no,
no
okay,
no,
it
hasn't,
oh
and,
and
so,
if
we
went
back
to
to
the
east
in
the
u.s.
A
A
What's
that
so
question
is
how
do
we
run
zookeeper
across
regions?
You
just
do
it
we
set
up.
We
actually
have
zookeeper
clusters
that
are
five
nodes.
We
distribute
them
the
same
way
that
we
actually
have
the
replicas.
So
it's
very
similar
to
our
replica
place
in
that
strategy
to
21
again
we
don't
have
like
an
ultra-high
load
on
it,
but
we
use
it
mainly
as
a
locking
thing.
So
there's
no
persistent
data
and
zookeeper
whatsoever.
It's
all
just
transient.
A
Is
there
any
specific
tunings?
Yes,
there
is
there's
a
specially
the
application
layer,
the
the
part
that's
tricky
about
zookeeper
and
locks
is
when
you
lose
connectivity
to
zookeeper
and
who
believes
that
you've
got
a
walk
because
you
might
still
believe
you've
got
unless
you're
doing
live
is
testing
on
the
connection.
So
there
are
it's
a
real
art
kind
of
in
picking
the
right
time
outs
on
the
connection
timeout
and
then
the
session
timeout.
A
E
A
We
don't
do
two
backups,
because
the
the
shelf
life
of
those
backups
would
be
probably
measured
in
like
tens
of
minutes,
and
so
it's
really
not
something
that
we
worry
about
very
much.
You
know
we're
only
really
worrying
about
the
in-flight
operational
data,
and
so
we,
you
know
as
long
as
that's
flowing
through
and
we
have
you
know
a
pretty
good
replication
factor.
A
We've
got
five
as
redundancy,
even
if
a
data
center
goes
down,
it's
still
fine,
and
by
the
time
that
we
mediate,
that
I
mean
the
data
is
going
to
be
pushed
through
the
system
anyway
and
processed.
We
don't
keep
historical
data,
we're
looking
at
some
other
applications
of
Cassandra
that
are
more
traditional,
where
it
actually
be
for
data
storage
and
those
cases
we
we
almost
certainly
would
be
using
a
probably
snapshotting
and
which
nodes
we'd
snapshot
from
that's
tricky,
because
once
you
get
into
like
a
larger
I
guess
you
could
use
the
yeah.
A
If
you
had
fewer
nodes
in
the
DCC,
you
would
have
fewer
replicas
that
you
have
to
snapshot
on
I'd.
Imagine
that
with
our
penchant
for
reliability
and
durability
would
probably
snapshot
in
multiple
DC's.
Since
we
wouldn't
want
that
snapshotting
to
go
offline,
if
a
DC
was
down
actually
reminds
me
a
little
bit
she
mentioned
earlier,
the
previous
presentation
was
about
reboot
day
last
year
at
AWS
and
I
think
it
was
West
to
where
they
kept
on
cycling
instances.
A
Our
multi
DC
setup
also
was
really
valuable
in
that
case,
because
we
didn't
care
because
we'd
never
lose
redundancy.
If
you
had
the
entire
Cassandra
cluster,
all
in
the
zone
that
they
were
rebooting
instances
in
and
they
took
away.
You
know
20
of
your
80
instances.
You'd
probably
have
data
that
you
had
zero
nodes
available
for
and
you
would
no
longer
have
redundancy
for
the
data.
Doing
the
multi.
Dc
replication
means
that
it's
really
a
non-issue
because
the
data
is
going
to
be
somewhere
else,
even
if
the
instances
in
one
region
are
being
rebooted.
B
A
We
don't
have
clusters
of
a
size
that,
and
we
don't
have
enough
clusters
that
we've
fully
automated
everything
like
we're,
not
really
a
Netflix
tile
in
that
regard.
Yet
we
have
for
the
most
part
through
you
can
call
it
either
trial
and
error
or
you
could
call
it
through
operational
learnings
and
experience
come
up
with
relatively
error-free
ways
to
do
that.
Generally,
we
don't
use
v
nodes.
We
still
stick
with
explicit
token
ranges.
So
we'll,
actually
you
know
calculate
new
token
ranges.
A
Bring
up
the
new
node
popular
with
data,
then
bring
it
into
the
ring
and
do
it
that
way.
And,
incidentally,
when
you
do
your
when
you
have
your
cluster
spanning
different
data
centers,
you
end
up
having
to
basically
treat
each
of
as
separate
cluster
in
terms
of
how
you
expand
it
and
how
you
scale
it
out.
So
it's
not
just
adding
one
node.
You
have
to
usually
add
one
node
in
each
of
the
data
centers
all
right.
Thank
you
very
much
and
if
you
have
any
questions,
I'd
be
glad
to
talk
to
you
afterwards.