►
Description
Speakers: Phillip Meredith, Application Developer & Jay Modha, Vice President, at Credit Suisse.
In this talk, I will demonstrate how we have built one of the most reliable services in Credit Suisse using Cassandra in an environment that is changing all the time — navigating the complex path of scheduled downtime, disaster recovery scenarios and the realities of using shared infrastructure in a corporate environment while still maintaining zero downtime for our users. We'll explore how a small team of 4 developers supports a user base of more than 600+ users globally and go through in more detail how we have deployed Cassandra and manage it from an operations perspective. We'll also discuss how we have used Cassandra’s TTL functionality to meet regulatory data retention requirements and go through some examples of how our Cassandra schema has evolved to deliver new functionality for our users. Finally, we will examine lessons learned and the future of Cassandra at Credit Suisse.
A
A
A
A
So
can
I
just
get
a
quick
show
of
hands
to
see
who
comes
from
a
risk
management
sort
of
evaluation
background
from
an
I.t
perspective?
Okay,
a
few
of
you
for
for
those
who
aren't
aware
of
what
a
risk
system
does.
Generally
speaking,
they
do
a
similar
thing.
In
our
case,
a
risk
system
is
used
to
to
risk
manage
a
portfolio
of
trades,
so
a
given
sort
of
valuation
within
the
system.
A
The
next
thing
that
happens
is
they.
The
trader
makes
some
decisions
about
that
data.
They
might
decide
to
bring
some
of
it
in
all
of
it
in
they
might
decide
to
override
some
of
that
data,
and
once
they've
made
that
decision,
that
data
is
then
tagged
and
I'll
go
into
a
bit
more
detail
about
what
what
a
tag
means
in
terms
of
of
hippo
and
a
tag
basically
represents
the
entire
universe
of
data.
That's
required
to
value
that
portfolio,
it's
quite
an
important
concept
within
hippo.
A
So
I
guess
the
point
I'm
trying
to
get
across
is
while
the
name
might
not
suggest
it.
But
hippo
is
a
very
a
very
important
component
within
this
risk
system
and
it's
involved
in
in
the
main
sort
of
aspects
of
risk
managing
a
portfolio.
The
only
thing
that
the
one
that
isn't
involved
in
is
obviously
the
human
sort
of
human
interaction
when
it
comes
to
making
decisions
about
the
data.
A
So
obviously,
you
know
developing
this
new
system.
We
came
across
a
number
of
challenges,
we're
going
to
cover
a
couple
of
them
in
this
talk,
the
two
main
ones
are
high
availability
and
scale.
Keep
in
mind.
You
know
we
we
are
going.
We
were
going
from
a
number
of
risk
systems
to
the
one
risk
system
and
in
the
past,
if
any
of
those
risk
systems
went
down,
then
it
would
really
only
impact
the
users
of
that
particular
risk
system
in
the
new
world,
where
there
is
only
one
risk
system.
A
If
that
one
system
goes
down,
it's
going
to
impact
all
of
our
users
and
the
other
thing
is
scale
as
well.
You
know
we
need
to
make
sure
that,
as
we
migrate
more
and
more
portfolios
onto
this
risk
system,
it's
not
going
to
impact
performance
and
also
from
a
cost
perspective
as
well.
So
now
I'm
going
to
hand
it
over
to
phil
he's
going
to
go
into
a
bit
more
detail
about
how
we
solve
some
of
these
challenges.
B
So,
as
jay
was
mentioning,
it's
obviously
very
important
for
the
traders
to
see
their
risk
because
if
they
can't
see
that
they
can't
hedge
positions
and
they'll
lose
potentially
a
lot
of
money.
So
for
us
the
availability
is
a
very
important
thing.
We
want
to
use
cassandra
to
solve
this
problem,
so
we've
got
two
types
of
downtime,
as
most
of
you
will
have.
B
So
we
approach
this
by
having
the
following
topology.
So
on
the
far
left
you
can
see,
we've
got
the
proxy
software
load
balancer
and
we
have
two
data
centers
in
a
region
and
we've
got
three
regions.
So
we've
got
three
setups
like
this,
so
the
hippo
is
a
java
app
as
joe
was
saying,
and
then
that
talks
directly
to
the
cassandra
clusters.
As
you
can
see
here,
we
run
an
active,
passive
approach
now.
B
The
main
reason
for
this
is
because
I'll
talk
a
bit
later
about
active
active,
but
we
found
initially,
there
was
a
bit
of
latency
and
we
couldn't.
We
weren't
always
getting
back
responses
that
we
needed
from
the
other
dc
and
then
in
the
cassandra
itself.
We
have
a
local
quorum
consistency
and
a
replication
factor
of
three.
B
So
with
this
kind
of
approach,
this
works
pretty
well
for
getting
that
kind
of
up
time.
Because
then,
if
the
trainer
notices
there's
some
kind
of
problem,
they
can
sort
of
mention
to
support,
and
then
we
can
come
and
manually
intervene
or
switch
flip,
the
dc's
and
that's
fine,
but
that
can
take
quite
a
long
time.
B
B
Then
when
we
lose
a
node,
the
the
hectare
driver
informs
the
hippers,
we've
lost
a
node
and
so
they're
aware
of
kind
of
how
many
they've
gone
down,
but
everything's
still
fine,
because
we've
got
that
replication
factor
of
three
and
the
consistency
of
local
quorum.
So
we've
got
enough
nodes
to
to
run
along
in
that
dc.
B
If
we
lose
another
one,
then
we
decide
it's
time
to
switch
over
to
the
other,
make
the
other
one
active
and
that
automatically
happens
all
the
hippos
decide
to
go
to
the
other
one
to
right
and
read
to
the
other
one
and
then
in
the
meanwhile,
we
can
figure
out
what's
wrong
with
the
passive
dc
and
fix
that
up
now.
This
has
worked
really
well
we're
a
small
team.
B
There's
four
of
us
developing
this
app-
and
this
is
like
a
really
important
kind
of
application
and
we've
not
really
had
to
do
much
any
intervention
in
this
kind
of
stuff.
So
when
it
comes
to
planned
down
time,
we
could
just
rely
on
this,
but
we
thought,
if
you
know,
you're,
going
to
have
to
take
some
dc
down,
for
whatever
reason
you
may
as
well
have
a
way
of
doing
that
ahead
of
time.
B
So,
on
our
little
status
page,
we
implement
a
little
switch,
so
we
can
just
switch
between
the
two
dc's
again
using
the
hippos
and
then
how
are
we
talking
about?
This
is
like
a
big
thing.
So
how
often
do
we
actually
do
this?
Well,
we
looked
at
a
three
month
range
and
you
can
see
where
the
colors
varies
when
the
switch
happens.
B
So
you
can
see
here
it's
actually
six
times
in
three
months,
which
is
quite
a
lot
when
you
think
about
it.
I
think
that's
like
once
every
two
weeks
or
something
but
in
terms
of
what
we
noticed
we
sort
of
came
in
and
go
look
at
this
graph.
It's
changed
and
there's
no
intervention
required
by
us,
because
cassandra
really
allows
you
to
be
up
all
the
time
with
man
with
very
little
effort
required.
B
So
of
course,
it's
really
great
having
an
app
that's
up
all
the
time,
but
if
you're
scaling
all
these
different
systems
into
one
system,
sorry
compacting
them
into
one
system.
You
really
need
to
handle
all
the
load
you're
going
to
put
it
on
it.
So
probably
a
more
complicated
challenge
is:
how
do
we
handle
all
the
scale
from
the
other
systems.
A
So
we've
sort
of
solved
the
high
availability
problem.
Our
next
challenge
was
keeping
in
mind.
We
were
migrating
a
number
of
portfolios
onto
this
risk
system.
We
wanted
to
make
sure
that
we
were
going
to
be
making
the
best
use
of
our
storage.
That
we
have
tco
was
a
total
cost
of
ownership.
It
was
a
big
sort
of
a
driver
in
some
of
the
design
choices
that
we
made.
A
So
one
of
the
options
that
really
wasn't
available
to
us
was
to
just
continuously
expand
our
cassandra
cluster
at
the
rate
of
the
same
rate
of
new
trades
getting
migrated.
So
we
had
a
look
at.
We
had
a
look
at
the
data
that
we
that
we
have
in
hippo
and
I'm
just
going
to
introduce
two
two
concepts
that
we
have.
The
first
one
is
called
gallerification.
So
this
is
this
is
when
we
effectively
store
market
data
within
hippo.
A
So
here's
an
example
of
what
that
market
data
might
look
like
in
this
case
we're
looking
at
an
apple
stock
price.
The
risk
system
would
tell
hippo,
you
know
here's
some
market
data.
Please
can
you
go
and
save
it
and
give
me
back
a
reference
to
it
and
behind
the
scenes?
What
hippo
does
is
it
will
take
this
and
it
will
check
to
see
if
it
already
has
an
apple
stock
price.
A
If
the
stock
price
hasn't
changed,
then
we
have
a
bit
of
an
optimization
and
say
well,
since
it
hasn't
changed,
we're
not
going
to
save
this
bit
of
data,
we're
just
going
to
give
you
a
reference
back
to
what
we
already
have.
So
that's
that's!
How
we
store
market
data
trades,
anything
I
mean,
and
I
guess
the
hippo
doesn't
really
model
any
of
this.
It
just
takes
the
json
data
and
stores
it
within
cassandra
and
cassandra's.
A
Sort
of
a
flexible
sort
of
schema
design
allows
this
to
happen
quite
quite
efficiently,
and
the
next
concept
is
tagging.
I
sort
of
referred
to
it
briefly
before
a
tag
is
effectively
everything:
that's
required
to
value
a
portfolio,
so
in
this
case
we're
looking
at
two
things
in
the
tag.
It's
it's
a
map
you
can
think
of
it
as
a
map,
we're
looking
at
the
apple
stock
price
in
the
gbp
usd
effects
spot
rate
and
every
time
a
valuation
is
done,
a
tag
is
created
and
that
tag
is
then
effectively
valued.
A
You
know,
in
this
case
we're
looking
at
just
two
elements
in
the
tag,
realistically
speaking,
there's
hundreds
of
thousands
of
elements
in
a
tag
that
make
up
all
the
tiny
bits
of
market
data
that
is
required
by
the
analytics
to
build
the
market
data
and
then
value
that
portfolio
and
this
this
data
that
we
have
sort
of
falls
into
two
categories.
The
first
category
is
short-term
data.
A
It's
quite
transient
and
if
you
in
a
real
world
example
during
intraday
usage
of
the
risk
system,
a
trader
will
value
the
portfolio
hundreds
of
times
so
he'll
value
his
portfolio
the
first
time
he
will
look
at
the
data
and
once
he's
finished,
looking
at
that
data,
he'll
value
it
again.
So
the
data
that
was
associated
to
the
first
valuation
is
now
effectively
stale
and
is
no
longer
won't
won't
be
referred
to
again.
A
The
other
type
of
data
is
long-term
data,
so
long-term
data
is
quite
special
because
it's
data
that
is
then
used
to
feed
down
to
other
other
risk
systems.
So
product
control
risk
management,
these
types
of
business
areas
so
long-term
data.
We
need
to
keep
for
a
longer
period
of
time
in
hipaa.
It
can
be
anywhere
between
13
months
to
sorry
30
days
to
7
years.
A
The
interesting
thing
about
long-term
data
is
hippo,
doesn't
know
it's
important
at
the
time
of
inserting
it
so
any
that
was
one
of
the
issues
that
we
had
to
work
around.
So
you
know
we
have
short-term
data
that
we
don't
want
to
keep
around
for
a
long
time
and
we
have
long-term
data
that
we
want
to
keep
around
for
much
longer.
A
So
we
thought
how
do
we?
How
do
we
handle
this
problem?
So
one
option
could
be
hippo
could
go
ahead
and
delete
the
data
right.
I
mean
something
you
can
schedule
something
within
hippo
and
it
would
try
and
figure
out
all
the
runs
that
were
done.
You
know
the
last
couple
of
days
or
whatever,
and
then
it
would
go
ahead
and
just
delete
that
data
out
of
cassandra.
A
However,
there's
you
know
this,
this
type
of
model
has
a
number
of
complexities
around
it.
Hey
you
know:
how
do
you,
how
do
you
handle
failure
scenarios?
How
do
you
handle
some
of
the
edge
cases
in
the
sense
that
if
one
data
is
referred
to
by
multiple
tags,
how
do
you
deal
with
orphan
data?
It's
quite
a
quite
complex.
It's
not
such
a
straightforward
problem
to
solve.
A
So
after
six
days,
if
nothing
is
done
to
that
data,
it
will
appear
to
be
deleted
to
the
users
and
then
behind
the
scenes.
Cassandra
will
mark
it
as
tombstone
data
and
then
it
will
eventually
get
compacted
away,
and
this
is
this
has
been
a
great
great
savior
for
us
in
terms
of
managing
the
amount
of
data
that
we
have.
You
know
as
it
turns
out
when
you
have
a
very
fast
efficient
data
store
that
can
save
data
very
very
quickly.
A
You
end
up
having
a
lot
of
data
very
very
quickly
and
ttl
is
a
great
it's
a
great
solution
for
us
now.
There's
also
this
problem
of
long-term
data.
I
did
mention
that
long-term
data
isn't
isn't.
This
is
sort
of
special
data.
We
don't
know
at
the
time
of
inserting
it.
So
we
introduced
this
concept
of
immortalization,
so
when
a
user
has
done
evaluation
and
they
decide
that
that
data
is
important,
they
immortalize
all
the
data
that's
associated
to
that
valuation
and
what
we,
what
hippo
does
behind
the
scenes?
A
B
So,
as
jay
was
mentioning
there,
having
a
minimal
set
of
data
can
really
help
keep
your
costs
down,
but
also
help
improve
your
performance.
I
don't
know,
I
think
I
recognized
some
of
you
from
the
course
of
performance
tuning
yesterday,
but
one
thing
they
kept
mentioning
was
the
data
model.
If
your
data
model
isn't
very
good,
then
your
performance
won't
be
very
good.
So
how
do
we
improve
performance?
We
have
to
make
sure
our
data
model
is
sound.
B
Then
after
that's
happened,
you
create
a
tag.
So
when
the
trader
wants
to
value
something,
he
makes
that
tag
that
universal
snapshot
of
all
the
data
that
he
wants
to
use.
Now
this
takes
150
seconds
on
average
and
then
finally,
that
gets
fed
in
to
the
calculations,
which
is
like
a
library.
Another
team
provides
us
that
takes
all
the
data
and
figures
out
what
their
actual
risk
is.
B
B
B
The
problem
with
this
as
well.
We
found
is
there's
a
lot
of
hot
spots,
trying
to
read
it
out
and
things
like
that.
So
we
introduced
another
table
and
we
sort
of
normalized
it
a
little
bit
so
the
next
column,
family
or
table.
We
grouped
a
lot
of
those
resources.
So
then
a
tag
is
able
to
reuse
all
of
that
data
efficiently,
and
so
you
can
see
there.
B
The
fx
rates
now
are
both
part
of
one
row
and
again
these
will
be
wide
rows
of
all
your
different
resources,
so
you
can
have
as
many
as
you
like,
and
you
can
see
now.
The
top
level
tag
links
to
those
resources
which,
in
turn,
would
link
to
the
gallery
and
give
you
the
actual
json
blobs,
because
those
json
blogs,
like
many
megabytes
big.
So
it's
not
a
trivial
thing
to
just
slam
them
all
into
one
massive
wide
row,
for
example,.
B
B
So
then
we
did
a
little
bit
of
performance
analysis
and
we
actually
came
down
to
five
seconds
from
that
50
seconds
just
by
changing
the
data
model.
To
do
this,
which
you
know
it's
like
100
and
145,
or
something
second
saving
and
in
terms
of
the
tagging,
that's
a
97
increase
in
speed
just
from
getting
that
data
model
a
bit
better.
B
Now
this
is
particularly
relevant
for
we've
got
a
different
uses
from
the
traders
where
overnight
people
will
run
like
an
automated
batch
of
writings
where
they
alter
one
bit
of
data
every
run,
so
they
might
run
a
thousand
of
those
overnight
for
one
profile
as
jc
we've
got
hundreds
of
profiles,
so
there's
hundreds
of
thousands
of
runs
happening
where
actually
only
one
bit
of
data
is
changing
each
time.
So
this
allows
us
to
be
very
efficient
in
terms
of
what
we're
storing
and
what
we
want
to
value
so
pass
over
to
jay.
A
So
that's
that's
allowed
us
to
do
basically
just
in
time
cluster
expansions-
and
this
is
really
like
the
secret
source
behind
hippo
in
in
allowing
it
to
to
scale
and
meet
demand
of
all
these
migrations
that
we
were
doing
exactly
when
we
needed
it
to
and
like.
If
you
were
to
compare
it
to
traditional,
I
guess
relational
technologies.
A
But
the
advantage
of
cassandra
is.
We
didn't
need
to
do
that
this.
This
sort
of
this
line
is
kind
of
like
a
trend
line
of
the
number
of
trades
that
we
that
sort
of
increased
on
our
system
as
time
went
along,
and
this
example
basically
shows
that
you
know.
Initially,
we
started
off
with
a
cluster
size
of
three
within
a
within
a
specific
data
center
and
then,
as
soon
as
we
knew
that
you
know,
we
were
we're
getting
quite
close
to
sort
of
the
limit
of
how
much
data
and
performance
we
can
handle.
A
A
A
The
other
issue
is
consistent
performance.
We
have
a
number
of
clients.
You
know
I
mentioned
before
the
risk
system,
the
trading
ui
the
results
store.
They
all
depend
on
a
hippo
being
there
and
b
it
being
returning
data
very,
very
fast
and
at
a
consistent
sort
of
rate,
so
just
some
tools
that
we
use
to
help
us
track
our
our
performance
metrics.
A
So
there
are
three
lines
there,
so
the
green
line
is
measuring
today's
performance,
the
the
purple
line
is
measuring
yesterday's
performance
and
the
sort
of
light
blue
line
is
measuring
the
performance
seven
days
ago.
So,
as
you
can
see,
you
know
this
is
it's
all
tracking
relatively
well,
so
today
it's
nine
milliseconds.
A
Yesterday
it
was
11
and
seven
days
ago
it
was
11
as
well,
and
the
below
graph
is
showing
you
the
throughput
as
well,
so
the
throughput
is
roughly
roughly
similar
as
well.
So
this
is
this
is
when
things
are
going
good
when
things
are
going
bad,
it
begins
to
look
a
bit
like
this,
so
you
can
see
here
the
around
about
tuesday
at
4.
00
am
the
green
line
started
to
trend
a
lot
more
higher
from
the
other
two
lines
and
then
ultimately
around
about
wednesday
at
12
p.m?
A
Incidentally,
this
is
down
to
this
particular
issue
was
down
to
an
issue
we
had
with
our
storage
layer.
We
we
are
using
sand
storage.
I
know
that's
a
big
anti-pattern
and
we're
we
are
actually
moving
off
of
that
onto
physical
local
storage.
So
if
you
want
to
know
what
how
sound
can
impact
your
app's
performance,
if
you
use
a
cassandra,
that's
a
good
good
picture
of
what
that
looks
like
from
a
maintenance
perspective.
A
A
Cassandra
generally
just
handles
itself
right,
I
mean
we
don't
do
a
lot
of
maintenance
and
you
know
phil
phil
said
we're
a
small
team
there's
only
four
of
us
who
look
after
who
develop
and
sort
of
support,
hippo,
and
we
have
a
number
of
other,
very
important
applications
as
well
that
the
the
team
looks
after
there's,
two
main
things
that
we
do.
We
run
regular
repairs
once
a
week
less
than
the
gc
grace
period,
and
we
also
take.
A
B
So
finally,
I
just
want
to
quickly
go
through
some
lessons.
We've
learned
we've
had
this
system
in
production
for
about.
I
think,
four
years
now,
and
one
of
the
main
things
is
that
cassandra
is
changing
very
quickly
and
he
saw
some
great
improvements
coming
out
from
jonathan
ellis
this
morning.
So
one
thing
is
to
constantly
kind
of
re-evaluate
what
you're
doing
and
how
you're
doing
it,
because
just
because
it
worked
a
couple
of
versions
ago
might
be
the
best
way
of
doing
it
in
the
new
way.
B
B
So
really
think,
I
think
you
need
to
look
into
cassandra
if
you're,
using
it
in
your
production
system
and
try
and
really
understand
how
it's
working,
but
in
terms
of
the
solution
for
our
problem,
it's
definitely
been
stable
and
reliable,
and
I
think,
if
you
with
using
cassandra,
you
could
definitely
make
an
application
that
will
last
four
years
five
years
or
maybe
a
zoho.
I
said
for
the
next
20
years,
even
and
with
that
I
want
to
thank
you
for
listening
to
our
talk
and
take
any
questions
you
might
have.
A
A
C
B
So
that
I
think,
do
you
mean
the
slide
at
the
beginning?
The
decision
making
is
done
by
the
trader.
Mostly
the
trade
will
decide
if
they
want
to
hold
data
or
not,
and
they
just
feed
it
into
the
hippo.
So
a
lot
of
the
kind
of
they
kind
of
look
at
what
they
think
is
going
to
happen
to
hedge
their
positions.
They
will
do
a
lot
of
the
decision
making.
So
it's
not
really
hippa
doesn't
really
make
the
decisions
for
the
traders.
A
And
often
the
general
use
cases
when
they
refresh
their
market
data.
They
don't
want
to
re-value
that
portfolio
with
brand
new
data
all
the
time
they
might
be
only
interested
in
vols
at
this
particular
point
in
time
and
then
move
on
to
spot
prices
and
things
like
that,
and
that's
it's
down
to
them
how
they
want
to
manage
that.
D
B
Him,
well
I
don't
mind
the
hippo,
they
all
write
to
the
hippo,
so
so
the
hippo,
the
app
is
like
the
only
client
for
the
cassandra
rings.
So
all
of
our
clients
talk
to
the
hippo
first
and
then
the
hippos
itself
will,
I
guess
they
propagate
through
to
each
other
through
the
hector
driver
talking
back
to
the
hippo,
so
it
tells
them
the
node
is
down
and
they
switch.
They
know.
They've
got
some.
They
know
kind
of
that
dc,
aware
of
where
the
nodes
are
so
then
they'll
switch
to
the
other
dc.
A
Also
on
the
hectare
side
of
things,
we've
implemented
our
own
load,
balancing
policy
to
detect
nodes
going
down
so
as
and
that
load
balancing
policy
is
aware
of
our
cassandra
topology
and
it
will,
if
it
detects
that
there
isn't
enough
nodes
to
maintain
a
quorum,
then
it
will
all
all
those
nodes
will.
Eventually.
You
know
in
a
matter
of
seconds
and
flip
over
to
the
other
dc
as
well.
A
E
C
A
D
A
A
I
think
in
our
case,
a
lot
of
the
data
doesn't
change
all
of
the
time.
So,
if
you
imagine
like
a
stock
price
or
maybe
maybe
evolved
surface,
for
example,
that's
not
going
to
change
all
of
the
time
they
might
change.
You
know
100
times
a
day,
but
that
sort
of
the
delay
in
between
that
that
data
changing
is
enough
for
that
for
that
replication
to
happen
across
the
other
data
center.
A
B
They
are
separate
clusters,
so
we
don't
have
one
massive
kind
of
global
cluster
so
in
terms
of
what
they
store,
they
are
the
distinct.
So
there
isn't
really
a
consistency
problem
between
like
us
and
apac,
for
example,
so
yeah.
So
it's
a
similar
so
that
setup
just
is:
we've
got
three
instances
of
that
set
up
and
they
are
independent.