►
From YouTube: Macy's: Changing Engines in Mid Flight
Description
Speaker: Peter Connolly, Senior Architect
This presentation recounts the story of Macys.com and Bloomingdales.com's migration from legacy RDBMS to NoSQL Cassandra in partnership with DataStax.
One thing that differentiates this talk from others on Cassandra is Macy's philosophy of ""doing more with less."" You will see why we emphasize the performance tuning aspects of iterative development when you see how much processing we can support on relatively small configurations.
This session will cover:
1) The process that led to our decision to use Cassandra
2) The approach we used for migrating from DB2 & Coherence to Cassandra without disrupting the production environment
3) The various schema options that we tried and how we settled on the current one. We'll show you a selection of some of our extensive performance tuning benchmarks, as well as how these performance results figured into our final schema designs.
4) Our lessons learned and next steps
A
B
First
I'm
going
to
talk
about
background
of
our
application
and
some
problems
that
we
had,
which
is
led
us
to
Cassandra
Peters,
going
to
talk
about
the
options
that
we
explored
along
the
way
and
how
we
migrated
Vaughn's,
going
to
dive
a
little
deeper
into
some
of
the
data
models
that
we
tried
along
the
way
and
what
worked?
What
didn't
work
and
why
and
then
we'll
sum
it
up
with
some
things
that
worked
and
didn't
as
well.
Overall,
ok,
so
we're
a
we're.
B
A
services
application
providing
rest
ap
is
what
we
provide
is
a
catalogue
store
and
in
for
Tory
data
to
the
website
to
mobile
applications,
store
applications
and
some
partners
we're
not
doing
any
big
data
or
analytic
stuff.
Yet
we're
focused
on
like
highly
scalable
services
that
can
take
lots
of
requests,
low,
latency,
'he's
and
horizontally
scalable.
B
B
So
based
on
some
problems,
we
were
having.
We
had
a
suspicion
that
our
existing
application
would
be
a
problem
and
wouldn't
be
able
to
meet
those
challenges
that
we
had.
Some
problems
we
had
was
a
heavily
normalized
relational
database,
believe
it
or
not.
We
had
51
tables
with
for
unions
that
we
had
to
read
to
load
a
complete
product
and
for
certain
products
that
could
be
seconds
in
response
I'm
just
from
the
database
level,
not
from
the
application
here.
So
we
put
some
caching
in
front
thought
that
would
solve
it.
B
We
started
with
local
caches,
but
as
our
application
scaled
out,
you
know
we
were
still
putting
a
lot
of
load
on
our
database,
so
we
moved
to
a
distributed.
Cache
that
limited
the
numbers
of
requests
into
the
database,
but
still
from
from
a
front
cash
on
the
actual
application
servers
to
the
kind
of
data
grid.
B
We
also
had
some
operational
problems.
We
were
when
we
had
this
narrow
window
of
time
to
get
the
data
loaded
up
for
the
next
day
and
the
latency
between
the
front
and
the
back
caches.
We
decided
to
preload
things
into
memory,
but
that
was
limited
because
of
a
short
window.
If
you
grew
it
by
10
times,
only
a
small
percentage
of
your
catalog
would
be
preloaded,
and
the
second
problem
was
that
if
we
had
data
issues-
and
it
took
that
long
to
load
all
this
data
into
memory,
you
had
no
way
to
recover.
B
So
what
we
did
was
kept
two
versions
in
memory.
If
you
had
those
problems,
instead
of
starting
from
scratch
and
having
to
lazy
load
during
high
traffic
volumes,
you
could
switch
back
and
forth
this
complicated
a
lot
of
release,
processes
and
keeping
inventory
and
sink,
and
data
in
sync
between
the
two
different
cache
clusters
and
also
using
a
proprietary
cash
would
have
been
really
expensive
to
scale
out.
B
A
Hi
everyone
we
were
at
a
unique
point
in
our
work
there
and
that
we
had
been
experiencing
growth
of
thirty
percent
year-over-year
organic
growth
on
our
website.
When
I
started
in
2012,
we
saw
revenues,
go
from
two
billion
a
year
to
3
billion
year
and
then
continue
on
towards
4
billion,
so
the
growth
was
phenomenal
and,
as
Andrew
pointed
out,
we
were
being
challenged
by
the
business
because
of
our
omni-channel
efforts
to
grow
the
catalog
and
grow
the
ability
to
serve
that
catalog
beyond
what
we
had
even
anticipated
with
organic
growth.
A
So
it
was
one
of
those
kind
of
disruptive
moments
in
our
history
there
that
where
we
knew
we
had
to
do
some
fundamental
changes,
we
were
on
relational
technology.
It
had
to
be
fronted
with
a
in-memory
data
grid,
making
any
sort
of
foundational
database
change
is
expensive.
So,
in
terms
of
options
that
we
looked
at
of
course,
the
very
first
thing
off
the
bat
would
be
well.
A
We
can
to
normalize
our
relational
database
and
how
many
of
you
here
have
actually
been
through
an
exercise
of
D
normalizing,
a
relational
database
for
performance
reasons
and
a
bit
you
were
profoundly
dissatisfied
with
that
effort.
Just
like
I
have
been
in
the
past,
and
many
of
us
have
been
in
the
past.
If
you
can
only
do
normalize
a
relational
database
so
far
and
you're
still
stuck
with
a
number
of
different,
you
know
numerous
inner
joins,
you
have
to
do
and
performance
is
always
going
to
be.
A
You
know
suboptimal
we're,
read
intensive,
so
we
wanted
to
look
at
a
series
of
alternatives
to
relational
that
work,
maybe
meet
our
use
case
and
be
slanted
towards
heavy
read
during
the
day
and
heavy
right
at
night,
who
will
probably
looked
at
a
dozen
different
things
and
by
looking
at
him
I
mean
we
research
them.
We
evaluated
the
features,
and
this
was
this
was
back
in
the
day
when
Cassandra
was
one
of
many
and
it
wasn't
real
clear
who
was
going
to
be
the
front
runner.
A
So
he
had
a
lot
of
questions
and
we
would
bring
in
vendors
one
by
one
and
say
make
sure
that
we
understood
their
architecture,
understood
the
strong
points,
the
weak
points,
and
we
were
not
the
federal
government.
So
we
had
to
get
to
a
kind
of
cut
to
the
chase
quickly
and
say.
Well,
we
need
to
benchmark.
It
would
make
sure
that
this
technology
works
in
our
environment
with
our
use
cases,
but
we
had
to
get
it
down
to
just
the
short
list
and
the
short
list
turned
out
to
be
MongoDB.
A
It
was
very
compelling
with
his
JSON
document
orientation.
The
sharding
really
looked,
hopeful
Cassandra
looked
like
it
was
truly
scalable
and
true
peer-to-peer
which
really
caught
our
eye
and
active
spaces,
but
you
probably
haven't
heard
of
before
it's
a
typical
proprietary
in-memory
data
grid,
but
it
it
was
on
our
short
list,
because
our
colleagues
on
in
the
Macy's
systems
and
technology
organisation
have
a
deep
relationship
with
tibco
and
they
wanted
to
try
it
out.
So
so
we
included
that
on
the
short
list.
A
The
way
we
did
a
test,
a
benchmark,
I,
don't
want
you
to
over
focus
on
the
on
the
actual
performance
numbers,
because
this
is
more
than
just
looking
at
a
bake-off
on
performance
numbers.
We
looked
at
what
happened
when
you
took
nodes
out
of
a
cluster.
What
happens
when
you
put
them
back
in?
How
long
did
it
take
to
snapshot
to
back
up
to
recover
things
like
this?
So
it
was
across
the
gamut.
A
We
looked
at
a
lot
of
different
aspects
to
this,
but
in
terms
of
hardware
and
the
actual
POC
benchmarks,
those
were
run
on
amazon
ec2.
We
use
the
same
five
servers
for
all
three
tests.
They
were
c
3,
2,
x-large
instances
which
I
mean
you
can
read
too
it's
a
eight
CPU
15
gig
to
buy
80
SSD
configuration
and
we,
if
you're
in
the
middle
of
this
type
of
evaluation.
I
recommend
that
you
look
at
your
specific
business
use
cases.
A
A
First
row
is
initial
load
and
with
this
is
the
150
million
rows
that
we
load,
we
were
achieving
around
sixty-five
to
seventy
two
thousand
TPS
for
tibco
and
for
Cassandra
I
should
have
mentioned
sorry.
We
had
the
vendors
for
these
tibco
datastax
and
10gen.
Remember
10gen.
They.
They
were
all
on
site,
helping
with
these
benchmarks.
So
when
we
say
that
we
could
not
complete
the
MongoDB
loads,
that
was,
in
spite
of
the
help
of
10
gem,
couldn't
complete
those.
A
You
see
that
the
up
certs,
the
you
know
the
right
activity
is
around
4000
TPS
3.2,
millisecond,
3.57,
millisecond,
latency
zon
active
spaces
in
Cassandra,
so
there
are
neck-and-neck.
You
see
this
happening
all
through
the
benchmarks
and
then
for
Reed.
We
didn't
do
a
lot
of
tuning
there,
but
we
saw
that
we
had
pretty
much
even
performance
between
again
active
spaces
in
Cassandra,
so
it
was
basically
now
a
bake-off
between
those
two.
A
This
is
more
just
to
show
you
the
various
types
of
criteria
you
should
consider
when
you're
evaluating
these-
maybe
that's
not
as
necessary.
Now
it
really
looks
like
you
know.
We
saw
this
wide
open
space
of
different
vendors
that
had
no
sequel
offerings,
but
now
only
a
few
are
really
kind
of
left
standing
with
really
big
followings
cassander
being
one
of
them.
But
if
you
were
to
do
this,
these
are
the
types
of
categories
and
criteria
might
look
at
to
say
how.
A
A
So
this
is
the
kind
of
the
core
of
how
we
change
the
the
engine
in
in
the
middle
of
flight.
We
have
a
fundamental
piece
of
our
business,
the
catalog,
and
we
can't
just
kind
of
throw
the
other
one
out.
The
old
legacy
went
out
and
there's
hope
that
the
new
one
works
on
the
first
try
apologies
for
these
diagrams.
They
were
very
wide
end
to
end
diagrams
and
I
wanted
to
chop
off
the
portions.
A
It
didn't
really
apply
to
the
Cassandra
migration,
but
there
was
a
series
of
phases
that
we
went
through
to
to
accomplish
this
without
rocking
the
boat.
This
just
shows
kind
of
the
starting
point.
We
had
our
end
memory
data
grid,
a
relational
database
feeding
it.
We
had
store,
inventory
being
sent
via
event
queues
and
a
messaging
service.
That
was
loading
that
into
a
datagrid
during
the
day,
and
then
we
have
our
old
monolithic
services
serving
all
this,
and
those
services
were
Hessian,
which
is
a
standard.
A
A
This
one
here
is
now
the
phase
where
we
introduce
Cassandra
into
the
into
the
mix
and
now
we're
at
this
point
we're
loading
the
Cassandra
data
in
from
the
relational
database,
and
we
did
that
because
it's
kind
of
a
cheat.
It
was
a
short-term
thing,
the
idea
being
that
we
already
had
the
requirement
to
load
our
in-memory
data
grid.
From
this
relational
database
might
as
well
steal
that
code,
that
was
Yvonne's
approach
and
it
saved
us
a
lot
of
programming
time.
A
We
took
that
same
extract
and
loaded
into
Cassandra,
so
that
helped
get
us
into
production
quickly
and
try
this
out,
and
you
see
the
addition
of
new
catalog
messaging
to
handle
the
store,
inventory
and
new
catalog
service
to
handle
the
rest
Jason.
That
was
going
to
go
directly
against
Cassandra
notice,
the
use
of
an
a
load
balancer
there
so
that
we
can
tweak
the
amount
of
traffic
that's
going
to
new
versus
old.
So
we
weren't
just
going
to
turn
the
spigot
on
completely
all
at
once
and
not
being
done.
A
We
could
then
turn
the
dial
further
and
further
to
see
how
it
handled
more
and
more
volume.
This
next
one
is
just
adding
additional
capacity.
Having
done
the
pilot,
seeing
that
at
work,
we
started
moving
all
of
our
product
feed
and
our
fulfillment.
Our
online
fulfillment
inventory
feeds
into
this
and
adding
additional
capacity
using
the
f5
throughout
to
to
kind
of
tune
in
and
see
how
well
it
was
working
and
then
once
that
was
there
and
stable.
A
You
see
we
can
just
basically
get
rid
of
the
catalog
footprint
of
the
relational
database,
the
end
in
memory
data
grid
and
now
it's
just
Cassandra,
the
catalog
messaging
and
the
catalog
services.
So
this
is
just
a
final
diagram
to
show
kind
of
how
we
had
the
switch
from
on
the
left,
the
old
legacy
services
servicing
most
of
the
traffic
and
then
on
the
right.
Switching
over
to
the
99
percent.
Plus,
you
know
hundred
percent
of
the
traffic
going
to
the
the
new
catalog
services.
A
C
Everyone,
so
there
were
several
use
cases
we
needed
to
implement
on
Cassandra
today,
I'm
going
to
talk
about
one
use
case
which
actually
required
us
to
model
hierarchical
dating
Cassandra.
You
can
see
simplified
the
main
model
on
the
slide.
Basically,
there's
product
object,
which
has
such
attributes
like
name,
description,
shipping
information,
there's
one
too
many
relation
to
its
attributes
used
by
moisturizing
managers
to
trigger
various
business
rules
around
how
to
shop
electrons
aside
and
other
selling
channels,
there's
one
to
many
relationship
with
a
tribute
value
and
value
itself.
C
It's
a
composite
object
which
has
like
effective
expiration
date
status,
which
could
be
enabled
disabled,
archived
and
so
on.
There's
one
too
many
relations
firm
product
and
UPC
and
in
our
data
model
product
is,
has
some
common
information
for
sellable
items
and
UPC
is
something
it's
basically
UPC's
a
sellable
item
which
it
has
combination
of
color
and
size
of
liability,
information
online
and
in
stores
price
information.
C
So
there
are
many
such
one
too.
Many
relations
are
nested
relations
which
starts
from
product.
We
wanted
to
aggregate
the
data
in
such
a
way
to
minimize
minimize
number
of
reads.
Ideally,
we
would
like
to
read
all
that
information
by
single
read
operation.
We
wanted
to
avoid
it
reads
before
rights
for
element,
use
cases
to
leverage
high
write
throughput
of
Cassandra,
and
we
wanted
to
keep
all
the
data
we
store
in
Cassandra
in
clear
text,
so
that,
if
there's
some
issue
with
with
service
response,
we
could
directly
to
connect
to
Cassandra
see.
C
Maybe
it
and
kind
of
clearly
identifies
it
caused
by
data
issue
or
there's
some
defect
in
business
logic
and
application
layer
back
in
2013
there
were,
it
was
new
release
of
Cassandra
1.2,
it
had
collections
compound
keys,
and
so
on
so
first
day
tomorrow
we
tried
to
implement
that
was
based
on
what
we
call
a
parallel
list.
Let's
consider
that
example
on
the
slide
there's
product
entity
which
has
ID
and
name
fields,
there's
one
too
many
relation
to
the
PCM.
The
pc
has
ID
color
fields
and
product
ID,
which
is
foreign
key
to
product.
C
Upc,
has
one
too
many
relation
with
attributes,
which
has
ID
name
value
fields,
an
epc
ID,
which
is
foreign
key
to
you,
PC
and
how
it
works.
Is
you
peein
name
fields
of
product?
They
go
as
is
to
the
product
table
and
secure
because
they
belong
to
top
level
entity.
I,
gene
color
fields
of
UPC.
They
go
into
you,
p,
CHD
and
epc
color
lists
in
such
a
way,
so
that
I'd
in
color
of
the
first
UPC
they
become
the
first
element
in
those
lists.
I
didn't
cover
the
Tekken
3
pc.
C
C
Obviously
we
could
read
all
the
thing
by
product
ID,
which
is
partition
key
here,
it's
in
clear
text,
although
we
cannot
we
couldn't
to
update
before
without
reads
because,
for
example,
if
you
want
to
change
color
for
you
bc
with
a
g33,
we
don't
know
whether
its
second
item
on
the
list
or
the
first
first
item
element
in
dos
para
la
lists.
So
we
need
to
read
it
first
and
then
issue
updated
duration.
Don't
work
on
the
tissue!
We
thought
about
using
maps
in
maps.
C
C
Recently
in
2.1
version
of
Cassandra,
there
is
third
version
of
the
native
protocol
and
those
limits
were
increased.
Now,
therefore,
bytes
used
to
encode
collection,
size
and
length
of
each
element,
so
theoretically
like
that,
could
be
leveraged,
although
wanted
to
mention
that
there
are
still
64k
limit
for
set
elements.
C
Mark,
sighs
and
map
keys,
max
size
and
also
official
documentation
from
datastax
doesn't
recommend
going
above
64k
for
any
collision
type,
so
kind
of
you
can
use
it
at
your
own
risk
just
wanted
to
briefly
go
through
why
they're
still
64k
limit
for
set
elements
and
the
map
keys?
Let's
consider
this
example
there's
collection
table
which
has
four
columns
Mikey
my
set
my
map
and
my
list
and
I
insert
sample
data
like
two
integers
1
and
2
into
the
set
and
the
list
and
to
enter
to
the
map
so
internally.
C
We
know
that,
like
each
piece
of
dating
Cassandra
stored
as
a
cell
and
each
cell
has
some
named
Sullivan
times
them
some
other
tributes
to.
But
those
are
kind
of
major
things
and
you
can
see
highlighted
in
red,
said
elements
as
well
as
map
keys,
they
actually
stored
in
the
cell
name,
part
of
the
cell
and
highlighted
in
blue
those
are
list
values
and
map
values.
Part
of
entries
and
in
the
internal
storage
there's
actually
two
bytes
are
allocated
to
encode
cell
name.
C
That's
why,
even
though
later
protocol
supports
longer
set
elements,
it
is
limited
that
limitation
is
still
comes
from
kind
of
internal
storage
structure
also
wanted
to
mention
that
except
hexadecimal
string
in
bold,
that's
16,
bytes
I
used
to
maintain
order
filaments
enlist,
and
that
may
be
pretty
significant
overhead.
If,
if
you
store
some
small
village
at
least
like
integers,
which
is
just
4
bytes
32
model,
we
try,
it
was
based
on
compound
keys
and
Jason.
Congenial
dishonest
is
a
way
to
represent
hierarchy.
It
supports
it.
C
Naturally,
the
only
question
was:
do
people
like
her
granola,
it
should
be.
We
want
to
put
everything
product
with
all
its
attributes,
upcs
and
everything
into
a
single
big
JSON
object,
or
it
should
be
more
granular.
Based
on
our
written
update
patterns,
we
decided
that
the
best
way
to
split
it
would
be
to
have
just
one
object
for
products
and
separate
just
one
object.
3Pcs
next
station
was
how
do
we
express
relations
between
those
and
they
use
compound
key
to
do
that
in
this
table.
C
There
are
two
fields
which
are
part
of
primary
key
product
ID,
which
is
partition
key
and
UPC
ID,
which
is
clustering,
key
object,
field
of
type
text
which
can
store
either
production
or
your
PC
JSON,
based
on
value
of
UPC
ige,
our
classroom
key,
so
that,
if
it's,
if
you
p,
SE
d,
equals
zero.
That
means
that
Sakura
is
protocol
and
restore
you
product,
ID
product
JSON
there.
Otherwise,
if
it's
a
positive
number,
which
would
mean
would
be
actual
UPC
number
in
that
case
in
the
case
that
Sakura
would
be
a
UPC
row
and
object.
C
Columns
of
store,
UPC
JSON
object,
so
that
data
model
also
worked
fine
like
very
presented
hierarchy.
Thanks
to
Jason.
We
could.
We
even
got
some
more
flexibility.
How
we
can
read
and
updated,
because
we
could
read
product,
allows
you,
pcs
altogether
by
just
partition
key
or,
if
you
know,
UPC
number
like
by
product
I'd
appreciate.
If
you
could
read
just
j-just
UPC
object,
not
just
product
object,
so
one
question
was,
since
we
use
json
its,
doesn't
support
partial
rates
on
updates.
C
If
you
want
to
change
some
small
piece,
let's
say
in
the
product
json,
you
would
need
to
read
full
json
for
product.
First
I
apply
imitations
and
then
store
it
back
for
our
case.
Actually
for
such
cases,
when
we
needed
partially
written
updates,
we
just
extracted
that
information
to
separate
column
like
review
column,
fourth
column
of
the
table
like
if
you
go
vomit
stores,
customer
ratings
data
and
that
comes
from
different
source
that
has
different
life
cycle.
So
we
just
put
that
into
separate
column,
and
that
was
fine
with
us.
C
So
when
we
had
the
tall
terms
of
data
models,
we
ran
some
performance
tests
to
see
what
works
better.
One
of
those
tests
was
data
load
artist.
Basically,
criteria
knows
how
long
does
it
take
performance
test
consists
of
three
data
loads.
There
was
data
loader
component,
which
was
loading.
Data
from
relational
database
was
assembling
data
in
such
a
way,
so
it
could
be
written
to
Cassandra.
C
First
of
all,
with
collections
based
approach
that
approach
produced
hundreds
of
thousands
cells
per
partition,
key
on
real
data,
for
example
like
if
we
have
product
with
10
new
PCs,
so
to
store
all
those
10
new
PCs.
We
would
need
collection,
size
10
if
each
GPC
has
10
attributes
to
store,
then
Oh,
letting
all
the
tribute
of
all
the
pieces
with
inhalation
of
science
hundred
if
which
attribute
has
10
values,
then
to
store
all
the
wells
of
all
attributes
of
Allah.
C
Pcs
collection
would
be
of
size
thousand
and,
as
you
know,
each
lamenting
collection
produced
a
separate.
It
becomes
a
separate
cell
internally
in
Cassandra
storage
model.
So
second
reason
is
storage
overhead
there,
as
formula
could
be
used
to
calculate
size
of
each
cell
so
for
regular
column,
it's
a
column,
name
plus
column
value,
plus
15
bytes
in
case
of
lists,
as
I
mentioned
earlier,
there
are
16
bytes
encoded
into
column,
name
to
maintain
orders
with
additional
16
bytes,
plus
your
column
name.
C
So
and
let's
see,
if
you
store
something
small
as
like
product
ID
or
attribute
value
which,
which
is
just
few
bites,
you
can
pay,
maybe
like
40
bytes
of
storage
overhead
to
store
that
data.
So
it's
really
bad
ratio
of
storage,
metadata
and
payload
and
for
the
song
based
approach,
it's
much
better,
because
if
you
start
something
like
10,
kilobytes,
JSON
object
or
highly
club
a
social
object,
you
don't
really
care
if
there's
some
extra
30
or
40
bytes
of
overhead.
C
Just
to
summarize,
based
on
our
experience,
we
think
that
this
one
based
approach
works
better
to
a
store,
hierarchical
data,
because
there's
better
ratio
between
storage
and
payload,
it
had
direct
correlation
with
throughput
and
latency
of
both
read
and
write
operations.
It
supports
helical
structures
with
reducing
partial,
written
updates,
but
there's
work
around
you
can
expect
like
in
some
cases
it
works
and
some
it
doesn't.
But
in
our
case
it
worked.
C
Fine,
okay,
so
just
a
start
date
in
two
separate
column
also
wanted
to
mention
that
it
complicates
Kim
abortion,
because
without
JSON
you
care
only
about
Sakura
schema.
But
if
you
have
JSON,
which
evolves
over
time
with
your
application,
you
need
to
take
care
of
a
proper
version
called
boss,
secure
schema
and
rizal.
Also
there's
user-defined
types,
which
we
introduced
a
thing
as
part
of
casado
2.1,
which
might
be
good
option
actually
to
express
such
hierarchy,
because
now
you
gt's,
then
they
can
have
column
types.
B
Okay,
I
just
want
to
quickly
summarize
and
overrun
a
ton
of
the
the
overall
journey
we
had
what
we
thought
worked
really
well
and
what
was
good
was
the
documentation
for
Cassandra
is
really
good.
Datastax
was
very
helpful
along
the
journey.
Cql
3
is
very
easy
to
learn.
Coming
from
a
relational
background,
query
tracing
very
useful:
it's
been
stable,
we've
had
very
few
production
incidents
in
the
the
big
one
for
me
was
able
to
simplify
our
architecture,
reduce
all
the
caching
mess
that
we
got
into,
but
still
get
the
same
performance
for
development.
B
B
We
had
deleting
and
creating
key
spaces
without
truncating
them.
First,
we
were
able
to
load
data,
but
then,
when
we
tried
to
read
there
were
his
references
to
old
files
and
follow
it.
I
0
exceptions,
like
most
people
like
compaction,
dealing
with
tombstones,
was
a
learning
process
and
we
definitely
learned
a
lot
from
that.
We
were
clustering,
a
lot
of
data
that
shouldn't
have
been
so.
Those
nodes
in
particular,
took
days
to
finish
compaction
is
a
von
talked
about
you,
your
data
model
that
you
use.
B
You
know
you
need
to
understand
the
underlying
storage
behind
it
to
optimize
your
performance.
We've
had
a
few
problems
with
drivers,
we're
using
two
one
for
when
we
were
doing
rolling
upgrades
of
the
cluster
11
come
back.
We
took
it
out.
The
driver
didn't
really
understand
that
that
note
had
left
and
monitoring
performance
with
off
center
we
were
in.
It
was
hard
to
get
like
average
values
when
your
peaks
are
so
high,
but
they've
gotten
a
lot
better
in
the
most
recent
version,
some
of
our
own
problems
that
we
had.
B
We
we
have
a
small
cluster
but
kind
of
beefier
machines,
and
that
requires
that
all
your
servers
perform
really
well
rather
than
having
a
larger
cluster
of
smaller
machines,
which
probably
would
have
been
a
better
in
the
outside.
Again,
what
a
bond
was
talking
about
was
versioning
in
the
JSON
schema.
You
got
to
make
sure
your
version
is
backward
compatible.
We
found
that
problem
of
we
updated
it.
We
could
write
to
it,
but
we
couldn't
read.
B
We
like
to
export
production
data
to
use
in
different
environments.
We
found
that
that's
a
very
expensive
operation,
causing
jumps
and
cpu
non-primary
key
access
was
difficult.
We
usually
do
look-up
tables,
and
then
we
have
a
lot
of
test
environments
for
merchants
to
look
at
what
way
the
catalogue
is
going
to
look
on
a
different
day,
and
we
have
so
many
that
we
try
to
keep
them
small
but
still
have
the
same
data
size
and
that's
been
a
real
challenge.
B
We
want
to
do
conditional
conditional
updates,
based
on
calling
timestamps
coming
to
not
have
to
read
before
you
write
again
and
that's
in
for
dot
7.
We
want
to
do
DSE
spark
for
some
reporting
and
analytics
on
our
data
like
finding
out
which
brands
or
which
sizes
or
which
colors
are
running
low
on
inventory.
That's
something
that
was
very
easy
in
a
relational
database,
but
now
we're
just
doing
primary
key
based
reads
that
becomes
a
challenge.
We
have
split
into
multi
data
centers,
but
we're
not
replicating
yet
they're
kind
of
totally
isolated.
B
We
want
to
replicate
properly
managing
the
schema
with
new
gen,
our
relational
database.
We
used
liquid
base
and
that
worked
well.
We
want
to
do
that.
We
want
to
try
to
figure
out
how
to
solve
the
JSON
schema
versioning
we'd
like
to
use
our
eggs
Java.
So
we
can
sync
personals
asynchronously,
load,
different
components
of
the
products
like
reviews
and
inventory
and
then
have
it
reassembled
after
we
think
they'll
improve
our
performance
and
we
use
spring
a
lot
at
macy,
so
explore
some
spring
data.
B
A
few
things
we'd
like
to
see
we'd
like
to
see
for
the
reporting
stuff
like
a
cabana
on
top
of
elastic
search.
We
did
talk
with
datastax
and
I
know
they
played
around
with
banana,
which
is
the
same
UI
on
top
of
solar,
so
that
might
work
well
when
we
move
to
a
multi
data
center
model,
where
we
actually
do
replicate
the
data,
we
have
some
use
cases
where
we
need
to
notify
our
consumers
that,
like
inventory,
changed.
B
So
we
don't
have
a
current
way
to
know
that
one
product
finished
replicating
so
we'd,
like
some
kind
of
listeners
on
that
and
moving
to
an
omni-channel.
We
actually
have
a
lot
more
data
that
resides
in
different
stores
and
being
able
to
search,
geospatially,
okay
and,
as
usual,
we're
hiring
we're
in
San
Francisco.
We
also
have
some
offices
in
Atlanta
and
Boise
Idaho.
So
if
you're
looking
for
work,
we'd
love
to
have
you
so
thank
you.
B
And
there
is
one
final
thing
I'd
like
to
mention
is
that
I
know
a
lot
of
people
here
know
Patrick
I
hope
you
saw
him
blending
raspberry
PI's
at
the
keynote
we've
identified,
that
we
think
he
has
a
secret
identity
and
it
was
Peter
I'm
figured
out
his
secret
identity.
We
want
to
expose
it
right
now,
so
he's
buddy.
D
B
D
B
C
Even
intentionally
why
we
had
so
many
unless
it
relations
I
mean
that
was
data
model
we
we,
which
we
used
in
previous
data
store,
which
is
which
was
relational,
so
there's
an
established
entities
and
relations
within
them.
Those
are
used
by
which
lesson
managers
to
set
up
business
logic,
it's
kind
of
old,
all
business
logic.
Is
data
driven,
so
there's
it's
easier
for
sizing
managers
to
to
express
like
what
they
want,
how
they
want
to
show
produce
on
site
to
give
them
the
best
flexibility
they
want.
C
C
E
B
E
Your
catalog
in
a
hierarchical
way
in
some
entities
right
now,
your
multiple
devices
have
different
requirements
on
what
attributes
to
show
or
how
much
information
you
want
to
show
on
which
page,
depending
on
your
in
the
product
details
page
or
on
the
checker
or
wherever
you
are.
How
do
you
handle
that
you
filter
or
all
the
data
in
the
application,
or
do
you
store
multiple
copies
of
it?
So.
C
Let's
see
that's
right
so,
for
example,
the
category
browse
page,
it
shows
kind
of
smaller,
aggregated
virtual
of
product
right
and
I'll
produce
sales
page.
There
is
big
product
image
and
so
many
details
we
actually
supported
both
those
use
cases
using
single
table.
So
in
every
time
we
read
the
whole
thing
from
Cassandra,
although
we
separated
kind
of
dynamic
and
more
static
data.
C
So
whatever
is
driven
by
availability
kept
what's
kept
separate
and
we
catched
all
that
information
in
application
layer
in
local
cache,
and
there
was
just
some
business
logic
we
will
be
playing
on
top
of
that
product
data.
We
were
creating
something
and
quadrature
like
different
data
based
on
request
type,
all.
F
We
are
high,
so
in
your
Jason
and
the
compound
key
example
that
you
had
for
the
second
data
model,
you
mentioned:
there's
a
lack
of
indexing.
Besides
the
primary
index
to
pull
out
data
from
your
catalog,
for
example,
so
one
suggestion
I
have
I,
don't
know
if
you
guys
explored
that
is
dec.
Cassandra
has
solar
integration
with
it.
So
if
you
have
the
right
columns
exposed,
not
your
entire
JSON
document,
you
could
index
on
those
columns
and
get
the
data
quicker
or.