►
From YouTube: C* Summit EU 2013: Denormalizing Your Data: A Java Library to Support Structured Data in Cassandra
Description
Speaker: Eric Zoerner, Senior Software Developer at eBuddy
Slides: http://www.slideshare.net/planetcassandra/c-summit-eu-2013-denormalizing-your-data-a-java-library-to-support-structured-data-in-cassandra
In this session you'll learn about the design and implementation of a new open source general-purpose Java library that supports storing structured data in Cassandra. Instead of mapping the data to multiple tables like an ORM would or embedding data using serialization, this approach decomposes structured data of arbitrary complexity into separate columns of simple values, allowing the data to be retrieved or updated in parts using hierarchical paths. Implementations are included for Cassandra using both the Thrift and CQL3 APIs. In addition, Eric's experiences are shared regarding the challenges of using CQL3 vs. Thrift for schema-less data.
A
What
I'm
going
to
talk
about
I'll
mention
briefly
Hui
buddy
is
and
what
we
do
and
then
I'll
introduce
the
library
which
I
call
c
star
path
and
go
into
the
details
of
how
does
it
work?
How
is
it
designed
and
what
were
some
of
the
challenges
in
implementing
it
and
go
into
the
details
of
how
the
data
is
modeled
in
Cassandra,
using
this
library
and
then
finally,
some
plans
for
the
future
of
what
I'd
like
to
do
or
what
we
would
like
to
do
with
the
library.
A
A
A
Also,
we
use
it
for
persistent
session
store.
This
is
a
user
session
when
they're,
both
when
they're
connected
and
when
they're
not
connected.
So
every
user
has
a
persistent
session
so
once
you've
you've
connected
for
the
first
time,
you're
always
quote
unquote
logged
in
the
only
difference
is
whether
you're
currently
connected
or
not,
and
we
we,
we
use
the
session
for
a
number
of
things
like
caching,
the
contact
list
and
various
pieces
of
data
that
we
need
to
track
the
users
and
we
use
Cassandra
to
store
the
those
user
sessions.
A
We
also
use
it
for
message
history,
so
when
you
connect
with
a
client
it
depending
on
there
may
be
some
state
on
the
client
or
there
may
not
be.
But
the
client
can
query
XMS
to
get
the
message
history
for
its
conversations
and
we
store
all
the
message,
history
and
Cassandra
as
well,
and
we
also
use
it
for
location-based
discovery
so
that
you
can
find
other
contacts,
XMS
contacts
that
are
nearby
and
we
do
that
through
geo
hashing.
And
so
we
when,
when
that
feature,
is
enabled
we
store
locations
in
Cassandra.
A
And
just
to
give
you
an
idea
of
our
data,
we
have
about
half
a
terabyte
of
actual
data,
which
is
replicated
16
million
sessions
approximately
right
now,
which
includes
both
sessions
for
users
as
well
as
for
group
conversations.
I
also
have
their
own
session
and
we
have
one
particular
column,
family,
which
is
kind
of
an
index
into
other
column,
families
which
has
almost
a
billion
rows
in
it.
A
But
what
I?
What
I
want
to
focus
on
in
this
talk?
Is
the
library
C
star
path,
which
we
have
recently
open
sourced,
because
I
think
it's
a
useful,
useful
library
to
help
in
Cassandra
its
applications
to
store
structured
data
in
Cassandra,
so
it
dresses.
A
classic
problem
is
that
I
have
complex
objects,
complex
structures
that
might
even
might
contain
collections
of
other
objects.
And
how
do
I
store
that
in
a
data
store
that
supports
simple
data
like
keys
and
values?
A
For
example,
you
could
serialize
your
data,
so
you
take
the
whole
complex
object,
convert
it
into
bites,
one
way
or
the
other
store
those
bites
in
the
value
of
a
key
value
in
a
Cell
I
could
be
binary
or
it
could
be
textual.
For
example,
XML.
You
could
convert
your
object
into
an
XML
structure
and
just
save
that.
So
that's
that's
a
very
popular
way
to
do
it.
A
A
A
Serialization
has
the
advantage
that
you
can
do
a
single
right
when
you
write
the
object
to
your
database.
You
package
it
up
into
a
blob
and
you
just
do
one
right
and
the
whole
thing
is
there.
You
can
read
the
back
with
a
single
read,
so
these
are
advantages,
especially
with
Cassandra
I.
Think
you
probably
have
all
heard
about
best.
A
And
it
also
can
be
a
pretty
good
guarantee
of
a
consistent
update,
so
your
reading
and
writing
an
entire
structured
object
in
one
go:
there's
no,
it's
not
easy
to
read,
get
a
partial
read
or
to
corrupt
that
object
with
a
partial
right,
so
it
gives
a
good
guarantee
of
a
consistent
update
so
you're
either.
Writing
the
whole
object
or
nothing
right,
structural
access.
A
What
I
mean
by
that
is
the
ability
to
access
parts
of
the
structure,
so
a
single
street
name,
a
single
name,
a
single
address
from
a
more
complex
structure,
so
serialization
doesn't
really
allow
you
to
do
that,
it's
all
or
nothing.
So
it's
kind
of
the
inverse
of
consistent
updates.
There
also
cycles.
A
A
So
that
is
a
possibility
with
serialization
normal
normalization.
You
split
the
object
into
multiple
tables,
so
you
lose
the
ability
to
update
the
structure
with
a
single
right
or
single
read.
So
this
is
the
main
reason
why
normalization
is
not
a
best
practice
with
Cassandra
it's
better
to
denormalize
it
and
normalization
can
give
you
some
some
surety
that
you're
doing
consistent
updates,
but
perhaps
perhaps
not
that's
a
little
bit.
A
So
yes,
in
some
cases,
it's
possible
structural
access.
Yes,
you
can
read
and
write
parts
of
the
object
just
by
reading
and
writing
to
a
particular
table
and
cycles.
Yes,
you
can
also
model
cycles
with
normalization,
because
you
can
have
links
use
identifiers
to
maintain
object,
identity
right
with
decomposition.
A
You
also
get
the
benefits
a
lot
of
the
same
benefits
of
serialization
with
single
rights
and
single
reads,
which
is
why
I
think
it's
good
for
Cassandra
is
and
but
you
will
talk
more
about
consistent
updates,
it's
not
really
enforced
by
the
library.
The
library
remains
flexible,
so
it
is
possible
to
for
an
application
to
shoot
themselves
in
the
foot
with
with
an
update
that
doesn't
make
sense.
A
A
A
So
the
I
think
some
some
plus
points
on
this
library
is
that
it's
very
easy
to
use
very
simple
API
and
it's
good
for
Cassandra
I.
Think
because
it
provides
structural
access.
You
can
read
and
write.
You
can
write
parts
of
the
object
without
having
to
read
the
whole
object
first,
so
that's
the
best
practice
with
Cassandra
good
for
Needy
normalizing
the
data.
A
A
So
in
this
API
example
like
I
call,
the
structured
data
object
or
two
sorry,
the
structured
data
support
object
dao,
just
because
it's
short
fits
on
the
fly
better,
but
it's
actually
more
of
a
doua
support
object.
It
isn't
actually
intend
to
to
be
a
dowel
object,
although
you
could
use
that
for
simple
cases
as
a
dowel,
so
given
a
row
key
in
a
pojo
which
is
an
object
in
your
application
domain,
I.
A
Where
do
you
want
to
write
it?
So
you
just
create
a
path
using
the
support,
object
and
specify
the
elements
in
the
path
so
XYZ
you
can
make
it
as
long
or
short
as
you
like,
and
then
with
that
path.
You
just
call
right
to
path
and
give
it
the
roki
the
path
and
your
POJO
and
it
does
it
does
the
right
thing,
with
a
little
bit
of
help,
perhaps
and
we'll
go
into
the
we'll
dive
into
that,
a
little
bit
to
read
an
object
from
a
path.
A
You
again
create
a
path,
and
then
you
say,
read
from
past:
give
it
a
rocky
the
path
and
a
type
reference.
So
this
tells
the
library.
How
are
we
going
to
recompose
this?
This
object
into
your
POJO,
so
if
you're
familiar
with
jackson
library
for
doing
Jason,
then
this
should
be
very
familiar
to
you.
It's
it.
Actually,
it
isn't
the
Jason
class,
but
it's
pretty
much
a
copy
of
it.
A
It's
a
it's
based
on
the
Jason
implementation,
where
you
basically
say
this.
The
object
I
want
is
of
type
pojo
and
the
reason
you
have
to
use
this
strange
syntax
has
to
do
with
the
idiosyncrasies
of
Java
generics
and
type
erasing
or
whatever,
so
the
generic
type
becomes
available
to
the
library
at
runtime.
A
The
library
comes
with
an
interface
called
batch
contact,
which
is
an
abstraction
so
that
the
library
can
be
used
for
mole,
multiple
implementations,
there's
there's
actually
a
cql
implementation
and
a
thrift
implementation
in
the
library.
So
batch
contacts
is
just
an
abstraction.
So
this
code,
you
write
to
this
API
and
it
will
work
with
either
cql
or
or
thrift
with
no
code
changes.
A
A
A
A
So
to
decompose
an
object,
a
structured
object.
The
first
thing
it
does.
Is
it
converts
that
domain
object
into
a
basic
structure
of
maps,
lists
and
simple
values
like
strings,
bullion's
and
numbers,
and
to
do
this,
it
actually
uses
the
jackson
library
so
an
advantage
of
using
the
jackson
library.
Is
that
not
only
do
I
not
have
to
write
it
myself?
A
It
also
means
that
it
will
honor
the
jacks
and
annotations
on
your
domain
objects.
So
you
may
already
have
jackson
annotations
on
your
domain
objects
for
converting
those
objects
into
jason
right.
So
the
same
annotations
are
honored
in
this
library
for
converting
your
objects,
breaking
it
down
into
basic
structured
structures,
maps,
lists
and
simple
values,
and
then
once
it's
done
that
it
will
take
those
that
basic
structure
and
decompose
that
into
a
map
of
paths
to
simple
values-
and
this
is
done
by
an
object
in
the
implementation
called
the
decomposer.
A
Example,
so
you
take
a
complex
object
like
a
person
with
addresses
and
phone
numbers,
and
then
you
simple:
it
simplifies
it
into
maps,
lists
and
simple
values.
So
you
get
a
map
named
John
birthday,
nickname
a
list
of
addresses
and
a
list
of
phone
numbers
and
addresses
are
also
broken
down
into
a
list
of
maps,
so
so
objects
become
maps
and
collections
become
lists
and
then
and
then
it
creates
a
map
of
paths,
mapping
to
simple
objects
like
strings
and
numbers,
etc.
A
To
read,
we
basically
read
the
pads
and
values
out
of
the
database
at
a
given
path,
which,
which
means
do
if
you
specify,
for
example,
here.
If
you
specify
the
path
addresses,
then
it
will
do
a
slice
query
to
get
all
the
pads.
That
begin
with
the
element
addresses,
so
it
gets
all
the
structure
underneath
as
well.
A
Reads:
path:
values
from
the
database:
okay
and
then
it
takes
those
paths
and
values
and
merges
them
back
into
the
basic
structure
of
maps,
lists
and
simple
values
by
an
object
called
composer.
So
we
have
D
composer
and
composer,
and
then
it
passes
these
basic,
this
basic
structure
back
to
Jackson
and
gives
it
a
Jackson
type
reference
and
says,
give
me
your
PO
Joe
back
so
yeah.
That's
the
leverages,
the
jackson
library
to
do
that
so
design
and
challenges
some
things
that
are
more
specific
to
use
with
Cassandra.
A
So
the
paths
themselves
are
stored
as
strings.
You
saw
that
there
were
forward
slashes
in
the
pads,
that's
hidden
by
the
path
API.
It
was
yeah.
It
was
a
design
decision
to
store
them
as
strings.
There
is
another
option
that
I'll
talk
about
a
little
bit
later,
that
we
decided
not
to
do
at
least
not
yet.
A
The
path
elements
are
internally
URL
encoded
so
that
we
can
use
some
special
characters.
That's
that
are
used
internally
by
the
library,
for
example,
the
@
symbol
is
used
for
list
indexes
index
into
a
list.
The
path
API,
also
by
the
way,
allows
you
to
specify
a
list
index
in
your
path.
So
it
is
possible
to
read
and
write
to
individual
list
elements
using
the
library.
A
A
A
A
That's
kind
of
a
problem
because
now,
if
I
read
from
the
path
and
say
I
want
a
list
of
strings
back,
I
get
3
D
2,
which
is
not
nobody
ever
wrote
a
list
that
was
32.
What
happened?
Was
the
list
shrunk
on
this
on
this
right
and
it
left
behind
an
element
from
the
longer
list?
It's
still
there
in
the
database.
So
how
do
we
fix
that
problem?
Well,
the
implementation
writes
a
list
Terminator.
A
A
Yes,
so
the
values
here
as
we'll
see
you
later,
the
values
here
also
always
typed
as
text.
So
I
used
a
special
text
value
here,
which
is
an
illegal
eunuch.
Unicode
string,
well
reserves
Unicode
string
to
indicate
that
it's
the
terment
the
Terminator
and
then,
if
you
write
the
three
back,
it
will
write
the
Terminator
at
at
one
and
then
you
read
the
whole
list
back.
You
get
the
right
answer.
A
A
A
You
can
avoid
that
if
the
client
that
the
caller
knows
that
that
could
happen,
and
so
it
could
do
a
delete
before
it
does
the
right.
Okay,
well,
yeah,
the
library
does
not
enforce
that,
but
so
it's
up
to
the
client
to
do
the
right
thing.
So
it's
also
up
to
the
client,
the
user,
to
know
what
they're
doing
and
understand
something
about
the
how
this
is
implemented
right.
So
the
shrinking
list
problem.
It's
it's
been
solved.
If
you
are
reading
and
writing
entire
lists
at
a
time.
A
A
A
Okay,
so
now
in
your
database,
you
have
X
address
Street,
X
name,
which
contains
a
string,
but
then
also
at
X
name.
You
have
more
structure,
so
you
have
both
a
simple
value
and
structured
data
at
the
same
path.
So
that
is
not
consistent
right.
You
cannot
have
a
simple
value
and
a
structured
value
at
the
same
path,
but
it
is
possible
to
store
it
that
way
in
the
database.
A
A
A
A
A
However,
if
you
try
to
look
at
the
data
with
dynamic
composites,
it
causes
cql
shell,
to
crash
and
in
general,
it's
not
as
nice
to
be
able
to
browse
the
data,
so
it
at
least
with
this
version
of
the
library.
We
decided
to
just
keep
it
simple
and
use
strings.
When
you
read
the
data
back
into
your
pojo,
if
your
pojo
is
a
sorted
map,
those
keys
will
naturally
get
resour
tada
a
client
right,
but
for
this
version
of
the
library
anyway,
we
always
store
the
columns
as
strings.
A
Yeah,
starting
to
run
out
of
time
already
I
want
to
take
a
look
at
the
Cassandra
data
model
in
thrift.
We
just
store
the
paths
as
column
names
and
the
values
as
values
as
in
the
column,
families
with
super
column
families.
It's
also
possible
well
possible
in
the
sense
that
it's
not
implemented
yet,
but
it
is,
it
is
planned
to
the
first
element
of
the
path
will
just
become
the
super
column
name
and
and
we'll
go
from
from
there
down
for
the
rest
of
the
of
the
path.
A
The
thrift
implementation
uses
the
Hector
client
and
to
create
a
thrift
structured
data
support
object.
You
give
it
a
template,
object,
that's
also
part
of
this
library.
It's
our
own.
It's
our
own
template
object
that
does
all
the
basic
operations
using
thrift
using
the
Hector
clients,
so
it's
kind
of
an
abstraction.
Well,
it
is
an
abstraction
on
top
of
the
Hector
client,
which
is
in
turn
and
abstraction
on
top
of
thrift.
A
So
you
create
a
template
if,
at
the
cert,
the
Hector
serializers
that
you
need
and
using
in
our
own,
the
structured.
The
structure
serializer
is
part
of
the
library
as
well,
and
that's
just
used
to
be
able
to
actually
use
a
serialization.
If
you
actually
were
well
it
that's
implementation
detail,
but
it
encodes
structured
objects.
A
Cql
there's
some
constraints
on
the
data
model.
You
can
use
it
with
any
table
with
it,
but
you
have
to
have
the
path
itself
is
a
column
and
it's
it's
it's.
We
use
a
dynamic
column
family
for
this.
We
cannot
use
the
path
itself
as
a
column,
name
and
cql,
because
because
it's
dynamic
right,
it's
not
fixed.
So
we
use
a
fixed
column
name
like,
for
example,
key
I'm.
Sorry
path.
He
is
there
is
the
partition
key
such
as
path
and
then
another
column
called
value.
A
So
we
store
it
as
key
value,
pads
home
or
path
values
in
the
table.
So
you
see,
would
have
to
set
up
your
table
in
this
way
and
give
it
the
key
and
the
path
as
the
primary
key
key
is
just
the
roki.
The
partition
key
and
the
path
has
to
be
a
clustering
key,
and
the
reason
for
that
is
we're
doing
range
queries
that
library
does
range
queries
on
that
that
path,
column,
us
so.
A
It
needs
to
do
arranged
like
our
slice
query
on
the
path,
so
the
path
must
be
a
clustering
key.
It
has
to
be
the
first
clustering
key,
because
otherwise
we
would
have
to
use
an
equals
condition
on
the
previous
clustering
keys.
In
order
to
do
the
query,
this
is
basically
the
constraints
on
what
kinds
of
queries
are
possible
to
do
with
cql.
A
This
is
very
similar
to
how
cql
would
actually
nest
sets
as
well
and
GQ
also
currently
only
supports
simple
values,
because
if
it's
a
set
of
complex
objects,
then
it's
not
really
possible
to
maintain
the
structural
access
of
those
objects
so
well.
What
I'd
like
to
do
is
with
sets
of
simple
values,
is
to
be
able
to
put
those
simple
values
into
the
column
names
into
the
key
part,
and
then
that
way
you
can
do
set
operations.
Access
do
contains
access
elements
of
that
set
individually.