►
Description
Speaker: Gordon Worley, Senior Software Engineer
At AdStage we have a large volume of data about ads and their relationships: campaigns, ad groups, keywords, bids, budgets, targeting info - the list goes on. We started out storing all this data in Postgres, but even before we reached public beta we were already putting too much strain on the largest Postgres instance we could run. We considered sharding, but instead we decided to embark on a project to store our data in Cassandra. Now, after more than a year of development, we present Monacella, a relational object database that uses Cassandra as the datastore. In this talk we'll examine the architecture of Monacella, its features and use cases, and plans for future development.
A
A
So
to
give
you
a
little
bit
of
the
origin
story
of
it,
we
need
to
import
a
lot
of
data
from
multiple
ad
networks
and
just
two
I
didn't
put
like
a
you
know.
This
is
us
right,
so
we
basically
have
you
know
AB
data
coming
in
from
lots
of
different
networks,
and
we
need
to
pull
all
that
data
in
and
then
put
it
into
a
unified
place
where
we
can
then
process
it
and
manage
it
all
together.
A
So,
in
order
to
import
all
of
this
ad
network
data,
we
have
a
highly
parallelized
ETL
pipeline,
and
this
etl
pipeline
brings
in
two
kinds
of
data
right,
one
kind
of
data
it
brings
in
we're
not
going
to
worry
about
here
today,
which
is
like
ad
performance
data.
So
this
is
data
like
how
many
clicks
and
add
had
how
many
impressions
it
had
conversions
and
so
on.
You
know
that
data
is
like
important
and
interesting
and
we
actually
spent
a
lot
of
time
building.
A
It,
but
what
I
actually
want
to
talk
to
you
day
is
about
the
other
kind
of
data
that
we
bring
in,
which
is
the
metadata
that's
on
the
ads.
So
this
is
data
on
the
ads
that
tells
you
about
say
like
what's
the
name
of
the
ad
or
what
it's
the
budget
or
the
targeting
criteria,
and
also
how
it
relates
to
other
entities
that
exist
within
that
ad
network,
because
you
typically
don't
just
have
ads.
You
have
ads
that
belong
to
ad
campaigns
right
and
then
maybe
there
are
keywords
associated
with
it
and
other
objects.
A
So
you
quickly
end
up
in
a
situation
where
you
have
a
relatively
complex
relational
data
to
deal
with
and
as
I
said,
we
have
highly
parallelized
ETL
pipeline.
That
was
an
architectural
choice
we
made
early
on
because
it
worked
very
well
for
the
performance
data.
However,
when
we
as
we
started
the
scale,
this
happened
really
when
we
were
still
in
alpha.
B
A
A
We
weren't
quite
sure
how
it
was
going
to
work
out,
but
because
we're
already
using
Cassandra
extensively
for
all
the
performance
data,
we
had
a
lot
of
experience
with
it
operationally
and
in
terms
of
like
data
modeling.
We
use
it
additionally
for
storing
a
lot
of
analytics
data
related
to
our
own
conversion
tracking.
So
you
know
we
have
our
own
pixel
that
our
customers
can
install
on
their
ads
that
allows
them
to
help
track.
You
know,
did
like
clicky
on
this
ad
result
in
conversion.
A
A
Limits
hard
and
we
knew
that
we
had
to
solve
this
problem.
You
know
we
didn't
have
a
whole
lot
of
operations
experience
within
the
company
or
anyone
that
really
would
be
a
good.
It
would
be
a
good
use
of
their
time
to
be
heavily
dedicated
to
operations,
so
we
wanted
to
build
something
that
would
primarily
put
the
complexity
in
the
application
layer
and
not
at
the
operational
layer
right.
So
that's
that's.
Another
thing
is
you
know
we
had
all
had
experience
working
in
environments
where
we
had
sharded
postgres.
A
We
knew
how
much
operational
overhead
was
involved
in
that
and
by
comparison
we
knew
how
relatively
easy
it
was
to
manage
a
Cassandra
cluster.
So,
given
that
we
were
said,
okay,
let's
give
this
a
try
see
if
we
can
build
something
that
works
well
enough
for
us
with
the
Stata,
which
is
what
led
to
the
original
version
of
Mona
chela,
which
looks
something
like
this.
So
didn't
really
have
a
whole
lot
of
stuff
in
it
right
it.
Basically,
even
today
it
still
has
a
relatively
simple
architecture.
A
Most
of
the
heavy
lifting
is
done
at
the
application
layer,
so
the
data
is
stored
in
a
fairly
simple
ways.
You
can
see.
You
know
each
column.
Family
is
partitioned
by
primary
key,
which
uniquely
identifies
the
entity
being
involved
in
this
case
I'm,
showing
you
an
ad
table,
but
there's
ones
for
multiple
things
I
have
just
did
I
listed
up
here
that
we
have
like
ads
campaigns
and
accounts,
because
those
are
the
biggest
things
that
we
typically
deal
with
in
the
ad
world.
A
Right
you
have
ads
ads,
belong
to
campaigns,
campaigns
belong
to
accounts,
you
have
this
hierarchical
structure
and
that's
partly
why
we
thought
we
could
get
away
with
doing
something
relational
like
in
Cassandra,
because
our
data
is
very
hierarchical,
for
least
most
of
the
relationships
we
care
about
our
relationships
and
some
of
those
expressed
in
the
way
that
we
decided
to
store
the
data.
So
you
can
see
here
on
an
ad
and
I'll
talk
about
the
ad
ID.
In
a
moment.
A
A
We,
the
the
ID
for
the
ad
stores,
a
relatively
you,
know,
richer
mount
of
information
in
it
about
what
the
thing
is
this
way.
Well,
this
serves
a
couple
of
purposes
one.
We
actually
already
had
this
ID
format
that
we
were
using
in
a
few
different
places
in
the
app
and
another.
It
allows
us
to
store
some
type
information
that
will,
you
know,
there's
very
useful
to
us
when
we
want
to
actually
work
with
the
data
later
so
here
you
can
see
that,
like
basically
we're
storing
things
like
oh
I
know
what
network
it
is.
A
So
they
all
have
some
common
things
right
like
ads.
Pretty
much
all
have
a
like
a
cost-per-click,
but
maybe
not
right,
because
it's
been
some
kind
of
added
in
so
maybe
it's
maybe
it's
you
know,
cost
per
million
impressions
or
something
you
know
they
have
headlines
right.
But
maybe
not
because
like
say
it's
twitter,
twitter
ads,
don't
really
have
headlines
right.
They
just
have
the
content
of
the
post
so
because
we
had
this
data
that
was
you
know.
A
These
things
are
very
similar
in
types
they
often
we
operate
on
them
in
very
similar
ways
in
the
application,
but
the
data
is,
you
know
like
different
enough
to
cause
complications
for
us.
We
couldn't
just
store
it
all
in
one
table,
so
you
know
being
able
to
use
a
map
and
go
schema
lists
on
that
stuff
was
extremely
helpful
to
us.
A
Now
there
is
some
I
should
say
that,
like
as
part
of
Monticello,
it
does
actually
have
a
concept
of
like
applying
data
schemas,
but
that
happens
at
the
application
level,
not
at
not
in
Cassandra,
and
that
gives
us
more
flexibility
to
decide
what
we
want
to
do
with
it.
A
lot
of
that
comes
down
to
like
how
we
want
to
serialize
the
data
and
things
like
that.
So
now,
initially,
when
we
built
this
thing,
I
built
it
using
secondary
indexes,
because
almost
as
soon
because
when
I
built
this,
we
look
through
the
application
we
said.
A
A
I
very
rarely
want
to
get
all
the
ads
in
it,
so
he
said:
okay,
great,
you
know,
that'll
make
this
kind
of
use
case
very
easy,
because,
although
it's
you
know
somewhat
normalized
in
that
you
know
yeah
ads
or
store
different
partitions,
the
campaign's
and
different
partitions
that
accounts
I'm,
going
to
be
fairly
limited
in
how
many
queries
I
have
to
execute
to
actually
get
this
stuff
back,
but
then
it
turned
out
that
yeah.
We
actually
had
to
worry
about
that
other
ten
percent
of
the
time.
A
What
we
did
want
to
go
the
other
way,
because
we
tried
to
re-engineer
the
way
some
things
we're
built.
We
just
realized.
Okay,
there's
not
a
way
to
escape
these
queries.
We
have
to
do
them,
so
initially
we
tried
to
do
that
using
secondary
indexes
if
you've
ever
tried
to
use
secondary
indexes.
You
know
that
they're
kind
of
weird
they
work
as
advertised
right
like
you,
will
eventually
get
back
the
stuff
you
know
from
your
secondary
index,
but
if
the
fan-out
is
too
high,
they
suffer
significant
performance
problems.
A
What
is
too
high
the
Cassandra
documentation
is
not
very
clear
on
this.
I
would
say
too.
High
is
basically
if
your
fan
out
is
higher
than
5,000
right
basically
past
that
point,
your
stored
filter
will
reach
a
state
where
it
now
will
start
taking
exponentially
longer
and
longer
to
return,
as
you
add
items
to
it,
so
secondary
indexes
are
not
great.
Because
of
that,
and
if
you
get
into
this
situation
with
secondary
indexes,
there's
nothing,
you
can
do
to
tune
a
secondary
index
right,
you're,
just
stuck
with
it.
B
A
A
So
we
had
this
mechanism
that
would
allow
us
to
basically
tackle
both
of
these
problems.
You
know
replace
secondary
indexes
with
something
that
would
work
better
for
us
and
deal
with
this
problem
of
basically
having
you
know,
arbitrary
queries
and
the
way
we
do
that
is
by
pre
computing,
these
things
right.
So
a
storage
filter
is
essentially
just
a
pre
computed.
B
A
Inverted
index
that
we've
generate,
while
at
the
application
layer,
as
we
insert
the
data,
we
basically
make
it
so
that
it's
part
of
the
right
path.
So
in
order
to
get
any
data
into
Cassandra,
it
has
to
go
through
this
chunk
of
application
code
and
it
manages
making
sure
that
you
know
stored
filters,
get
updated.
Their
structure
is
pretty
simple.
It's
just
an
inverted
index
right,
so
it
has
like
this
facet
ID.
That's
to
identify
what
thing
you
know
your
data
is
faceted
by.
Is
your
campaigns
going
to
ads?
A
A
If
it's
longer
than
that,
what
it
has
to
do
is
go
back,
scan,
grab,
more
scan,
grab
more
scan,
grab
more
and
depending
on
your
access
patterns.
This
can
become,
you
know,
extremely
prohibitive,
especially
if
you
don't
get
it
all
at
once
so
or
if
you
try
to
like
page
through
it,
it
can
just
take
forever
to
like
run
over
the
length
of
your
entire
row,
and
I
mean
that
relatively
right,
if
your
primary
concern
is
through
plate,
it's
fine
right.
You
can
have
very
wide
rows
work
with
them.
Just
fine
throughput
will
be
fine.
A
However,
response
times
will
not
be
great
so
to
get
around
this,
what
you
can
do
is
basically
internally
shard
your
data
at
the
application
layer,
and
so
we
do
this
by
inserting
just
a
short
ID
into
the
primary
key
in
our
deployment.
It's
there's
32
shards,
so
it
basically
just
takes
the
other
stuff,
and
then
you
know
calculates
modulo
32
on
it
and
assigns
it
a
shard
ID,
oh
I'm.
Sorry,
that's
I
said
that
wrong.
A
The
it's
storing
this
entity
ID
so
basically
like
modulo,
30
tues
that
entity
ID
chooses
a
shard,
puts
it
in
that
one
and
that
what
that
allows
us
to
do
is
okay.
Fine!
Now
we
have
32
of
these
rows,
but
now
these
rows
are
distributed
around
the
cluster.
I
can
query
for
all
of
them
at
once,
and
although
the
total
amount
of
work
the
cluster
has
to
do
is
more
I
get
a
much
better
response
time
by
breaking
up
that
work.
So
this
is
one
thing
you
can
do
to
like
improve
response
times.
A
If
you
care
a
lot
about
reads-
and
you
have
a
lot
of
stuff
to
read
back
so
and
then
the
final
part
of
it
is
it
just
stores,
we
call
it
this
nad
ID,
it's
just
whatever.
The
thing
is
that
right,
you're,
looking
at
four
in
this
inverted
index
in
this
case
they're
stored,
is
the
clustering
column.
I
really
don't
know
why
I
decided
to
do
that.
I
think
it
was
3am
and
I
just
decided
that
that
was
okay.
A
A
So
that
was
basically
are
sensible
architecture,
and
this
is
actually
what's
in
place
today
and
once
we
built
this
thing,
we
didn't
actually
at
first
really
know
we
had
we
we're
still
sort
of
thinking
like
well
I
guess
this
is
something
like
no
sequel
thing
that
we
built
right.
I
thought
well,
it
kind
of
looks
like
maybe
a
graph
thing,
but
if
you
look
at
most
graph
databases,
they
actually
can't
handle
data
like
what
we
have,
because
most
grafted
well,
essentially
all
graph
databases
that
I've
worked
with
operate
efficiently.
A
A
We
have
type
based
relationships
between
the
objects
and
we
have
support
for
object
relationships,
but
not
arbitrary
joints.
So
then
I
realized,
oh
I've,
just
accidentally
reinvented
an
object
relational
database.
Now.
Why
would
we
want
this
right?
Because
the
thing
is
almost
nobody
uses
object,
relational
databases
right,
like
almost
nobody
has
ever
used
them.
A
They
are
like
strictly
inferior
to
relational
databases,
almost
always
right,
and
the
reason
for
that
is
because
they
just
offer
a
like
more
limited
feature
set
that
what
you
could
do
in
a
relational
database,
so
yeah.
Why
would
we
even
want
this
right?
Well,
the
reason
might
be.
You
need
horizontal
scalability,
but
you
don't
really
need
all
the
features
of
relational
database
and
you
need
object
relationships
so
the
difference
here
being
that
you
know,
typically
in
a
relational
database,
you're
able
to
do
arbitrary,
joins
between
data.
A
You
know
if
it's
expressible
in
relational
algebra,
you
can
do
that
in
your
relational
database.
It
doesn't
matter
how
complex
your
sequel
query
is
how
deeply
nested
the
joins.
Are
you
can
do
it
whatever
you
want
here,
however,
that's
not
possible,
but
in
most
applications
you
don't
need
to
do
that
or
it's
not
necessary
to
do
that.
A
If
you
structure
your
data
correctly,
so
in
some
cases
you
know
you
have
to
do
a
lot
of
deep
joins
just
because
your
data
is
highly
normalized
here
by
treating
the
data
as
objects,
are
you
know,
as
attributes
of
objects,
you're
able
to
keep
the
data
model
relatively
simple,
and
this
sort
of
thing
becomes
possible
all
right
so
made
this
object
relational
database?
Now?
What
are
we
doing
with
it?
Well,
you
know
so.
A
I
actually
hope
that
by
this
time,
I
would
actually
have
Monticello
in
a
state
where
I
could
open
source
it
and
just
say:
okay,
here's
the
repo,
you
guys
come
check
it
out.
Unfortunately,
we're
also
a
start-up
and
trying
to
like
you
know,
make
money
right,
so
I
sometimes
have
to
do
other
things,
but
these
are
the.
This
is
basically
the
roadmap
for
Mona
chela
of
what
needs
to
happen
before
it's
sort
of
like
ready
for
GA.
A
One
thing
is
to
simplify
the
data
model
right
now.
The
data
model
is
highly
like
the
internal
data
model,
I
mean
in
Cassandra.
It's
a
it's
actually
fairly
tightly
tied
to
the
way
our
to
our
domain
model.
That's
because
originally
we
were
expecting
to
use
well,
as
you
saw
like
we
use
like
the
stored,
the
secondary
indexes
and
stuff,
and
that
stuff
basically
requires
that
you
build
like
new
tables
for
each
type
in
order
for
it
to
make
sense.
A
A
There's
a
couple
of
ancillary
tables
related
to
things
like
paging
and
stuff
like
that,
but
we're
not
you
know,
I,
don't
count
those,
so
the
data
model
needs
to
be
simplified.
So
that
way
it
can
be
easily
like
used
by
anyone
else,
and
you
don't
have
to
do
a
lot
of
like
you
know,
weird
configuration
stuff
or
like
really
understand
deeply,
how
the
whole
thing
functions
in
order
to
be
able
to
use
it,
storage,
filter,
optimizations.
This
is
by
far
obviously
the
slowest
part
of
this
design.
A
Not
surprisingly,
because
we're
doing
exactly
the
sort
of
thing
you
are
not
supposed
to
do
with
Cassandra
right,
we
have
not
fully
denormalized
the
data
and
then
we
get
something
back
and
then,
when
we
go
to
execute
one
of
these
storage
filter
queries,
what
we
have
to
do
is
ok,
we
go
to
the
database,
we
get
back
to
stored
filter.
It
gives
we
get
this
inverted
index
right.
It
gives
us
a
whole
bunch
of
ids
for
objects,
and
now
I
say
oh,
I
would
really
like
those
objects.
So
now
I
issue
to
Cassandra.
A
So
we
have
made
some
improvements
to
the
way
stored
filters,
work
and
the
plan
would
be
to
make
some
additional
optimizations
around
that,
primarily
by
moving
a
lot
of
this
logic,
much
closer
to
Cassandra
and
one
option
might
even
be
to
operate
within
the
same
jvm.
So
this
is
not
like
unheard
of
people
have
written
applications
that
basically
are
built
so
that
they
run
inside
the
same
jvm
as
Cassandra,
and
this
eliminates
a
lot
of
the
you
know
back
and
forth
that
goes
on.
A
When
you're
doing
network
operations,
it
won't
totally
eliminate
it
because
we'll
still
have
situations
where
you
have
data
on
other
nodes
and
you
have
to
go
retrieve
it,
but
you
know
it
will
cut
out
as
much
as
it
can
and
finally,
another
option
that
we
might
try
is
to
actually
change
the
data
store.
So
if
you
noticed
I
didn't
really
do
anything
that
took
much
advantage
of
the
big
table.
Features
of
Cassandra
right
because
remember,
like
Cassandra,
is
really
like
two
things
merged
together
right.
A
It's
big
table
at
dynamo,
so
I'm
using
lots
of
the
dynamo
features
for
scalability
and
ease
of
operations,
but
almost
none
of
the
big
table
features.
So
it
would
be
very
easy
for
me
to
move
to
essentially
anything
else
that
can
function
as
a
key
value
store,
but
also
operates
under
the
dynamo
paradigm.
So
I
can
get.
You
know
the
operational
advantages,
so
one
thing
I've
looked
at
and
haven't
fully
finished.
Validating
is
netflix
released
a
thing
called
dynamite
which
essentially
allows
you
to
build
a
dynamo
database
on
top
of
any
other
underlying
data
store.
A
So
one
a
couple
of
compelling
options
would
be
to
try
Redis.
Maybe
is
an
underlying
data
store
for
this,
because
we
care
a
lot
about
latency
in
our
application,
or
you
know
actually
using
postgres
as
the
data
storing
each
one
of
the
charts
and
then
being
able
to
do
like
say,
like
some
chard
local
stuff
in
Postgres.
A
Before,
like
we
actually
move
up
the
application
layer.
Another
thing
that's
available
that
should
mention
is
de
rattus.
Soda
rattus
is
essentially
the
same
thing
as
what
we
built,
except
it's
actually
available
today
to
you
guys.
So
this
is
the
thing
out
of
Dell's
professional
services
arm
and
they
open
source
day.
So
you
can
find
it
on
github
I,
think
it's
just
like
I
think
their
dell
OSS
and
then
it's
like
taratus
right.
A
This
does
very
much
the
same
thing
that
Mona
chela
does
except
it's
already
available
to
you.
It
comes
in
two
parts,
spider
and
OLAP.
The
spider
part
is
essentially
the
same
thing
as
what
we
built
it
uses
slightly
different
technology
than
we
did.
However,
it
also
supports
a
lot
more
functionality
than
we
do
because
they
have
needed
other
things
than
we
have
right.
Mona
chela
is
primarily
restricted
by
right
now,
what
I've
needed
it
for
and,
as
I
said,
also
has
an
OLAP
layer
which
is
interesting
to
us
too,
because
that's
also
something
we
built.
A
So
you
know
if
you're
thinking
about
dealing
with
like
an
have
an
OLAP
application
and
you're
trying
to
figure
out
okay,
how
am
I
going
to
store
all
my
data
and
all
this?
This
actually
provides
fairly
straightforward
way
to
manage
it
and
then
finally,
another
option
that
we
could
go
back
to.
You
is
actually
sharted
postgres,
because
a
big
part
of
the
reason
we
didn't
go
shard
to
postgres
originally,
is
because
at
the
time
a
year
ago
the
state
has
charted
postgres
was
not
really
that
great
for
us
right
operationally
speaking.
A
However,
the
story
has
started
to
change
a
little
bit
in
the
last
year.
Now,
there's
like
this
PG
shard
extension
now
that
automates
a
lot
of
the
sharding
logic
that
you
would
have
to
do
as
part
of
sharding
postgres.
You
know
terms
of
like
making
sure
your
data
ends
up
on
the
right,
shard
moving
it
around.
All
of
that,
so
that's
also
another
possibility
is
that
we
might
reconsider
that.
A
B
A
A
I'm
actually
very
excited
about
both
of
those
things.
So
at
the
time
I
put
together
the
slide.
I
guess
I
hadn't
really
looked
at
like
what
was
coming
in
to
two
and
three
or
maybe
it
wasn't
announced
I,
don't
know,
I'd,
never
know
if
I'm,
just
not
paying
it
or
if
it's
not
been
announced,
but
yeah
I'm
excited
about
that
and
materialized
use.
Both
of
those
things
would
actually
simplify
a
lot
of
we're
currently
doing
and
would
attack
a
lot
of
bottlenecks
that
we
currently
have
that
exist
around.