►
From YouTube: Hailo: Building A Global Startup using Apache Cassandra
Description
Speaker: Dave Gardner, Software Developer at Hailo
Building a successful startup is hard work. This talk will explain how Hailo has succeeded by putting C* front and center - building an award-winning, hundred-million-dollar startup that operates on three continents. In this talk, I will cover our journey from MySQL to Cassandra, and the challenges involved in migrating from MySQL to Cassandra, from single data-center to multi-data center, and upgrading versions of Cassandra -- all with zero down time. I will talk about the myriad use cases to which we have applied C*, including simple entity storage, time series indexes, and realtime analytics using HyperLogLog. Finally I'll cover some of the operational and organizational challenges of running Cassandra.
A
A
Things
have
come
a
long
way
since
then
I
work
for
a
company
called
halo
and
they
have
an
app
that
is
similar
to
a
us-based
competitor,
you're,
probably
more
familiar
with
over
on
the
west
coast,
and
it's
basically
an
app
that
you
can
press
a
button
and
we'll
dispatch
a
car
to
come
pick.
You
up,
take
you
where
you
want
to
be,
and
then
you
can
jump
out
the
other
end
and
we'll
we'll
do
with
the
payment
and
send
you
a
receipt,
and
this
is.
A
This
is
a
little
bit
like
the
the
appletalk
now
so
I'm,
actually
launching
a
brand
new
feature
on
stage
right
now,
slightly
fortuitous
in
that
it
just
happened
to
launch
this
morning,
but
I
feel.
Like
you
know,
this
is
this
is
kind
of
almost
the
flint
center
now,
so
we
just
launched
a
brand
new
feature
where
you
can.
A
We've
just
announced
that
we're
launching
Singapore,
with
a
partnership
with
SMRT
and
we're
also
in
Osaka
and
Tokyo,
and
it's
kind
of
that
that
geographical
distribution
of
our
business
was
one
of
the
reasons
kind
of
the
drivers
into
using
Cassandra
the
other.
The
other
things
that
we
looked
at
when
when
adopting
Sandra
were
resilient.
So
so
we
wanted
to
be
able
to
have
great
availability
and
also
to
cover
future
growth,
but
we
halo-halo
isn't
a
company
that
are
kind
of
solving
big
data
problems,
particularly
you
know,
there's
a
lot
of
companies
around
he'll
talk
about.
A
You
know
millions
and
millions
of
data
points
and
terabytes
or
whatever.
We
don't
really
have
that
problem,
but
cassandra
has
got
such
a
good
set
of
features
that
it
actually
covers.
Quite
a
lot
of
use
cases
and
what
I'm
going
to
be
talking
about
is
really
kind
of
our
Cassandra
adoption
timeline
Howie
how
he
started
halo,
how
we
went
from
my
sequel
back
end
when
we
launched
and
kind
of
why
we
did
that
and
through
to
today,
where
we're
running
we're
running
systems
that
are
solely
backed
by
Cassandra.
A
Now
so,
with
my
grade,
everything
on
to
Cassandra
and
I'm,
going
to
talk
a
little
bit
about
how
we
use
Cassandra
the
different
use
cases
we
have
for
it
and
why
we
think
it's
good
and
I'm
also
going
to
talk
about
I'm,
going
to
try
and
sort
of
talk
about
the
the
general
problem
of
if
you're
starting
a
startup.
What
what
storage
back-end
should
you
choose?
A
So
this
tweet
floated
past
a
few
weeks
ago?
This
is
from
Al
Toby
who's
won
only
one
of
the
guys
who
works
at
datastax,
and
he
said
he
said:
dream
big
or
go
home.
Your
startup
will
end
up
with
an
AP
datastore.
So
why
not
start
out
with
three
nodes
of
cassandra
or
react?
And
this
got
me
thinking
because
I
guess
the
alternative
viewpoint
is
that
you
should
start
simple
with
something
like
postgres,
something
that
you
know.
A
It's
me
sort
of
more
immediately
familiar
and
more
basic,
but
not
basic
in
terms
of
functionality,
but
it's
basically
in
terms
of
its
not
distributed
just
runs
on
one
node
and
then
scale
out
as
and
when
you
need
to
so
I
was
thinking
about.
You
know
the
decision
for
halo,
so
we
started
with
mysql
and
when
the
company,
when
the
company
was
founded
in
2010,
cassandra
is
a
version
nor
point
6,
as
I've
alluded
to
already.
I
actually
used
Cassandra
naught
6
and
it
was.
It
was
reasonably
painful.
A
It
was
kind
of
drivers
to
start
a
meetup
group
about
it,
because
that's
that's
kind
of
how
difficult
it
was.
If
you
fast
forward
to
today,
I
think,
potentially
you
would
do
things
differently,
but
but
back
then
it
was
definitely
the
right
decision
to
go
with
MySQL
in
in
halo
and
so
so
this.
So
we
ended
up
with
a
kind
of
story
of
adoption
of
migration
from
from
MySQL
through
to
Cassandra,
and
while
we
were
migrating
the
database,
there
are
other
migrations
happening
within
the
company
as
well,
so
the
architecture
kind
of
changed
as
well.
A
At
the
same
time
that
we
changed
our
datastore.
So
this
is
what
we
started
off
with
a
halo.
This
was
kind
of
our
law
infrastructure
on
AWS.
We
ran
in
one
region
in
the
EU
region,
and
we
had
we
have
two
apps,
so
halo
works
by
having
two
apps
is
one
out
that
the
drivers
have
in
their
in
their
calves
as
they
drive
around
and
there's
another
app
that
the
passengers
have
to
hell
taxis.
A
We
had
two
applications
written
in
PHP,
that
kind
of
SAT
behind
those
apps
and
then
below.
It
was
a
sort
of
bigger
java
application
that
kind
of
did
some
of
the
heavy
lifting
and
all
of
these
things
talk
to
mysql
and
there
was
a
kind
of
sprinkling
of
Redis,
and
this
is
what
we
launched
with
it
had
kind
of
it
had
some
minimal
resiliency
in
that
we
had
a
mysql
multimaster,
so
we
could
lose
11.
Other
databases,
halo
is
a
company
that
kind
of
set
off
with
a
growth
story
we
wanted
to.
A
So
what
we
did
was
we
introduced.
We
got
rid
of
MySQL,
basically
from
the
kind
of
customer
appt
side,
so
the
customer
and
we
hollowed
out
all
the
functionality
that
that
component
of
the
application
stack
had
and
we
replaced
it
with
these
kind
of
broad
services.
So
here
we've
got
ya,
for
instance,
the
customer
service
stored
customer
records.
So
you
know
when
you
register
or
update
something
about
your
account
that
would
be
stored
in
there,
and
this
is
where
we
introduced
Cassandra.
A
First
of
all,
so
we
introduced
Cassandra's
the
backend
storage
mechanism
for
those
components,
and
this
was
kind
of
a
nice
pattern
because
we
could
run
you
know
on
that
side
of
the
picture.
We
could
run
more
than
one
of
everything
we
could
run
them
on
in
multiple
regions
and,
if
anything
died,
it
would
carry
on
working.
On
the
other
side
of
the
picture
on
the
left,
we're
sort
of
very
either
there's
a
green
stripe
that
you,
you
can't
really
see
brilliantly
on
the
on
the
slides,
but
we
basically
duplicated
what
we
had
in
London.
A
A
A
So
no
we
did
you
get
to
migrate,
all
of
the
kind
of
that
relational
and
worldview
into
a
kind
of
non-relational
worldview.
So
what
we
ended
up
with
was
something
looks
like
this.
This
is
kind
of
what
our
architecture
looks
like
today,
and
we've
taken
a
lot
of
inspiration
from
netflix,
and
they
can
really.
They
can
be
thanked
for
the
kind
of
showing
us
that
this
could
work
really
or
given
us
the
confidence
to
do
it.
A
So
we've
replaced
on
the
kind
of
broad
services
that
we'd
written,
which
were
HTTP
services,
and
instead,
we've
got
this
kind
of
micro
service
architecture,
so
we
write
services
in
go
or
Java
and
they
all
talk
to
each
other
via
rabbitmq
and
in
fact,
in
production.
We've
got
about
a
hundred
and
thirty
of
these
things.
A
Every
single
one,
limb
that
needs
to
store
stuff
will
use
cassandra.
Cassandra
is
kind
of
the
one
thing
that
joins
the
region's
together:
that's
how
data
gets
between
regions,
so
you
can
see
at
the
bottom
in
a
picture
there's
kind
of
an
hour
ago
and
left
and
right.
So
we
can
run
this
in
architecture
the
we
run
it
on
multiple
AWS
regions.
The
whole
thing
is
kind
of
fully
automated,
so
we
can
spin
them
up
and
down
on
demand.
A
Like
having
no
master
node,
not
you
know
not
being
able
to
kill
anything
and
it
will
carry
on
working
and
to
kind
of
give
you
an
idea
of
what
it
looks
like.
So
this
is
our.
This
is
a
very
washed
our
version
of
our
I
kind
of
network
diagram.
This
shows
how
all
of
130
services
are
kind
of
talking
to
each
other,
so
we
trace
the
traffic
as
it
goes
to
the
through
the
SOA
and
the
this
is
a
chord
diagram
that
kind
of
shows
how
the
services
talk
to
each
other
and
I.
A
Think
it
just
gives
you
a
kind
of
quick
glimpse
as
to
the
kind
of
scale
of
what
we've
ended
up
with
and
one
of
the
things
we
did
at
the
same
time
as
we
kind
of
applied
be
the
same
world
you
to
kind
of
how
we
do
testing.
So
you
know
again,
you
know
lots
of
automation,
lots
of
being
able
to
kill
things
of
them
carry
on
working
and
we
use
this.
We
have
a
kind
of
testing
set
up
there.
A
It's
kind
of
running
soak
test,
so
it's
it's
not
it's
pushing
through
in
a
continuous
level
of
medium
load
through
a
pretend
system.
So
we
have
a
bunch
of
pretend
drivers
who
drive
around
and
they
do
pretend
jobs
with
pretend
customers,
and
this
is
kind
of
a
map
showing
sort
of
20,000
I'm
just
round
into
a
very
small
square
in
a
city,
and
this
is
kind
of
how
we
can
assert
that
the
130
service
system
is
still
up
and
running.
A
So
behind.
All
of
this
is
cassandra.
Cassandra
is
the
the
data
back
end
for
a
better
fit
a
whole
system?
Now
and
we've
we've
ended
up
and
there's
some
been
most
of
migration
here
as
well,
and
what
we're
running
today
is
we're
running
two
clusters,
which
we've
kind
of
named
premium
and
economy.
The
premium
cost
to
cluster
is
running
on
SSDs
the
whole
thing's
on
amazon,
so
we're
completely
kind
of
cloud
native.
We
run
everything
on
amazon
and
the
economy.
Cluster
is
on
on
the
ephemerals.
A
But
the
reason
for
the
split
was
that
we
found
that,
as
we
migrating
more
and
more
stuff
onto
Cassandra,
we-
and
in
fact
one
of
the
one
of
the
kind
of
designs
of
having
this
this
SOAs
microservice
SOA
was
the
was
the
kind
of
idea
of
freedom
and
responsibility.
We
want
people
to
be
able
to
build
a
new
service
and
ship
it
to
production
quite
quickly,
and
ideally
that
won't
have
much
impact
on.
A
What's
there
already
and
what
we
were
finding
was
that
some
of
the
some
of
the
patterns
of
use
were
kind
of
conflicting
knew
each
other.
So
we
would
have
right
heavy
workloads
that
would
cause
lots
of
compaction
in
Cassandra,
and
that
would
impact
a
read
performance.
So
it
make
it
kind
of
lesson
less
less
flatline,
more
more,
more
more
variable.
So
what
we
did
was
we
split
it
out
and
the
premium
cluster
is
a
cluster
that
we
put
read
heavy
workloads
on
and
the
economy
class
or
is
very
put
right.
A
Heavy
workloads
and
the
way
it
also
generally
breaks
down
a
halo
is
that
the
the
right
heavy
workloads,
a
normally
kind
of
secondary
thing.
So
one
example
is
that,
when
you're
doing
a
job,
we
we
storm
the
drivers
location
every
five
seconds,
so
he's
peeing
as
his
location
and
we
put
that
into
Cassandra
in
a
row
so
that
we
can
do
things
like
drawn
out
for
customer
services
to
show
you
where
you've
been.
But
if
that
thing
doesn't
work,
it
doesn't
really
matter.
It's
not
going
to
stop
you
getting
in.
A
So
we're
using
the
graph
I
instrumentation
system-
and
this
is
a
graphene
which
is
kind
of
grabbing
on
the
graphs
together
for
us,
and
this
is
what
we're
using
to
monitor
our
clusters.
This
is,
this
is
kind
of
a
live
view
of
them
and
we
found
that
that's
one
of
the
major
advantages
we've
got
from
going
to
to
do.
A
So
as
we've
as
we've
been
migrating
all
of
the
things
over
there,
we
we
found
that
the
they've
generally
break
down
into
four
four
main
use
cases.
We've
got
entity
storage-
that's
like
this
is
kind
of
the
bread
and
butter
I
guess
of
most
apps,
it's
the
kind
of
boring
stuff
of
you
know.
I
am
craving
a
customer
update.
My
name
change
my
email.
What
you
know
jobs
it's
that
kind
of
stuff,
then
we've
got
time
series
data.
So
you
know
the
points
are
a
good
example.
A
The
drivers
sending
his
location
update
every
five
seconds
and
we're
storing
that
as
a
time
series
list
of
data
that
we
can
read
back
and
there's
a
lot
of
things
that
kind
of
fit
into
that.
That
kind
of
worldview
of
time
order
data.
Do
you
want
to
be
able
to
query
by
time,
then
the
search
which
for
halo
is
is
mainly
for
management
portals.
So
it's
not.
We
don't
need
that
to
work
to
get
you
a
taxi
and
then
finally,
there's
analytics,
which
is
you
know,
kind
of
how
many?
How
many
times
has
this
happened?
A
A
So
we've
got
the
ids
for
each
customer
are
being
generated
by
mysql
by
auto
increment.
The
index
constraints
are
automatically
being
checked
by
mysql,
so
it's
going
to
work
out
if
someone's
already
using
an
email
or,
if
someone's
already
using
a
phone
number
and
it's
kind
of
a
familiar,
it's
a
familiar
pattern.
You
know
it's
something
that
developers
are
familiar
with
when
you
move
to
Cassandra
some
of
this
stuff.
Some
some
of
this
stuff
gets
quite
a
bit
harder.
A
So
for
IDs
we've
we've
switched
to
a
system.
You
know
we
haven't,
got
my
school
or
two
increments
anymore,
so
we've
switched
to
a
system,
that's
like
snowflake.
So
snowflake
is
an
open
source
project
by
Twitter,
which
will
generate
you
guaranteed
unique
64-bit,
integer,
IDs
kind
of
in
a
distributed
setting.
A
What
we
do
is
like
the
canonical
example
of
why
he'd
written
this
feature.
So
it's
kind
of
the
flawed
application
level,
zookeeper,
locking
followed
by
you,
know,
checking
stuff
in
Cassandra
and
then
and
then
making
updates,
and
this
is
kind
of
still
how
we
do
it.
So
this
is.
This
is
not
kind
of
not
the
way
to
do
it.
Basically
and
the
reason
it's
not
the
way
to
do
it
is
because
there's
there's
lots
of
kind
of
edge
cases.
You
know
you're
kind
of
trying
to
get
some
strong
consistency
out
of
an
eventually
consistent
database.
A
So
the
big
problem
is
if,
if
something
goes
wrong,
so
you
know,
if
your
query
fails
it,
what
it
normally
means
is,
it
may
actually
have
worked.
It
means
that
it
means
that
Cassandra
doesn't
know
whether
it's
finished,
so
it
might
mean
that
it,
you
know,
if
you
ask
for
a
local
quorum
or
a
quorum
read,
it
means
it
sent
it
to
some
replicas,
but
it
didn't
get
an
answer
from
enough
replicas
to
consider
the
whole
operation
of
success.
But,
crucially,
you
don't
really
know
whether
it
worked
or
not.
A
It
might
be
SAT
there
in
on
some
of
the
boxes
and
eventually
make
it
out.
So
the
kind
of
ideal
solution
is
the
lightweight
transactions
that
are
introduced
into
version
2,
which
you
can
which
you
can
do
via
cql.
Now,
we've
only
just
we've,
only
just
a
halo
migrated
to
version
two.
We
don't
think
we
did
it
about
two
weeks
ago
before
that
we
were
running
version
11,
something
so
we've
only
just
got
the
ability
to
do
this
and
as
yeah
we're
still
we're
still
kind
of
in
the
thrift
world.
A
I
still
have
a
little
soft
spot
for
thrift,
but
but
yeah
we
need
to
we'd
need
to
migrate.
Cql
to
make
use
of
this
update
is
relatively
straightforward,
I
mean
this
is
how
we
would
have
done
it
in
our
first
first
infrastructure
and
c
ql
or
thrift
really
are
the
same.
You
know
you
can
with
Cassandra.
The
only
rule
really
for
updating
stuff
is
that
you
want
to
just
change
the
columns
that
you're
changing.
A
I
look
now,
but
these
are
a
couple
of
examples.
So
in
our
app
you
can
look
at
your
trip
history.
This
is
actually
my
tree
of
history
and
then
on
the
portal.
You
can
see
that's
kind
of
a
list
of
all
jobs.
This
is
from
our
automated
QA
environment.
So
we
we
have
a
system
that
pushes
a
jobs
through
in
production,
every
30
seconds
to
verify
the
whole
system
is
working
and
it
runs
on
heard
island,
which
is
in
the
middle
of
the
Atlantic
middle
of
Pacific.
A
One
of
the
things
is,
you
need
to
put
the
data
in
the
way
you
want
to
get
it
back.
So
what
we've
ended
up
with
is
a
system
where
we've
kind
of
built,
a
library
that
kind
of
handles
the
guts
of
storing
stuff
as
a
time
series
in
Cassandra,
and
this
is
go
and
and
the
way
it
basically
works
is
that
you
define
a
time
series
and
say:
hey
I,
want
to
store
this
entity
and
I
want
the
rose
to
be
kind
of
30
days.
Big
or
you
know,
or
seven
days,
big
or
one
day.
A
A
So
if
a
customer
say
does
one
drop
a
month
and
you
want
to
be
able
to
read
back
three
months
worth
of
jobs
but
say
that
you've
busted
it
so
that
each
each
row
is
one
day
you
don't
really
want
to
have
to
go
and
look
for
90
rose,
you
kind
of
just
want
to
be
able
to
look.
You
just
want
to
know,
there's
only
three
rows,
so
we've
kind
of
build
that
in
the
ideal
solution
is
probably
against
cql.
A
You
know
tell
me
about
your
data
structure,
whereas
the
way
we
do
is
be
more
opaque,
it's
just
blobs
of
stuff,
so
you
have
to
go
and
look
at
the
code.
So
I
think
probably
this
is
the
ideal
solution.
The
only
doubt,
the
only
downside
really
potentially
is
that
I
mean
the
example
at
apsara
blacked
kind
of
row
partitioning
by
time,
so
you
might
have
to
put
that
back
in.
So
if
you
look,
the
partition
key
here
in
this
example
is
the
weather
station
ID.
A
A
The
key
thing
for
halo
is
that
search?
Isn't
that
important?
Really
it's
not
in
the
critical
path
we
we
do
asynchronous
indexing
of
things
and
then
the
search
is
really
for
customer
portals
and
and
stuff
like
that.
So
it's
not
super
important.
This
is
kind
of
how
it
would
have
looked
in
version
one.
A
So
again
we
just
looking
at
the
data
in
the
database
yeah
and
what
we've
moved
over
to
is:
we've
used
elasticsearch
now,
I,
don't
know
what
anyone
reads:
the
the
kind
of
the
gypsum
series
of
posts,
but
elasticsearch
was
one
of
the
things
investigated.
So
we
we
kind
of
follow
the
we
basically
treat
it
as
a
secondary
index.
We
don't
store
it's
not
the
canonical
source
of
any
data
anytime.
We
can
throw
the
whole
thing
away
and
I'm
rebuild
it.
A
The
key
thing
is
that
a
primary
data
store
is
still
cassandra,
so
so
we
we
feel
that
the
data
is
kind
of
safe,
as
opposed
to
put
in
elastic
search,
analytics.
I
think
this
is
still
the
hardest
thing
to
get
right.
I,
don't
know
what
anyone
else
tried
to
go
to
the
Apache
spark
talk.
That
was
in
the
kind
of
smallest
room
that
you
could
ever
imagine.
A
Maybe
that
kind
of
shows.
That
is
something
that
people
have
still
got
a
lot
of
interesting
in
the
in
the
community.
It
was
heaving,
so
this
is
an
example
of
halo.
Again,
this
is
our
QA
environment,
so
the
lines
are
quite
kind
of
flat
because
we
just
pushed
through
jobs
every
30
seconds
to
make
sure
things
are
working,
but
the
kind
of
thing
we
want
to
be
able
to
do
is
say:
hey.
You
know
how
many,
how
many
people
are,
how
many
drivers
are
around
right
now.
A
You
know
how
many
drivers
are
in
being
on
shift
in
the
last
hour.
You
know
we
want
to
be
able
to
do
things
like
count.
The
number
of
distinct
customers
who've
tried
to
book
a
job
in
the
last
hour,
so
the
to
rule
use
cases
here
are
count
and
Candice
ting
to
big
things
we
wanted
to
do
and
what
we've
done
is.
This
is
kind
of
how
it
would
have
looked
in
the
olden
days.
This
is
this
is
I
actually
Douglas
out
or
I'll
source
code.
A
A
What
we've
ended
up
moving
to
is
a
system
where
we've
kind
of
decoupled
the
analytics
pipeline
from
the
rest
of
the
application.
So
what
we've
moved
to
now
is
we've
on
the
left-hand
side.
You've
got
these
services,
so
you
can
imagine
there
130
services
to
make
up
halo
and
they're
all
doing
little
jobs,
each
one's
doing
a
different.
You
know
tiny
job
and
what
happens
is
as
they
as
they
do
their
staff.
They
h
out
event
so
they're
emitting
events.
A
These
events
will
go
on
to
the
the
fire
hose
which
we
use
n
SQ,
for
which
is
a
go
project.
Sort
of
kafka
would
be
another
good
choice
for
that
for
that
component
and
all
these
events
kind
of
flow
down,
and
then
we
have
another.
You
know
we
have
other
services,
basically
that
scrape
the
events
out
there
and
do
stuff
with
them.
So
the
most
basic
version
of
this
is
that
we
get
every
event
and
put
on
s3
that's
kind
of
the
most
basic
versions.
That's
that
kind
of
archive.
A
That's
our
raw
data
are
potentially,
but
we
also
do
other
think
so
so
for
surfer
kind
of
bi,
we
put
it
all
into
redshift,
which
is
an
amazon
service
which
are
from
what
I
can
understand
it
just
like
a
massive
postgres
thing
that
they've
somehow
made
work,
so
we
load
it
all
into
redshift
and
actually
that
was
kind
of
a
big
change.
Like
last
last
summit,
I
did
a
similar
talk
and
I
would
be.
A
I
was
talking
about
how
we
at
that
time
we
were
trying
to
store
all
these
events
in
Cassandra
and
what
was
happening
was
we
was
we
were
storing
morning
Sandra
and
they
weren't
all
I
useful,
Inc
Sandra.
At
that
time.
You
know
we
didn't
really
have
any
way
of
using
them,
particularly
so
we
would
write
them
in
and
they
would
take
up
an
enormous
amount
of
space
cost
a
little
money
and
he
didn't
really
achieve
anything.
So
we've
moved
to
this
system
of
pin
or
minus
three,
which
means
we
can.
A
You
know
potentially
running
a
mile
jobs
if
we
need
to
we
put
them
in
red
shift,
which
kind
of
makes
them
useful
and
then
the
hob
activity
services.
The
thing
that
that
draws,
those
graphs
that
I
showed
you
on
the
previous
page,
so
really
lots
of
different
things
can
do
whatever
they
want
with
these
events.
So
it's
quite
a
nice
pattern,
and
this
is
this
is
one
example.
So
this
is
a
service
that
we
wrote
to
to
kind
of
do
real-time
analytics
in
memory
in
memory
and
kind
of
roll
ups.
A
So
what
it
does
is
it's
basically
a
bit
like
stats
d,
which
is
the
thing
you
put
in
front
of
graphite.
So
it
listens
to
the
events
off
of
the
fire
hose.
Then
it
rolls
them
up
in
memory
into
a
data
structure
which
is
like
a
call
hyper
log
log,
which
is
a
probabilistic
data
structure
for
counting
distinct
things
and
the
interesting
property
of
it
is
that
you
can
merge
two
of
them
together.
A
So
if
you've
got
two
of
these
data
structures
with
different
things
in
you
can
kind
of
merge
them
and
then
get
the
total
count
distinct.
So
what
we
do
is
we
read
all
the
event
put
them
into
these
data
structures
and
then
black
the
whole
thing
into
Cassandra's
like
a
blob
of
binary.
So
this
is
kind
of
where
thrift
works
well,
because
you
can
just
put
behind
into
cassandra
and
then,
when
you
read
it
back,
you
you
can
you
know
you?
Can
you
can
kind
of
group
together
column?
A
So
you
know
we
flushed
to
disk
every
30
seconds,
I,
think
and
if
you've
got
say
two
or
three
different
machines.
They're
all
doing
this
one
job,
because
we
run
in
an
environment,
that's
distributed,
then
that's
great
because
it
doesn't
matter
when
you
read
them
back.
You
just
join
them
all
together
and
in
return,
the
results.
A
The
only
possible
downside
is
that
can
make
lots
of
column.
So
if
you,
if
you,
if
you
know
if
you're,
if
you
wanted
to
say,
show
me
like
you-
know
three
months
worth
of
data,
this
system,
one
really
work
and
in
fact
we
only
use
if
we
only
use
it
for
seven
days.
So
we
we
used
cassandra
TTL
to
flush
the
columns
out
of
cassandra
after
seven
days,
and
it
works
quite
nicely
for
that.
A
I
think
the
ideal
solution
here,
really
is
just
the
nicest
thing
probably
we
ever
did
was
to
split
the
two
split
out
that
events
fire
hose
and
kind
of
D
couple.
Our
analytics
work
flow
from
the
rest
of
the
application,
because
it
means
that
people
can
use
the
data
in
different
ways,
and
this
is
kind
of
one.
Other
example
of
this
is
effectively
real-time
analytics
as
well.
So
someone
someone
in
the
company
went
away
and
built
this
guy
called
Steven.
A
This
is
all
in
memory,
so
there's
actually
nothing
to
do
X
and
row,
but
and
then
what
you
can
do
it
is.
You
can
basically
say
right
how
fast
is
traffic
moving
in
central
London
or
wherever--wherever
operating?
This
is
kind
of
a
view
of
this
running
in
one
of
our
test
environments,
and
the
interesting
thing
about
this
was
that
he
could.
He
could
build
this
service
completely
independently,
because
the
because
the
analytics
layer
is
decoupled
and
then
he
could
use
this
data
to
enhance
our
estimate
EAS.
A
So
when
you
say
hey,
how
long
is
it
going
to
take
to
get
me
a
cab?
We
can
go
well.
Actually,
it's
good,
you
know
traffic's
not
moving
in
London,
so
it's
going
to
take
a
little
bit
longer
and
that
makes
our
service
better
okay.
So
to
finish
up
I'm
just
going
to
talk
about
a
few
of
the
kind
of
operation
on
organizational
challenges
we
had
in
this
process
of
going
from
MySQL
to
Cassandra.
A
This
is
kind
of
a
view
of
kind
of
how
Cassandra's,
as
morph
term,
in
Halo
over
the
years,
so
we
started
off
with
them.
Sandra
version
1.0
point
nine
and
we
were
using
PHP
casa
with
thrift
and
we
were
only
in
one
region,
EU
west,
and
what
we've
been
able
to
do
with
Cassandra.
This
is
this
is
kind
of
one
of
the
stories.
This
is
one
of
the
things
that
I
think's
best
about
Cassandra,
so
we
were
able
to
expand
into
the
u.s.
with
no
downtime.
A
So
we
could
fire
up
an
entirely
new
cluster
in
the
u.s.
join
the
two
clusters
together,
get
the
data
replicating
between
them
without
any
downtime,
and
then
eventually
we
expanded
into
into
Asia
as
well.
So
we
fired
up
a
data
center
in
AP
northeast
one
and
the
same
thing.
No
downtime
and
then
eventually
we
I
think
we
shot
that
one
down,
so
he
went
back
to
two,
so
you
can
kind
of
up
and
down
scale
and
and
cassandra
is
able
to
cope
it
all
that
we
we
tried
out
different
clients,
so
we
use
it.
A
A
Over
and
again,
we
managed
to
do
that
with
no
downtime
one
of
the
main
things
we've
done
since
last
year
and
I
did
similar
similar
similar
ish
to
last
year
and,
at
the
end
of
our
said,
we
were
going
to
try
and
hire
someone
in
New
about
Cassandra,
which
at
the
time
was
very,
very
difficult
to
find
anyone
who'd
come
to
our
organization
with
any
prior
knowledge
of
Cassandra,
but
we
did
manage
to
hire
someone.
So
that's
kind
of
made
life
easier
for
the
rest
of
us.
A
This
was
the
this
was
the
go
client
that
we
picked
so
when
we,
when
we
started
using
go
we
wanted.
We
wanted
an
easy
upgrade
path
from
what
we
were
already
doing
with
Cassandra
in
with
thrift,
so
he
picked
the
client
that
was
thrift
based
and
then
the
guy
who
who
built
it
kind
of
announced
they'd
done
with
it.
He
didn't
want
anything
to
do
it
anymore
and
he
did
have
some
significant
bugs
and
actually
as
well.
So
we
had
to
fix
all
those.
A
Luckily
I
put
this
slide
in
this
morning.
So
this
morning,
Matt
stump
from
datastax
release
this
a
brand
new
driver
for
go
which
is
kind
of
like
wraps
the
data
stack
c++
driver.
So
I
think
I've
been
doing
talks
on
sounder
about
four
years
and
pretty
much
every
talk
would
be
mon.
The
kind
of
state
of
the
drivers
in
the
you
know
in
the
community
they're
all
rubbish
basically,
but
datastax
have
basically
fixing
that
now.
So
that's
really
awesome.
A
One
of
the
other
challenges
is,
is
an
it's
kind
of
a
the
process
of
Education
and
getting
people
into
the
into
the
mindset
of
Cassandra
and
how
it
works,
and
there's
really
two
sides
to
that.
I
see
one
of
them
is,
is
people
are
terrified
of
D
normalizing
people
really
really
don't
like
it.
So
we
had
we
had.
A
We
have
people
who
will
who
will
create
a
column
family
and
then
add
like
thousands
of
secondary
indexes,
because
then
it's
a
bit
like
MySQL,
but
it
doesn't
work
at
all,
and
you
know
Jonathan
alluded
to
that
when
he
said
oh
yeah,
you
could
use
his
fellow
volume
and
yeah.
They
do
actually
really
mean
that
if
you
actually
want
to
be
able
to
query
stuff,
you
need
to
write
the
data
more
than
once,
and
so
you
can
read
it
back,
and
this
was
another
problem
we
had
where
we
had.
A
We
were
writing
in
time
series
data
and
we'd
bucketed
the
data
by
day.
So
every
single
row
represented
one
day's
worth
of
stuff
and,
as
we
had
more
stuff,
so
it
grew
until
we
ended
up.
We
had
actually
we
actually
our
immortan
come
in
and
give
us
some
consultancy.
This
is
a
while
ago
now
and
he
pointed
out
that
one
of
our
roads
was
sixteen
point,
six
gigabytes
and,
and
it
would,
the
problem
of
this
is
that
that
this
was
kind
of
when
we
were
trying
to
store
all
the
raw
events
in
Cassandra.
A
And
what
we'd
find
is
that
every
now
and
again
a
driver
would
turn
on
a
phone
that
it
had
off
for
a
couple
of
weeks
and
they'd.
Send
us
like
10
events
and
then
send
us
ten
events
from
sort
to
weeks
ago,
and
what
happened
is
we
have
to
ten
new
events
would
need
to
be
merged
into
a
row
that
was
kind
of
you
know
ten
gigabytes
wide
and
then
so
they're.
Basically,
the
entire
cluster
spent
his
entire
life
compacting
everything
because
he
was
trying
to
kind
of
squash
together.
A
A
Another
interesting
problem
we
had
was
we
used
to
earn.
We
were
running
DSC,
the
not
DSC,
we're
running
the
monitoring
thing
that
the
name
this
gave
to
me
off
center
that
one
yeah
and
it
actually
stores
all
the
data
for
the
list-
operators
it
all
in
Cassandra.
So
it's
kind
of
like
the
thing
that
you're
monitoring
is
is
also
basically,
you
know
adding
to
your
cluster
load,
and
we
got
to
the
point
where
actually
we
realized
that
the
main
the
main
traffic
in
our
entire
cluster
was
op
center.
A
So
I
think
it
was
like
you
know.
Ninety
nine
percent
of
all
our
traffic
was
off
center
and
I
think
it
was
just
because
we
have
a
like
a
lot
of
key
spaces
it
because,
because
of
the
Micra
service
architecture,
people
basically
have
to
have
their
own
key
space,
that's
kind
of
way,
we've
gone,
which
other
well,
that's
a
good
idea
or
an
optic.
A
You
know
I'm
so
having
an
RA,
but
we
probably
have
more
than
we
should,
and
so
anyway,
we
built
we,
luckily
the
220,
think
I
now
and
we
could
instrumented
better
and
we
put
it
into
graphite
and
we
built
this
tool,
which
is
called
see
top.
So
it's
like
top
you
just
running
on
one
of
the
nodes,
and
it
tells
you
what's
going
on,
and
this
is
kind
of
useful.
A
So
you
know
that's
one
allele
of
operation
challenges
is
kind
of
understanding
how
this
thing's
running
and
having
enough
knowledge
to
be
able
to
diagnose
problems
and
fix
them.
Luckily,
with
Cassandra,
it's
getting
to
the
point
now
where
for
us
certainly
has
really
gone
wrong,
I
think
if
you,
if
you're
at
the
absolute
bleeding
edge,
then
you
probably
get
your
fingers
burned
a
bit,
but
we're
kind
of
safely.
A
Just
behind
that,
let
other
people
fix
all
the
bugs,
which
is
a
nice
place
to
be,
and
then
probably
the
biggest
challenge
of
all
was
was
kind
of
the
job
of
the
the
operational
job
of
gains
people's
minds
into
the
idea
of
using
a
database
that
wasn't
relational
and,
at
the
same
time,
we've
kind
of
moved
from
this
monolithic
db2
a
distributed.
Db
we've
also
moved
from
the
monolithic
app
to
the
distributed.
Apps
we've
got
the
my
craft
services
architecture
and
and
we've
also
moved
from
one
Olympic
team
to
distributed
team.
A
So
what
I
mean
by
that
is
that
when
I
joined
the
company
we
you
know
we
had
a
back-end
engineering
team.
We
all
pretty
much
words
together.
The
way
we
operate
now
is
we
have
lots
of
small,
independent
self-organizing
teams
that
are
kind
of
more
cross-functional,
so
kind
of
a
more
typical
scrum
thing.
A
So
you
know
the
interesting
thing
that
I
think
is
that
all
the
things
on
the
right
kind
of
go
very
well
together.
So
having
the
independent
self-organizing
teams,
with
a
big
monolithic,
app
I
think
would
be
quite
challenging
and
before
we
migrated,
we
did
have
some
challenges
there,
with
people
stepping
on
each
other's
toes
and
and
trying
to
coordinate
changes.
You
know
if
you've
got
seven
teams
going
to
change
one
project.
Basically,
it's
quiet,
you
need
a
lot
of
communication,
so
the
distributed
team
works
really
well
with
a
distributed
app.
A
So
each
team
can
kind
of
work
on
their
little
tiny
subset
of
services,
and
sometimes
people
need
to
change
the
same
service,
but
it
reduces
the
communication
and
and
and
some
of
the
friction
and
the
process
and
then
the
distributed
app
works
really
well
with
a
distributed
DB
as
well,
so
wrapping
up
last
last
minute
or
two,
so
this
is
kind
of
where
we
started
and
and
I
think.
If
we,
if
we
started
halo
today,
I
think
that
we
would
go
with
C
with
Cassandra
and
the
factors
in
that
decision.
A
You
know
what
one
of
the
main
factors
would
be.
That
halo
has
started
with
a
view
to
being
on
multiple
continents
from
day
one
that
was
always
kind
of
a
require
and
Cassandra
just
makes
that
job.
So
easy
and
I
think
it
would
have
been
wrong
to
choose
Cassandra
version
or
point
6,
because
it
would
have
been
too
painful
to
make
work.
But
with
two
point:
oh
it's
it's.
A
Actually
it
just
works
and
then
finally,
this
is
something
that
I
think
we've
come
to
realize,
a
halo
which
is
that
Cassandra
and
microservices
go
together
really
well,
because
the
one
thing
that
you
you
can't
do
with
Cassandra
well
I'm,
not
the
one
thing.
One
of
the
things
you
can't
do
with
Sandra
is:
is
you
can't
and
you
can't
join
so
people
come
from
that
view
of?
Let
you
know
I'm
going
to
you
know:
I'm,
writing
some
feature
I'm
going
to
join
all
of
the
things
together
and
then
create
this
endpoint
and
with
Cassandra.