►
Description
Speaker: Julien Anguenot, VP Software Engineering at iland
iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information.
In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc.
Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.
A
A
Good
afternoon,
thank
you
for
joining
today,
so
my
name
is
Julian.
Angry
nut
and
I
will
be
telling
you
about
Island
and
I'll.
Let
you
know
about
the
company
or
a
bit
after,
but
basically
we
are
putting
customer
into
action
in
a
multi
data
center
deployment
and
I'd
like
to
show
you
a
bit
and
a
little
bit
about
it.
So
we
will
start
by
checking
a
little
bit
what
we
do
and
the
story
behind
the
use
of
Kazan
work,
because
I
think
it's
pretty
important
that
you
understand
why
we
actually
choose
casts
on
one.
A
What
we're
doing
with
it,
then
we'll
take
a
look
at
the
data
and
all
the
constraints
that
we're
having
a
wrong
those
use
of
the
data
as
a
cloud
provider
and
we'd
be
focusing
mostly
on
the
analytics
data
in
the
in
that.
In
that
part,
then
we
take
a
look
at
our
deployment.
The
configuration
give
you
some.
You
know
hard
wisconsin
consideration
and
enough
in
architecture
overview
of
the
wall,
application
in
a
multicenter
deployment,
then
we'll
take
a
look
at
a
couple
of
lessons
around
her
own
configuring
Cassandra.
A
The
problem
we've
had
the
real
things
to
you
that
that
we
found
where
we're
important
and
then
some
future
platform
extensions,
namely
a
spark
and
Kafka
and
you'll,
see
why
it
makes
sense
for
us
to
actually
get
there
at
this
point.
So
let
me
let
me
start
by
telling
you
a
little
bit
more
about
island
cloud,
because
you
probably
don't
know
us,
you
guys,
probably
use
a
sure
or
on-premises
iws
a
word
cloud
provider,
and
what
we
do
is
a
public
public
cloud
and
private
cloud.
A
We
are
when
known
for
our
disaster
disaster
recovery
as
a
service
offering,
and
we
are
cisco
certified
with
vmware
partner
for
about
seven
years
and
we've
been
20
years
in
business,
had
quoted
in
houston
texas.
So
why
would
you
actually
host
with
us
and
not
with
a
zero?
You
heard
a
speech
this
morning
we're
so
we
are
much
more.
So.
A
The
big
reason
why
you
would
want
to
us
is
because,
if
you're
looking
for
custom
SLA
because
you'll
not
like
one
of
the
top
like
20
biggest
company
in
the
world,
we
can
provide
like
costa
mesa
day.
We
can.
We
have
compliance
data
center.
So
we
can
help
you
being
compliant
and
we
guarantee,
on
our
actual
date,
on
your
actual
data
and
order
process
around
it.
We
have
a
global
data
center
footprints,
we're
in
us.
We
have
suite
at
the
center
in
the
u.s.
A
two
in
the
UK
and
one
in
Asia,
and
another
thing
is
our
platform
and
our
web
console
management,
which
is
the
reason
why
I'm
here
is
because
I'm
going
to
show
you
like
how
its
bothered
and
and
of
cooker
sore
eyes
d
is
borrowing
that
so,
let's,
let's
take
a
look
at
a
platform.
What
are
what
are
we
been
building
our
island?
A
We
do
basically
billing
for
billing,
during
correlation,
so
you'll
be
able
to
see
like
real-time
bidding,
matching
your
your
usage
and
some
predictive
analytics.
That
will
tell
you,
for
instance,
that
you
know
the
fifteenth
of
the
more
so
at
the
end
of
the
month.
You
will
be
your
cloud
will
cost
you,
like
example
of
dollars,
so
the
web
console.
Let's
take
a
look
at
it.
I
think
it's
going
to
speak
for
itself,
so
you
will
find
here
and
pretty
traditional
web
management
console
for
the
cloud.
A
You
have
alerts
and
that's
a
big
deal,
so
you
can
actually
be
alerted
when
you
of
you,
your
vm
is
using
more
resources
than
you
expected,
or
if
it's
costing
you
more
money
than
you
were
anticipating,
initially
plus
other
type
of
errors,
but
that
would
be
the
domain
ones.
Then
you
have
all
kind
of
reporting
available.
You
can
get
you
know.
A
Even
history
are
on
your
environment,
we're
doing
like
vulnerability
scanning
and
tomorrow
we
are
scanning
all
kind
of
different
cool
stuff
and
you
you're
able
to
actually
pull
reports
that
will
help
you
undergo
your
potential
compliance,
certifications
or
compliance
with
it.
Then
you
have
words
like
typically.
This
is
one
feature
that
I've
seen
announced
by
Google
platform
last
week
is
the
actual
ability
to
set
you
know
some
threshold
and
get
arrested
on
Yemen
of
money,
you're
spending
on
York
Road
in
on
a
on
a
particular
period
of
time.
A
We
have,
of
course,
on
a
cpu
memory,
memory
balloon,
except
what
all
the
commands
I'll
show
you
that
it
will
be
later
and
you
may
be
wondering
so
why
do
we
built
all
that
right?
Well,
so
we
were
using.
We
were
one
of
this
member
of
this
club
provider
using
a
VMware,
vCloud
director
and
week
later,
also
walking
right.
It
has
some
some
some
some
limitation,
for
instance,
the
performance
analytics
is
something
that
you
will
not
find
inside
that
software.
Typically,
this
is
something
that
you
have
when
you
use
vSphere
or
in
premises
deployment.
A
So
let's
take
a
look
at
the
data
for
the
analytic
spot,
so
we
start
with
the
constraints.
So
what
were
the
constraint
here?
So
the
consequent
we're?
Basically,
write
latency
in
highlight
and
sheer
violence,
basically
mutti
data
centers,
so
think
about.
You
know:
100,000
1000,
thousands
of
vm
like
running
and
the
particularities
that
we
are
not
forcing
our
customer
to
install
agent
the
basic.
I
have
to
pull
the
information
and
write
it
and
make
sure
that
you
know
it
comes
in
real
time
and
it
gets
aggregated,
etc.
So
we
talked
about
that.
A
It
has
to
be
available
and-
and
the
roku
is
pretty
high,
especially
when
you
see
the
amount
of
data
data
data
points
and
the
performance
counters
with
collecting
on
a
per
diem
basis.
So
that
was
basically
the
initial
constraints
for
genetics
part.
So
what
we
do
with
those
data
we
basically
collect
the
real
time
data
on
the
vm
and
again
no
agent.
We
do
that
transparent
here
without
performance
cause
for
the
the
customer
or
any
other
two
with
an
agent
you
to
be
installed.
A
We
stole
them,
we
aggregate
them
and
then
we
correlate
them,
meaning
that
castle
one
not
only
have
design
latex
data,
it
has
your
configuration.
It
has
the
task
that
you've
been
a
Cushing
on
your
environment.
It
has
no
configuration
user
permissions,
everything
everything
in
there.
Then
we
are
to
war
up,
because
when
you
have
the
real-time
data
you
will
want
to
see,
you
know
on
a
yearly
basis,
was
the
tran.
A
So
we
have
a
one
minute
one
hour
one
week,
one
month's
roll-ups,
the
happening
I'll,
explain
a
bit
more
about
that
and
then
the
important
passage
the
processing
processing
is
about
basically
getting
alerted
when
something's
going
on
that
that's
user
defined
and
actually
have
some
real
time
feedback
on
what
what
your
cost
is
costing
you,
your
cloud
is
costing
you
and
then
you're
reporting
on
a
crime.
So
let's
take
a
look
at
those
real
time:
performance
samples,
so
they're,
20
seconds
into
our
this-
is
the
smallest.
A
We
can
get
from
the
underlying
management
software,
which
is
vmware
vsphere.
This
is
the
smallest
we
can
we
can
get
so
we
have
like
three
data
points
for
purple
punta
per
minute.
We
collect
that
for
all
the
compute
storage
and
network
piece
of
the
of
our
cloud,
and
we
have
a
sorry.
We
have
our
own,
like
15
performance,
const
Iran,
a
vm
on
peruvian
basis.
A
So
it's
on,
like
50
data
points
per
minute,
for
a
particular
vm
they're,
basically
time
series
very
simple
stuff,
I'll
show
you
like
the
actual
customer
table
after
that,
and
they
have
couple
of
we
have
couple
of
metadata:
do
you
need
Intel
or
exit
rod,
those
the
real
time
one?
We
keep
them
for
a
year.
So,
on
a
one-year
period
of
time
you
are
able
to
get
20
seconds
interval
performances
on
you
infrastructure
on
our
cloud.
So
let's
take
a
look.
Just
quickly.
Add
one
of
these.
These
actual
path
path,
samples.
A
So
if
you
check
they
come
they
identified
by
33
different
value.
So
you
got
the
group
that
will
be
CPU
memory,
disk
or
network.
Then
you
have
the
name
of
it.
That
will
be
the
usage
like
percentage,
the
CP
already
uses
MacGyver's
and
then
the
type
of
product
that
they
are
doing
on
the
run,
so
average
summation
latest
etc.
So
I
put
on
this
slide,
like
the
actual
full
list
of
contours,
we're
getting
on
a
vm
on
a
per
diem
basis.
A
If
you
want
to
see
after
this
this
tube,
you
can
get
the
slide
online
and
see
for
yourself.
We
have
liked
more
contests,
not
gonna,
get
into
that
that
are
coming
from
so
party
services.
Typically
a
disaster
recovery
solutions.
You
will
want
to
see
like
how,
like
the
performances
of
the
soul,
patches
services
are
doing
excess
fat,
so
we
have
more
than
that.
So
then,
let's
take
a
look
at
the
time
series.
A
What
we
do,
then,
is
that
we're
binding
the
actual
sorry
you
ID
is
very
simple-
is
managed
at
the
application
level
we
can
identify
the
actual
sorry
by
constructing
uid,
based
on
the
actual
performance,
simple
information
that
I
saw
that
I
suppose
that
I
show
you
page
previously,
and
the
theory
itself
is
very
easy.
You
have
a
new
ID
that
you
can
match
with
the
actual
performance,
samples
and
and
a
primary
key
does
that.
A
A
So
if
that
was
only
what
we
are
doing,
that
that
that
would
be
really
really
not
something
simple.
This
is,
but
this
is
something
that
the
castle
was
aware
or
wait
for
us.
So
there's
not
really
any
big
challenges
around
that
now.
The
difference
in
our
bidding
model
is
that
we
don't
price
our
customer
on
a
per
instance
basis.
A
We
price
on
a
pair
resource-poor
basis,
meaning
that
you
don't
buy
one
vm
to
vm
3pm
and
your
annual
inverse
against
that
you're
going
to
buy
into
a
hollow
gigabyte
of
ram,
20,
gr,
cpu
and
two
terabyte
of
storage,
for
instance,
and
then,
with
that
with
that
we
saw
spool,
you
can
create
how
many
VMS
you
have
that
your
resource
pool
can
accommodate.
So
if
we
take
a
look
on
the
left,
you
have
basically
like
a
kind
of
hierarchy
and
at
the
top
level,
is
basically
you
I'm
Jared
months.
A
Then
you
have
the
resource
pool
that
you
have.
Let's
say
it
has
again
a
certain
amount
of
ram,
cpu
and
storage
below
that
you
have
a
group
that
in
vmware
they
called
V
app.
It's
a
kind
of
see
that
as
a
kind
of
folder
that
contains
vm
and
a
Hospitaller
type
of
property,
one
of
those
properties
would
be
the
some
particular
network
configuration
but
I'm
not
gonna,
get
into
details
into
that.
A
Right-
and
this
is
this
is
basically
the
little
challenge
with
that
is
that
you
have
to
go
through
those
steps
of
aggregation
before
even
thinking
about
doing
historical
role,
herbs
and
all
that,
of
course,
it's
separated
tables
is
basically
following
the
same
well
as
you
saw
before,
but
just
you
need
this
operation
to
actually
I
get
the
data,
so
this
toy
can
roll
up
I
mentioned
them
earlier.
Those
we
keep
them
a
little
bit
longer,
especially
you
one
hour.
You
want
our
samples
because
they'll
use
for
the
billing
and
they
actually
necessary
for
compliance
reason.
A
So
usually
we
keep
them
in
between
three
or
four
years
right.
All
that
all
that
is
that
application
at
the
application
level,
so
the
one
minute
forever,
as
you
can
imagine,
this
is
how
we
able
to
alert
on
a
one
minute
interval
if
something
goes
wrong
with
you
and
faster
to
you
or
if
you're,
you
know
using
too
much
on
something
that
you've
been
defining.
Basically,
all
that
we
are
detecting
is
actually
going
wrong
in
joint
pass
voucher,
and
this
is
all
real
time
and
the
one
our
laps.
A
They
are
basically
used
for
final
billion
computation,
and
this
is
this
is
what
we
use
to
charge
charge
back
the
customer
and
it's
basically
using
Salesforce.
So
there's
a
challenge
here
where
we
have
to
actually
always
sync
up
the
latest
like
contract
information
in
service,
for
that
are
basically
moving
because
again,
remember.
We
have
a
customer
surveys,
so
we
don't
have
one
single
customer
having
the
same
rate
on
a
particular
vm,
so
that
adds
a
little
bit
in
complexity.
A
A
So
our
both
based
foundation
for
cloud
is
cisco,
ucs,
hard
percent
with
esxi
and
vsphere
for
the
management,
the
management
layer.
So
here
you
have
to
understand-
is
that
not
only
that
is
powering
our
castle,
rock
roster
and
the
application?
And
I'm
talking
to
you
guys
about
today,
but
it's
actually
pouring
our
war
public
road
infrastructure.
This
is
exact
information.
A
So
this
is
how
it's
how
it's
looking
like
on
a
pair
data
center
basis,
so
we
got
the
actual
wiggidy
actual
clients,
which
is
an
angularjs
client
right
using
like
api
of
the
White
House,
we
have
a
DMZ
with
a
combination
of
hg
proxy,
keep
alive
the
apache
initially
taking
those
requests.
We
use
a
fantastic
product
called
key
clock,
that's
running
in
a
wildfire
application
server
that
allows
us
to
do
all
those
two
and
managing
our
education,
an
authorization
and
you
will
see
on
the
right,
I
actually
put
it.
A
We
have
a
post,
West
database
here.
This
is
the
last
SQL
database
that
we're
having
on
that
on
that
infrastructure.
So
there,
after
we
have
a
layer
web
application,
server
layer
which
is
the
prevailing
like
rest
api,
and
then
this
is
where
the
action
is
happening.
We
have
a
big
job
as
a
war
ii's
like
cluster
and,
of
course,
are
coming.
People's
gonna
tell
me
that
this
is
where
they
succeed
and
spa
or
something
but
like
you
have
to
understand.
A
It
is
a
huge
cluster
like
if
we
take
the
size
of
this
cluster
is
probably
as
large
as
the
actual
apache
deployment
in
term
of
hip
size
and
memory
cpu
xfr,
because
it
does,
although
roll
up,
etc
and
I'll
be
I'll,
be
talking
about
it,
because
this
is
central
in
the
evolution
that
we
consider
right
now.
That's
basically
the
despot
that
that
will
undergo
some
changes,
you'll
see
as
well.
Something
is
ready
and
I'll
talk
a
bit
about
Freddie's.
A
After
on
top
of
you
using
along
with
Cassandra,
then
we
have
apache
Lucene
for
the
actual
full
tactical
food
tech
survive,
NFS,
messing
crazy
here,
and
then
we
have
like
Cassandra
acting
as
a
foundation.
As
we
said
this
morning.
This
is
really
the
big
foundation
for
all
the
application,
and
this
is
the
one
actually
replicating
over
to
other
dissenters.
So
I'll
tell
you
mobile
data
center.
We
got
six,
and
at
customer
really
is
the
one
responsible
for
that
strong
from
the
acting
a
strong
foundation.
Sequel
is
a
synchronous.
A
Second
izing
data
all
over
the
place,
so
there
on
the
right
is
an
actual
public
code.
So
it's
a
remember:
it's
a
cisco
ucs,
it's
a
vmware,
esxi,
vsphere
machine
manager,
etc,
switch
lots
of
vmware
technologies
and
what
we
have
is
basically
those
this
white
flag
cluster
actually
pulling,
as
I
said
earlier,
on
new
agent
we're
putting
information
from
the
actual
card,
but
as
we're
we're
getting
information.
So
we
have
a
name
QP
rabbitmq
like
running
here
and
that's
not
for
analytics
is
just
for
events
like
knowing
what's
happening
on
on
the
on
the
cloud.
A
You
know,
new
user
create
a
new
vm
change
of
EMG
81
and
then
syslog-ng.
We
use
it
for
network
traffic
we
actually
able
to
root
through
through
a
syslog
ng,
then
I'm,
QP,
network
traffic
for
further
analysis
and
I'll
be
talking
about
that
little
bit.
After
in
one
of
the
reason
why
we
need
to
extend
this
passenger.
A
A
Think
something
like
that:
Stevie
sistani
acts,
but
using
6
equals
3,
which
was
added
at
the
time
in
assonet.
We
started
with
an
empty
cluster.
No
legacy
data
are
nothing
to
migrate,
no
seem
to
worry
about
me
early
2014.
We
switch
to
the
Java
Sea
co-driver,
big
improvement.
As
far
as
like
performance,
he
was
concerned
and
actually
a
network
usage
and
request.
That
was
really
a
fantastic,
a
break
and
let
2014
I-
remember
I've,
been
here
being
a
here
last
year
and
getting
all
excited
about
21
and
I'll.
A
Tell
you
one:
we've
been
like
jumping
on
to
one,
especially
for
the
actual
UDT
solid.
You
know
about
that.
So
now,
let's
take
a
look
at
some
number
of
our
cluster,
so
we
had
six
data
center.
We
have
only
one
key
space
right
now
around
27.
That's
the
navarre
between
25,
to
30,
notes
on
a
piano
basis,
1.5
to
2
terabytes,
depending
on
the
teacher
and
depending
in
free
free,
doing
some
kind
of
project
of
some
data
that
we
consider
are
not
like
pretty
critical
I
guess
that
speaks
for
itself.
A
Basically
have
the
circus
on
roaring,
discuss,
wha
rights
that
are
running
in
the
define
data
center
and
getting
sync
outs
all
over
the
place
on
a
panel
on
a
pair
data
center
basis.
We
have
one
of
Q
Castle
Rock
of
three
Cassandra
nodes
and
I'll,
explain
why
we
might
we
we
have
to
sometimes
we're
using
the
rack,
inferring
snitch
freud's.
A
We
have
a
virtual
private
network
addressing
across
the
data
center
and
we
use
our
education
factor
of
three,
which
means
like
the
data
set
is
actually
on
every,
although
every
single
node,
because
we
still
can
do
it
and
perform
well
right
now,
even
even
when
we
went
to
2.5
terre
by
to
switch
our
bite
at
a
time,
nothing
crazy.
It's
a
it's!
A
A
rock
in
flowing
snitch
you'll,
see
like
the
private
addressing
on
the
left
that
our
Center
identified
by
the
second
octet
so
pretty
classic
here
and
let's
talk
about
the
nose,
so
each
node
first,
its
vm
right
the
same
way
you
win
2015
right
the
same
way.
You
don't
run
an
oracle
to
do
that
or
shouldn't
be
any
reason
for
anybody
to
run
like
some
to
run
on
a
barometer
if
you
can
do
it
on
the
implied
so
we're
using
Ubuntu
using
the
open-source
castleroy
distribution.
A
A
So
I
will
just
say
that
is
that
we
are
we
in
running
on
10k
SAS
for
a
good
I'd,
say
one
year,
one
year
and
an
hour
two
years
and
we've
been
able
to
achieve
some
pretty
decent
performances
up
to
a
certain
point
and
that
up
to
a
certain
point,
I
see
Marie
rich.
When
we
went
over
like
1.5
terabyte,
we
started
actually
using
hybrid
arise
in
some
of
the
in
some
of
the
data
center
in
some
of
the
data
center
that
were
the
most
busy.
But
honestly
we
are
still
having
actually
a
10k
SS.
A
These
days
we
are
basically
switching
Roger
early
too
two-hybrid
accelerated
SSD,
but
it's
working
party
where
so,
when
it's
not
walking.
Where
is
when
you're
actually
putting
a
note
when
you
need
to
perform
like
operation
air
when
you
want
you
need
to
stream
so
that
that
usually
doesn't
doesn't,
doesn't
really
work
well,
but
Cassandra.
What
it
needs,
cpu
and
run
it's
all
you
can
eat.
Of
that,
you
can
give
CPR
on
to
cassandra.
Is
gonna
eat
everything.
A
What
we
see
as
well
is
that
the
the
rights
are
actually
super
where
handler
in
Cassandra,
and
we
see
that
the
even
if
you
have
a
high
write
throughput,
you
see
like
some
very
low
percentages
of
right
on
those
nodes,
because
it's
probably
handle
I
mean
idea
this
level
on
the
iOS,
a
DI
ops
level.
We
see
a
lot
more
read
than
right,
adidas
level,
so
it's
working
quite
right
network
wise
we've
been
running
with
one
king
line
for
almost
two
years:
we're
switching
basically
electronium
10
gig
lines.
A
So
it's
very
chatty
Cassandra
you
have
the
divorce
water,
which
is
the
one
we
are
using
right
now.
If
you
extend
this
water
is
going
to
use
everything
that
you
tell
it
to
use,
but
it's
constant,
so
you
can
count
on
it.
We're
basically
using
the
20
20
25
megabit
second
swatter
at
the
castle
wall
ever
to
be
walking
right
and
the
thing
I
mentioned
earlier
that
we
don't
use
the
ssl
built-in
feature.
A
We
have
a
virtual
private
network
in
between
the
actual
data
centers
to
show
you
how
constant
it
is
is
going
to
use
that
all
the
time,
no
matter
what
two
together
so
we're.
So
what's
the
idea
all
hit?
It's
basically
having
an
API
that
that
can
do
everything,
meaning
everything,
meaning
compute
storage,
network,
backups
or
services,
we're
providing
and
we
available
everywhere
so
everywhere
in
the
US,
everyone,
UK
and
everywhere
in
Asia,
with
all
the
data
there,
because
when
when
you
in
in
the
in
Europe,
you
probably
have
some
data
center
in
America
right.
A
So
if
you,
if
you
in
Europe-
and
you
have
your
footprint
spread
out
in
different
on
different
continents,
probably
you
want
the
same
response
time.
Then
then,
on
your
UK
are
footprints
that,
on
your
us,
/
explained
so
typically,
that's
that
that's
what's
happening
is
that
if
you
remember
like
the
overall
architecture,
I
showed
you
it's
on
a
pair
data
center
busy.
A
A
Sometimes
we
have
nodes
that
are
dedicated
for
reads,
especially
the
ones
like
serving
api's
on
a
personal
basis,
and
this
is
why
you
would
see
them
only
being
unused
for
read,
while
the
other
is
basically
write
and
read
for
the
actual
aggregator
and
more
up
stuff.
So
let's
take
a
look
at
all
right.
A
So
what
was
the
pain
point
because
globally
I
have
to
say
we
super
happy
with
the
way
things
single
and
has
been
walking
right,
but
so
we
had
to
cover
those
things
that
that
were
like
more
complicated
now
than
others
to
to
configure
so
fine
standard,
the
heap
size
and
all
the
Java
gvm
set
up
quite
complicated.
So
you
have
a
rule
of
thumb.
Remember
I
told
you
so
d
2
gigabyte
run
for
the
node.
They
say
one
force
is
a
good
start,
but
it
doesn't
always
work
like
that.
A
That
will
be
really
helpful
is,
of
course,
the
concurrent
white
and
concur
on
wait,
they're,
one
of
the
causes
for
the
customer
to
go
out
of
memory
and
crush.
If
you
go
too
crazy
on
that,
the
compactor
very
interesting
I,
don't
I
haven't
checked
the
documentation
for
a
while,
but
they
were
advertising
very,
very
few
compaction
threads,
even
with
ten
KSS
disk
we've
been
able
to
use,
and
we
actually
had
to
use
in
between
four
and
five
compaction
threads.
A
Otherwise
the
compaction
would
get
like
really
late
and
the
node
would
not
perform
basically
crashing
out
of
memory.
At
some
point,
when
you
wear
like
thousand
compaction
behind
your
tone
snapshot,
we
ve
be
aware
of
that.
It's
enabled
by
default.
If
you
have
big
tables,
unless
you
have
like
twice
the
size
of
your
actual
disk,
you
may
get
bad
surprises,
which,
like
snapshot
like
getting
there
and
then
between
you,
because
you,
you
know
cleaning
them
up
or
not
taking
care
of
them.
A
Alright,
so
let's
get
back
a
radius.
Why
do
we
have
read
it
right
while
Casanova
rice
noodles
is
really
great
and
we'd
is
good
and
not
good,
as
like
average,
like
really
good,
is
just
like?
If
you
compared
with
rights,
then
it's
easier
a
bit
under,
so
we're
not
storing
a
lot
in
Reggie's
we're
studying
with
restoring
basically
bindings
and
common
queries,
likely
stove
Williams
to
correct.
You
know,
remember
this
binding
in
between,
like
the
series
I
I
showed
you.
A
This
is
something
that
we
would
instead
of
like
doing
like
queries,
we
basically
stores
on
wizards
and
then
invalidate
that
on
regular
basis.
So
it's
not
really,
like
you
know
a
latex
data.
It's
more
older
configuration
and
a
correlation
of
those
data
that
we
basically
like
load
in
memory
refresh
and
regular
basis,
but
that
really
minimized
a
lot.
The
new
queries
that
we
have
to
do
in
Cosmo
and
we
don't
use
on
his
memory.
I
only
know
three
transient
data.
It
can
be
restarted.
It
doesn't
matter
migration,
so
this
was
a
good
one.
A
So,
as
I
said,
we
were
the
customer
was
to
mid
last
year
and
when
the
UDT
got
gotten
ours,
we
were
like
really
waiting
for
it
because,
as
you
can
imagine,
we
have
all
these
services
that
coming
from
south
party-
and
it
was
really
time
consuming.
We
come
up
with
tables
every
time
very
specific
and
have
all
these
bindings,
like
you
know
like
two
three
tables:
/
service,
etc.
When
you
did,
he
actually
came
in.
We
started
leveraging
them
like
right
away
and
I
really
helped
us
like
drastically
to
be
really
fast
at
integrating
support.
A
Services
and
I
was
the
main
reason
why
we
switch
over
to
that
one.
So
we
went
live
with
211
assume
it
was
an
end
of
november.
You
know
good,
like
three
weeks
before
Christmas.
Well,
it
was
walking
right.
We
hit
couple
of
bugs,
mostly
at
the
driver
level
or
on
UDT
I,
mean
honestly,
nothing
really
crazy
to
say
about
about
the
data
release,
but
the
big
problem
is
that
we
had
to
retune
the
work
roster.
So
all
the
set,
the
settings
that
are
actually
like
showed
you.
A
When
we
switch
to
two
dot
one,
we
started
to
see
diagnose
going
out
of
memory.
We
started
to
select
crashes
and
I,
wasn't
because
necessarily
that
Cassandra
was
buggy
it's
because
when
you
do
not
go
out
like
that,
actually
you
have
to
go
and
retune
everything
that
you
had
to
before,
and
that
was
that
was
the
most
painful
part
of
that
we
got
that
migration.
A
We
use
the
Python
driver
for
that
I,
bread,
so
I
know
the
release
a
new
version
I
recently,
but
that
was
really
slow
for
youtube
for
the
for
that,
I
migration,
I,
haven't
actually
tested
the
letters
that
supposed
to
be
a
much
better
in
term
of
performances,
but
we've
been
experiencing
some
very
long
migration
time
with
the
pattern.
Privates
are
also
what
you
not
do
at
all
at
the
application
level.
Second
row
index
is
so
we
actually
started
using
them.
I
guess
like
everybody,
when
we
initially
like
lady
with
our
model,
Wyatt
bad
ID.
A
As
soon
as
you
get
like
to
a
lot
of
data,
that's
not
going
to
walk
so
you
can
still
use
it.
Don't
use
it
to
rely
on
something
that
your
application
on
a
regular
basis
require
use
it.
For
you
know
you
want
to
do
some
particular
type
of
queries
that
with
that
are
not
planned.
It's
a
one-time
thing.
Then
yeah,
you
put
your
sugano
indexes
you're
on
your
crying
and
then
you
remove
them
that
that
works.
A
The
in
operator
I,
don't
really
have
like
data
on
it
right
now,
but
we
actually
like
removing
those
in
operators,
because
we
find
them
like
20
through
at
the
moment
on
some
tables
that
are
pretty
busy.
So
that's
something
that
started
happening
with,
but
I
can
it's
a
really
really
more
by
that
the
TTL?
That
was
very
interesting.
So
when
we
started
the
idea
was
to
broadcast
on
one
owed
a
lot
so
that
we
could
actually
make
sure
that
Casanova
was
handling.
A
You
know
the
volumes
of
data
we
wanted
and
at
some
point
we
actually
had
to
do
some
pervs.
The
problem
is
that
there's
no
range
delete
in
a
in
Cassandra,
so
it
makes
that
operation
like
pretty
slow,
especially
with
the
pattern
passing
driver
so
think
about
TTL
upfront.
Think,
like
you
know,
how
long
are
you?
A
Data
was
being
kept
because
it
will
avoid
you
having
like
some
issues
later
on
and,
of
course,
the
complex
ratio
type
of
model
you
to
not
do
which
bring
us
to
the
Jews,
which
is
basically
designing
very
simple
data
model
rights
are
cheap
duplicate
data
and
use
basically
batch
statements
to
make
sure
that
your
mother's
consistent,
but
don't
go
crazy.
The
simplest
model,
that's
what
we'd
perform
the
best
with
with
Cassandra
minimize
weed
with
from
customer
a
little
bit.
A
If
you
really
read
intensive
and
the
uddi
awesome,
we
really
like
them,
pain,
stuff,
I'd,
say
for
us,
like
the
most
allowed,
the
biggest
one
that
was
describing
new
nodes
and
remember.
We
started
with
swapping
most
of
our
data
center
in
the
two
dot
0
early
to
load
of
actions
and
only
two
dot.
One
he's
been
really
bad,
I
mean
we
had
to
start
and
then
stop
and
start
and
stop
until
having
those
tuning.
Those
parameters
and
the
news
performing
well
like
I,
showed
you
earlier.
A
It's
been
complicated,
so
customer
22
I
haven't
tried
it,
but
they
have
a
booster
presume
that
should
otherwise
that
problem
and
I'm
sure
is
going
to
be
like
Whedon
or
something,
because
that
was
probably
the
most
complicated
thing
that
happened
to
us.
Initially,
repairs
repair
saw
a
big
deal.
It
takes
a
lot
of
time.
It
hits
your
cluster,
the
incremental
repair
they're
walking.
Now
in
castle
watch,
one
they've
been
broken
I,
seeing
to
14
to
15
major
improvement.
A
You
can
run
them
without
like
hitting
your
cluster
and
without
actually
padding
for
resources
on
your
notes
to
support
its
operational
regular
basis,
all
right.
So
having
said
that,
what
what
are
we
thinking
of
doing
so,
as
you
saw
like
the
Zebras
cluster,
is
really
at
the
center
of
everything
it
does
order
roller
aggregation,
etc.
So
we
think
that
the
problem
we're
having
is
not
that
it's
not
walking,
because
it's
actually
walking
pretty
right,
but
it's
very
hard
to
trace.
We
are
very
few
visibility.
A
We
have
like
hundreds
of
job
like
thousands
of
sweat
running
it's
very
hard
for
us
to
know
exactly
what's
happening
in
it
and
also
we
could
go
in
with
a
solution
like
New,
Relic
and
stuff.
Like
you
know,
like
putting
information
there
and
see
you
have
those
friends,
we
believe
that
we
might
have
like
some
room
for
to
introduce
Park,
actually
have
spark
start
to
tackle
a
couple
of
these
issues.
A
A
You
know
network
analysis,
pattern,
checks,
access,
rot
and
introduce
some
machine
learning
on
it
and
try
to
leverage
the
data
we
have
and
as
well,
because
our
code
base
it's
a
role
like
Ronald
K
of
Java,
and
that
would
be
really
good
to
start
decrease,
decrease
that
and
get
to
two
more,
like
you
know,
smaller
code
base
for
the
actual
White
House
and
platform
and
the
rafting
and
I
think
also
took
tomorrow
and
the
Finke
will
be
there
it's
about
compliance
reasons.
So
it's
very
easy
with
Cassandra
to
replicate
everywhere
and
it
does
it
really.
A
Where
does
it
fast
super
really
able?
The
problem
is
that
sometimes
you
have
some
constraint
when
it
comes
to
the
actual
data
you
having
banking
information,
health,
information
and
I
think
we
will
start
like
looking
at
having
like
multiple
key
space
and
see
if
we
can
actually
have
like
a
subset
of
the
data
replicated
or
not
depending
on
the
location
and
the
compliance
constraints.
We
we
would
have
just
talked
to
more
I,
don't
remember
what
time
but
certainly
looks
interesting
and
of
course,
during
the
network
traffic.
A
If
you
replicate
less
everywhere,
then
you
don't
need
as
much
network.
Thank
you
very
much
guys
I'd
like
to
sing
that
as
tax
for
the
opportunity
to
press
on
here.
We
happy
to
be
here
and
I'd
like
to
thank,
like
my
team
I
have
like
some
ninja
over
there
like
developing
and
doing
all
that-
and
this
is
a
really
a
great
pleasure
to
to
to
to
be
let
up.
So.
Thank
you
very
much.