►
Description
In this talk, Tim Moreton, Founder and CTO at Acunu Analytics, and Nicolas Favre-Felix, Software Engineer at Acunu Analytics, share the concept, implementation and benefits of virtual nodes in Apache Cassandra 1.2 & 2.0. They also go over why virtual nodes are a replacement to token management, and how to use Acunu Analytics to collect event data, build OLAP-style cubes and ask SQL-like queries via a RESTful API, on top of Cassandra. This talk was recorded at the DataStax Cassandra SF users group meetup.
Understanding and (Not) Managing Virtual Nodes in Cassandra
Virtual nodes were added to Cassandra in version 1.2 and are now the default distribution model for the newly released Cassandra 2.0. This talk will explain the concept and implementation of virtual nodes in Cassandra, and the numerous benefits it brings. We will show you how virtual nodes make token management a thing of the past, improves the failure characteristics, improves bootstrapping and decommission speed, make incremental cluster growing and shrinking possible, and much more.
Under the Hood of Acunu Analytics
Cassandra is a great fit for building real-time analytics applications — but getting from atomic increments to live dashboards and streaming queries is quite a stretch. In this talk, we'll talk about how and why we built Acunu Analytics, which allows you to collect event data, build OLAP-style cubes, and ask SQL-like queries via a RESTful API on top of Cassandra. We'll dive into how it works and show how it follows Cassandra's spirit of denormalization under the hood.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
I
will
tell
you
a
little
bit
about
the
the
background
of
a
kunu,
so
we
so
we
started
out
building
a
storage
engine,
in
fact
a
storage
system
for
virtual
machine
environments.
So
looking
at
block
storage-
and
this
was
sort
of
2009
2010-
and
we
recognized
that
sort
of
pretty
soon
after
that
we've
got
some
great
technology.
A
It
sort
of
it
all
worked
quite
nicely,
but
we
recognized
that
there
wasn't
really
a
market
for
the
sort
of
ability
to
to
to
manage
large
volumes
of
structured
data
in
the
context
of
a
block
environment.
You
know
at
one
end
you
have
sort
of
ebs
and
at
the
other
hand
you
had
emc
and
there
wasn't
really
a
middle
there.
So
we
thought
hey.
What
should
we
do
with
this
cool
technology?
A
So
we
sort
of
offered
cassandra
with
the
storage
engine
for
a
little
while,
and
that
was
all
going
well,
but
we
sort
of
really
observed
that
one
of
the
challenges
that
people
had
around
using
cassandra
was
was
how
do
you?
How
do
you
build
analytical
applications
on
top
of
it?
So
you
know
cassandra
a
lot
of
the
use
cases
that
we
were
seeing
were
people
trying
to
get
back.
I
guess
sort
of
rich
query,
semantics
and
framework
that
people
were
familiar
with
and
coming
from
a
relational,
a
relational
background.
A
So
that's
I
guess.
The
genesis
of
acuno
analytics
so
I'll
tell
you
a
little
bit
about
that
and
then
go
on,
and
maybe
maybe
show
it
to
you
if
if
the
wind
is
coming
from
the
right
direction
and
and
so
on,
so
I
guess
the
first
thing
is
like
analytics
on
nosql.
Does
it
even
make
sense?
Why
would
you
not
just
use
hadoop-
and
I
guess
the
the
sort
of
answer
to
that
is
that?
Well,
you
should
use
hadoop.
A
So,
if
you're
trying
to
find
the
unknown
unknowns-
or
you
know
the
sort
of
so-called
needles
in
a
haystack
to
help-
you
inform
some
business
process
six
months
down
the
line.
You
know
you're
really
interested
in
how
meteorological
data
is
going
to
affect
your
widget
sales,
and
you
need
to
tell
your
sales
force
to
do
something
subtly
different
and
send
a
memo
out
at
the
end
of
the
week,
then
hadoop
is
a
great
fit
for
being
able
to
tackle
those
sorts
of
problems,
and
I
sort
of
call
that
discovery
analytics.
A
You
know
that
from
your
own
domain,
knowledge,
right
and
so
operational
intelligence,
I
guess,
is
about
being
able
to
build
dashboards
and
build
systems
which
react
to
changes
or
potential
problems
in
those
metrics,
rather
than
sort
of
discovering
new
metrics
in
their
entirety.
So
and
in
particular
often
that
contains
a
sort
of
real-time
element,
so
in
particular,
particularly
you're.
A
Looking
at
sort
of
use
cases
where
five
minutes
matters,
you
know,
if
there's
a
business
decision,
you
could
make
better
on
the
back
of
fresh
data
or
if
there
is,
I
guess
you
know
something
you
can
do
to
sort
of
raise
the
alert,
raise
the
alarm
and
go
fix,
something
because
you've
spotted
that's
broken
or
because
something's
gone
wonky
or
even
better.
Do
it
predictively
before
something
that
goes
out
of
out
of
out
of
band
or
out
of
out
of
bounds,
then
that's
that's
great.
A
A
So
you
know,
we've
been
contributing
to
cassandra
for
a
long
time
and
we've
been
using
it
and
like
it's
great
to
see
to
see
you
know
it's
continued
adoption
it
is.
Has
it
has
some
great
properties?
So
in
particular
it's
it's
it's
highly
scalable.
You
can
just
add
machines
and
increase
capacity,
increase
performance
and
the
multi-master
architecture.
That
cassandra
has
means
that
it
doesn't
create
you
operational
headaches
as
you
grow
it.
A
It's
very,
very
fast
in
particular,
it's
very
fast
for
writes
and
that's
really
useful
when
you're
doing
operational
analytics,
because
what
you
tend
to
do.
There
is
you're
collecting
this
big
stream
of
data
or
perhaps
you're
collecting
or
trying
to
record
information,
perhaps
you're,
counting
stuff
about
this
big
stream
of
data
and
you're,
potentially
reading
it
back
showing
on
dashboard
or
or
raising
alerts.
Much
less
frequently,
so
your
write
rate
to
read
ratio
is
is
often
like
a
hundred
or
a
thousand
to
one.
That's
typical
for
the
sort
of
customers
that
we
work
with.
A
So
the
fact
that
cassandra
is
very,
very
good
at
writing
fast
is
is,
is
important.
It's
also
highly
available.
I
think
it's
one
of
the
only
stores
out
there
where,
within
sort
of
10
minutes,
you
can
set
up
a
cluster
and
tell
it
to
spam
three
data,
centers
and
it'll.
Just
do
it
in
a
sensible
way
and
that's
a
that's,
a
really,
really
really
useful,
useful
thing.
A
A
Actually,
it
is
still
pretty
hard
to
query
cassandra,
so
you
know
you
put
your
data
in
and
then
you
have
to
pull
it
out
roughly
in
the
same
way
that
you
put
it
in
and
the
sort
of
whole
process
of
denormalizing
your
data,
which
cassandra
encourages,
means
that
you
need
to
think
in
advance
about
how
you
want
to
read
data
out
of
cassandra
and
that's
fine
unless
you
get
into
the
situation
as
as
as
bob
and
his
mate
do
here
is
mate
at
britishism.
I
think
it
might
be
anyway.
A
Bobbins
may
have
got
into
the
situation
where
you
know
bob
wants
to
query
the
database
himself
right.
You
know
in
fact,
there's
a
middle
step
right,
which
is
that
you
build
this
great
application
and
then
the
requirements
change,
and
you
then
have
to
query
in
some
other
way,
and
you
have
to
change
your
cassandra
data
model
to
adapt
to
that
and
that's
a
real
pain
and
in
particular,
if
you
then
need
to
roll
that
back
over
historical
data.
A
That's
really
hard
as
well,
because
you
have
to
probably
write
scripts
or
you
have
to
munch
it
in
some
way
and
then
the
step
beyond.
That
is
when
bob
wants
to
be
able
to
ask
queries
himself
and
has
to
become
an
expert
in
cassandra
data
modeling,
which
is
getting
considerably
easier
with
the
advent
and
improvements
in
cql3.
But
it
you
know
it
still
ain't
at
the
position
and
I
don't
think
it
ever
has
really
intended
to
be
where
you
can
just
give
it
to
a
business
user
and
let
them
let
them
loose
on
it.
A
A
So
we
built
this
thing
called
a
kunu
and
I'll
dive
into
a
bit
of
technical
detail
about
it
in
a
sec.
But
basically
you
take
events,
so
we
have
a
http
api
where
you
can
just
throw
a
json
at
it,
with
flume
integration,
kafka,
integration
and
and
a
range
of
other
things
now,
and
basically
it
just
sort
of.
Does
your
data
modeling
for
you,
you
describe
tables
and
sort
of
dimensions
like
the
the
sort
of
fields
that
correspond
to
the
keys
in
your
in
your
in
adjacent
objects,
say
and
say.
A
This
is
a
time
or
this
is
a.
This
is
a
path
like
a
sort
of
hierarchical
structure
or
it's
a
number,
and
we
basically
turn
those
tables
and
sort
of
the
templates
of
the
queries
that
you
want
to
ask
on
them
into
into
operations
on
cassandra.
So
an
event
turns
into
rights
on
cassandra
and
we
use
cassandra
to
store
both
the
raw
events
and
these
intermediate
and
these
intermediate
results
as
well,
and
the
reason
we
do.
A
A
So
we
let
you
build
olap,
cubes
and
update
them
in
a
continuous
fashion
and
rather
than
you
coming
back
to
your
data
warehouse
the
next
morning
and
getting
the
results
from
yesterday
and
getting
a
sort
of
cubing
environment
set
up
from
yesterday,
where
you
can
sort
of
slice
and
dice
your
data.
Actually
that
just
happens
continuously,
and
we
can
do
that
because
of
the
great
features
that
cassandra
has,
and
on
top
of
that,
we
also
built
a
bunch
of
dashboards
so
like
a
browser-based
set
of
dashboards.
I'll
show
you
to
sort
of
help.
A
You
visualize
all
this
stuff
in
real
time.
So
how
does
this
work?
So?
The
first
step
is
that
you
define
aggregate
cubes.
This
is
sort
of
logical
progression
right,
so
you
define
you
say,
you
know
we
keep
the
spirit
of
denormalization
that
cassandra
has
and
we
we
basically
ask
you
to
say,
tell
us
how
you're
going
to
be
querying
your
data,
but
in
a
sort
of
much
more
sort
of
templatized
sense.
A
So
a
cube
consists
of
an
aggregate
function,
something
you
want
to
do
with
the
data
and
a
set
of
dimensions
that
you
might
want
in
a
where
or
a
group
by
clause.
So
here
we're
going
to
build
a
cube
which
takes
the
approx,
the
prox
top
five
or
ten
or
whatever
keywords,
and
we
are
gonna-
want
to
be
able
to
specify
and
filter
by
browser
and
by
time,
and
then
we
wanna
be
able
to
group
by
time
buckets
that
are
sort
of
granularity
there.
So
what
happens?
Is
we
as
we
collect
these
events?
A
We
do
the
work
to
update
all
of
the
potential
places
in
the
materialized
view
that
you
that
that
event
features
so,
for
example,
if
you're,
counting
by
day
or
counter
by
hour
of
day
or
counting
uniques
by
hash
tag
or
something
like
that
and
the
event
features
in
a
range
of
different
places.
In
these
cubes.
We
will
convert
that
single
event
into
multiple
rights,
which
is
fine,
because
cassandra
has
great
ride
performance.
A
We
also
stored
the
raw
events
as
well
I'll
come
back
to
why
in
a
sec.
But
when
you
ask
a
query
in
the
the
kunu
query,
language
which
looks
as
you
can
see,
cunningly
like
sql
you.
Basically,
what
we're
doing
there
is
just
decomposing
that
into
reads
basically
get
slices
in
cassandra
on
these
on
on
these
cubes
and
the.
A
I
guess
the
benefit
of
doing
that.
The
key
observation
here
is
that
we're
never
having
to
touch
that
original
data.
If
you're
going
to
ask
a
query.
So
if
you
go
to
hive-
and
you
do
select
count
from
your
whole
data-
set,
that's
going
to
take
a
while,
because
it's
going
to
go
over
every
record
and
it's
going
to
have
to
pull
all
of
that
back.
So
the
whole
purpose
here
is
that
by
you
guys
specifying
what
you
the
sort
of
shape
of
queries
that
you
want
to
build
in
advance.
A
A
So
if
you
spot
a
particular
outlier
in
your
latency
distribution,
and
you
want
to
drill
into
that
and
find
out
particularly
why
why
that's
occurred,
you
can
still
see
the
raw
events
through
that
and
of
course,
then,
when
your
boss
comes
and
changes
their
mind
about
the
requirements
and
says
it
would
be
great
if
we
could
have
this
different
cube
or
if
we
could
group
by
this
other
dimension.
Or
could
you
show
me
this
other
dashboard?
A
We
allow
you
to
oh
and
by
the
way
I
want
results
for
yesterday
as
well,
because
I
want
to
compare
them.
We
allow
you
to
take
the
raw
events.
We
diff
the
schemas
work
out
the
underlying
data
that
we
need
to.
We
need
to
change
in
order
to
be
able
to
get
that
historical
data
and
backfill.
We
basically
backfill
by
pumping
that
those
raw
events
into
the
new
cubes
does
that
make
sense
cool.
So,
okay,
let's
do
a
concrete
example.
A
So,
basically
imagine
you've
got
some
data
like
this,
so
we
break
every
cube
is
represented
in
cassandra.
By
not
every
cube,
I
guess
a
table
will
be
broken
down
into
a
set
of
column,
families
in
cassandra.
So,
under
the
hood,
we
are
maintaining
a
set
of
materialized
views,
so
each
cube
is
basically
a
materialized
view
and
the
basic
way
of
thinking
about
it
is
a
where,
in
your
cube,
is
a
projection
onto
a
particular
row.
A
Each
of
those
represents
a
sort
of
a
step
of
a
group
in
your
columns
and
a
set
of
rows
as
well.
So
here,
for
example,
just
to
sort
of
understand
that
right
what
you
want
to
be
doing
is,
when
I
say
something
like
select
count
where
time
is,
is
within
an
hour
and
group
by
minute.
I
want
to
be
able
to
do
as
little
work
in
cassandra's
possible
to
pull
those
that
data
out.
A
So
the
way
we
organize
that
is
to
to
have
the
hour
as
the
row
the
row
key
there
and
to
be
able
to
have
the
minute
as
the
column
key.
So
a
single
get
slice
operation
in
pre-cql3
terms
allows
you
to
allows
you
to
get
that
answer
and
of
course
the
original
event
has
to
update
a
whole
bunch
of
different
counters.
So
just
talking
about
cancer.
A
So
that's
the
sort
of
very
basic,
a
basic
example.
So
there's
a
number
of
other
things
that
you
can
do
with
this.
So
we
support
hierarchical
aggregation
as
well.
So
you
can
break
urls
or
paths
of
something
like
data
center
rack
machine
down.
We
support,
arithmetic
expressions
limits.
You
can
do
filtering
and
query
time.
We
deal
with
time
zones.
A
So
if
you
are
inserting
events
with
a
whole
range
of
different
time
zones,
and
then
you
want
to
come
back
and
query
10
o'clock,
1003
pacific
through
to
1157
utc,
we'll
piece
together,
the
buckets
that
are
available
with
with
a
minimal
amount
of
effort
and
just
return,
you
the
overall
results
and
then
you
can
do
inner
and
outer
joins
that
warning.
That
happens
at
query
time.
A
It's
designed
as
a
convenience
thing
for
you
to
do
on
sort
of
low
cardinality
dimensions
is
not
designed
to
replace
sort
of
massive
into
machine
joins
in
a
data
warehousing
context.
So
and
then
you
can
do
sort
of
sql
sub
query
type
things,
but
I
personally
can't
get
my
head
around
sql
sub
queries.
So
I
think
a
pipe
for
syntax
is
much
easier
to
think
about
where
you
can
take
the
result
of
one
query
on
a
cube
and
then
ungroup
or
regroup
or
summarize
by
by
something
else.
A
If
you
see
what
you
have
here,
it's
basically
still
a
tool
for
developers.
So
if
you
look
at
this
sort
of
workflow
of
install,
create
tables,
define
cubes
like
plug
in
your
data
feed
and
then
build
widgets
well,
only
the
only
the
widget
building
bit
is
really
doable
by
bob
right.
Everything
else
bob
is
sort
of
still
has
to
go
and
pester
this
other
guy.
A
For
so
one
things
we
wanted
to
do
is
change
that
a
little
bit,
so
we
released
analytics
v5
last
week
and
one
of
the
things
we
so
bob
still
has
to
that.
The
other
guy
has
to
still
install
it.
One
of
the
things
we
did
there
was
really
try
and
improve
the
workflow
for
business
users.
So
try
and
give
you
the
equivalent
of
one
of
the
rich
bi
tools
that
you've
had
for
years
in
sql.
A
A
We
allow
business
guys
to
build
widgets
by
adding
a
sort
of
drag
and
drop
ui,
which
sort
of
does
an
ad
hoc
query
and
then
works
out.
What
cubes
you
need
to
fulfill
that
creates
those
cubes,
backfills
them
and
then
updates
for
real-time
data.
Of
course,
as
it
comes
in,
you
still
need
to
get
this
other
guy
back
involved.
When
you
turn
on
the
data
feed,
but
that's
getting
a
bit
easier.
A
We've
built
a
set
of
ingestors
and
and
other
sort
of
add-ons
so
that
you
can
just
from
the
command
line
plug
in
different
data
sets
and
and
that's
definitely
getting
easier.
And
then
you
can
re.
You
know
that
your
developer
then
can
get
out
of
the
picture
and
get
on
with
more
useful
stuff,
and
you
can
leave
bob
to
go
and
iterate
on
the
analytics
and
dashboards
that
he's
been
building.
A
So
I'll
show
you
I'll
show
you
what
they
look
like
so
they're,
basically
just
json
events
right,
so
they're
json
objects
with
like
keys
and
values
inside
them.
In
fact,
the
keywords
here
have
been
split
out,
so
maybe
we're
cheating
a
little
bit,
but
we
you
basically
just
give
it
a
set
of
events,
and
we
will
then
look
at
those
events
and
work
out
what
the
structure
of
them
is.
So
we
sort
of
correctly
infer
the
time
stamp
as
a
time
we
completely
incorrectly
a
further
keywords
as
a
path.
A
So
we
also
support
this
type
bag,
which
is
a
bit
like
a
set
of
tags.
So
if
you
send
an
event
with
a
sort
of
an
array
of
strings
in
it,
we'll
sort
of
process
those
and
pull
them
out
for
as
a
bag,
if
we
go
over
here,
use
a
country
string,
great,
that's
good,
tweets
path.
Here,
that's
wrong
should
be
a
string.
A
So
I
don't
really
need
to
know
very
much
about
cubes
or
or
analytics
to
be
sort
of
doing
this.
The
sort
of
target
audience
is
somebody
who
can
use
a
pivot
table
in
excel,
so
I've
just
created
my
enormous
big
data
set
of
40
tweets
from
my
file,
and
now
I
want
to
do
some
big
data
analytics
on
all
40
of
those
tweets.
A
I
think
they're
a
fortune
there
anyway,
so
I
can
drag
count
up
here.
Yeah
40.
and
I
could
this
is
just
doing
an
ad
hoc
query
from
the
raw
events
at
the
moment.
There's
no
cubes,
no
cubes
here
at
all,
and
so,
let's
group,
my
country,
shall
we
so
I
group
my
country,
I
get
a
picture
which
breaks
that
count
down
by
my
country.
A
There
we
go,
and
at
this
point
maybe
I'm
happy
with
it,
so
you
can
see
underneath
by
the
way
that
there's
a
there's,
a
query
that's
going
on
here,
so
we
do
an
event
store
query
we
pipe
it
through
and
we
sort
of
do
a
fake
cube
query
and
by
the
way,
for
every
query
in
here
you
just
have
to
say
all
the
ui
is
doing
is
just
using
our
http
api,
which
is
available
to
your
applications
and
we'll
just
give
you
jason
back.
Yes,
okay,
so
you.
B
A
Yeah,
so
that's
what's
happening
here,
so
I've
just
uploaded
a
file
and
there's
no
cubes
in
here
yet
so
this
is
just
a
so
if
you
watch
when
I
click
add
to
dashboard
here.
Basically,
this
is
the
cube.
I
need
right.
I
need
to
count
from
tweets
group
by
country,
but
I'm
missing.
I'm
missing
that
cube.
So
when
I
so
new
cubes
will
be
created
to
support
instant
updates
as
widget.
A
So
when
I
click
add
widget
that
cube
will
be
created
and
with
that
tick
box
field,
it
will
populate
it
with
my
vast
amount
of
historic
data.
So
if
I
refresh
that
yeah
there
we
go
so
now
it's
on
a
dashboard
and
I
can
drop
down
to
pull
in
some
twitter
dates
some
twitter
data.
We
we
put
together
this
twitter
ingester,
which
I
will
attempts
to
start.
A
Let
me
just
grab
the
so
we
have
this
plug-in
framework
now
for
ingestors,
where
you
can
basically
write
your
own
or
download
off
github
different
types
of
ways
of
getting
data
into
a
cuny.
So
you
have
this
notion
of
an
ingester
and
then
a
flow
so
flow
basically
takes
data
of
a
particular
sort
and
pipes
it
through
to
a
table
in
in
analytics.
So
the
table
that
got
created
I'll
just
go
and
show
you
that
the
table
that
got
created
was
basically
have
these
fields
that
we
created,
and
it's
now
got
one
cube
in
it.
A
Fingers
crossed
it'll
go
to
twitter
and
pull
us
in
some
events,
although
it
did
try
and
rate
limit
my
api
key
earlier.
So
there's
no
guarantee
that
that's
going
to
happen.
Oh
there
we
go,
we've
got
some
okay.
So
if
we
go
back
to
the
dashboard
we
just
made,
you
know
what,
let's,
so
I
can
change
the
visualizations
on
it
later
as
well,
so
change
your
pie
chart
if
I
make
that
live,
let's
see
what
happens
all
right,
so
we're
getting
some
twitter
data
about
countries
and
then
talking
about
drilling
down
to
raw
events.
A
A
So
so,
by
taking
the
definitions
of
the
cubes
that
you
set
up,
we
work
out
what
we're
going
to
need
to
insert
in
rows
and
what
we're
going
to
need
some
certain
columns
and
build
a
data
model
that
maps
maps
that
so
under
the
hood,
we're
creating.
We
have
create
a
key
space
used
by
default
called
analytics,
and
then
we
create
column
families
within
that
which
match
the
cubes
that
you
set
up,
and
then
we
have
to
do
because
cassandra's
got
some
limitations
on
your
ability
to
change
the
schema.
A
After
the
fact,
we
sort
of
have
to
do
a
bit
of
funky
serialization
of
our
own
to
to
to
sort
of
make
that
work
so
say
you
want
to
change
a
field
which
you
first
declare
as
a
number
into
a
string,
and
then
you
want
to
create
a
different
cube
on
it.
So
we
at
that
point
we
end
up
having
to
do
a
bunch
of
our
own
serialization,
but
more
or
less
all
we're
doing
is
mapping
sort
of
higher
level
constructs
about
like
I
want
to
cr.
A
A
So
what
happens
if
your
event
definitions,
change?
Well,
you
can
go
and
change
these
things.
If
you,
if
you
like
so
for
example,
I
can
add
a
field,
so
I
can
go
and
change
that
and
I
can
add
a
new
field
if
I've
already
been
collecting
events
with
that
field.
That's
absolutely
fine.
We'll
have
been
storing
those
events
and
then
you
can
come
and
backfill
them
and
pick
up
that
data
from
those
events
you've
already
collected
so
yeah.
A
A
So
at
that
point
we
will
then
take
a
slice
of
data
from
the
past
and
basically
do
this
distributed
process
where
we
roll
over
the
raw
events
and
pull
them
back
into
this
new
cube.
So
you
can
get
historical
context,
but
if
you
don't
care
about
history
on
this
new
cube,
you
don't
have
to
do
that.
Obviously,
that's
a
bit
like
a
map
produced.
It
takes
a
while.
A
Different
cubes
yeah
yeah,
if,
if
the,
if
the
graphs
are
showing
different
things,
so
you
might
have
graphs
which
are
showing
you
know,
clicks
from
people
in
california,
clicks
from
people
in
oregon
and
maybe
they're
the
same
cube,
but
just
with
different
parameters
filled
in
so
cube.
A
cube
represents
like
a
sort
of
unparameterized
shape
of.
A
Right
so
you're
right,
so
there
is
definitely
a
space
blow
up
by
maintaining
materialized
views,
so
I
mean
the
question
just
for
the
recording.
The
question
was:
what's
the
space
blow
up,
if
you've
got
a
terabyte
of
data,
however,
cassandra
has
built-in
compression,
which
is
actually
really
really
good,
and
also
all
we're
doing
is
just
for.
Most
of
these
things
is
counting,
so
numbers
don't
really
take
up
a
lot
of
space,
it's
actually
the
keys
that
are
the
the
large
the
large
element.
A
So
we
do
have
some
data
structures
which
aren't
counts
like
so.
We
support
a
prox
count,
distinct,
for
example,
which
uses
a
sort
of
probabilistic
sketch
algorithm.
To
give
you
a
better
performance
on
count,
distinct,
which
is
quite
hard
to
do,
and
there
the
sort
of
space
is
sort
of
customizable.
You
can
choose
how
how
how
big
they
are.
In
fact
there
you
go,
there's
the
size
field,
just
there.
A
A
C
Okay,
welcome
to
this
talk
on
virtual
nodes,
the
feature
that
we
call
the
operational
aspirin
for
focus
on
I'd
like
to
start
with
a
one
minute
recap
on
how
sonic
distributes
data
in
a
cluster.
So
cassandra
is
a
distributed
database.
It
tries
to
spread
your
data
set
across
multiple
nodes
and
the
way
that
it
does.
This
is
using
a
technique
called
consistent.
Hashing
nodes
are
clustered
in
in
a
structure
that
we
call
the
ring.
So
this
is
a
picture
that
I
will
reuse
in
this
stock.
C
C
So
the
way
you
configure
this
in
cassandra,
at
least
before
virtual
nodes,
is
by
giving
a
token
to
each
node.
So
in
this
case
the
blue
node
has
token
0.
The
yellow
node
has
token
max
token,
divided
by
4
max
token,
divided
by
by
two
three
quarters
etcetera.
So
if
you
have
ten
nodes,
you
would
take
this
maximum
token
divide
it
by
ten
and
then
give
zero
times
this
to
the
first
node
one
times
this
to
the
second
node
etc.
C
This
is
something
you
have
to
do
by
hand
when
you
set
up
your
cluster.
The
way
that
exoner
used
to
do
it
is
using
md5
reduced
to
127
bits.
Instead
of
128
now
uses
murmur
3,
which
is
160
bits,
I
think,
but
it
doesn't
really
matter
which
hash
function
it
uses.
The
point
is
that
this
will
distribute
data
all
across
the
cluster
evenly
as
long
as
your
partitions,
obviously
are
are
set
up
properly.
C
If
you
set
up
one
partition,
which
is
just
between
zero
and
one
o'clock,
your
your
one
note
out
of
four
will
only
receive
one
key
out
of
twelve,
only
okay,
so
this
has
some
limitations
and
in
particular
the
main
problem
is
the
operational
complexity
of
the
system.
There
are
some
problems
in
setting
this
up.
It's
quite
error-prone.
C
We
frequently
see
people
getting
this
wrong
and
ending
up
with
unbalanced
clusters
which
can
cause
problems.
It's
also
a
liability
when
you
try
to
rebuild
the
cluster
in
a
way
that
we'll
see
in
a
little
bit.
But
the
main
point
is
that
when
you
try
to
add
a
node
or
remove
a
node,
you
will
not
involve
your
whole
class
type,
only
a
local
fraction
of
it,
which
is
not
exactly
what
you
would
want
from
a
peer-to-peer
master,
less
decentralized
distributed
system
in
terms
of
maintenance
as
well.
C
There
are
some
some
processes
that
you
need
to
run
in
cassandra,
in
particular
something
called
repair
in
which
the
nodes
will
synchronize
themselves
in
a
distributed
fashion,
and
this
involves,
without
virtual
nodes,
only
in
the
same
way
a
small
fraction
of
the
cluster
when
you
add
or
remove
nodes
when
any
sort
of
topology
changes
changes
happen.
When
you
add
the
data
center,
this
is
a
manual
process,
it's
quite
tedious
and
right.
This
is
now
a
modern
way
of
distributing
data.
C
C
So
the
way
we
we
would
add
it
in
the
first
place
is
to
insert
the
new
node
in
the
cluster.
This
new
node
would
come
up
at
around
five
o'clock.
Let's
say
this:
would
this
is
a
random
random
pick,
but
if
it
comes
at
five
o'clock,
it
only
has
a
twelve
of
of
the
whole
ring,
which
is
not
what
we
want.
We
want
all
these
notes
to
have
something
like
well
to
have
20
of
of
of
the
ring.
So
there
is
a
manual
process
in
which
we
need
to
move
each
note.
C
C
C
If
you
do
it
node
by
node,
you
can
see
that
it
only
involves
the
immediate
neighbors
of
of
the
new
node
in
the
ring.
This
is
time
consuming
it's
resource
intensive
as
well,
and
it
is
generally
a
pain
to
do
at
least
in
small
clusters.
So
what
people
tend
to
do
is
at
least
for
small
clusters
is
to
double
the
number
of
nodes.
C
C
This
doesn't
really
scale
when
you're
growing.
For
for
from
more
than
eight
notes
right,
you
can't
really
justify
spending
double
the
money
every
single
time.
Usually
what
people
do
above
eight
or
ten
nodes
is
to
start
a
new
data
center.
So
they
push
this
problem
a
little
bit
further
in
time,
but
is
still,
in
general,
an
operational
pain.
C
So
the
way
to
add
or
remove
nodes
is
to
insert
this
this
new
node
with
a
new
token
and
rebalance
those
ranges.
So
ranges
are
these
these
partitions,
these
slices
of
the
the
whole
data
set
that
belong
to
a
node
or
another
in
general.
This
is
something
that
people
do
wrong:
either
they
make
mistakes
in
the
calculation
of
of
the
new
tokens
or
they
will
move
more
data
than
is
optimal
right.
If
you
add
a
new
node
from
from
four
to
five,
maybe
you'll
leave
the
one.
C
That's
at
zero
and
you'll
move
the
one
that's
at
25
to
20
percent,
so
you're
moving
five
percent
of
your
data
to
the
new
node
to
to
another
one,
but
in
general
you
will
not
go
through
the
most
optimal
process.
So
some
there
are
some
commercial
tools
that
will
give
you
a
rebalance
option.
I
will
try
to
calculate
the
minimum
amount
of
work
for
for
rebalancing,
but
if
you
do
it
by
hand
you'll
probably
do
it
wrong
at
least
a
little
bit.
C
If
you
have
a
hundred
nodes,
all
for
some
of
our
customers,
two
thousand
nodes
moving
a
few
out
of
several
thousands
is,
is
not
really
the
the
most
efficient
way
of
of
balancing
of
spreading
the
work
among
those
nodes.
It's
all
right.
It's
manual,
tedious,
air,
prone
painful
and
in
general,
it's
now
a
thing
of
the
past.
C
Removing
the
node
is
is
a
pain
in
the
same
way,
and
it
is
pretty
much
the
same
operation
in
reverse
in
a
way
that
people
will
for
small
clusters
divide
the
number
of
nodes
by
two
and
removing
one
node
will
involve
the
same
process
except
in
reverse.
When
you
have
your
five
notes
with
20
each,
if
you
remove
one
of
them,
you
will
have
to
rebalance
all
of
the
the
surviving
ones
to
25
right.
C
C
C
C
So
virtual
nodes
means
having
more
than
one
token
per
node.
There
are
different
ways
to
do
this.
It
means
that
now
we're
splitting
the
token
space
the
key
space
into
more
than
four
partitions
into
more
than
four
ranges
for
four
nodes,
and
there
are
different
strategies,
so
the
one
that
christian
roman
point
two
uses
is
to
have.
If
you
consider
n
the
number
of
nodes
and
b
the
size
of
your
data
set,
the
number
of
partitions
here
would
be
proportional
to
the
number
of
nodes.
C
Typically
in
cassandra
1.2
and
over,
you
would
have
256
partitions
206
ranges
per
node,
so
in
our
full
node
cluster,
that's
1024
ranges
that
are
spread
all
over
the
ring.
It
means
that
the
partition
size
in
this
case
grows
in
proportion
of
data,
but
also
is
is
lowered
by
having
an
increased
number
of
nodes.
C
It's
a
bit
different
from
the
way
that
react
distributes.
Data
react
has
virtual
nodes
as
well,
but
you
have
a
fixed
number
of
virtual
nodes
in
the
whole
cluster,
and
this
is
defined
when
you
install
rare
and
mongodb.
Although
I'm
not
I'm
not
certainly
not
an
expert
on
on,
but
I
believe
this
is
how
it
works
with
auto
shouting
that
you
have
a
certain
number
of
partitions
which
depends
on
the
size
of
the
data.
C
C
So,
let's
see
how
it
works.
When
you
we
start
a
single
node,
it
will
start
with
256
tokens.
Well,
this
one
has
about
10,
or
so
I
said
that
these
tokens
are
generated
at
random,
so
they
split
the
ring
into
256
ranges
of
various
sizes.
As
you
can
see
here,
some
of
these
ranges
are
larger
than
others,
sometimes
a
lot
larger
than
others.
C
Adding
a
new
node
will
also
generate
256
new
tokens
and
spread
those
across
the
cluster.
So
by
injecting
its
own
tokens
into
the
ring,
a
new
node
will
steal
some
data
from
the
existing
nodes.
In
this
case,
there's
only
one,
so
the
one
that
joins
will
then
start
to
own
the
the
ranges
that
are
here
represented
in
green.
C
C
You
can
see
how
adding
more
nodes
to
the
cluster
will
lead
to
a
more
balanced
version,
a
more
balanced
distribution
in
this
case.
So,
in
practice
it
is
very
close
to
1
over
n
for
all
your
nodes.
It's
true
that
it's
not
exactly
that
you,
it
defers,
often
by
a
few
percent,
but
really
is
not
it's
not
a
big
deal
in
practice.
It
doesn't
really
matter
at
all.
C
Adding
a
new
node
so
that
this
process
called
bootstrap
will
then
assign
a
host.
256
tokens
split
ranges
from
existing
nodes,
and
it
means
that
instead
of
having
data
sent
just
from
the
local
neighbors,
you
have
the
new
node
steal
data
from
all
the
existing
nodes,
and
this
will
stream
data
in
from
you
from
your
whole
cluster.
It
will
involve
all
your
nodes
in
bootstrap
right,
so
what
you
get
from
this
is
right,
no
need
to
rebalance.
C
You
just
add
one
note
to
grow
your
cluster
and
you
could
remove
one
node
one
at
a
time
as
well.
So
if
we
remove
the
one
that
has
the
blue
ranges,
we
end
up
in
the
same
situation
as
as
before
it
will
release
some
of
the
nodes
that
some
of
the
ranges
that
it
stole
from
the
green
node
to
the
green
node
and
and
same
thing
for
the
right
node.
C
C
C
It
becomes
replaced
by
noodle
status,
in
which
you
can
see
this
host
id,
and
you
can
see
the
the
columns
called
owns,
which
is
the
the
the
sum
of
all
the
sizes
of
the
ranges
that
this
node
is
is
responsible
for
over
the
size
of
the
whole
ring.
So
in
this
case,
I
just
started
five
nodes
at
random.
I
didn't
change
anything
in
the
configuration
just
made
sure
that
the
virtual
nodes
were
set
up
properly,
and
you
can
see
that
I've
got
pretty
much
an
even
distribution.
C
C
C
C
Okay,
some
math,
you
run
some
some
simulation
on
what
it
means
to
have
random
ranges.
So
this
is
just
running
simulated
token
assignment.
If
you
pick,
let's
say
10
nodes
and
you
give
them
a
thousand
tokens
each.
You
just
insert
them
at
random.
In
this
token
space
and
you
look
at
the
distribution
of
range
sizes-
and
you
can
see
that
well,
there
are
some
ranges
that
are
much
larger
than
others,
and
some
ranges
are
much
smaller
than
others,
but
this
is
a
distribution
that
applies
not
on
the
whole.
C
So
we
run
the
test
in
simulating
data
insert
into
this
cluster,
and
this
is
roughly
what
we
got.
You
can
see
that
on
the
x-axis
you
have
the
virtual
node
location,
that's
really
the
the
token
itself
between
zero
and
one
billion.
Let's
say
in
this:
if,
if
that
was
your
token
space
and
in
on
the
y-axis,
you
have
the
amount
of
data
stored
per
node
and
as
you
can
see,
some
of
them
are
much
much
larger
than
others
in
practice.
C
C
You
would
see
that
some
of
them
are
really
close
to
each
other,
and
some
of
them
are
quite
far
apart
when
you
put
this
together
in
in
the
whole
cluster,
you
end
up
with
distribution,
which
is
very
much
centered,
which
is
very
much
a
normal
distribution
centered
on
the
expected
amount
of
data
that
you
would
have
for
n
ranges
in
that
space,
so
you
have
an
nth,
is
centered
on
one
end
and
being
the
total
number
of
virtual
nodes.
A
total
number
of
tokens
in
in
the
ring.
C
So
in
this
balanced
cluster
cluster,
we
now
have
keys
that
are
randomly
distributed
evenly
distributed
in
the
cluster
by
the
random
partitioner
and
in
practice
it
does
tend
to
balance
itself
out.
So
I
run
the
test
just
before
before
the
stock
started
two
nodes,
they
started
with
48
and
51
percent
three
nodes,
almost
a
third
four
nodes,
almost
exactly
25
in
practice.
If
you
have
60
nodes,
they
will
be
completely
equal.
C
C
C
C
C
C
C
The
way
it
works
is
it
will.
You
would
stop
one
node
at
a
time
and
then
start
it
again
with
virtual
nodes.
Enabled
so,
if
you
think
of
your
four
nodes
in
this
equally
divided
quadrant
of
25
each,
if
you
stop
one
node,
then
enable
virtual
nodes
there
and
then
started
it
again.
What
sonic
will
do
is
identify
that
there
was
a
running
cassandra
cluster
cassandra
node
there
with
25
of
the
data.
It
will
take
this
range
and
split
it
into
26
tokens.
C
C
I
don't
think
I've
ever
seen.
Anyone
use
shuffle
successfully
have
you
yes
well,
you're,
the
one
so
eric
who
used
to
work
at
okuno
and
who
wrote
shuffle.
I
said
that
it
worked
on
his
laptop.
That
was
the
the
only
guarantee
we
had
so
far.
So,
okay,
I'm
glad
to
hear
that
someone
managed,
but
it
is
quite
broken.
I
would
say
so.
C
So
you
should
really
be
using
versions
by
now.
If
you
haven't,
then
this
there
is
a
migration
path,
migration
path.
It
is
worth
it
because
managing
tokens
and
shuffling
data
together,
moving
data
together
by
hand
is,
is
really
a
pain,
and
this
will
let
you
grow
your
cluster
and
and
in
a
much
easier
way
right.
Thank
you.