►
From YouTube: C* Summit 2013: Cassandra at eBay Scale
Description
Speakers: Feng Qu, Principal DBA and Anurag Jambhekar, Senior Manager of Database Infrastructure
Slides: http://www.slideshare.net/planetcassandra/5-feng-qu
We have seen rapid adoption of C* at eBay in past two years. We have made tremendous efforts to integrate C* into existing database platforms, including Oracle, MySQL, Postgres, MongoDB, XMP etc.. We also scale C* to meet business requirement and encountered technical challenges you only see at eBay scale, 100TB data on hundreds of nodes. We will share our experience of deployment automation, managing, monitoring, reporting for both Apache Cassandra and DataStax enterprise.
A
Good
afternoon,
everyone
I
hope
everyone
is
doing
great
this
the
day.
I
hope
your
a
fun
time
running
lot
of
things.
My
name
is
an
oral
exam.
Baker
I
manage
the
database
infrastructure
team
in
eBay
marketplace
and
I'm
here
with
Franco,
who
is
a
principal
in
BBA
in
my
team,
and
we
are
going
to
talk
about
Cassandra
at
ebay,
skill
and
and
before
I
before
you
delve
into
Cassandra.
I
just
want
to
take
a
few
minutes
to
talk
about
the
environment,
the
environment
we
have,
which
basically
runs
the
all
the
ebay.com
infrastructure.
A
So
our
site
is
at
ebay.com
site
predominantly
runs
on
on
our
calendar
as
this
with
the
emergence
of
the
no
sequel
technologies.
Slowly,
we
are
migrating
piece
by
piece
to
the
newer
technology
where
it
makes
sense.
We
started
off
with
with
one
Oracle
database
back
in
1998,
and
now
we
have
about
hundreds
of
Oracle
databases,
which
runs
the
all
the
major
of
functionalities
and
features
of
the
ebay.com
site.
A
Our
philosophy
of
running
oracle
database
is
pretty
simple
and,
and
that-
and
this
is
helper-
helped
us
to
basically
introduce
newer
database
technologies
in
the
infrastructure.
We
can
seamlessly
transition
from
one
database
technology
to
the
to
the
other
technology.
Due
to
the
way
we
have
designed
our
application
and
the
abstraction
layer.
So
what
what
we?
The
way
we
run
our
Oracle
infrastructure
is
that
we
use
it
as
a
plane
data
store
which
provides
durability
and
high
concurrency
and
the
high
throughput,
and
we
don't
use
any
of
the
relational.
A
The
true
relational
feature,
the
foreign
keys
and
then
and
the
stored
procedure
and
all
those
all
that
which
comes
with
the
traditional
or
DBMS.
Our
application
is
predominantly
breed
based.
We
have
eighty
percent
read
and
twenty
percent
right
and
and
and
or
all
most
of
our
reads,
our
primary
key,
lookup
type
and
so
far
the
Oracle
layer
has
scaled
very
well.
For
us,
we
have
upward
of
about
140
billion
in
a
day
on,
and
that
turns
on
hundreds
of
oracle
server.
A
A
So
what
we
have
found
is
that
the
each
technology
has
its
pros
and
cons
and
we
evaluate
that
technology
based
on
its
merit
and-
and
then
we
see
our
use
case
and
based
on
that
use
case,
we
decide
on
which
technology
to
use.
So
what
we
have
found
with
the
MongoDB
is
that
MongoDB
is
good
for
very
real
intensive
use
cases
where
you
need
a
thousands
of
reach
from
a
single
nodes
and
and
and
where
the
the
active
data
set
fits
nicely
into
the
memory.
A
A
So
the
basic
foundation
of
ebay
databases
is
logical,
logical
host
and,
and
we
we
divide
our-
are
we
basically
divider
application
into
multiple
logical
host
and
the
reason
for
doing
that
is
to
provide
a
transparency
of
the
physical
database
layer
to
the
application.
So
what
we
have
is
it
is
one
physical
having
multiple
logical
hosts
map
to
it,
so
you
can
think
about
it
as
a
as
a
shards
of
a
of
an
application
which
are
mapped
to
which
can
map
to
a
single
physical
host
or
it
can
map
to
multiple
databases.
A
So
this
this
is
a
foundation
for
our
for
scaling
or
database
infrastructure.
The
once
we
have
split
our
application
into
multiple
logical
hosts.
We
can
basically
start
off
with
a
small
number
of
physicals
and
then,
as
our
application
starts
scaling,
we
can
move
this
logical,
cross,
various
physical
without
any
any
change
to
the
application.
This
transparently
helps
us
scale
our
application,
without
actually
involving
our
developers
to
make
any
changes
to
the
application,
and
we
also
have
multiple
rate
copies,
where
a
single
logical
host
cannot
basically
sustain
the
traffic
from
from
the
application.
A
We
periodically
archive
the
data
from
from
our
site
to
basically
keep
our
oltp
data
platform
smaller,
and
we
have
about
400
million
active
items
ons
on
site,
which
is
which
is
huge
and
and
and
we
have
archival
of
all
the
all
the
items
played
items
and
related
data
on
the
archive
database
and
with
the
emergence
of
the
new
sequel
technology,
the
shouting
and
the
scalability
has
become
easier.
You
can
basically
add
the
capacity
and
then
the
software
does
the
magic.
A
As
I
said,
logical
host
is
a
is
a
foundation
of
our
data
model
where,
where
we,
when
the
application
is
designed,
we
assign
a
logical
host
to
it.
If
there
is
existing
logical
host,
then
it
basically
uses
that,
if
not,
we
played
a
new
logical
host
and
then
we
have
a
mapping
configuration
where
we
map
this
logical
host
to
an
existing
physical
hosts
are
are
to
a
newer
physical
host.
A
There
are
many
advantages
of
using
logical
host,
especially
in
the
in
in
the
previous
world,
when
we
had
a
big
beefy
hardware
and
and
and
you
are
mapping
multiple
applications
to
the
same
physical.
If,
if
you
had
that
configuration
hard-coded,
then
anytime,
you
basically
change
any
configuration.
You
have
to
basically
make
the
application
changes
anyway,
because
of
that
abstraction
we
have
been
able
to
freely
move
the
logical
from
physicals
to
physicals
and
depending
on
the
various
requirements.
A
It
has
also
helped
with
the
with
the
failover
scenario,
so
there
are
many
logical
swear.
We
have
that
failure
functionality
so
where
our
dialer
application
code,
dial
application
layer
has
that
intelligence
to
to
move
their
traffic
from
a
primary
primary
logical
to
its
logical
when
the
primary
logical
is
not
available-
and
this
is
prior
to
dust,
a
lot
of
scalability
for
our
read
traffic,
where
the
right
host
goes
down
and
and
and
but
the
reeds
continues
on
the
on
the
failover
host
and
then
the
logical
host.
A
We
are
able
to
quickly
turn
off
the
less
critical
site
features
in
the
event
that
our
demand
goes
our
capacity
projection,
especially
on
the
holiday
traffic
season.
There
are
couple
of
sundays
where
we
see
this
spike
of
traffic
and-
and
we
do
a
our
best
effort
to
to
basically
project
that
demand
and
then
right,
size
or
infrastructure,
for
that
there
are
certain
times
where,
where
we
need
to
basically
take
a
turn
off,
less
critical
site
feature
to
basically
not
impact
the
main
site
functionalities.
A
So
so
this
is
the
this
is
the
way
we
have
designed
our
always
available
breed
databases.
So
we
have
a.
We
have
a
we
have
a
load
balancer,
which
is
a
software
load.
Balancer
is
called
database
access,
layer,
a
load
balancer
which
basically
directs
the
traffic
to
one
of
the
many
rate
copies
of
the
of
the
right
database
and
we
have
in
each
year
center.
A
We
have
this
set
of
read
copies,
so
our
data
layer,
which
it
translates
that
access
to
that
logical
host
in
each
day,
I
said
this
dares
enter
separately
to
its
own
localized
set
of
read
databases,
and
this
provides
a
low
latency,
read
response
to
the
all
the
data
requests
from
from
a
given
pool-
and
this
is
this-
is
the
model
we
have
for
Oracle.
As
you
see,
we
have
a
read/write
oracle
database
and
then
we
have
a
multiple
read
copies,
but
this
is
a
general
architecture
which
we
follow
for
for
this
way
of
scaling.
A
A
There,
when
we
started
off
with
this
logical
host,
we
thought
that
we
have
broken
down
our
functionality
into
smaller
chunks,
but
we
were
wrong.
We
soon
found
out
that
even
a
sudden,
logical
for
us
to
be
need
basically
capacity
bigger
than
what
what
what
is
provided
by
a
by
a
single
machine.
So
we
further
broke
down
those
logical
post
into
multiple
mods,
and
then
we
basically
had
split
that
into
multiple
mods
and
then
we
map
each
each
Mart
to
its
a
set
of
physicals.
A
So
now
we
can
basically
split
up
or
logically
spray
the
logical
host
of
too
many
logic,
many
sub
logical
host,
using
this
architecture.
Initially,
when
we
designed
this,
we
started
off
with
the
mod
10.
It
was
easier
on
eyes
to
basically
find
out
that
if
user
a
user
ID
ends
with
0,
then
we
basically
assign
it
to
mod
0
and
if
it
ends
with
for
it,
it's
get
signed
to
mod
4.
But
we
found
out
that
from
a
from
when
you
had
us
scale
it,
it
does
not
provide
the
even
scalability.
A
We
started
off
with
10
logic
technological
host,
and
then
we
split
into
five
and
five
and
map
five
to
the
first
physical
52
second
physical.
Then
we
found
out
that
now
share
split
further.
Now
we
have
this
uneven
configuration
now
we
have
to
basically
do
23
32
math
or
we
have
to
do
to
22
22.
So
we
change
that
later
on
to
mark
12,
which
provides
us
a
distribution
if
it
starts
splitting
it
across
multiple
physical
hosts.
A
But
again,
this
has
a
drawback
that
once
you
have
once
you
have
decided
a
mod
scheme.
You
cannot
change
it
later.
So
what
if
you
find
out
that
we
have
12
physicals
with
the
marked
well
approach,
and
then
we
run
out
of
capacity
on
those
12
physical?
Now
we
cannot
split
it
any
further
without
making
some
changes
to
application
on
our
data
to
redistribute
that.
So
so
this.
A
Basically,
we
learn
some
lessons
from
this
approach
of
scaling
our
database
infrastructure
and
then
we
moved
on
to
a
range
based
model,
and
this
is
the
way
we
have
split
our
item
database,
which
is
where
or
when
a
user
goes
to
ebay.com
site
and
listen
item.
This
is
the
way
we
basically
assign
a
particular
database
host
to
the
seller,
who's
listing
your
item
and
then
it
provides
us
some.
The
application
I
mean
to
basically
find
out
we're.
That
item
is
based
on
the
item
id
provided
by
the
application.
A
Now,
how
do
you?
How
do
we
follow
from
a
user
perspective?
Which
item
holds
that
user
goes
to
so
we
have
another
way
of
actually
doing
that
mapping.
It's
something
called
user
look
up
where
we
have
for
each
user.
We
have
a
record
which
basically
points
to
wear
that
for
that
particular
user,
where
the
item
host
is
and
and
so
when
a
user
is
or
seller's
listing
an
item.
Basically,
application
goes
to
this
lookup
table
find
out
that
which
is
the
item
host
for
this
particular
seller
and
then
goes
to
that.
A
That
particular
item
hose
and
then
then
basically
use
that
sequence
generated
ID
to
basically
select
the
item
ID
for
that
item
and
then,
with
this
item
ID
can
it
can
be
easily
accessed
from
the
application,
because
application
has
that
mapping
of
all
the
item
ID
ranges.
So
this
is
scaled.
Our
item
infrastructure.
A
And
then
a
large
purse
enough
for
our
access
is
based
on
a
user,
that
what
is
the
items
listed
by
a
user
or
what
is
the?
What
are
the
preferences
for
this
user
or
what
is
the
personalization
data
for
this
user?
So
for
those
type
of
data
where
the
key
is
always
the
user
ID
we
have,
we
have
something
called
a
lookup
based.
We
are
splitting
ordered
ice,
called
user
split
where
we
have
this
a
mapping
database
where
we
have
a
mapping
of
each
user.
To
this.
A
This
logical,
logical
host,
where
the
defining
query
is
always
based
on
that
user
ID.
So
we
have
tens
of
these
logical
host
where
the
defining
key
is
always
based
on
user
ID
and
an
application
goes
to
this
user.
Lookup
database
finds
out
where
that
user
is
and
from
from
there.
It
basically
goes
to
that
dal,
basically,
with
data
access
layer,
translate
that
to
a
physical
database
and
an
application
goes
to
that.
A
To
query
that
data,
and-
and
this
is
this-
is
also
infinitely
scalable
model,
because
we
can,
we
can
basically
add
a
new
new
host
and
then
basically
change
the
either
migrate.
The
data
from
the
existing
databases
to
this
new
host
or
we
can
start
creating
the
new
users
on
this
new
horse.
So
this
is
again
another
foundation
for
scaling.
Our
read
in
frustration
processor,
which
has
gone
from
less
than
50
50
billion
sequel
calls
calls
to
150,
plus
billion
sequel
calls
in
last
several
years.
A
But
again,
all
this
thing.
Actually
they
were
designed
when
there's
no
sequel
options
were
not
available
and
lot
of
these
things
actually
are
now
natively,
provided
by
no
sequel.
The
way
the
the
data
is
distributed,
cross
multiple
node,
the
mapping
of
the
data,
all
that
is
basically
natively
handled
by
the
the
database.
Software
itself.
A
What
are
the
challenges
of
the
traditional
art
of
unisys?
So,
although
we
use
very
basic
features
of
the
relational
databases,
we
use
it
only
to
basically
store
data,
we
have
simple
primary
killa
cups,
we
don't
use
primary
keys,
stored
procedures,
functions,
etc,
but
the
volume
is
so
so
huge
that
that
that
we
sometimes
beta
the
performance
penalty
of
maintaining
the
acid
features
of
the
relational
databases.
As
I
said,
the
traditional
image
does
not
have
the
natives
shouting
and
replication
features
in
built
into
it.
A
A
We
have
a
higher
cost
of
commits
due
to
the
fact
that
it,
the
traditional
our
viewers,
does
not
provide
the
flexibility
in
terms
of
how
the
data
gets
written
from
the
memory
to
disk,
and
this
is
where
I
think
the
newer
technologies
like
Cassandra,
comes
into
play
where
we
can.
Basically,
we
are
not
off
control
over
how
the
application
interact
with
the
database
and
lot
of
control
over
how
the
database
basically
perform.
B
B
One
this
one:
okay,
okay
before
I,
to
really
start
a
hope
of
pologize
about
my
work
yesterday
morning
was
so
cold.
Basically,
I
got
a
stake.
Had
a
half
favor
over
last
night
anyway,
just
one
to
apologize,
make
sure
you
know
every
people
can
hear
me
clearly
before
we
go
to
the
Cassandra
itself,
just
one
of
about
a
few
words
about
myself
about
my
experience,
how
to
I
get
to
the
no
secret
world.
I
start
with
our
dbms
almost
more
than
20
years
ago.
B
So
in
the
beginning,
when
I
started
at
Oracle,
fire
I
still
remember
when
I
first
time
in
Seoul
order,
five
I
got
about
more
than
twenty
floppy
disk
it
hard
it
first.
It
took
a
very
long.
We
do
start
to
install.
Secondly,
if
you
have
a
one
with
the
floppy
with
bad
in
the
middle,
you
have
to
start
over.
So
that's
very
cute
me.
That's
why
you
know
I
keep
still
keep
it
running
after
so
many
years.
B
I
still
remember
all
these
things
for
passed
over
almost
decade,
I
primary
were
castle
or
over
DBA,
and
also
work,
and
my
microsoft,
SQL
server
and
also
some
of
the
Mexican
stuff,
as
you
can
see
most
of
the
time
and
working
on
the
database
I'd.
These
are
all
traditional
database
and
I
worked
most
of
work
it
down
as
a
double
click
yahoo
and
include
in
the
last
few
years,
with
the
no
single
database
become
a
more
and
more
popular
I
realized.
B
B
Let's
talk
about
why
you
know
we
talked
about
the
cassandra
in
the
ebay
there's
a
few
things
give
us
a
benefit.
No,
instead
of
the
traditional,
a
DBMS.
Whilst
we
keep
hurting
from
the
developer,
is
they
always
come
down
the
DBA
to
make
a
schema
change,
a
creditable
add
a
new
column
before
they
can
release
their
code
or
feature?
This
is
a
not
very
efficient
and
also
at
of
a
DBA,
the
always
in
a
cannibal
headache
to
push
the
schema
change
in
the
p'rushim
arm.
B
One
problem
we
have
here
is
a
you
know
for
large
system.
You
need
application
some
time
to
put
some
schema.
Otherwise,
you
can
now
get
a
lock
release
an
object.
Secondly,
when
you
make
us
changes,
it's
very
easy
to
change
the
underneath
extrusion
plan,
so
there
could
severely
improve
it
impact
of
our
system
of
performance.
What
can
I
dad
is
kind
of
a
schema-less
schema,
so
it
gives
us
a
developer
flexibility,
10,
angry
and
change.
You
know
whatever
the
column
is
there
and
also
for
TBA.
It's
definite
resolve.
B
One
of
the
issue
with
no
longer
worried
about
you
know,
push
the
schema,
change
and
call
some
negative
impact.
Also
consider
whether,
based
on
our
benchmarking
with
we
found
out,
it
has
good,
very
good
performance
for
both
read
and
write.
Even
Cassandra
in
the
beginning
was
well
known
for
the
high
performance
right,
but
the
in
the
regional
releases,
as
you
can
see,
based
on
from
the
compression
from
the
level
compassion
and
also
move
a
lot
of
object
of
the
Java
heap
space.
So
they
give
us
a
lot
of
improvement
in
Rita
Brown.
B
Once
you
relax
on
the
most
is
the
horizontal
linear
scalability.
If
you
think
about
a
normal
side,
when
you
try
a
new
skill
beyond
you
know
one
physical
machine,
so
we
can
use
the
oracle
RAC,
but
the
skill,
the
skill
factor
will
be
definitely
is
not
as
lenient
as
well.
So
when
you
go
two
for
one
node
to
atone
to
know,
drag
you,
you
are
at
the
best
Republicans
see
seventy
percent
increase.
B
If
it
goes
from
two
to
three
nodes
ran,
then
the
scale
factor
will
reduce
to
two
point:
five
or
even
less
same
thing
for
the
other
Mexican
side
of
same
thing
and
the
manual
center,
so
only
Cassandra
from
where
our
experiences
can
cables
or
horizontal
linear
scalability.
Let's
make
our
life
much
easier,
especially
for
capacity
planning.
Whenever
we
see
the
capacity
increase
we
can
easily
to
allocate
the
more
note
add
a
model
to
the
system,
and
we
know
how
many
loads
we
need.
B
We
know
how
much
number
you
can
gather
from
the
loads
one
feature
also
coming
from
Cassandra
is
a
building
high
availability
for
ebay.com
kind
of
side.
We
definitely
want
provide
a
have
ability
in
case
of
the
machine
failure
or
he
dated
cinder
baby
Cassandra
itself,
because
it
is
a
peer-to-peer
architecture.
First
easily,
mated,
a
single
point
of
failure.
We
no
longer
worry
about
a
single
notes.
Fear,
even
though
the
field
for
like
hours
or
days
is
basically,
we
are
not
impacted
picture
basically
for
Humanity.
B
Another
thing
is:
when
we
do.
The
multiplication
center
department
has
to
be
always
through
ebay.
It
gave
us
a
automatic,
chill
load.
Balancing
because
of
as
you
see
from
a
previous
slide,
we
have
a
application
server
in
multi
data
centers
when
we
have
Cassandra
T
problem
to
market
center,
which
always
can
enforce
those
application.
Server
will
talk
to
the
local
casino
knows
that
gamers,
where
it
could
really
write
latency.
You
never
need
to
go
beyond
same
data
center.
B
So
another
thing,
the
like
an
automatic
asharti.
This
is
a
pen
nature
itself.
The
data
will
randomly
distributed
across
the
margin
notes,
so
you
don't
have
to
design
as
implemented
complete,
complicated
sharding
on
your
own
another
benefit
we
carried
for
the
data
analytics
function,
just
like
a
taste
and
in
the
price
it
comes
with
the
building,
hedule
finality
and
also
solar
from
identity.
This
will
give
us
a
flexibility
result
from
a
previous
job
with
other
builders
separate
the
Hadoop
cluster.
B
So
also
you
can
eliminate
all
the
etl
one-fifth
last
feature
will
tell
about
it
from
the
TB
perspective,
if
you
think
about
in
other
databases,
when
the
data,
if
you
have
a
data
after
you
fire,
the
only
way
you
can
do
is
writing
some
of
my
kind
of
script
to
delete
the
data.
First,
you
can
reduce
the
story
using
secondary.
You
can
improve
the
performance,
but
Cassandra
just
using
TTL
can
easily
implement
this
feature.
B
Let's
talk
about
the
Cassandra
a
bit
using,
we
started
Cassandra
about
two
years
ago.
We
used
Apache
Cassandra,
also
that
they
stay
in
the
price
when
we
have
a
project
required.
You
know
when
the
developer
want
to
use
the
analytic
function
to
the
MapReduce
job
then
lets
the
place.
Women
we
provide.
The
data
in
the
price
ebay
is
a
primary
is
a
java
application.
B
We
are
using
the
mostly
the
Hector
client
in
the
building
into
the
our
common
access
layer.
We
can
call
it
their
data
access
layer.
So
far
we
have
a
more
than
10
/
camp
production
cluster
on
the
Cassandra
proposed
Cassandra
and
the
DSE.
Overall,
we
have
a
more
than
100
apparel
nodes
and
among
these
nodes
we
have
a
more
than
200
pack
of
terabytes
storage,
a
provision
in
terms
of
the
user
data.
We
have
already
have
a
more
than
one
terabyte
across
multiple
dates.
B
B
B
Here's
about
you
know
the
stack
of
the
software
and
hardware.
We
are
using
the
Cassandra
over
the
time.
We
have
a
1
point,
0
100
one
and
also
we
recently
started
using
one
point
to
the
TS
inside.
Where
is
the
one
few
cards?
So
we
have
in
the
tsc
sleep
on
the
whole,
the
piece
of
under
operations
stiff
operands
inside
we
are
using
the
Red
Hat
Linux.
B
We
used
to
the
sixth
column,
one
sequin
to
the
latest,
lead
0613
and
Hector
client
that
we're
using
1.1
ebay
with
so
far
we
haven't
used
the
virtual
machine
yet
because
the
our
scale,
because
it's
very
hard
to
implement
no
such
a
volume
of
data-
and
we
have
butter
for
majority
of
them-
were
using
the
physical
hardware.
The
first
two,
as
you
can
see,
we're
using
the
HP
dl380.
B
The
water
hydrate
is
basically
just
like
a
fan.
Notes.
General
summation
in
the
yesterday
kingdoms.
Those
machines
are
pretty
powerful
machine,
have
a
12-core
international
60,
180
gb
ram
and
either
1.1
thoroughbreds
of
five
points
headers
and
after
the
mirror
in
regarding
to
the
flash
memory,
so
we
are
using
the
sofa
we're
using
the
shared
track
memory.
Basically,
we
have
a
vendor
called
violin.
We're
using.
There
are
6
260
600
flash
memory
for
the
smallest
cases
potential.
We
are
also
working
on
vm
solution.
B
B
Ok,
I
want
to
share
some
number
about
ebay
scale,
how
much
data
were
italian
model
here.
First
thing
is
a.
We
have
a
three
days:
syndromes
from
majority
application.
We
deployed
to
data
center
because
that's
the
primary
center,
our
application
server
there
so
far
production
cluster.
We
have
mullen
teen,
we
have
a
modern,
100
physical
notes,
total
dataset,
more
than
hundred
terabytes
largest
cluster
started
to.
B
B
In
case
we
have
a
more
than
40
terabytes
in
single
column,
family,
one
of
the
biggest
one
largest
carbon
family
/
number
of
roles
of
perky's,
either
more
than
thirty
two
billion
kids
in
the
single
column,
family,
the
biggest
cotton
family,
based
on
the
rate,
is
it
that
the
1.6
billion
per
day
in
terms
of
the
rice,
the
business
account
family?
We
are
doing
more
than
6
billion
per
day.
Regarding
on
the
right
side,
we.
B
Do
so,
let's
talk
about
living
on
the
department
side
because
of
all
the
email
machine
will
behind
a
firewall,
so
it's
very
hard
for
us
to
get
it
back
in
the
front
row,
for
example
from
this
tag
a
PM
servers.
So
what
do
we
do?
Is
we
build
our
own
up
here
and
packaged
and
put
our
own
repository
center
on
the
server?
So
we
can
still
run
the
yum
install
to
the
EV
for
integration
and
for
the
upgrade
we
build
a
piano
for
both
cassandra
and
also
the
DSE.
B
The
killers
are
very
consistency.
We're
using
the
same,
build
same
battery
across
all
the
machine
as
part
of
the
packaging.
We
also
choose
of
a
parameter
in
general,
we
tune
a
few
on
the
colonel
said
and
also
be
under
Cassandra
set,
even
seeing
the
dissipate
with.
Let
us
see
example
about
the
world.
We
have
down
under
Colonel
opportunity
and
also
GBM
hip
side
and
also
on
the
compaction
side,
and
the
Hector
said
one
thing:
I
want
to
talk
about
it
on
the
compassion
said:
either
we
started
with
the
set
here.
B
The
compaction
will
also
try
to
level
compassion
in
the
100
0
80
release,
but
we
at
that
time
we
have
very
bad
a
performance
issue,
especially
when
we
have
a
one-hour
number
of
a
sensible
girl
around
200,000.
So
at
that
time
this
that
particular
Kaiba
someone
just
patch
to
resolve
that
problem
in
the
new
one
pan
to
release
we're
going
to
enable
the
level
compassion
for
most
of
the
use
case.
B
Well,
we
are
going
to
talk
about
it
about
scalability.
We
did
a
lot
of
print
marking
the
reason
to
break
marking
is
we
want
to
avoid
the
production
surprised
you
don't
want
to
deposit
cluster
about
feature?
Then
you
realize
the
system
cannot
media
requirement.
So
what
do
we
do
is
before
we
on
board
any
major
application?
We
ask
the
developer
to
run
the
load
and
performing
testing
in
the
pre
production
environment,
which
we
have
also
a
node
cluster.
So
families
are
loading
you
mark,
load
and
repeat
testing.
B
We
can
easily
to
project
whether
how
many
nodes
we
needed
to
support
the
user
traffic
in
general
for
the
cassandra
is
most
people
singing.
Ok,
it's
very
easy
to
scale
back
scale
out
when
you
see
the
traffic
increase.
What
it
does
just
add,
more
notes
for
us
use
cases
is
a
sometimes
we
do
finding
each.
In
some
cases
we
screwed
to
the
scale
up.
First,
why
you,
damn?
B
How
result
is
either
networker
saturation
machine
we
use
started
early,
is
only
have
a
single
click
network
card
instead
of
Adam
or
physical
nodes,
which
is
a
more
expensive
that
clingy
network
hard.
What
we
do
is
we
replace
the
machine
with
a
beta
mesh
with
the
different
machine,
have
a
tongan
network
card,
so
in
Stanford
purchasing
more
expensive
physical
nodes,
we
can
easily
to
increase
the
network
traffic
by,
like
a
rounded
like
a
one-thousand-dollar
yourself,
a
ten-thousand-dollar.
B
B
B
Few
notes
on
the
operational
side,
because
a
lot
of
use
cases
is
a
read-only
logging,
so
in
that
case,
with
remote
run
the
routine
repair
as
the
most
people
know,
that
repair
is
very
intensive
operation
only
we
run
a
repair
once
a
machine
down
for
whatever
reason
for
more
in
the
handle
of
the
window.
In
that
case,
when
we
bring
machine
off,
we
run
the
repair
make
sure
we
can
get
the
meat
data
from
the
remaining
nodes
we
use
with
interminable
Clank
a
tradition.
We
don't
give
them
an
actual
physical
host
name.
B
What
we
do
is
the
recency
name.
This
cname
can
easily
transfer
the
between
the
physicals.
What
the
benefit
of
altering
this
is
well,
when
the
Machine
get
it
replaced.
Normally,
it
has
a
different
IP
address,
a
different
host
name
with
a
cname.
We
can
easily
move
the
signal
from
existing
one
to
the
new
one
and
without
changing
cancel
configuration.
This
was
the
benefit
under
regarding
to
the
TTL
data
once
evening
why
you
should
be
headed
early.
B
The
moving
to
the
capacity
planning
we
didn't
include
the
cheesy
crispier
grease
in
the
beginning,
for
example,
especially
for
the
data
with
the
short
ppl
one
problem
we
had
a
lot
early
is
we
have
a
use
case
with
the
seven-day
TDL.
We
keep
seeing
the
data
increasing
beyond
our
initial
projection
later
on,
with
realized
for
the
g
series
period
p
for
10
days.
That
means
we
actually
store
17
days
of
data
install
seven.
So
if
you
have
a
similar
case,
you
need
to
bring
to
the
put
a
capacity
planning
either
you
need
to
reduce.
B
B
Another
usually
see
before
you,
the
bootstrap
here
most
time
is
no
whatever
at
a
new
node.
We
with
our
replacement.
We
do
the
bootstrapping
feel
the
time
we
did.
We
realize
you
know
prescribing,
didn't
work.
First
time
didn't
work
second
time,
the
help
keep
training
until
it's
succeed.
So
if
you
ever
have
this
case,
just
try
a
few
more
times,
it
won't
work
regarding
the
real
cash
most
people
already
know.
Luke
has
only
worked
for
worse,
small
dataset.
B
B
Still
on
the
operational
side,
monitoring
is
very
important
for
the
Prussian
system.
We
have
a
multiple
ways
to
monitor
Cassandra.
First,
we
integrated
with
our
network
operations
center,
which
is
operate
from
the
post,
US
and
China
standing
by
24.
We
interviewed
Cassandra
with
listen
love
as
long
as
along
as
along
with
the
Oracle
my
circle
mongodb
together.
This
is
the
first
year.
Whatever
there's
a
problem.
There
is
a
to
install
click,
the
containers
status
up
or
down
in
almost
real
time.
Every
few
seconds
is
a
query.
B
Cassandra
notes.
We
went
our
own,
my
small
HTTP
server,
so
it
pinned
up
hvs
d
mail
every
few
seconds
to
see
it's
Cassandra
up
or
down.
So
if
you're
at
machine
ever
have
a
problem
in
a
few
seconds,
you
can
notice
on
the
North
screen.
I'll
show
you
in
the
infinite
we
also
for
the
low
latent
priority
event
non-critical
one.
We
also
have
a
self-developed
that
email
alerts
for
the
pending
compaction,
capitulation
frequency,
etc.
B
We
also
use
a
dis
data
of
enterprise
of
absent
or
enterprise.
First
thing
we
can
see
have
a
high-level
dashboard
for
more
than
one
cluster
at
same
time.
In
one
place,
second
thing
that
we
are
also
used
for
certain
administration
now
operation,
for
example,
to
a
ruling
restart
it's
a
very
coming
in
this-
is
a
screenshot
of
the
nest
of
center
I
have
to
mash
up
a
few,
the
number
of
nodes
or
step
on
the
name.
B
This
is
our
not
real
time
monitoring
would
so.
This
is
the
only
the
Cassandra
side
screen
in
the
real
world.
It
can
also
show
the
Oracle
medical
staff,
so
this
is
an
easy
math
formula,
one
of
the
controller
15
notes
to
wonder
when
the
colonel
knows
to
actually
fill
the
holes
and
about
load
area
and
the
sessions
so
one
when
the
one
machine
have
a
performing.
B
The
problem
is:
first,
it
started
at
a
yellow
in
the
top,
which
means
a
container
of
soup
function,
but
if
I
take
a
longer
time
to
pin
the
Cassandra
second
case,
either
even
a
machine
it
down.
So
this
machine,
the
machine
will
pop
out
on
top
as
a
red
color.
That
means
I
couldn't
connect
to
the
container
notes
from
here.
B
Just
a
few
use
cases
on
the
ebay,
we
have
a
Randy
intensive
metric
use
cases
long,
a
metric
collection.
We
also
have
a
reading
intensive.
We
call
a
recommendation
backhand
these
two
out
go
over
in
the
next
few
slides.
We
also
have
a
mixed
workloads
it
coming
to
the
production.
We
call
personalization
here
in
the
screenshot
of
our
metric
clashing
picture.
What
is
the
what
it
does
is
the
storing
and
serving
real
time
metric
from
tens
of
thousands
ebay
devices
that
include
the
server
sweet,
strawberry,
etc.
B
Those
can
be
used
for
the
dashboard
in
reporting
and
also
the
trade
automation,
for
this
particular
caster
is
to
have
a
16
node
102
to,
and
we
only
have
a
to
copy
of
data
across
to
this
center.
The
reason
we
cannot
most
katie,
we
have
to
copy
the
because
of
design,
and
as
for
this
particular
case,
we
have
a
model,
600
million
kids,
and
every
day
this
is
a
most
already
intensive
case.
It
does
more
than
six
billing
rights
include
the
tutor
also
to
the
raw
app
within
the
Cassandra.
B
This
is
a
really
intensive
case
that
we
call
a
recommendation:
backhand
ebay.
You
know
we
have
a
model:
few
hundred
meaning
users
and
also
every
day
there
are
multiple
hundred
several
hundred
meanings
item
listed
and
every
day,
there's
a
lot
of
billions:
billions
user
activity.
For
example,
when
you
Peter
Adam,
we
watch
Adam
when
you
actually
buy
them
all
these
translated
to
a
event
at
the
backhand.
So
what
it
does
basically
this.
B
B
For
these
use
cases,
we
have
a
two
different
cluster
while
have
a
32
knows
another
one
have
48
nozzle.
So
totally
we
have
a
40
notes
combined
over
with
the
English
cluster.
We
have
a
model
20
billing,
please,
and
we
have
a
story
to
tell
buyers
SSD,
because
the
viral
reading
turns
image.
They
have
a
high
drama
on
the
reader
latency.
This
is
the
above
600,
meaning
writes
per
day
about
three
bidding
reads
per
day.
B
Lasting
want
to
talk
about
is
a
vendor
support.
It's
very
critical
for
a
company
like
a
eb
side
to
have
a
good
partner
when
we
adopt
a
new
technology.
We're
working
for
you
know
to
have
a
taste
act
as
a
partner
where
we
start
using
the
Cassandra
under
in
the
past
film
a
few
years
we
want
to
send
you
know
the
dates
that
cut
a
service
team.
They
always
give.
You
know
when
we
ever
open
use
cases,
they
always
give
very
good.
You
know,
services
provide
a
solution
or
either
provide
a
workaround.
B
B
B
B
Another
factor
we
want
to
scale
better
is
a
separate
story
from
compare
notes.
This
givers
are
more
flexibility
to
scare
a
particular
component,
but
especially
for
the
story.
While
we
don't
have
to
decide
the
number
of
nodes
based
on
of
storage
in
anymore,
we
want
two
morrow
missions.
We
want
to
do
more,
like
a
soft-serve
model,
just
like
a
PD
right
now,
where
TBE
install
the
diploid
cluster,
so
PD
can
use
going
forward
that
we
want.
The
PV
can
have
click
a
few
buttons
in
few
minutes,
maybe
hours
you're
going
to
have
a
ready-to-use
cluster.
B
We
also
want
to
basically
have
a
more
you
know,
Rumble
kind
of
a
practice,
so
we
can
leverage
knowledge
across
the
different
teams.
This
will
kind
of
wrap
up
for
our
presentation
before
we.
If
you
have
any
Q&A,
we
can
always
be
here
for
few
more
minutes.
You
can
ask
any
question:
we
want
Thank
You
time
and
also
ebay,
always
looking
for
the
no
secret
talent.