►
Description
Speaker: Joseph Streeky, Manager of Search Framework
Company: Career Builder
Today we use Cassandra along with RabbitMQ and Sinatra to feed 5,000 documents a second across our 600 solr instances. We will talk about how we manage a 21 node, 59 TB Cassandra ring and how we use this to send updates to all of our Solr clusters. We have setups in both our private datacenter and AWS.
A
Okay,
so
where's
our
tongue,
about
how
we
use
a
central
and
double
the
technologies
to
feel
are
a
variety
of
stellar
clusters
and
relatively
large
scale.
So
little
of
me,
my
name,
is
Ricky.
I
will
be
measure
of
the
search
framework
development,
so
we
own
the
plural,
servers
and
data
us
and
feeding
and
searching,
and
all
that
this
is
josh
smith,
our
database
administrators
and
he
specifically
blackness.
A
So
so
a
little
bit
about
thrillers
of
company
for
those
who
do
go
in
human
capital
solutions
find
Tennessee
Walker.
We
have
a
little
points22
unique
visitors.
Every
month
we
have
more
three
thousand
alerts:
post
more
than
a
million
jobs
per
month,
our
main
site.
We
have
operational
Brendan's
in
the
ad
states,
your
Camtasia,
so
in
total
lacrosse
book
still
terrific
its
acquisitions.
We
have
presidents
in
developing
countries
are
on
vote
so.
A
Simply
search
of
her
other,
which
is
what
I'm
concerned,
that's
have
been
better
jobs.
Each
month
we
have
60
million
searchable
resonates
above
the
United
States
and
abroad
do
about
5,000
search
servers.
Individual
solar
new
builds
across
both
our
private
data
centers
and
AWS.
We
have
thousands
of
deeply
dynamically
generated
certain
deserts,
so
each
when
a
customer
creates
a
document
they
decide
at
the
type
of
feeding.
Let
that
addiction
to
look
like
I'm,
so
I'm,
really
powerf--
is
your
fault.
A
B
A
A
B
A
A
A
There
happens
on
solar,
and
that
was
through
my
step,
so
it's
not
a
little
about
other
stuff,
so
we
are
setting
up
our
feet
and
platform.
We
have
some
specific
requirements.
We
want
to
feed
more
than
a
thousand
documents.
A
second
and
then
requirement
came
from
the
concept
we
want
to
do
a
refeed
all
of
our
resume
corpus,
which
is
about
60
million
documents.
It
is
edge,
and
so
that
was
where
we
start.
We
end
up
getting
significantly
higher
coats
in
that,
but
that
was
like
our
initial
encounter
case.
A
A
You
have
million-dollar
sand
and
you
want
a
bigger
standing.
There's
often
some
two
million
dollars
and
f
is
not
a
particularly
appealing
investment.
We
wanted
to
be
high
all
of
our
resources
to
be
highly
available,
because
feeding
is
infinitely
corridor,
our
business
does
and
we
want
to
be
easily
deployable
to
multiple
locations
right.
So
we
want.
A
Out
to
AWS
smoother
and
we're
going
to
the
process
of
transitioning
a
lot
of
our
resource,
a
bonanza
lies,
so
here
are
some
technologies
that
power
of
a
step
to
make
use
of
us,
Sinatra
and
Ruby
on
top
of
engine
X
as
a
very
small
web.
App
to
how
all
this
logic,
music
in
Sandra,
is
our
game
store,
grab
and
Q's
our
community
structure?
A
So
you
have
a
visualization
of
how
our
process
works
right.
We
get
data
from
sources
like
sequel
Edna.
Do
they
come
to
our
feet
and
staff
that
feeds
back,
which
is
Sinatra
rice
to
the
center?
That's
correct,
then
we
will
process
in
tier.
That
is
watching
the
radical
Hughes
and
as
soon
as
it
sees
a
message,
speaks
about
it
works
it.
We
do
a
couple
through
publican
stages
of
minutes
that
allow
us
to
flow
transform
data,
that's
right
to
the
pressure.
B
A
A
We
then,
then
goes
on
to
assuring
step
if
you're
ensuring
step,
we
decide
where
the
docket
is
going
to
be
rabbited.
There
are
a
couple
different
approaches
to
take
ceramic
Tom
summer.
Bigger
clusters
behind
all
of
our
surgeons
are
streaming
across
multiple
charts,
so
we
have
to
always
route
the
same
got
into
the
safeguarded
wise.
You
are
not
going
to
we're
problem
silver,
so
that's
one
of
the
possible
Charlotte's
friends
that
we
make
yourself
another
one.
A
B
A
Property
together,
we
totally
five
minutes
in
back
for
documents
that
have
to
be
gathered
together.
Mostly
pistols
really
prefers
them
to
match.
Feeding
on
our
experimentations,
we
normally
want
about
50
documents
per
match
for
solar.
We
can
optimally
happen
within
cinematics.
Install
opting
out
is
the
Mafia
verify
you,
the
very
few
weights
about
10
minutes.
We
ought
to
commit
the
source
of
your
five
minutes.
A
So
it's
about
10
minutes
because
in
checks
to
see
if
the
batch
success
limited
into
solar,
it
has
no
real
in
we
specific
design
has
no
real
way
to
tell
we
don't
let
go
like
spray
like
solar,
like
it
love,
it
differs
like
that.
We
just
put
the
document.
If
you
find
the
doc
me
the
document
is,
then
that
means
the
back
successfully
updated.
We
use
that
timestamp
to
verify
the
capacity
to
change
as
well
be
fair.
The
dog
was
in
place,
we're
good
without
me.
If
we
have
some
kind
of
crop
failure.
A
Any
of
these
steps,
especially
verified.
We
have
a
reprocessing
step
that
will
take
the
cake,
the
gets
that
failed
and
try
to
figure
out
where
why
did
a
delivery
costs
them
and
then,
after
a
couple
rounds
of
that
immunity
process
down
to
send
them
off
to
a
dead
Q
in
essence,
for
enterica
investigation
on
like
wide
fail,
two
sisters,
so
all
of
our
Buster's
our
master
slave
today,
and
so
we
have
a
block
suppose
you
can
imagine
well
master
slave
setup.
A
A
So
talk
a
little
bit
of
the
far
south
of
the
same,
so
we
have
to
college,
it
pulls
like
a
logical
classification
of
documents.
Customers
come
to
us
and
stay
with
us.
We
want
to
use
maybe
surge
application
on
top.
Your
blacklist
cool
will
make
them
to
kind
of
isolate
the
customers
from,
and
so
we
have
two
compounds
per
cooling
example.
We
have
an
initial
data
pool
and
they've
been
translated.
We
store
both
stations
in
Cassandra
because
having
the
transitive,
also
his
head
Gorillaz
and
it
repeats
much
more
quickly.
Our
large.
A
Case
to
refuse,
when
we
upgrade
our
followers
lower
box
in
between
we
go,
are
perfect:
five
hundred
silver
boxes.
You
don't
want
to
go
back
and
ask
you,
know
20
or
30
different
customers
day,
because
you
go
every
feed
your
data
from
your
data
store.
So
we
have
this
intermediate
or
score
that
allows
us
to
that.
We
feed
those
comments.
A
Both
both
colleges
have.
A
document
is
the
key,
which
is
a
unique
dated.
Thought
unique.
Identifier
will
document
that
the
customer
business
we're
not.
We
don't
care
about
how
the
customer
makes
the
decision
phase
difference
the
initial
column,
family,
nokia,
that
a
number
of
fields
based
on
the
data,
its
dynamic
priscilla,
since
you
can
tell
us
about
a
this,
this
document
has
these
fields
in
SF.
It
has
a
different
set
of
fields
where
you
have
to
specify
them
on
insert
and
than
their
back
moment,
but
it
allows
you
to
make
kind
of
whatever
decisions.
A
Without
Nadine
and
then
the
single
transferable
content
feel
you
should
have
three
they're
just
aluminium
metadata
around
where
the
doctor
is
private,
so
we
can
verify
the
rat
was
correct,
come
after
the
devil,
so
talking
about
how
we
read
it
right
from
Cassandra
week
or
remain
calm
in
all
cases
as
most
because
our
feeding
pipeline
doesn't
self
heal.
If
adopted,
failed
to
load
successfully
so
Wilo
an
older
version,
rescue
totally
screwed,
there's
nothing.
A
We
can
do
today
to
recover
it
Rick
as
in
essence,
as
far
as
we
view
the
data
they're
just
streams
coming
in
to
us
from
customers-
and
you
know
the
event,
screens
and
jobs
because
of
resumes
screams
and
use
your
actual
experience
of
whatever
we
know
we
care
so
because
of
that
you
will
form
reading
form
right
thing,
the
time
little
bit
the
actual
service.
Yesterday.
B
So
on
the
deviator,
Cassandra
lovely
design
makes
a
really
super.
Tv
is
so
here
talk
about
our
I
faure,
mechanical
problems
and
we
have
conditionally.
So
we
have
universe
rips
that
we
have
a
3
New
Testament,
which
is
what
the
original
purpose
is,
what
they
use
to
test
new
products.
Then
we
have
our
21
note
production
rate,
which
has
exist
terabytes
in
it
we're
running
10
steps
Cassandra
to
2025.
We
are
running
virtual
virtual
nodes,
which
are
jobs,
that
all
the
tables
are
replication
factor
3
and
are
normally
our
average.
B
With
120
gigs
of
ram
he's
watching
giving
to
us,
but
we
have
a
35
1
terabyte
poop
160
of
my
SSD
is
a
grade.
5
SSDs
actually
work
on
sent
to
us.
So
when
I
reached
out
to
vote
for
Cassandra,
we
were
actually
rivers
wound
up
and
I've.
Actually,
the
DBA
efforts
on
for
about
two
years
now
so
or
there's
a
lot
of
issues
we
had.
We
had
issues
with
passion,
coal,
trains
and
cars.
B
Prussia
compaction
like
several
running
version,
one
deviled
egg,
originally
in
a
single
tourniquet,
passionate
and
the
Compassion's
were
in
beating
up
the
heap
space
until
we
don't
get
an
out
of
memory
error
for
job.
So
it
took
us
up
a
little
bit
to
face
that
prepare
what
the
problem
was.
So
basically,
we
gently
figure.
B
Installed
as
we
install
that
and
enabled
it
reduce
the
mem
table
and
cache
size
which
helped
which
helped
with
the
app
membrane
and
the
thing
that
killed,
it
was
a
increase
in
heaps
of
12
gigs
because
they
add
memory
you
slowly
eat
up
throughout
the
days
and
then
trying
to
shut
down
each
of
us
on
turn
up.
So
throughout
the
rate
of
twenty
one
knows
two
or
three
will
go
down
each
day
and
then
doing
all
this
actually
fixed
Vista
problem.
B
So,
as
joe
said,
very
meaning,
we
had
a
cobra
problem.
This
is
that
one
on
spending
days,
so
whenever
they
read
feed
they
take
every
single
growth
is
out
of
a
queue
and
just
push
it
through.
That
means
each
world
we
run
all
the
times.
So
I
means
SS
tables
that
have
been
touched
in
a
while.
It
has
to
go.
He
has
going
to
disk
to
read
them,
so
I
mean
lost
seats
and
spinning
this
hate
seats.
So
the
way
we
face
that
was
throwing
money
at
man
geez.
A
B
With
these
entities,
we
ended
up
doing
these
everyone
else
across.
To
be
honest,
maybe
this
was
killer.
Can
we
do
one
slick
today?
Yes,
it
decreased
the
right
latency
time
from
15
miles,
XM
too
hot,
for
which
is
higher
than
we
have
to
have
think
we're
gonna
have
to
point
to,
but
one
thing
I
did
was
went
up
when
I
with
SSDs
etc
are
changing
the
current
reason
with
current
rights.
That
was
a
big
issue.
B
I'm
charging
them
with
that.
So
right
now
we're
using
combination
of
game
eyes
are
any
figure
with
Cassandra
installed
and
reason
chef
to
figure
the
various
looking
at
all
for
each
for
each
instance.
So
we
can
theoretically
push
push
out
a
new
instance
for
the
clinical
body
now
and
it
will
automatically
join,
join
the
ring
and
everything
will
be
good
right
now.
We're
using
a
our
three
guests.
Garages
with
EB
is
optimized.
B
We
originally
we
had
you're
using
a
fairy
storage.
We
had
an
issue
with
Amazon.
We
had
a
66
ring
cluster
and
Amazon
Sun
decommissioned
for
the
same
signal
and
it
dropped
everything.
So
that's
what
what
EB
is
optimized
we
so
far.
We
haven't
really
done
it
at
high
volume,
but
the
performance
seems
comparable
to
local
data
set
living
late,
local
data
center.