►
Description
Speaker: Ken Krugler, Big Data Consulting at Scale Unlimited
Slides: http://www.slideshare.net/planetcassandra/c-summit-2013-suicide-risk-prediction-using-social-media-and-cassandra-by-ken-krugler
In this presentation, Ken will describe a portion of an early-phase project that uses social media data (tweets, Facebook posts, etc.) from service personnel to predict suicide rates. There's a lot of motivation to provide better data for military psychologies, since more military wind up taking their own lives than are killed in the line of duty. By analyzing social media data that is voluntarily provided by personnel, plus a predictive analytics system, we can provide assessments that help mental health workers focus their time and energy on the most at-risk individuals. This project uses Cassandra as the scalable storage system for this social media data, which is then analyzed in a distributed environment using Hadoop. The project also uses the Solr search support from DataStax Enterprise to provide ways for users to dig into the underlying data, which is critical when understanding the assigned risk levels.
A
All
right,
so
it's
after
lunch,
so
everyone's
nice
and
awake
have
you
noticed
that
the
Oracle
boat
is
still
going
back
and
forth
back
forth
I'm,
not
making
this
up.
You
guys
should
go
out
there
and
check
this
out.
The
Oracle
yacht
is
going
back
and
forth.
So
I
was
joking
before
about
us
mounting
up
some
sort
of
an
attack
by
the
end
of
the
day.
Maybe
not
I
I
think
we're
gonna
get
some
rowboats
and
some
gun,
or
not
guys
can't
do
that.
This
is
heresy.
A
We're
just
gonna,
go
take
a
boarding
party
over
there
and
we're
gonna,
take
it
back,
but
yeah,
so
that
I
mean
that's.
That
is
by
the
way,
the
Oracle
yacht.
That
probably
a
lot
of
you
help
pay
for
and
Larry
says
thanks,
but
that's
gonna
be
in
the
America's
Cup.
That's
the
America's
Cup
is
going
to
be
here.
So
that's
the
only
redeeming
quality
of
the
whole
thing.
So
our
next
talk
is
by
a
friend
of
mine
colleague,
Ken
crew.
Glar.
A
This
is
it
really
I.
Don't
know
this
this
use
case.
We
were
talking
about
it
and
when
we
were
talking
about
sessions,
this
is
probably
the
one
that
people
should
look
at
is
like
the
real
good
use
of
big
data.
You
know
we
could
talk
about
education
and
things
like
that,
but
there's
some
really
fascinating
hard
problems
that
affect
people's
lives
that
can
be
affected
by
big
data
so
with
that
can
sell
yours.
Thank
you.
B
Alright,
as
patrick
said,
I
am
Ken.
Crew
blur
and
I
am
I
was
really
happy
when
I
got
this
project,
I'm
also
really
happy
with
the
stage
I
feel
like
I'm
like
at
a
beyoncé
concert
or
something
or
you
can
walk
out
into
the
crowd.
I've
never
done
a
presentation
with
this
kind
of
setup.
So
a
lot
of
the
projects
that
I've
worked
on
over
the
years
haven't
been
so
socially
compelling.
B
For
example,
I
worked
on
a
project
where
what
we
were
doing
was
helping
celebrity
stalkers,
get
real-time
updates
on
sightings
of
like
Brad,
Pitt
and
Justin
Bieber,
etc.
You
know
it's
like,
oh
the
shame,
but
my
daughter
needed
braces.
You
know
what
you
gonna
do,
and
this
then
was
was
a
product,
a
project
that
I
was
really
excited
to
work
on
because
it
actually
was
socially
redeeming.
It
had
some
real
value
to
it.
B
Oh
they
said
don't
hit
that
button
all
right,
so
so
what
I'm
going
to
talk
about
today
is
kind
of
the
basics
of
the
project,
how
we
use
Cassandra
to
store
social
media
data,
nothing
really
exciting
there,
because
everybody's
looked
at.
Of
course
the
Tewa
sondra
samples,
so
you
understand
how
to
take
a
Twitter
feed
and
run
it
into
Cassandra.
One
of
the
things
are
going
to
be
talking
about
is
using
Hadoop
with
Cassandra,
so
we
process
the
data
that
we've
saved,
using
a
Hadoop
based,
workflow
and
also
but
wait.
There's
more.
B
You
also
get
solar
search
and
fundamentally
like
how
do
you
do
good
with
big
data
obligatory
background?
This
is
a
picture
of
a
river
near
where
I
live.
I
live
in
nevada
city,
which
is
nowhere
close
to
Silicon
Valley
and
people
always
ask
me,
like
you,
do
all
this
stuff
with
big
data.
Why
don't
you
live
in
Silicon,
Valley
and
I?
Thought
I'd
throw
this
in
today
to
show
you
why
I
live
in
nevada
city
because
of
the
beauty
there.
So
I
have
a
boutique
consulting
company.
B
Boutique
sounds
better
than
really
small,
there's
just
four
of
us
and
it's
scale
unlimited
and
we
do
consulting
for
companies
on
primarily
on
you
know:
Big
Data,
workflows,
so
machine
learning
in
search
and
I
also
teach
classes.
In
fact,
I'm
teaching,
the
solar
component
of
the
datastax
environment
on
Thursday.
B
So,
let's
get
into
the
actual
project
Durkheim.
What
is
this
Durkheim
project
and
I'm
gonna?
Make
it
really
clear
in
this
section
right
here,
I'm
talking
about
a
lot
of
stuff
that
I
had
nothing
to
do
with
its
this
project
is
a
is
a
team
effort
involving
lots
of
different
companies,
lots
of
different
people?
We
got
brought
in
to
deal
with
scaling
issues.
What
I'm
going
to
go
over
here
is
a
lot
of
the
stuff
that
other
people
worked
on,
which
is
is
basically
what
is
the
Durkheim
project?
B
So
in
doing
this
project,
I
learned
a
couple
things
that
I
didn't
know
like
more
more
soldiers
died
from
suicide
than
from
combat
and
in
fact
there
are
more
suicides
than
homicides
every
year
in
the
US,
and
the
suicide
rate
is
climbing,
both
among
military
and
civilians,
and
one
of
the
things
I
didn't
understand.
Is
that
often
the
way
people
find
out
that
somebody
has
an
issue
that
they're
suicidal
is
something
bad
happens?
B
They
crash
their
car
there's
some
there's
some
event,
and
the
issue
is
that
by
the
time
that
event
happens,
intervention
often
isn't
successful
like
at
that
point.
The
person's
already
made
a
decision,
and
so
the
key
is,
can
you
intervene
earlier?
Can
you
intervene
before
they
get
to
that
point?
And
how
do
you
do
that?
How
do
you
know
so?
B
Thus
the
Durkheim
project
Emile
Durkheim,
it's
named
after
this
guy.
He
was
a
french
sociologist,
in
fact,
maybe
the
first
person
who
coined
the
term
sociology
late
1800s.
He
wrote
a
paper
about
suicide
and
in
that
paper
he
was
talking
about
how
you
could
look
at
the
writings
of
people
to
help
assess
their
level
of
you
know,
suicidality,
and
thus
it's
very
similar
to
the
goals
of
this.
It's
a
DARPA
funded
project
and
the
the
ultimate
goal
is
to
give
military
physicians
the
data.
B
They
need
to
be
more
successful
at
intervening
right
at
getting
involved
early
enough
to
make
a
difference.
So
the
way
it
does
that
is,
it
collects
things
that
people
say
through
social
media
through
tweets
through
Facebook
posts,
etc
and
uses
predictive
analytics
on
that
data.
To
try
and
come
up
with
a
risk,
green,
yellow
red
like:
are
they
fine,
not
so
sure,
or
serious,
trouble
and
sort
of
a
confidence
level?
In
that
assessment?.
B
The
original
project
DARPA
funded
Raytheon,
was
the
lead.
You
know,
ultimately,
it
involved
patterns
and
predictions
patterns
and
predictions
as
a
company.
It's
really
this
guy
Chris
Poulin,
who
has
the
lead
researcher.
His
background
was
using
predictive
analytics
in
financial
markets,
so
taking
financial
news
and
using
that
to
try
and
predict
what
is
going
to
happen
in
the
future,
those
that
same
code,
machine
learning
code
is
used
here
and
now.
It
also
involves
a
collaboration
with
Dartmouth,
Medical
and
Facebook
and,
in
fact,
I
think
they
were
supposed
to
be
this
huge
announcement.
B
It
would
have
been
great,
you
know
supposed
to
come
out,
I
think
last
week
it
didn't
happen,
stay
tuned.
You
should
hear
something
about
it
in
the
next
week
or
so
we
finished
phase
one
which
was
really
small
scale,
trial
and
now
the
next
phase
is
how
you
roll
it
out
to
a
bigger
audience.
How
do
you
get
more
people
involved
and
there's
details
of
it
on
this
website
here
Durkheim
project,
org
that
you
can
check
out.
B
So
what
is
the
predictive
analytics?
Basically,
it's
if
you
take
text
that
people
write
and
you
do
some
natural
language
processing
on
it
and
you
break
it
up
into
terms
into
you
know
one
or
two
or
three
N
word
phrases.
Then
you
can
build
statistical
models
that
try
and
predict
things
from
that
text.
So,
for
example,
like
the
text,
there
are
very
few
people
in
the
world
that
know
the
real
me.
B
Okay,
that's
something
my
13
year
old
daughter
says
you
know,
because
she's
in
high
school
or
not
in
high
school,
yet
but
she's
sort
of
pre
high
school.
Getting
there.
That's
probably
okay.
On
the
other
hand,
if
you
know
I
lay
down
to
go
to
sleep,
but
all
I
do
is
stare
at
the
blackness.
You
We
Need
to
Talk
right.
So
as
a
human
reading
that
you
can
got
to
get
a
sense
of
it.
How
do
you
have
a
program
sort
of
doing
the
same
kind
of
assessment
from
the
text?
B
Well,
the
key
like
with
lots
of
machine
learning
stuff
is
you
have
training
data?
You
have
labeled
training
data
and
the
data
for
this
project
came
from
VA
hospital
records.
Obviously
highly
sensitive
information.
I
never
saw
the
data
I've
never
got
anywhere
close
to
the
data.
I,
don't
want
to
get
close
to
that
data
where
they
have
pure
military
personnel
things.
They
wrote
things.
B
They
said
where
they
didn't
try
and
commit
suicide
and
people
who
did,
and
there
you
have
your
labeled
training
data
set
for
coming
up
with
this
predictive
analytics
model
and
the
results
of
phase
one
were
promising.
They
were
much
better
than
random
in
terms
of
predicting
outcomes
from
the
text
and
it
uses
an
ensemble
method
which
is
like
really
common.
B
These
days,
like
we
read
about
the
people,
the
team
that
won
the
netflix
contest
for
improving
the
recommendation
engine
by
ten
percent,
and
it
was
an
ensemble
method,
you
have
multiple
different
models
or
predictors
all
running
at
the
same
time,
and
you
combine
the
results
together
to
get
a
more
accurate
result
and
the
end
result
of
this
thing
is
so
it's
called
a
cleanse
clinician
dashboard.
So
this
is
what
the
military
positions
would
use.
B
They
have
a
person
it
gives
them,
in
this
view,
a
risk
yellow
and
it
gives
them
a
confidence
level
for
it,
and
it
gives
them
some
additional
data
like.
What
exactly
is
the
text
you're
using
you're
extracting
from
what
they've
written?
What
are
the
most
interesting
words
or
phrases
that
seemed
to
indicate
there's
a
problem
here,
all
right
now
we're
finally
getting
to
something
that
involves
cassandra.
So
they
had
a
system
in
place
that
worked
built
on
top
of
my
sequel.
Part
of
the
project.
Was
it
had
to
be
able
to
scale
right?
B
One
of
the
goals
of
the
project
was
this
thing
has
to
work
with
millions
of
people
using
it
on
a
daily
basis
and
at
that
scale
their
current
system
wouldn't
work.
So
that's
what
we
got
brought
in
it
says
come
up
with
a
solution.
That's
going
to
scale
and
as
easily
as
possible
replaces
the
existing
pieces
we've
got
in
our
infrastructure.
B
So
fundamentally,
the
question
is:
if
you
had
a
billion
pieces
of
text,
where
would
you
put
them
so
a
system
that
continuously
monitors
social
media
and
just
saves
the
results
away?
You
could
write
that
out
to
a
file
right.
You
could
just
stuff
it
somewhere
in
a
file
that
gets
uploaded
to
s3
and
we're
all
good.
B
You
know
I'm,
so
glad
I
didn't
directly
follow
Adrian,
because
whenever
he
gives
to
talk
about
Netflix,
it's
like
oh
and
here's,
our
you
know,
5,000
node
cluster,
with
our
80
petabytes
of
day.
You
know
and
I'm
talking
about
a
system.
Here,
that's
got
five
servers
all
right.
It's
got
to
data
collector
front
ends
and
like
I
Cassandra
cluster
of
three.
B
But
one
of
the
key
points
here
is
that
this
is
an
it
was
an
easy
system
to
set
up
and
it
worked
just
fine
for
the
limited
size
of
data
during
this
test
phase
and
it
will
easily
scale
up
right.
So
if
they
decide
it's
time
to
go
for
a
big
deployment,
we
don't
have
to
make
any
architectural
changes.
We
can
just
up
the
number
of
nodes
in
our
cluster
tweak
a
few
other
things
and
we're
good
to
go.
There's
no
redesign
required
and
that's
one
of
things.
I
love
about
Cassandra.
B
We
do
consulting
projects
and
I
used
to
do
a
lot
of
stuff
with
HBase.
When
you
do
a
consulting
project
in
a
company.
Often
we
me
wind
up
being
the
ops
person
we
wind
up
being
like
the
first
line
of
defense
for
anything
that
goes
wrong
and
setting
up
and
configuring
Cassandra
was
way
easier
than
doing
that
same
thing
with
HBase.
So
that's
why
primarily
now
I've
switched
all
my
work
over
to
using
Cassandra,
and
this
is
an
example
of
it.
B
It
was
really
easy
to
set
up
a
three
node
Cassandra
cluster,
that's
running
in
AWS
and
hook
it
up,
and
it
just
worked.
So
this
data
collection
there's
a
there's
a
at
the
top
there
there's
that
durkheim
app,
which
is
a
facebook
app
and
what
it
does
is
when
somebody
goes
there
to
sign
up
to
opt
in
voluntarily
to
this
program.
It
tells
us
okay,
we've
got
a
new
user
and
that's
a
call
to
this
Durkheim
social
API
that
we
created
and
that
updates
an
entry
in
a
Cassandra
users
table
and
then
independently.
B
There's
a
demon
running,
that's
using
this
gigya
service.
I
don't
know:
does
anybody
here
use
gigya
and
we
know
about
it
one
or
two
people?
Basically,
it
gives
you
a
unified
view
of
a
whole
bunch
of
social
media
data
streams,
so
we
can
use
the
same
API
and
we
can
say,
get
us
facebook
data
get
us
tweets,
get
us
Google+
posts,
you
know
even
linkedin.
B
If
we
wanted
it,
though
I
don't
think
people
are
going
to
be
exposing
their
suicidal
thoughts
on
linkedin,
but
we
can
go
through
the
giga
service
to
make
requests
and
it
will
go
out
and
actually
grab
the
data
for
the
user
from
whatever
that
social
media
cloud
of
data
is
the
and
then
it
writes
it
out
to
an
activity
table
again,
a
Cassandra
table.
So
at
a
high
level
this
is
kind
of
the
the
architecture
we've
got.
I
keep
doing
that
so
in
designing
the
column,
families.
For
this
we
followed
Patrick's
advice.
B
You
know
he
always
says
you
got
to
think
about
what
queries
you're
going
to
have
to
answer
from
your
data
and
that's
how
you
design
your
schema
and
in
our
case,
the
queries
that
we
need
to
handle
are
always
by
a
user.
It's
like
from
the
user
table,
give
me
all
the
data
about
this
one
user,
like
their
account
ID
for
Twitter
and
Facebook,
etc,
or
we
want
from
the
activity
table
all
of
their
activity
or
maybe
a
range
by
date
of
their
activity.
B
B
So
this
user
table
it's
like
a
like
a
traditional
database,
basically
we're
replacing
what
would
be
my
sequel,
Oracle
here
with
the
user
table,
so
we
create
a
unique
ID,
a
uuid
for
each
user
when
they
join.
We
got
like
static
columns
about
you
know,
with
their
first
name
last
name,
Facebook,
ID,
etc.
Key
point.
Another
reason
why
I
like
Cassandra
is
ok.
Now
we
want
to
start
pulling
Google+
data.
B
Now
we
got
the
more
interesting
table,
which
is
this
activities
table
and
similar
to
you
know
anybody
who's
looked
at
to
a
Sandra,
etc.
Roki
is
still
this
uuid
that
we
create
per
user.
But
now
we
do
is
we
have
composite
columns,
and
so
we
do
is.
We
have
essentially
four
columns
for
each
event.
That
happens
where
the
column
name
is
a
timestamp,
a
source
like
Facebook
or
Twitter,
and
then
data
or
ID
or
provide
rd
etc
like
static
components
to
it.
So
we
get
essentially
four
columns
for
each
event
that
occurs.
B
So
we
got
a
wide
row
here,
but
if,
even
if
somebody
posts
like
10
times
a
day,
let's
say
so-
that's
40
columns
a
day
over
the
course
of
a
year,
12
K
columns.
That's
nothing!
Alright,
that's
really
easy
to
work
with.
So
the
wide
ro
format
works
well
here
and
it's
helpful
when
we
think
about
these
schemas
to
think
about
it.
Two
different
ways:
one
is
the
traditional
row
column
view.
So
here's
an
example
of
row
column
deal
have
got
my
uuid
out.
B
There
is
the
row
key
and
then
I
have
these
column
names
I'm.
Using
short,
very
short
time
stamps
213,
you
know
so
the
way
at
time,
2
13
in
facebook,
the
data
was
I,
feel
tired
and
the
Facebook
post
ID
was
number
32,
etc.
But
you
can
also
think
about
it
as
as
key
values
where
that
roki.
That
uuid
is
essentially
a
reference
to
a
map
where
the
keys
in
the
map
are
the
column
names
and
the
values
are
the
values
right.
So
that's
the
second
view
down
here.
It's
a
key
value
map.
The
keep.
B
The
important
point
is
even
though,
if
you
think
about
it
as
a
key
value
map,
you
have
to
understand
that
the
keys
are
ordered.
It's
like
a
linked
hashmap
and
that
ordering
is
important
because,
since
we're
ordering
by
timestamp,
we
can
say
I
just
want
to
see
the
things
from
a
week
ago
to
a
month
ago
and
it'll
give
us
a
slice
of
that
data,
so
implementation
details,
so
the
stroke
I
map,
it's
going
through
an
Amazon
load
balancer
and
it's
talking
to
t
1
micro
servers
and
we
get
them
in
different
regions.
B
So
if
one
region
goes
away,
you
know
we
still
keep
running
and
on
those
T,
1
micro
servers,
we've
got
the
actual
API
running
and
the
API
then
is
talking
to
a
three
server
Cassandra
cluster
right
and
we
use
asti.
Onyx
I
was
kind
of
joking
like
they
say,
Oh
Asti
onyx
is
the
I,
don't
know
what
the
son
of
Hector
or
something
like
that
you
could
just
make
up
a
name
and
no
one
would
argue
with
you
or
you
could
be
like.
B
So
we
have
these
two
parallel
API
services
running
and
load
balance
between
and
then
on
one
of
them
we
have
an
active
and
on
one
of
them
we
have
a
standby
for
the
demon.
That's
the
thing:
that's
continuously
through
gigya,
getting
the
actual
social
media
day
and
writing
it
out
to
this
cluster,
and
then
we
can
do
manual
essentially
back
up,
so
we
can
take
a
snapshot
and
we
can
upload
the
SS
tables
to
s3.
B
B
B
What
we
have
is
we've
got
for
a
given
user.
All
of
this
activity,
all
right,
we've
got
all
these
posts
for
them.
We've
gotten
models,
these
predictive,
analytic
models
that
are
defined
elsewhere.
We
have
nothing
to
do
with
the
models
and,
in
fact
it's
a
pretty
interesting
point.
A
lot
of
people
think
when
they
think
about
scalable
machine
learning,
they're
thinking
about
building
the
models,
but
building
the
models
is
typically
something
you
don't
need
to
do
its
scale
like
the
amount
of
labelled
actually
labeled
data
that
you've
got
to
train.
B
These
models
is
usually
almost
always
small
enough
to
fit
on
one
server
and
in
fact,
the
systems
in
place
to
build
these
models,
often
assume
they're
not
running
in
a
distributed
environment,
they're
running
on
one
server,
so
the
model
generation
is
not
distributed.
What
happens
is
the
models
once
they're
built?
Those
are
the
things
that
you
distribute,
so
you
can
run
them
in
parallel
across
all
your
data.
So
in
our
case,
we've
got
these
models
that
we
actually
read
out
of
a
my
sequel
database.
B
We
get
the
text
for
a
given
user.
There's
some
magic
feature
extraction
for
extracting
terms
from
the
text.
We
we
give
them
to
the
model
and
the
model
returns
back
result
like
for
this
particular
model.
Braiding
is
two
point:
oh
I
mean
the
rating
is
green
and
the
probability
is
you
know,
point
75
and
here's
some
of
the
keywords
that
we
thought
were
really
important
for
giving
that
result
back
and
then
we
can
persist
the
model
and
that's
it.
Another
reason
why
this
approach
works.
B
Well,
for
this
and
I'll
talk
more
about
this
in
a
bit
is
that
means
we
never
have
to
see
the
training
data
and
the
training
data
is
the
real
sensitive
stuff
at
the
training
data
here,
we're
talking
about
VA
hospital
medical
records
of
real
people
with
real
issues,
and
that's
not
the
kind
of
thing
that
I
want
anything
to
do
with.
So
we
get
these
statistical
models
that
have
been
generated
somewhere
else
and
that's
what
flows
into
the
system
along
with
the
social
media
activity
data.
B
So
why
do
we
use
to
do
you?
Wouldn't
if
you
just
had
like
a
single
user
and
you're,
applying
a
single
model
to
it?
You
don't
need
to
do
if
you
have
like
five
models
and
you're
running
it
on
a
single
user,
you
don't
need
to
do,
but
what
happens
when
a
model
changes
and
this
model
is
being
used
on
a
bunch
of
different
users.
B
Now
you've
got
to
reprocess
all
of
those
users
with
that
new
model,
either
a
modified
model
or,
let's
say
a
model
that
was
added,
and
this
is
happening
all
the
time,
because
these
models
are
under
constant
evolution.
So
when
that
happens
now,
you've
got
a
bulk
data
processing
problem,
it's
okay
to
do
it
batch,
but
you
don't
want
it
to
take
forever
all
right.
So
here's
how
you
can
leverage
the
dupe
to
do
that
right.
We
we
can
use
the
Duke,
because
latency
is
ok,
I
Hadoop,
as
a
batch
processing
system.
B
So
when
we
do
Hadoop
workflows,
we
don't
write
Java
code
at
the
Hadoop
API
level,
because
if
you
do
that
you're
making
a
mistake
like
it's
like
writing
microcode,
like
nobody,
writes
microcode
or
machine
language
anymore.
You
know
you
use
a
higher-level
language,
so
we
use
something
called
cascading,
which
is
an
open
source
project.
It's
a
Java
API.
On
top
of
a
dupe
for
building
workflows.
We
then
read
data
from
the
activities
table
using
what's
called
a
tap-in
cascading
terminology,
a
source
of
data,
there's
a
cassandra
tap.
B
So
on
the
left
side
up
there,
you
can
see.
I've
got
the
user
profile
data
because
I've
got
in
there.
I've
got
information
about
the
person
their
age,
the
rank,
you
know,
sort
of
the
static
data
about
the
user
is
in
that
user
table
and
then
I've
got
the
activity
table
that
has
that
stream
of
social
media
text
for
them,
and
this
joining
of
those
two
data
sets
is
a
really
really
common
thing
if
you're
doing
machine
learning,
because,
typically
what
you've
got
is
some
data?
B
That's
fairly
static,
that's
per
user
like
any
time
you
go
to
a
shopping
site
and
they
know
something
about
you.
What
they're
doing
is
they're
taken
the
static
data.
They
know
about
you
with
the
clickstream
data,
which
is
typically
a
larger
data
set
and
they're
joining
those
data
sets
together,
because
machine
learning
typically
wants
to
get
these
Willy
really
wide
records.
When
you're
doing
machine
learning
stuff,
you
typically
want
to
get
a
record.
That's
got
everything
in
it
like
it's
got
all
the
details
about
you
that
are
staticky
and
then
all
the
data.
B
That's
kind
of
this
continuous,
like
scaling
stream
of
data,
and
you
give
it
to
it
all
at
once.
So
there's
a
join
that
has
to
happen.
Then
we
run
these
classifier
models
against
that
data.
We
get
the
result
back.
That
thing
of
you
know,
here's
the
prediction:
you
know
green
twenty
percent
and
we
write
that
out
right.
So
the
workflow
itself
is
pretty
straightforward.
There's
a
couple
interesting
issues
here
when
you're
doing
bulk
processing
in
Hadoop
there's
a
challenge
because
Cassandra
is
not
designed
for
row
scans.
B
Alright,
that's
fundamentally
like
anytime,
you
say
I'd
like
to
do
a
roast
can
like
on
the
mailing
list.
They
like
don't
do
that,
because
unless
you
bail
on
the
random
partitioner,
you
know
you're.
How
do
you
get
the
list
of
ID's
that
you
want
to
scan
over
in
each
node?
And
it's
really
easy
to
create
hot
spots
on
your
node
and
like
all
kinds
of
things,
don't
work
very
well.
B
So
this
workflow
besides
this
issue
of
Rose
scans
has
a
couple
other
issues
around
it
like
one
is
right
now
it's
a
manual
operation
where,
ultimately
you
want
it
to
be
triggered.
Oh
there's,
a
new
model.
You
know,
boom
I
got
a
scheduler
that
runs
the
workflow.
We
run
the
workflow
in
a
separate
cluster
all
right.
So
we've
got
this
one
cluster
here
which
has
all
of
the
data
in
it.
The
activity
data,
the
workflow,
runs
in
a
separate
cluster
and
it
pulls
the
data
over.
B
If
we
were
using
the
data
sacks
Enterprise
Edition
those
could
be
combined.
And
finally,
when
you
do
this
kind
of
thing,
in
Amazon,
you've
got
issues
where
you
want
things
locked
down
as
much
as
possible,
but
not
too
much,
and
so
Cassandra
and
Hadoop
have
all
these
different
ideas
about
what
ports
need
to
be
open.
And
so
you,
wind
up
spending
a
lot
of
time
in
Amazon
security
groups,
making
sure
that
you've
got
exactly
the
right
set,
but
only
those
set
of
ports
open
between
these
two
clusters.
B
All
right
so
now
we're
on
to
solar
search
and
I
call
this
poking
at
the
data.
So
here's
the
issue
when
you
get
these
model
results
back
you'll
see
something
like
the
key
phrases
feel
angry
and
it
has
some
weight
associated
with
it.
But
do
you?
Is
it
really
like
I
feel
angry
or
maybe
the
text
was
you
know,
I,
don't
get
angry
when
people
misinterpret
what
I
write
in
my
Twitter
posts
it?
How
do
you
know
what
the
actual
source
was?
B
You
want
to
go
back
and
you
want
to
find
all
the
places
where
for
this
user
they
use
this
text
to
understand
and
I
call
this
poking
at
the
data,
because
Hadoop
is
great
at
taking
petabytes
and
turning
it
into
pie
charts,
but
you
can
take
a
dip
and
you
can
throw
all
this
data
in
the
big
funnel
and
I'll
pops
a
pie
chart
okay,
great
now,
I
got
a
pie
chart
like.
Is
it
correct?
What
is
it
really
telling
you
and
it
used
to
be
in
the
old
days?
B
You
know
somebody
threw
some
slide
up.
You
know
with
some
grad
some
numbers
on
it
and
you
disagreed.
You'd
say
well,
that's
wrong
and
maybe
go
and
you'd
find
the
spreadsheets.
You
know
whatever,
and
you
can't
do
that
anymore,
because
it
would
be
Satan
spreadsheet.
It
would
have
like
a
billion
rows
in
it.
So
you
can't
argue
over
the
data
using
traditional
methods.
So
what
do
you
do?
B
B
But
there's
a
problem
here
because
remember
in
our
cassandra
table
our
column
family.
We
had
wide
rows
right.
The
regular
support
for
solar
in
Cassandra
and
DSC
says
I've
got
solar
fields
that
correspond
to
my
columns
in
Cassandra.
But
when
you
have
a
wide
row
like
that
with
essentially
what's
dynamic,
you
know
constantly
changing
column
names.
You'd
have
to
use
solar,
dynamic
fields
which
a
don't
really
work
that
well
or
actually
don't
work
in
DSC
and
you're
limited
to
a
thousand
24
per
document.
So
it
just
doesn't
work,
so
what
does
that
mean?
B
What
do
you
do?
Well,
you
make
a
copy
of
your
data
and,
as
patrick
also
says,
don't
be
afraid
of
making
copies
like
this
space
is
cheaper
than
programming
time.
So
what
we're
going
to
do
is
some
of
the
data
that
we
write
out
to
that
activity
table
we're
going
to
write
out
to
a
separate
table.
That's
specifically
set
up
for
search.
So
what
does
that
look
like.
B
Well,
here
you
see,
we've
got
a
activity,
column,
family
right
where
we've
got,
let's
say
an
example
here
that
same
one,
I've
got
a
uuid,
as
my
roki
and
I've
got
some.
You
know,
fields
for
an
event:
I
transform
that
into
the
search
column
family,
where
the
row
key
is
built
up
from
the
uuid,
the
timestamp
and
the
source
of
the
data.
So
here
in
the
search
column,
family,
the
entry
would
be
uid
one
time
stamp,
two
and
three
providers
FB
and
then
the
data
which
is
a
fixed
column
there.
It's
a
single
column.
B
The
data
has
the
actual
data
from
the
event
like
I
feel
tired.
So
now,
when
we
do
a
search
and
we
get
that
hit,
we
get
back
the
row
key
and
from
the
roki
we
know
the
user,
and
we
know
all
the
other
information
we
need
to
actually
get
at
that.
Full
data
set
in
the
activity
table
makes
sense.
So
basically,
what
we're
doing
is
just
creating
a
copy
of
our
data,
which
is
transformed,
so
it
works
well,
solar
and
that's
the
kind
of
thing
I'm
going
to
be
talking
about
more.
B
If
any
of
you
are
taking
the
Cassandra
training
on
thursday
I'll
be
going
over
these
kinds
of
things
in
more
depth,
then
so
the
solar
schema
is
really
simple,
and
that's
how
you
want
it
right
if
you
have
complicated
solar
schemas,
that's
not
a
good
thing.
So
it's
really
simple.
It
only
has
two
fields
in
it:
a
key
in
a
data
field
which
essentially
match
up
one
for
one
with
that
search,
column,
family
and
because
the
the
key
field
is
stored
and
I
get
it
back.
B
B
Now
I'd
mention
that
you
know
we.
We
actually
have
a
supper
to
do
cluster
that
pulls
data
from
the
Cassandra
cluster
and
processes
it,
but
you
can
actually,
with
DSC,
have
a
combined
cluster
where
you've
got
like
every
node
has
a
sander
on
it,
and
you
can
say
some
nodes
have
solar
on
it,
and
some
nodes
have
a
dupe
on
it.
I
guess
you
could
say
some
notes.
Have
solar,
Anna,
doop
I,
don't
know
why
you
do
that.
B
So
that,
then,
would
let
you
is
essentially
reduce
your
infrastructure.
Overhead
I'd
have
one
cluster
that
you're
using
for
all
this
stuff.
It
wait.
It's
tough,
it's
a
floor
wax
and
it's
a
dessert
topping
like
it's
your
search
cluster
and
your
data
processing
cluster.
One
of
the
reasons
we
actually
had
to
dupe
as
a
separate
cluster
was
it's
transient
like
we
would
run
a
job
and
then
we
would
shut
it
down
in
amazon's
cloud.
We
wouldn't
use
that
cluster
anymore.
B
We
didn't
need
to
have
it
running
all
the
time,
so
those
one
advantage
to
having
them
separate
all
right
security,
normally
security,
sort
of
almost
like
a
side
thing
for
us
like
either
our
clients
really
care
about
security,
and
they
have
all
their
data
locked
up
inside
their
own
firewall
or
you
know
its
web
crawl
date
or
whatever
it's
completely.
You
know
who
cares
it's
just
up
in
s3,
public
s3
buckets
up
in
the
cloud
this
is
sort
of
in
between,
and
so
I
want
to
talk
a
little
bit
about
the
security
issues.
B
So
the
most
important
thing
about
security.
The
key
point
is
we
don't
have
any
personal
medical
data,
we
don't
have
any
personal
medical
data
and
we
don't
have
any
personal
medical
data.
Those
are
the
three
key
points
because,
as
soon
as
you
start
processing
personal
data
you're
talking
about
a
whole
nother
level
that
we
did
not
want
to
get
involved
with.
B
So
we
care
about
the
data
security,
but
we
don't
care
at
the
same
level
of
like
medical
records
bank
records
that
kind
of
thing.
So
what
are
the
three
aspects
of
security?
Well,
one
is
the
standard
one
where
all
these
servers
that
we're
running
up
there
in
amazon's
cloud,
you
know
they're
all
locked
down.
You
can
only
log
in
with
ssh
and
a
key,
a
private
key
that
we
restrict
access
to.
B
The
second
thing
is:
we
have
that
API
that
the
facebook
app
is
using
to
both
create
users
and
two,
let's
get
some
information
back
about
users.
So
there
we
do
a
thing
that
is
really
common
in
like
Amazon,
where
every
request
that
goes
in
gets
a
hash
created
using
a
private
key
shared
shared
key
that
we
both
know,
which
is
private
and
a
sha-1
hash.
And
if
the
signature
doesn't
match,
then
we
know
it's
not
valid
and
we
reject
the
request.
B
First
key
point
is
that
you
can
effectively
use
Cassandra
as
a
repository
for
social
media
data,
so
that's
nothing
new
again.
Twiss
andhra,
like
all
these
different
projects,
do
exactly
that.
The
interesting
thing
is,
then:
you
can
also
use
it
as
the
source
and
the
sink
for
work,
clothes
that
are
taking
this
data,
doing
something
useful
with
it
and
then
generating
new
results
right.
B
So
you
can
combine
it
as
a
data
store
and
as
a
as
part
of
your
workflow
infrastructure,
and
that's
an
area
of
Cassandra
I
think
you're
going
to
see
more
and
more
about
in
the
future,
and
the
last
thing
is
that
you
can
also
then
leverage
the
DSE
support.
If
you
have
that
for
solar,
so
now,
you've
got
don't
have
two
different
so
to
infrastructure
stacks
that
you're,
trying
to
end
where
you're
trying
to
keep
that
data
in
sync.
It
just
naturally
is
kept
in
sync
for
you,
you
right
into
cassandra
and
magically
boom.
B
It
appears
in
your
solar
search
index,
which
is
a
beautiful
thing,
and
the
meta
point
is
that,
yes,
it's
possible
to
do
more
with
bigger
than
optimized
ad
yields
like
working
on
this
project
felt
a
lot
better
than
working
on
a
project
where
it's
like
giving
me
ads
for
this.
You
know
bald
spot,
that's
showing
up
on
the
top
of
my
head
and
and
that's
a
good
feeling
and
I
think
there's
going
to
be
more
and
more
of
these
kinds
of
projects.
B
It's
going
to
raise
interesting
issues
about
how
much
data
are
you
willing
to
share
in
order
to
get
some
kind
of
positive
benefit,
because
often
you
have
to
have
a
fair
number
of
people
opt
in
to
gather
enough
information
actually
do
something
with
that
data.
I'm
participating
in
the
Durkheim
project,
I
think
they're
going
to
be
opportunities
like
that
in
the
future.
I
think
people
are
giving
their
data
way
now
for
lots
of
reasons
that
are
not
nearly
as
useful
and
so
I
encourage
everybody.
B
You
know
if
you're,
especially
if
you're
a
vet,
please
participate
in
this
project.
Please
go
to
the
website,
check
it
out
and
sign
up
so
that
when
it
gets
rolled
out
you
can
participate
because
the
more
people
that
are
part
of
the
project
that
we're
successful,
it's
likely
to
be
I,
think
that's
it.
Thank
you.
B
So
normally,
when
I,
when
I
teach
classes,
I
insist
people
ask
questions
in
the
middle
of
the
class
because
it
makes
it
way
more
interesting,
but
I
guess
when
you
do
presentations,
you
got
to
wait
for
the
end.
So
if
there
are
any
questions
about
anything
up
here
that
I
can
talk
about,
I'd
be
happy
to
answer
them.
B
So
the
question
is
we
model
the
validity
of
the
prediction,
the
half-life
so
I,
don't
think
they're
at
that
level.
Yet
because
the
you
know
again,
these
are
the
details
of
the
stuff
I
didn't
directly
work
on,
but
I
know
that
it
was
a
pretty
small
sample
set
for
the
initial
assessment
and
I
know
there
are
things
that
they
were
looking
at
doing
that
probably
they
talked
about
model
decay
over
time,
like
you
know
things
as
things
shift,
but
I
don't
know
the
details
about
that.
B
So
so
the
question
is:
what's
that
black
box
thing
I
talked
about
where
we
were
taking
like
the
social
media
data
and
the
from
the
activity
table,
and
also
the
information
about
the
user
and
putting
it
into
a
black
box
and
then
getting
result
back
so
that
black
box
is
a
combination
of
code
and
actually
its
native
code,
that
we
had
to
C++
code.
We
had
to
interface
to
and
one
or
more
models
which
are
just
data,
and
so
assuming
we've
got
this
code
and
these
models
that
forms
that
black
box.
B
So
how
was
the
model
created?
That's
the
part
where
it
was
done,
offline
to
the
side
inside
of
I
believe
VA
hospital,
where
they
used
records
medical
records
for
a
bunch
of
personnel
and
essentially
wound
up
being
labeled.
By
had
they
tried
to
commit
suicide
or
not.
That
was
the
the
data
set
and
it
was
the
text
I'm
not
sure
of
all
the
details
here,
but
it
was
like
the
text
of
their
conversations
with
the
physicians
and
things.
B
Maybe
they
wrote
the
text
that
I
showed
you
up,
there's
examples
of
like
you
know
when
I
try
and
fall
asleep.
I
always
use
blackness,
that's
actually
from
teen
a
teenage
suicide
poetry
which
that
was
one
of
the
downsides
to
this
project,
because
you
read
through
this
data
and
you
just
depressed
like
you,
you
know
we
have
this
like
test
data
and
it's
all
dark
poetry.
You
know
it's
it's,
that
was
the
one
bummer,
but
that
was
not
data
coming
from
military
personnel.
That
was
just
public
source
data
from
a
poet
recites
somewhere.
B
B
So
if
we
were
running
this
so
again,
I
think
that
does
the
question
is:
what
happens
when
you
take
those
activity
data
and
you
run
it
through
the
modeling
you
get
a
result
back.
Is
that
also
sensitive
data
and
I
think
the
answer?
The
answer
that
is
yes,
and
so
that
happens
inside
of
Dartmouth
Medical
firewall,
that's
not
happening
up
here
in
AWS
we
did
our
proof-of-concept
tests
up
in
Amazon
and
that's
the
point
I
didn't
make,
but
I'm
a
big
fan
of
any
time.
B
I
have
a
client
who's
working
on
a
big
data
project
I
tell
them
synthesized
your
data
as
accurately
as
you
can
but
create
mock
data
and
then
run
workflows
against
that
mock
data
in
Amazon.
Because
that's
going
to
give
you
that
information,
you
need
about
scaling
like
how
many
servers
are
what
size
do
we
need
in
order
to
fulfill
our
SLA
s,
and
so
essentially,
that's
when
we
were
running
these
two
clusters
in
amazon's
against
mock
data,
not
real
data.
B
B
B
Triggers
okay,
so
we
could
probably
use
triggers
as
the
thing
that
that
causes
the
workflow
to
run,
and
that
would
be
totally
fine
that
that
should
work,
I
haven't
looked
at
how
we
would
integrate
triggers
into
cascading
workflows,
which
are
kind
of
you
know
what
we're
using
like.
What's
the
right
way
to
cause
that
to
kick
off
a
series
of
jobs,
but
yes,
I'm
very
hopeful,
that's
the
great
thing
about
open
source.
It's
like
oh
man,
I
wish.
We
had
some
way
to
read:
SS
tables
as
an
input.
B
B
So
right,
so
the
question
is:
is
anything
else
used
for
the
model?
Besides
text
I
think
the
current
models
in
phase
1
were
text
and
time-based
all
right.
So
it's
a
both
are
attributes
signals.
I
know
that
we
are
collecting
data
about
things
like
image
posts,
but
not
the
actual
images
themselves.
I.
Imagine
that
would
be
an
interesting
area,
but
I
haven't
heard
of
them
doing
anything
with
like
image
data
as
far
as
analytics
go.
B
So
how
do
you
measure
the
success
or
error
rate?
Well
again,
one
of
the
interesting
things
here
is
that
what
it's
being
compared
against
is
nothing
right.
There's
no
signal
right
now,
essentially
for
in
advance,
so
all
it
has
to
do
is
be
significantly
better
than
random
and
it's
a
win
now
what
they
did
is
they?
B
B
B
So,
on
the
solar
side
of
things,
performance
is
fine,
so
it
you
know
it
winds
up
generating
a
solar
index
just
like
any
other
solar
index.
You
know
their
issues
where
we
could
tune
if
we
had
a
single
standalone,
solar
server.
I
could
do
more
tuning
on
it,
so
I
could
probably
get
more
performance
out
of
it
if
I
had
to
hasn't
been
an
issue
yet
using
cassandra
as
the
source
for
data
in
a
work
flow
through
the
current
thrift.
B
Api
that
we
were
using
is
not
so
performant
versus
like
if
that
data
was
just
stored
in
HDFS
as
files,
so
there's
definitely
performance
it
there
and
that's
where,
when
we
scale
up
I'm
hoping
that
there's
an
SS
table
input
format,
you
know
that
we
can
directly
use
because
that
right
now
is
the
bottleneck
right.
Getting
the
data
like
at
scale
out
of
Cassandra
into
a
Hadoop
workflow,
but
the
solar
side
was
fine.
Of
course
you
know
with
synthetic
data
so
who
knows
in
real
world
what
it'll
be
but
I'm
reasonably
confident
about
that?
B
So,
what's
the
giveaway
for
project
like
this
I
think
in
this
case
right
here
there,
the
giveaway
is
you're
helping
fellow
bats,
alright,
basically
by
participating
you're,
improving
the
ability
of
the
system
to
help
those
people
who
really
need
help
by
improving
the
accuracy
of
the
models.
I.
Think
that's
the
angle,
though
again
that's
part
of
the
PR
for
this
project.
I'm
not
involved
with
you
know,
and
what
they
with
a
you
know,
give
away,
would
be
it's
an
instron
question.