►
From YouTube: Proofpoint: Fraud Detection and Security on Social Media
Description
Speaker: Harold Nguyen, Data Scientist
Social media has become the new frontier for cyber-attackers. The explosive growth of this new communications platform, combined with the potential to reach millions of people through a single post, has provided a low barrier for exploitation. In this talk, we will focus on how Cassandra is used to enable our fight against bad actors on social media. In particular, we will discuss how we use Cassandra for anomaly detection, social mob alerting, trending topics, and fraudulent classification. We will also speak about our Cassandra data models, integration with Spark Streaming, and how we use KairosDB for our time series data. Watch us don our superhero-Cassandra capes as we fight against the bad guys!
A
Yes,
good
morning,
everyone
so
welcome
to
the
second
day
of
casinos
summit.
It's
pretty
good
summit
so
far,
I
thought,
because
no
single
conference
in
the
world
today
I'm
going
to
talk
about
some
of
the
casino
use
cases
at
Proofpoint,
and
hopefully
they
might
inspire
some
of
your
use
cases.
So
there's
a
bunch
of
different
use
cases.
We
have
a
proof
point.
A
Let
me
start
off
by
just
talking
about
a
little
bit
about
me:
I'm,
a
data
scientist
and
data
engineer
and
I'm
responsible
for
the
content
classification
and
some
of
the
fraudulent
and
nominally
detection
and
security
research,
I
work
with
the
engineering,
marketing
and
research
teams
at
Proofpoint.
So
about
a
year
ago,
I
gave
a
talk,
I
christen,
her
Summit.
At
the
time,
I
was
looking
for
a
next
gate,
so
next
gate
was
a
social.
A
Next
gate
is
a
social
media
security
and
compliance
company
about
a
few
months
after
that
talk
next
gate
was
acquired
by
proof
point
and
in
hindsight,
I
can
say
that
it
was
due
in
large
part
to
the
stable
infrastructure
and
reliable
performance
of
Cassandra.
That
was
next
gate
success
and
its
continued
success
with
entry
point.
A
quick
outline
of
this
talk.
I'll,
give
it
a
little
bit
of
a
background
about
proof
point
and
then
talk
about
the
many
use
cases
of
Cassandra.
A
At
proof
point
you
might
notice
that
I've
separated
into
two
sections
so
email
security
and
then
social
media
security
and
compliance
of
which
I'm
more
involved
with
before
I
begin,
though
I
have
to
give
credit
to
a
lot
of
the
people
who
whose
work
is
being
talked
about
in
these
slides.
So
I
want
to
mention
the
following
people:
my
Proofpoint
colleagues
Brian
burns,
Brian
Hawkins,
Wayne,
Lewis,
San
Dimas
and
on
some
money,
Greg
Celia,
rich
sudden
poof
point
is
a
cybersecurity
that
focuses
in
the
email,
social
media
and
mobile
space.
A
It's
been
around
since
2002
and
there's
over
a
thousand
employees.
So
it's
a
sizable
number
of
people
at
Cassandra
and
as
such
there's
a
lot
of
use
case
is
for
Cassandra
Proofpoint,
and
some
of
these
use
cases
might
seem
different
from
each
other,
although
they
fall
on
the
umbrella
under
the
umbrella
of
security.
But
I
hope
you
take
away
that
there.
We
all
the
use
cases
take
advantage
of
casinos
features.
A
So,
let's
start
with
the
email,
security
and
compliance
space.
How
do
you
use
Cassandra
and
email
before
I
begin
with
the
first
use
case
of
Cassandra
and
email
I
want
to
talk
about
the
product
that
it
supports
to
gain
a
better
appreciation
for
that
product.
So
there's
a
product
in
Proofpoint
called
tap.
It's
it
stands
for
targeted
attack
protection
and
a
targeted
attack
is
an
attack.
That's
organized
around
an
individual
or
an
organization.
A
So
the
way
it
works
is
it
analyzes,
the
URLs
attachments
and
user
clicks
in
an
email
message
and
there's
a
go
senator
table
will.
Let
me
talk
about
an
article
that
recently
came
up
that
highlights
the
powerful
pinus
of
tap.
So
just
last
week
there
was
a
campaign
against
the
Russian
military.
So
imagine
you're,
Russian,
militant,
a
part
of
the
Russian
military.
You
receive
an
email,
you
receive
the
email
and
the
in
the
email,
there's
a
attachment
to
a
work
document.
So
you
open
up
that
word
document,
because
your
antivirus,
all
your
antivirus,
has
passed.
A
There's
no
virus
on
that
document.
You
open
it
up
and
you
read
it
and
it
talks
about
Russian,
military
history
and
you're
interested
because
you're
part
of
Russian
military
once
you
finish
reading
it,
you
close
it
and
then
you
never
think
about
it
again,
so
it
actually
happened.
When
you
closed
that
document,
was
it
triggered
a
second
file
to
download
a
third
file,
and
this
third
file
is
where
the
nasty
stuff
comes
in
now
the
antivirus
would
have
not
caught
that
because
there's
no
virus
in
the
document.
This
is
a
comment.
A
Anti
send
boxing
technique
that
a
lot
of
sandboxes,
miss
but
tap
is
a
product.
I
would
have
caught
that
so
the
way
because
tap.
The
reason
why
tap
is
so
powerful
is
because
it's
supported
by
Casandra,
so
the
link
is
saying
is
used
with
tap.
Is
that
it's
basically
used
as
an
indexer,
it
hashes
the
URL
and
then
stores
the
metadata
of
the
email
message
in
the
columns
and
these
email
messages
can
be
used
to
determine
what
are
the
email
messages
across
all
the
customers
that
contain
this
URL.
A
So
if
the
URL
is
malicious,
we
can
find
out
all
the
messages
that
had
that
you
malicious
URL.
Furthermore,
we
can
use
the
metadata
and
the
email
message
to
map
back
to
the
content
itself
so
that
we
can
display
on
a
dashboard
what
kind
of
message
it
was
in
case.
There
are
other
messages
similar
to
that
message,
but
with
a
different
URL.
A
The
Kissena
infrastructure
for
a
tab
is
a
40
node
cluster
in
AWS,
with
c32
x
larges,
it's
about
2,
terabytes
of
EBS
storage
on
each
there's,
a
replication
factor
of
4
and
data
has
increased
about
hundred
percent
since
last
year,
Carlos
DB,
which
is
a
time
series
database
built
on
top
of
Cassandra,
plays
a
huge
part
in
our
met,
our
monitoring,
our
metrics.
So
it's
become
pretty
critical
to
proof
point
to
be
able
to
monitor
jmx
metrics
BMXer.
A
A
Another
thing
we
have
for
email
and
email
is
something
we
call
internally
threat
database
now
we're
getting
a
lot
of
data
from
email
and
then
from
tap,
but
we
don't
have
a
way
of
correlating
all
these
data
points.
So
we
decided
to
build
a
custom
graph
database
on
top
of
Cassandra
and
this
graph
database.
A
We
used
wrote
key
as
the
vertex
and
their
columns
as
the
edges,
there's
18
nodes,
24
terabytes
of
data
and
then
its
highest
peak,
there's
about
a
million
events
per
second
and
it
was
actually
successfully
used
to
detect
campaigns
that
couldn't
have
been
detected.
Otherwise,
so
there
is
actually
net
word
effect
out
there.
What
one
of
the
campaigns
we
detected
you
might
have
heard
of
or
might
not
might
not
have
heard
of-
is
the
dry
decks
campaign.
A
It
wasn't
as
much
integration
with
Titan,
DB
and
Cassandra,
and
that
is
apparently
coming
and
in
one
of
the
future
versions
of
data
stacks
Enterprise
Titan
is
going
to
be
integrated,
Cassander,
so
I'm
looking
for
where
we
are
looking
forward
to
that,
but
just
to
dive
a
little
deeper,
though
at
the
time
a
lot
of
these
graph
databases
generated
their
own
IDs
and
that
caused
us
to
have
a
lot
of
unnecessary
querying,
and
so
it
killed
our
insert
performance.
So
we
just
wanted
to
write
something
that
was
custom
and
fit
for
our
use
case.
A
A
Another
use
case
we
have
for
Cassandra
and
email
is
as
a
general-purpose
infrastructure
storage,
so
we
store
attachments
and,
as
large,
encrypted
blobs
in
a
product
called
secure
share
which
allows
customers
to
securely
share
files
and
emails
with
each
other.
Some
of
these
files
range
between
20
megabytes
and
2
gigabytes.
It's
also
used
as
an
identity
store,
so
we
might
store
users
and
customer
information
in
there.
A
It's
about
a
there's
6
nodes
across
two
data,
centers
three
nodes
in
each
data
center
and
the
idea
why
we
could
have
stored
this
in
sequel,
but
we
wanted
to
use
Cassandra
because
of
its
distributed.
Nature
it'd
be
easy
to
scale
up
if
we
needed
to
and
lastly,
one
of
the
use
cases
we
have
an
email
for
Cassandra.
Is
a
research
focused
one?
This
is
a
project
that
came
out
of
hack
day,
which
people
wanted
to
use
the
ability
of
spark
and
consider
together.
A
In
this
case,
we
wanted
to
cluster
email
topics
together
and
the
way
we
did
that
was
by
using
a
spark
streaming
algorithm
using
spark
library
and
the
ML
Lib.
We
use
the
k-means
clustering
algorithm
using
the
word
to
Veck.
As
a
means
of
clustering,
so
we
started
with
k,
equals
20,
50
and
100,
just
because
we
weren't
sure
what
kind
of
clusters
were
the
best,
but
it
turns
out
that
using
war,
two
vet
was
great
for
clustering,
because
a
lot
of
the
topics
that
are
synonymous
with
each
other
live
in
the
same
vector
space.
A
A
So
now
I
want
to
jump
into
Kissena
for
social
media
so
which
I
am
more
directly.
A
part
of
a
main
motivation
behind
Cassandra
for
social
media
is
that
we
wanted
to
store
everything
one
of
the
main
applications
in
our
product.
The
next
gate,
social,
social
media.
Product
of
proof
point
is
classification.
A
So,
in
order
to
have
a
great
classification
system,
we
needed
to
have
an
accurate
classification
system
and
it
would
have
an
accurate
classification
system.
We
needed
to
train
it
on
a
lot
of
the
data
and
in
order
to
have
a
lot
of
data,
we
need
to
store
that
data
and
so
Cassandra
being
able
to
store
all
that
data
was
a
great
choice
for
us,
because
Kasana
is
easy
to
use
easy
to
scale.
I
had
a
powerful
data
model
and
there's
a
great
community
behind
it.
A
A
A
So
this
is
what
we
currently
have
in
production
on
the
social
media
side.
It's
about
two
terabytes
of
data
across
three
data
centers.
Our
current
rate
is
hundred
fifty
rights
and
fifteen
reads
per
second,
which
is
great
because
cosider
has
made
for
fast
writes.
So
it's
great
that
our
rights
are
better
faster.
More
than
our
reads
and
it's
interesting
to
talk
about
how
we
got
here
when
we
started
it
with
Cassandra
in
2012,
we
were
using
Cassandra
one
one,
six
with
a
boo
to
1004.
A
A
We
are
using
solid
state
drives
and
we
have
over
half
a
billion
content
stored,
so
Chuck
Norris,
while
drinking
his
Pepsi
would
have
proved
that
that
it
never
gone
down
so
because
Santa
has
never
gone
down
for
us
and
I've
noticed
that
a
lot
of
the
some
of
the
people
at
Kissena
summit
are
wearing
shirts.
That
say,
he'd
never
gone
down
and
I
really
wish
I
couldn't
go.
I
could
have
got
one
of
those
shirts
for
this
slide,
but
I
swear
to
you
that
I
wrote
this
aloud
reforest
all
those
shirts.
A
A
A
So
just
to
give
you
an
example,
we
have
something
called
a
global
spammy
users
list
and
the
spammy
users
list
calculates
authors
that
have
that
are
spammers
in
their
local
region.
So
each
local
region
can
benefit
from
another
local
regions-
spamming
users
list
by
talking
to
that
global
data
center,
that
you
see
that's
connected
by
the
red
arrows.
Another
thing
we
use
for
global
data
centers
is
a
clever
user.
A
Authentication
system
can't
go
into
too
much
of
the
details,
but
it's
similar
to
two
FA
and
we're
using
the
nice
feature
of
Cassandra
that
it's
distributed
and
also
its
TTL
feature,
and
this
is
what
I,
Google
Data
Center
looks
like
right
now,
there's
two
nodes
and
three
data:
centers,
there's
not
a
lot
of
data,
but
the
important
thing
is
that
it's
enabling
our
application
solving
our
feature,
requests
and
doesn't
take
a
lot
of
set-up.
Time.
A
Okay,
our
sixth
use
case
we're
set,
is
on
social
media,
is
detecting
spam,
so
an
email
in
the
email
world
spam
is
very
familiar.
It
may
not
be
a
completely
solved
problem,
but
at
least
it's
familiar
and
in
the
email
world.
Basically,
whenever
you
send
one
message,
it's
a
one:
send
one
recipient
come
kind
of
model,
so
one
to
one,
but
in
social
media
a
single
post
on
a
active
Facebook
page
can
get
up
to
a
million
views.
A
A
So
this
simple
casino
data
model
allowed
us
to
query
in
real
time
the
repeated
content.
So
this
is
how
we
did
it.
We
take
the
social
media
content
hash,
it
put
it
as
a
row
key
and
then
the
column
names
are
the
metadata
of
that
social
media
content
so
that
it
could
include
things
like
the
timestamp
and
then
the
item
identifier,
which
can
map
back
to
other
things
about
that
content.
A
It's
not
enough
that
we
find
out
how
many
times.
So
what
with
this
data
model,
you
can
count
the
number
of
columns
on
that
row
key
and
you
get
back
how
many
times
that
content
has
been
seen.
It's
not
enough
to
see
how
many
times
that
content
has
been
seen
to
flag
it
as
spam.
So
that's
why
we
include
this
metadata
in
a
column
names.
We
go
back
into
offline
analysis.
We
might
be
able
to
do
time,
series
analysis
to
find
patterns
or
content
analysis,
which
is
important,
because
sometimes
you
might
get
a
multiple
choice.
A
Question
on
the
social
media
page
and
the
answers
could
be
a
b
b,
b,
b
c
BB
b.
So
you
get
a
lot
of
the
repeated
content
as
the
letter
B,
but
that
that's
not
pretty,
that's
not
bad.
So
we
want
to
go
back
and
retune
or
repeated
content
spam
algorithms
to
perhaps
put
a
threshold
on
the
number
of
words
a
number
of
characters.
Things
like
that.
So
that's
been
pretty
powerful.
A
Another
use
case
we
have
for
social
media
with
Cassandra
is
trending
topics,
so
sometimes
a
brand
or
company
might
announce
the
release
of
a
product.
Maybe
if
you're
a
your,
your
a
movie
and
there's
a
DVD
release
announcement
coming,
you
might
get
a
lot
of
people
coming
onto
that
Facebook
page
to
post
their
views
and
opinions
about
yeah.
This
DVD
is
coming
out
in
a
certain
date,
so
the
spammers
take
advantage
of
the
fact
that
there's
a
lot
of
eyes
around
this
activity
and
there's
so
there's
a
burst
in
activity.
A
We
wanted
to
create
a
center
data
model
that
could
detect
this
person
activity.
So
we
we
call
it
social
mob
or
a
spike
in
their
tivity,
but
it's
not
enough
to
know
that
there's
a
spike
in
activity
because
sometimes-
and
this
happens
quite
often
for
us-
it's
very
benign
spike-
you
might
have
someone's
birthday.
So
you
get
a
lot
of
people
saying
happy
birthday
and
there's
a
huge
spike
in
activity,
but
it's
not
so
bad.
A
So
it's
not
enough
to
know
that
there's
a
spike
in
the
time
series
data
we
also
are
interested
in
the
content
of
it
too,
and
the
cool
thing
about
this
use
case
is
that
a
single
sick
casino
data
model
can
solve
both
of
those
cases
you
can
find
out
when
the
spike
is
happening
and
what
it's
talking
about.
So
let's
go
into
that
data
model
a
little
bit.
It's
actually
just
a
textbook
use
case
for
Co
Center
in
time
series
modelling
the
goal
is
to
get
the
number
of
times.
A
A
topic
has
been
talked
about
in
an
arbitrary
period
of
time.
So
the
way
we
do,
that
is
for
the
we
use
as
a
we
use
a
partition
composite
partition,
key,
which
is
the
account
ID
and
the
year
month
that
the
message
was
posted.
This
ensures
that
all
the
data
from
the
same
account
in
the
same
time
period
will
live
on
the
same
note,
so
that
makes
for
faster
querying
and
then
the
rest
of
the
column
names
will
will
include
things
that
we
would
like
to
curry
on,
so
that
we
can.
A
We
include
the
timestamp
down
in
the
minute,
which
means
that
we
can
now
clearly
the
time
range
down
to
the
minute,
not
just
the
month
on
the
road
key.
We
also
include
the
topic
so
that
we
can
now
know
what
topics
are
being
talked
about,
and
then
we
include
the
item
identifier,
so
that
we
can
map
back
to
what
the
topic
actually
is.
The
contents
of
this
message,
things
like
I,
said
before.
In
the
other
slide,
we
can
look
back
to
who's
posted
the
time.
Do
time,
series
analysis,
offline,
etc.
A
A
Okay,
so
our
last
use
case
it
was
is
with
archive
search
basically
by
using
some
of
the
tools
and
data
stacks
Enterprise
we've
been
able
to
build
a
lightweight
version
of
archive
search
on
social
media,
so
this
has
fulfilled
a
lot
of
our
compliance
requesters
from
customers.
They
can
go
ahead
and
search
there
throughout
all
the
posts
they've
ever
made
or
comments.
That's
been
made
on
their
page
just
by
doing
open-ended
searches.
A
So
I
want
to
mention
finally
that,
with
this
use
case
by
combining
this
use
case
with
the
previous
use
case,
we've
actually
created
a
very
powerful
moderation
tool.
What
happened
just
last
week,
that
was
that
we
found
a
spam
campaign
through
our
training
topics
about
weight
loss
pills,
so
it
occurred
on
some
account.
There
was
a
spike
in
activity,
weight,
loss
pills
and
then
we
use
archive
search
to
search
for
some
of
these
keywords
and
that's
a
campaign
and
we
were
able
to
moderate
all
the
all
the
other
accounts
that
also
had
this
campaign.
A
Some
of
these
took
advantage
of
distributed
nature,
and
then
we
other
tools
can
easily
plug
in
like
solar
and
spark
the
fact
that
it's
easy
to
use
and
that
there's
amazing
community
support
so
that
that
concludes
my
talk.
So
thanks
Davis
extra
for
hosting
the
event
and
thanks
for
everyone
joining
me
today,.
A
C
C
A
So
I
can
speak.
I
could
definitely
speak
on
the
social
media,
so
I
think,
but
not
for
the
email,
but
for
social
media.
How
do
we
check
our
Danica
Center?
So
our
our
application
is
built
on
top
of
rails
and
we
we
can.
If
we're
installed
on
a
Facebook
page,
we
will
we
have
real-time
updates
and
as
soon
as
the
content
comes
in,
we
just
write
a
straight
to
Cassandra.
D
This
was
a
good
presentation.
Thank
you
very
much.
Thank
you.
I
wanted
to
go
back
to
the
second
last
use
case.
You
had
on
the
benign
spikes
between
the
difference
between
a
happy
birthday
and
a
nasty
comment,
and
you
you
said
something
about
the
way
you
parse
it
or
which
I
missed,
that
how
you
differentiate
between
benign
in
and
non
benign
right
and
I.
A
A
That
yeah
sure
okay.
So
the
question
to
that
is
that
it's
actually
being
sent
as
an
email
alert
and
so
we're
doing
this
manually
right
now
and
we're
gonna
find
a
way
to
look
at
more
features
to
determine
when
a
topic
is
benign
versus
when
it's
bad,
but
at
the
moment
we're
getting
alerts
on
trending
topics,
so
we're
looking
at.
So
we
get
the
top
a
list
of
top
ten
topics
when
they're
whenever
there's
a
spike
and
then
we
can
alert
to
the
user
that
Oh
something
is
happening
so.
D
D
A
D
D
So
you
have
persistently
stored
the
data,
say
a
hundred
thousand
tweets
or
Facebook
posts
in
the
last
one
hour,
for
example.
Okay,
now
do
you
go
back
and
look
at
it
again
to
find
out
benign
non
benign
or
does
the
model
live
in
memory
and
as
the
data
is
going
in,
is
parsing
it
and
building
a
pattern
or
a
histogram?
D
A
A
D
B
A
F
The
threat
database
you're
storing
a
graph
right
and
you're
storing
each
row
has
a
key
and
that's
the
vertex
and
each
of
its
edges.
Outbound
are
columns
right,
so
you
haven't
you're
using
the
wide.
The
wide
row
is
correct.
I
was
just
wondering
if
you
were
storing
back
edges
to
each
of
the
vertexes.
That's.
A
A
B
So
sadly,
no
we
don't
use
the
encryption
from
Cassandra
from
data
stacks
and
part
of
the
reason
of
that
is
because,
as
you
saw,
we've
been
deployed
with
Cassandra
for
a
long
time
in
production
right
since
2012.
So
that's
one
of
the
aspects
of
data
stacks
Enterprise
that
we
have
not
deployed.
We
use
third-party
tools
to
do
disk
encryption
of
in
this
case.
B
The
question
is:
what
is
the
use
case
that
drives
the
encryption?
The
use
case
is
number
one.
It's
just
a
bad
idea
to
not
encrypt
your
data
on
disks
in
Amazon
right
those
disks
get
recycled.
There
are
reports
of
sometimes
an
instance
attached
storage,
getting
attached
to
a
new
instance
and
has
the
old
data
on
it.