►
From YouTube: Wibidata: Using the Open-Source Kiji Project to Build a Personalization Application on Cassandra
Description
Speaker: Clinton Kelly, Member of Technical Staff at Wibidata
Cassandra’s scalability and robust feature set make it a natural choice for building personalized Big Data Applications such as product recommenders, personalized search engines and fraud detectors. However, creating such applications requires a lot of time, resources, and expertise to build the additional functionality needed in addition to the Cassandra platform. Enter: the Kiji Project. Kiji is an open-source, modular platform that provides developers a head start in building real-time, Big Data Applications on Cassandra. Created by engineers with experience building personalized applications at companies like Google, Cloudera and Amazon, Kiji includes modules for capturing data, analyzing data, training machine learning models, and applying machine learning models in real time. Let their expertise work for you! In this talk, we provide an overview of the Kiji Project, detail a case study of how one Fortune 500 retailer uses Kiji in production for product recommendations, and discuss how Kiji works with Cassandra.
A
Okay,
I'm
gonna
go
ahead
and
get
started
good
afternoon.
Everyone,
my
name,
is
clint
kelly.
I
am
a
member
of
technical
staff
at
wikidata
and
I'm
going
to
speak
for
the
next
30
or
40
minutes
about
kiji.
Kiji
is
an
open
source
platform
that
you
can
use
to
build
personalized
big
data
applications
on
top
of
cassandra,
first
I'll
step
through
the
agenda.
For
today,
I'm
going
to
begin
by
describing
the
problem
we're
trying
to
solve
when
we
build
kiji.
That
is
what
it
means
to
build
a
big
data
application
and
why
it
can
be
difficult.
A
Next
I'll
talk
about
how
kg
works
and
how
we
can
use
this
open
source
platform
to
more
easily
build
the
kinds
of
applications.
I'm
going
to
describe
third
I'll
talk
a
little
bit
about
how
we
use
kegee
in
production
and
describe
some
real-life
use
cases
where
we
use
kiji
to
build
real-time,
big
data
applications
and,
finally,
I'll
talk
about
how
we've
added
support
in
kiji
to
run
on
top
of
cassandra.
A
What
our
data
model
looks
like
and
some
of
the
pitfalls
we
ran
into
and
how
we
got
around
them
just
a
little
context.
We
originally
developed
kiji
to
sit
on
top
of
hbase
and
not
cassandra.
We
added
support
for
cassandra
this
year,
so
this
gives
you
a
little
flavor
of
what
it's
like
to
add
support
in
a
system
that
was
designed
for
hbase
for
cassandra
as
well.
A
So
first
the
problem.
Let's
say
that
you're
a
developer
and
you
want
to
build
a
personalized,
real-time
big
data
application.
I've
shown
a
few
well-known
canonical
examples
here
to
be
a
little
bit
more
specific.
Let's
say
that
you
work
at
a
retailer
like
this
acme
clothing
company
and
you
want
to
personalize
your
customers
experiences.
A
A
We
could
begin
most,
obviously
by
looking
at
the
contents
of
a
user's
shopping,
cart
and
what
products
he
or
she
has
purchased
or
viewed
in
the
past.
For
example,
if
I
buy
a
pair
of
trousers
or
if
I've
just
added
a
pair
of
trousers
to
my
shopping,
cart,
probably
makes
sense
to
recommend
to
me
a
shirt
or
a
pair
of
shoes
that
will
match
those
trousers.
A
A
A
These
might
be
users
whom
we
want
to
recommend
inexpensive
products,
and
we
can
continue
personalizing
in
this
manner
by
doing
things
like
looking
at
what
users
have
searched
for
previously
on
our
site,
how
they've
interacted
with
the
mobile
applica
with
our
mobile
application
and
so
on,
and
just
to
be
clear,
we
can
personalize
our
application
in
ways.
Far
beyond
just
product
recommendations,
as
a
couple
of
examples,
we
could
serve
personalized
content
to
the
user's
mobile
device.
A
A
A
good
place
to
start
is,
with
all
of
the
proven,
scalable
open
source,
big
data
software-
that's
available
out
there
if
we're
going
to
be
building
a
personalized
application,
we're
going
to
need
to
record
a
lot
of
user
interactions.
So
all
those
things
I
discussed
before
like
search
terms,
things
you've
added
to
your
shopping,
cart,
price
filters
and
so
on
a
good
place
to
store
that
data
is
apache.
Cassandra
which
you've
all
heard
today
is
great
for
this
kind
of
high
volume
high
velocity
time
series
data.
A
Once
we
have
this
user
interaction,
data
stored
in
cassandra
we'll
need
a
way
to
analyze.
It
hadoop
is
a
great,
is
the
natural
choice
for
analyzing.
This
kind
of
high
volume
data
and,
along
with
cassandra
and
hadoop,
there's
a
pretty
large
ecosystem
of
other
proven
scalable
big
big
data
technologies.
We
can
use.
A
A
So
these
technologies
give
us
a
good
start,
but
we
still
have
a
lot
of
functionality
that
we
need
to
build
on
top
of
them
to
get
to
our
final
personalized,
big
data
application.
So
let's
talk
a
little
bit
more
about
what
that
functionality
is
that
we
need
to
build
to
begin
with.
We
need
a
way
to
get
all
this
information
about
our
users
into
cassandra.
A
We
may
design
our
applications
such
that
we
record
a
lot
of
user
interactions
to
log
files
or
relational
databases.
We
can
use
hadoop,
mapreduce
or
similar
technologies
to
bulk
load.
This
data
into
cassandra,
we'll
likely
also
want
to
design
some
kind
of
a
restful
interface
that
will
allow
us
to
stream
user
interactions
as
they
occur
in
real
time
and
write
them
to
our
database.
A
A
The
data
scientist
again
oops
again
could
go
about
doing
this
training
with
standard
data
science
tools
like
r
or
python.
These
are
familiar
and
very
accessible,
but
they
usually
don't
scale
out
to
the
quantity
of
data
that
we
can
hold
in
a
data
store
like
cassandra
running
analyses
on
these
large
data
sets
again.
A
Hadoop
is
a
natural
choice,
but
writing
mappers
and
reducers
is
usually
not
the
best
job
for
a
data
scientist
and
anyone
who's
worked
with
this
technology
knows
it's
not
terribly
accessible,
so
a
big
part
of
our
personalized
application
and
building
it
is
going
to
be
providing
some
tools
that
provide
a
good
balance
between
the
accessibility
and
familiarity
of
r
and
python,
and
the
scalability
of
in
hadoop
analysis.
So
here
are
a
couple
open
source
examples
that
we
might
find
useful
getting
started
with
such
an
application.
A
As
an
aside,
when
I
talk
about
this
trained
machine
learning
model
for
the
purposes
of
the
example
I'm
showing
today,
we
can
think
of
it
as
a
function.
The
inputs
of
this
function
are
all
the
data
that
we've
gathered
about
how
a
given
user
has
interacted
with
our
application.
So
again
everything
that
the
user
has
added
to
his
or
her
shopping,
cart,
clicks,
impressions,
views
add
to
wishlist,
etc.
A
The
output
of
this
function
is
a
set
of
personalized
product
recommendations
for
this
user.
Again,
in
the
general
case,
this
could
be
something
totally
different,
but
for
the
purposes
of
this
application,
I'll
talk
about
product
recommendations,
in
any
case
after
the
our
data
scientist
has
trained
this
machine
learning
model,
we
then
want
to
apply
it
to
generate
product
recommendations
for
our
users.
A
So
how
do
we
do
that?
One
pretty
straightforward
way
to
do.
It
is
again
to
use
hadoop.
We
can
pretty
easily
write
a
mapreduce
job
that
will
read
all
the
information
we
have
about.
All
of
our
users
out
of
cassandra
in
a
large
batch
operation,
apply
the
machine,
the
machine
learning
model
and
generate
a
set
of
personalized
recommendations,
one
set
for
each
user.
This
is
really
straightforward.
This
is
good
enough
for
a
lot
of
applications
and
it's
pretty
easy
to
make
this
scale
and
make
it
reliable.
A
A
Our
application
will
talk
via
rest
to
cassandra
and
make
a
query
for
the
latest
set
of
recommendations
for
this
user
to
show
them
on
the
screen.
Cassandra
will
have
some
of
these
recommendations
for
every
user
from
the
batch
job
I
described
earlier
so
it'll
serve
them
back
via
rest
to
the
application.
A
The
application
will
serve
them
to
the
user
and
he'll
see
his
his
shirts
and
trousers
and
so
on.
So
the
disadvantage
of
this
system
is
that
we
derive
these
customers,
these
product
recommendations
in
batch,
so
they're,
based
on
all
the
information
we
had
about
a
given
user
up
until
the
time
we
ran
that
batch
operation.
So
they
probably
don't
include
things
that
the
user
has
done
in
the
last
five
minutes,
or
maybe
even
in
the
last
hour,
and
if
we're
building
an
application
like
the
one
I've
described
here.
A
A
This
logic
sees
this
request
for
recommendations,
and
it
knows
that
recommendations
are
something
that
are
derived
from
applying
a
machine
learning
model,
so
the
application
logic
will
trigger
a
real-time
application
of
this
model.
These
models
are
usually
pretty
expensive
to
train,
but
not
very
expensive,
to
apply
or
score.
So
we
can
do
this
in
real
time
within
the
latency
of
our
applications
typically
very
easily.
A
So
now
we
get
the
freshest
possible
set
of
recommendations.
We
write
them
back
to
cassandra.
We
also
send
them
back
to
our
application
via
rest,
our
user
sees
higher
quality
content
and
hopefully
we
get
a
promotion
for
designing
a
better
application
to
review.
There
are
three
main
types
of
functionality
that
we
want
to
build
in
an
application
like
this.
A
This
is
a
fair
amount
of
tech
functionality
to
build,
on
top
of
these
proven
open
source
technologies
like
hadoop
and
cassandra,
and
it
was
with
this
in
mind
with
bridging
this
gap
between
these
open
source
tools.
In
your
final
application
that
we
at
wibby
data
developed
to
the
kiji
project.
This
is
an
open
source
platform
that
you
can
use
to
get
a
head
start
and
avoid
reinventing
the
wheel
when
you're
building
an
application
like
this
and
instead
as
developers,
get
to
concentrate
on
whatever
it
is.
That
makes
your
application
unique.
A
First,
a
little
bit
of
history
about
the
kiji
projects
we
developed
it
at
wikidata,
where
I
work
we're
a
startup
company
in
san
francisco,
founded
a
few
years
ago
by
folks
from
google
and
cloudera,
who
had
a
lot
of
experience.
Building
these
kinds
of
personalized
big
data
applications
on
top
of
open
source
technologies
like
hadoop.
A
Again,
if
you've
designed
your
application,
such
that
you
capture
a
lot
of
user
interactions
into
log
files
or
relational
databases,
or
maybe,
if
you
have
some
legacy
data,
that's
still
in
flat
files
on
hdfs,
you
can
use
a
component
called
kg
mapreduce
to
bulk
load.
This
data
into
kiji,
using
hadoop
mapreduce.
A
A
Again
we
can
train
the
machine
learning
model
using
our
python
that
has
the
downsides.
I
discussed
earlier
to
train
the
machine
learning
model
over
the
full
amount
of
data
we
can
store
in
kigi
schema
in
kiji.
We
have
a
couple
of
components.
The
first
is
kiji
mapreduce
in
kigi
mapreduce.
We
have
a
library
of
training,
algorithms
for
things
like
k-means,
k-means,
naive,
bayes,
collaborative
filtering
and
so
on.
A
Again,
if
mapreduce
is
not
the
most
accessible
technology,
so
if
you'd
like
something
easier
to
use,
we
provide
kg
express
pg
express,
is
a
scala
based
dsl
for
describing
mapreduce
jobs.
It's
based
on
scalding,
which
is
an
open
source
project
from
twitter.
It
provides
a
way
for
describing
fairly
complicated
chains
of
mapreduce
operations
in
scala
in
a
way
that
resembles
applying
functional
programming
techniques
to
scala
collections.
Libraries,
it's
also
very
similar
to
spark.
A
A
A
A
We
support
this
kind
of
batch
scoring
in
kiji,
but
we
also
naturally
have
a
mechanism
for
real
time
scoring.
There
are
a
couple
of
additional
components
we
use
for
this
they're
called
kg
scoring
and
the
kiji
model
repository
once
we've
trained
a
machine
learning
model
we
upload
the
associated
and
we're
ready
to
use
the
associated
score
for
real-time
scoring.
A
Now,
when
again,
if
we
go
through
our
previous
example,
when
a
user
visits
our
website,
our
application
makes
a
request
to
kiji
rest
for
some
data.
That's
stored
in
kiji
in
this
case
kiji
scoring,
is
that
application
logic
that
sees
a
request
for
a
column.
That's
a
result
of
applying
a
machine
learning
model.
A
A
We
want
a
way
to
inspect
our
data
and
then
train
machine
learning
algorithms.
For
this
purpose.
In
kiji,
we
provide
kiji
hive,
qg,
mapreduce
and
kg
express,
and
finally,
we
want
a
way
to
do
scoring
with
this
model
and
to
do
so
in
real
time
and
to
this
end,
in
kiji,
we
provide
the
model,
repository
and
kg
scoring.
A
One
other
note:
this
architecture,
I've
shown
is
highly
modular,
so
if
some
of
you
are
building
applications
that
resemble
this
but
use,
maybe
your
application
is
a
subset
of
what
I've
shown
or
some
kind
of
a
superset.
It's
really
easy
to
use
only
the
components
that
you
need
and
to
design
new
components,
maybe
some
kind
of
streaming
operation
with
kafka
and
storm,
for
example,
and
plug
it
into
this
architecture.
A
Now
I'll
talk
briefly
about
how
we
use
it
in
production
with
real
commercial
customers,
a
couple
of
use
cases
I
chose
the
highlight
today
we
use
kiji
in
one
fortune,
150
retailer
and
also
in
a
top
luxury
retailer
for
as
a
personalization
engine.
This
covers
some
of
the
use
cases
I
described
earlier.
Things
like
product
recommendations
and
so
on.
A
One
thing
one
other
note:
so
this
luxury
retailer
uses
kiji
and
also
uses
some
of
our
enterprise
software
that
we
build
at
wibby
data.
This
enterprise
software
is
an
example
of
the
kind
of
end-to-end
application
that
you
could
build
on
top
of
this
platform,
I'm
describing
today.
So
it
includes
a
lot
of
retail,
specific
functionality,
machine
learning,
models,
table
layouts
as
well
as
uis
consoles
and
so
on.
For
folks
that
work
at
companies
like
this.
A
To
get
data
into
kiji
with
this
particular
customer,
we
bulk
load
a
lot
of
historical
data,
so
these
are
things
like
product,
catalogs
or
customer
order
histories.
We
wrote
mapreduce
jobs
in
kigi,
express
that
take
these
flat
files
on
hdfs
and
load
them
into
cassandra
and
the
table
layouts
we're
using
cassandra
are
a
specific
retail
tale.
Excuse
me,
specific
retail
layouts
that
are
part
of
wiki
retail.
A
We
capture
real
time
interactions
that
users
have
with
this
application
using
kg
rest.
We
have
some
specific
excuse
me
some
javascript,
that
we
use
to
instrument
the
customer's
website.
So
whenever
you
do
things
like
clicking
on
something
or
adding
a
product
to
a
cart,
it'll
stream,
this
information
in
real
time
through
kg
rest
and
right
at
the
kiji
schema.
A
Our
data
science
team
or
the
data
science
team
at
the
customer
excuse
me,
does
all
kinds
of
analyses
using
jobs
that
are
written
in
kg.
Express
some
of
these
are
pretty
straightforward,
algorithms,
that
all
of
you
are
familiar
with
for
things
like
recommendations.
We
also
have
a
number
of
retail
specific
algorithms
that
are
part
of
wiki
retail.
A
And
finally,
we
serve
personalized
content
in
real
time
using
a
system
built
on
top
of
technologies.
Like
kg
rests,
so
we
have
an
endpoint
and
again
if
a
user
visits
a
web
page,
for
example,
from
this
customer,
this
endpoint
will
read
all
the
information
we
have
in
kigi
schema
about
that
customer
trigger
the
appropriate
score
and
generate
some
personalized
content
that
we
write
back
to
kiji
schema
and
also
serve
via
kiji
rest.
A
A
For
those
who
are
familiar,
bigtable
is
the
kind
of
ancestor
of
technologies
like
cassandra
and
hbase
in
kiji,
we
store
all
of
our
data
in
tables.
Tables
are
made
up
of
rows.
Rows
have
a
row
key
as
well
as
some
kind
of
a
data
payload
the
row
key.
We
call
in
kiji.
We
call
an
entity
id
an
entity.
Id
can
be
made
up
of
multiple
components:
some
hashed
and
some
unhashed.
This
is
very
similar
in
cassandra.
Now
you
can
have
a
partitioning
key
or
sorry
a
primary
key
that
has
a
partitioning
key
and
clustering
columns.
A
A
It
differs
in
that
we
offer
explicit
support
for
different
time
stamped
versions
of
every
cell
in
a
kiji
table.
This
is
similar
to
how
hbase
behaves
and
also
how
bigtable
behaves
under
the
hood.
Cassandra
also
stores
different
time-stamped
versions
of
every
cell,
but
as
users,
we're
not
exposed
to
those
and
cassandra
uses
them
instead
to
resolve
different
data
from
different
replica
nodes
within
each
cell,
so
users
of
kiji,
by
the
way,
can
use
these
different
timestamp
versions
for
doing
things
like
storing
time
series
data
within
each
cell.
A
A
As
a
motivating
example,
let's
say
that
you're
building
an
application
in
which
you
have
some
column,
families
that
contain
columns
that
you'd
like
to
use
for
batch
operations
and
some
that
contain
columns
that
you'd
like
to
use
for
real-time
operations.
Some
of
these
you'd,
probably
want
to
store
in
memory,
have
cache
all
the
time.
Others
you
might
want
to
compress,
for
example,
but
as
a
user,
you
might
find
it
more
convenient
to
think
of
these
columns,
as
all
logically
being
in
the
same
table.
A
So,
as
a
kegee
user,
you
could
set
up
a
couple
of
locality
groups,
one
of
which
you'd
use
for
your
batch
column,
families
and
one
for
your
real-time
column,
families,
and
these
will
get
these
column.
Families
will
now
get
stored
separately
on
disk,
so
we
can
assign
separate
attributes
to
the
column
families.
A
A
A
In
kiji,
we
also
allow
compare
and
set
operations
across
locality
groups.
This
is
something
that's
possible
in
hbase,
so
as
part
of
our
original
kiji
api-
and
this
is
one
of
the
few
examples
of
where
we
had
to
break
api
compatibility
and
just
not
support
this
in
the
cassandra
version
of
kiji,
for
what
it's
worth
in
production
with
our
enterprise
customers.
This
has
never
been
an
issue.
A
The
version
of
kiji
that
runs
on
top
of
hbase
can
have
filters,
for
example,
that
will
return
rows
only
that
have
data
values
that
match
a
certain
regular
expression,
and
this
is
something
that's
really
not
possible
in
cassandra
and
it's
kind
of
against
a
lot
of
the
design
philosophies
of
cassandra
itself,
where
we
usually
want
to
denormalize
our
tables
such
that
we're
only
doing
very
simple
reads
that
scan
contiguously
over
disk,
wherever
we
could.
We
use
where
clauses
to
implement
these
filters
in
the
cassandra
version
of
kiji.
A
So
if
you
have
filters
that
are
filtering
based
on
the
column
name
or
the
timestamp
value,
for
example,
those
are
things
we're
able
to
support.
We
ended
up
using
a
lot
of
allow
filtering.
Originally
you've
probably
heard
a
lot
of
pretty
bad
things
about
this
today
and
we
also
experimented
a
lot
with
secondary
indices.
A
This
didn't
work
out
so
well
when
we
started
dealing
with
production
level
payloads
on
big
clusters,
and
so
we
got
rid
of
all
of
those
and
our
our
fallback
solution
was
really
just
to
do
as
much
filtering
as
we
could
with
simple
where
clauses.
Otherwise
we
just
do
it
on
the
client
side
and
really
we
just
optimized
the
way
that
we
design
our
tables
for
things
like
the
retail
applications
I
described
earlier,
such
that
our
queries
never
have
to
use
these
kinds
of
they
never
have
to
use,
allow
filtering
and
never
require
secondary
indices.
A
A
A
A
A
A
I
wanted
to
thank
the
cassandra
community.
Those
of
us
all
of
us
at
whitby
data
really
didn't
have
experience
with
cassandra
before
this
year.
All
of
the
folks
on
the
mailing
list,
at
the
meetups
and
at
all
the
various
different
conferences
have
been
a
big
help.
It
was
also
really
great
to
have
all
the
webinars
conference,
videos
and
so
on
online
and
all
the
great
training
from
data
stacks.