►
Description
The first session of the Big Data Special Interest Group, expands on a a regular Commons session also titled Big Data and Apache Spark on OpenShift.
In the previous session, Red Hat’s Will Benton gave us a vocabulary for talking about data-driven applications and outlined some example architectures for building data-driven applications with microservices. In this session, he’ll provide a tutorial introduction to Apache Spark and walk through an example data-driven application. You’ll learn how to incorporate data-driven behavior into your applications and start building these capabilities with Apache Spark.
William Benton leads a data science team at Red Hat, where he has applied analytic techniques to problems ranging from forecasting cloud infrastructure costs to designing better cycling workouts. His current development focus is contributing to open-source distributed computing projects, but he has also conducted research and development in the areas of static program analysis, managed language runtimes, logic databases, cluster configuration management, and music technology. Benton holds a PhD in computer sciences from the University of Wisconsin.
For the latest information on OpenShift 3.1 and available briefings, please visit http://commons.openshift.org or subscribe to the OpenShift Blog (https://blog.openshift.com).
The OpenShift Commons exists to provide a platform for customers, partners, developers and other open source technology initiatives to collaborate, share and accelerate the pace of innovation and adoption of OpenShift globally.
The OpenShift Commons represents a new open collaborative community model designed to facilitate communication and sharing of best practices, feedback and development across the many open source initiatives that integrate with OpenShift. The best way to get involved is to join the conversation today at http://commons.openshift.org
A
A
So
hello
and
welcome
everybody.
This
is
our
first
official
Big
Data
sig
meeting.
We
had
a
we
co-opted,
a
Commons
briefing
a
month
or
so
ago
and
had
we'll
who's.
The
speaker
today
give
an
introduction
to
big
data
and
all
things
about
big
data
in
relationship
to
open
ship
and
new
cloud
architectures,
which
was
absolutely
wonderful
and
that
you
are
well
off
into
the
cat
room
without
reporting.
It's
also
available
on
the
open,
shipped
YouTube
channel.
But
today
what
we
wanted
to
do
was
follow
it
up
and
kick
off
the
open
shift.
A
Ominous
big
data
sting
with
part
two
talking
about
big
data
and
Apache
spark
in
relationship
to
open
running
it
on
open
ship
and
we'll
have
graciously
said
he
would
be
here.
Even
though
August
is
the
month
of
everybody's
vacations,
so
without
further
ado,
I'm
gonna,
let
will
talk,
give
his
presentation
and
talk
through
all
this.
The
format
will
be
him
talking
for
20-25
minutes
and
then
we'll
open
it
up
to
Q&A
during
his
off.
Please
enter
any
questions
into
the
chat
room
after
his
talk.
B
Thanks
Diana,
so,
as
Diane
mentioned
last
time,
I
gave
one
of
these
talks.
I
talked
about
architectures
for
data-driven
applications.
Today,
I'm
gonna
give
a
quick
overview
of
Apache
spark
and
show
how
you
can
put
it
into
applications
now,
because
this
is
a
short
talk,
I'm
gonna,
sort
of
get
things
get
things
started,
give
you
an
introduction
and
sort
of
show
you
where
you
can
go
to
learn
more
and
give
you
a
place
to
explore
some
of
these
ideas.
B
I'm
also
gonna
show
a
simple
application:
that's
a
data-driven
application
using
Apache
spark,
but
nothing
I
do
in
that.
Application
would
be
something
you
want
to
do
in
production.
In
a
future,
my
colleague,
Mike
McCune
will
present
on
how
we
can
actually
put
an
application
with
a
spark
component
into
production
in
a
sensible
way
on
OpenShift.
B
So
before
we
get
started,
there
will
be
an
audience
participation.
Part
of
the
talk
later
on
and
I'd
like
to
ask
that
you
pull
down
this
docker
image
that
I
have
linked
to
on
this
slide
and
run
it
with
the
command
line
that
I
have
listed
here
and
I'll.
Give
everyone
a
few
seconds
to
do
that
if
you,
if
you
run
this
and
pull
it
down
now,
then
it
will
be
available
for
you
when
we
actually
get
to
the
part
of
the
talk
where
we'll
do
some
audience:
participation,
I'm.
A
A
B
And
then
I'm
also
going
to
paste
a
URL,
which
is
what
you
can
visit
after
you've
run
that
command
line
to
verify
that
everything
is
working
and
it
will
look
look
like
a
notebook
when
you,
when
you
get
that
running
okay,
so
a
little
bit
of
background.
Some
of
this
will
be
review
to
those
of
you
who
were
at
the
last
talk.
I
gave
on
application
architectures.
But
you
know
in
the
middle
of
the
last
decade,
MapReduce
was
a
popular
idea.
It
was
a
system
that
Google
published
a
paper
about
in
2004.
B
It
took
some
very
old
ideas
from
the
world
of
functional
programming
and
applied
them
to
data
processing,
and
the
idea
was
that
you'd
have
a
lot
of
relatively
modest
commodity
machines
with
slow
networks
and
slow
disks.
But
if
you
could
formulate
your
problem
in
such
a
way
that
you
could
distribute
the
storage
and
distribute
the
compute
to
go
to
the
storage,
you
could
scale
out
relatively
easily.
So,
as
we
know,
developing
distributed
applications
is
hard,
but
with
MapReduce.
The
idea
was
that
you
define
three
components.
B
So
this
was
a
lot
easier
than
writing
distributed
applications
in
the
old
way,
but
if
we
and
we
can
solve
a
lot
of
a
lot
of
problems
that
scale
out
naturally.
So
if
we
wanted
to
say
count
all
of
the
words
in
a
very
large
file
that
was
spread
across
multiple
computers,
we'd
map
each
file
to
a
sequence
of
say,
our
files
look
like
this
we'd
map,
each
file
to
a
sequence
of
word
occurrences.
So,
instead
of
a
word,
we'd
have
a
word
appearing
once
on.
The
key
is
the
word.
B
The
value
is
a
count
of
one
and
then
we'd
shuffle
these
pairs
around
the
network,
so
that
all
the
pairs
with
the
same
key
for
the
same
word
are
going
to
be
on
the
same
machine
and
then
finally,
we'd
reduce
these
by
adding
the
values
for
every
pair
together
with
the
same
key,
going
from
a
sequence
of
word,
occurrences
to
a
shorter
sequence
of
word
counts
a
year
a
year
after
the
MapReduce
paper
is
published.
The
Hadoop
project
produced
open
source
implementations
of
the
eye's
ideas.
B
B
Now
this
programming
model
was
so
beautiful
that
it
is
set
a
course
for
a
huge
segment
of
the
industry
for
the
last
decade,
I'm
kidding.
Of
course,
no
one
really
wants
to
write
code.
That
looks
like
that,
but
the
power
of
HDFS
is
a
data
Federation
solution
and
the
promise
of
eventually
realizing
massive
scale
out
were
extremely
attractive.
A
lot
of
smart
people
accepted
those
trade-offs
and
began
storing
their
data
in
HDFS
and
developing
code
and
libraries
atop
this
abstraction.
B
B
Hadoop
is
that
if
you're,
if
you're
thinking
about
Hadoop
you're
thinking
about
an
execution
model,
you're
thinking
about
a
way,
that's
easy
to
run
parallel
programs
and
you
sort
of
as
a
programmer
have
to
put
the
effort
into
translating
your
problem
into
that
model
with
SPARC,
you
start
with
a
programming
model
that
looks
like
something
you
would
use
as
a
programmer
and
figure
out
how
to
execute
this
across
multiple
machines
in
the
distributed
environment.
So
the
fundamental
abstraction
in
SPARC
is
called
a
resilient
distributed
data
set,
and
this
is
a
partitioned,
immutable
and
lazy
collection.
B
Partitioned
because
it
can
be
spread
across
multiple
machines,
it's
immutable
in
that
nothing!
You
do
to
one
of
these
changes
it
you
just
create
a
new
one,
and
it's
lazy
in
that
computations
only
happen
when
they
have
to
and
because
computations
only
happen
when
they
have
to
and
because
we
don't
actually
change
the
underlying
collection.
B
We
have
two
kinds
of
operations
on
these
resilient
distributed,
datasets
transformations
which
create
new
resilient
distributed,
datasets
that
encode
a
graph
of
dependency
information
to
a
parent
distributed
data
set,
and
we
have
an
action
which
is
where
the
actual
work
gets
done.
This
is
something
that
schedules
a
cluster
job
to
compute
values
in
the
RTD,
so
that
you
can
get
an
ultimate
value
out
of
it.
B
These
are
resilient
because
when
we
have
a
distributed
application,
we're
inevitably
going
to
have
failures
and
failures
mean
that
individual
partitions
of
these
datasets
can
disappear,
but
because
of
immutability
and
laziness,
we
always
know
how
to
reconstruct
partitions
when
they
go
away
and
when
we
get
a
new
machine
to
create
them
to
it,
to
run
that
job.
So
I'm
going
to
give
you
just
a
quick
overview
of
some
of
the
operations
we
can
do
on
these
things.
B
The
first
step
is
creating
an
RDD
and
you
can
create
an
RDD,
backed
by
a
local
programming
language
collection
from
a
file.
That's
either
on
a
local
file
system
or
in
an
object,
store
or
from
a
distributed
file.
That's
possibly
partitioned
across
many
machines.
Now
there
are
other
ways
to
create
an
RDD,
but
these
are
just
a
few
of
the
few
of
the
first
ones.
We'll
look
at
we're
talking
about
the
rest
of
this
programming.
Interface.
I
want
to
take
a
quick
sidebar
and
talk
about
some
type.
B
Another
useful
operation
is
the
flat
map
operation
where,
instead
of
mapping
each
element
to
another
element,
a
sort
of
one-to-one
mapping.
We
map
each
element
to
a
sequence
of
zero
or
more
elements,
and
we
concatenate
those
together.
You
can
do
a
lot
of
really
powerful
and
interesting
things
with
that.
B
B
We
can
combine
values
together,
as
we
did
in
that
sort
of
word
count.
Example
by
taking
a
commutative
and
associative
binary
operation
on
the
value
type
and
applying
it
so
that
we
will
wind
up
with
one
value
for
each
key.
That's
the
result
of
combining
all
of
the
values
for
that
key
in
the
parent
ID
D,
and
we
can
also
do
a
database
style
join
now
again.
These
things
all
operate
lazily,
so
we
haven't
done
anything.
Yet.
B
B
We
can
get
ount
of
values
in
the
RDD.
We
could
reduce
in
a
similar
way
that
we
would
reduced
by
a
key
by
combining
all
the
elements
in
an
RDD
with
a
commutative
and
associative
binary
operation,
and
we
can
save
the
file
save
to
a
file
either
locally
or
as
a
partition
file
on
a
distributed
file
system.
B
So
that
word
count
example
that
we
saw
in
Hadoop
earlier
that
took
a
whole
bunch
of
lines
of
code
is
actually
pretty
simple
in
spark,
and
here
we're
here
we're
using
this
in
Python.
There's
a
there's
a
typo
on
this
slide,
I
apologize
for
that.
But
if
we
create
an
RDD
from
a
text
file
on
the
first
line,
I'll
be
building
up
a
dependency
graph
along
the
side
of
this
slide.
B
Then
we
map
each
word
into
a
word
occurrence,
so
we're
saying
we've
instead
of
saying
we
have
a
word
we're
saying
we
have
a
word
that
we've
seen
once
and
then.
Finally,
we
use
that
reduce
by
key
function,
which
moves
data
around
the
network
to
get
all
of
the
records
for
each
given
word
together
and
adds
together
all
of
those
counts.
B
B
Alright,
so
we
we
actually
have
not
computed
anything
at
this
point,
because
the
only
things
we've
done
to
this
distributed
data
set
are
transformations
when
we
get
to
the
next
line,
where
we
have
an
action.
That's
when
spark
will
actually
schedule
these
jobs
on
our
cluster
to
run
this
computation,
and
that
will
look
something
like
this.
B
It
will
coalesce
together
all
of
the
things
that
can
do
at
once
on
on
a
given
node
in
order
to
minimize
communication
time
and
run
those
and
then
finally
will
get
a
a
file
with
all
of
our
all
of
our
results
in
it.
So
this
RTD
is
a
pretty
convenient
way
to
write,
distributed
applications,
but
it's
a
very
low
level
and
it's
not
it's
not
the
most
popular
way
to
use.
Sparc
I
just
introduced
it
because
it
is
really
fundamental
to
how
SPARC
works
and
understanding
it
will
make
understanding.
B
Other
things
fork
does
easier,
but
we
can
go
beyond
rdd's.
A
lot
of
people
want
to
do
interesting
things
with
their
data.
Different
applications
and
one
example
of
something
you
might
want
to
do
is
train
predictive
models
with
machine
learning
on
SPARC
has
a
lot
of
implementations
of
classical
machine
learning,
algorithms
and
models
that
you
can
evaluate,
and
it
has
some
primitives
for
efficiently
implementing
your
own
machine
learning
models.
B
So
my
team
has
had
success
both
using
the
ones
built
in
SPARC
and
implementing
our
own
machine
learning,
algorithms
on
top
of
SPARC,
unlike
a
lot
of
systems
that
have
different
api's
for
you
for
a
processing
batch
data
that
you
have
all
at
once
and
streaming
data
that
arrives
a
record
at
a
time.
Sparc
says
well
what,
if
we
looked
at
each
window
of
a
stream
has
its
own
tiny
batch.
B
It
might
give
you
it
might
wind
up
evaluating
a
more
efficient
program
so
with
spark.
If
you
write
arbitrary
programming
language
code
in
your
RDD
operations,
spark
can't
know
anything
about
those
and
rearrange
them
to
improve
the
performance.
But
if
you
use
a
high
level
data
specific
language
for
queries,
spark
can
generate
optimized
execution
plans
for
your
code
and
just
as
an
example,
let's
say
we
have
an
RDD
program.
B
If
we
just
sort
of
do
this
naively
we're
looking
at
the
cross
product
of
two
very
large
sets
and
we're
going
to
reject
almost
everything
if,
instead,
we
rearranged
this
program
so
that
we've
first
filtered
out
and
only
took
those
people
with
extremely
high
net
worth
and
then
join
that
with
only
the
people
from
the
criminals
table
who
had
felony
convictions,
it
would
be
a
much
more
efficient
program
and
spark
offers
a
few
different
ways
to
do
these
structured
queries.
You
can
actually
write
sequel
statements
against
spark
and
that
will
work.
B
If
you
have
a
syntax
error,
your
program
will
crash,
but
that's
pretty
much
the
case
with
with
any
database
there's
also
a
couple
of
Dino
main
specific
languages
for
doing
these
sorts
of
queries
in
a
safer
way
and
we'll
look
at
those
now
I
those
of
you
who
went
to
that
docker
image.
Let's
pull
that
up
and
I
will
pull
it
up
on
my
on
my
screen
here.
Can
everyone
see
this.
B
So
I'm
not
going
to
go
through
all
of
this
I'm,
just
gonna
sort
of
point
you
to
how
you
can
explore
this.
This
is
really
a
self-contained
notebook.
You
can
see
there's
a
lot
of
a
lot
of
stuff
in
there,
but
I'm
just
gonna
go
through
and
sort
of
show
some
of
the
stuff
we
talked
about
earlier.
So
what
we're
doing
at
the
top
here
is
we're
creating
a
spark
context,
which
is
how
we
sort
of
deal
with
a
spark
scheduler,
create
these
distributed
collections
and
and
schedule
computations
on
them.
So
we
have
some.
B
So
we
can
run
some
of
these
actions
on
the
RTD
as
well
and,
as
you
can
see,
we
get
we
get
the
results
we
would
expect
from
those-
and
here
is
next
I'm
going
through
this
sort
of
quickly,
because
I
expect
people
will
want
to
revisit
this
on
their
own
time.
But
I
just
want
to
provide
some
context,
so
it
makes
sense
and
then.
B
If
we
want
to
use
this
structured
query
support,
we
can
load
a
structured
data
file
in
this
case.
I
have
a
file
in
the
park'
format,
which
is
an
efficient
format
for
storing
relational
style
data,
with
some
log
messages
from
the
Fedora
projects
infrastructure.
And,
as
you
see
here,
we
can
do
a
basic
database
style
aggregation
with
pretty
straightforward
code,
and
we
get
results
of
all
of
these
messages.
B
B
So
as
you
go
on
in
this
example,
there's
a
sort
of
overview
of
how
you
could
do
some
data
cleaning,
which
is
an
important
part
of
most
real-world
data-driven
applications
and
also
at
the
bottom,
a
building
a
training,
a
machine
learning
model
on
this
on
this
data.
So
that's
great
for
experimenting
with
SPARC
I
hope
that
people
will
be
able
to
try
that
out
and
have
some
fun
with
it.
But
I
want
to
show
what
an
an
actual
application
that
has
a
data-driven
component
in
it
looks
like
so
we'll
go
on
to
that
now.
B
The
model
we're
going
to
use
in
our
data-driven
application
is
called
word
Tyvek
and
word
Tyvek
is,
is
a
really
cool
technique
in
natural
language
processing
that
encodes
words
as
vectors?
Hence
the
name
word
avec.
The
cool
thing
about
these
vectors,
though,
is
that
they
encode
some
meaningful
semantic
information.
B
It
learns
words
that
are
synonyms
because
those
words
appear
in
similar
contexts.
So
if
we
have
words
that
are
synonymous,
they
will
have
vectors
with
similar
angles,
and
the
other
cool
thing
about
these
vectors
is
that
we
can
use
basic
linear
transformations
on
them
to
find
analogies.
For
example,
if
we
take
the
vector
for
Madrid
and
subtract
the
vector
for
Spain
but
add
the
vector
for
France,
we
get
a
vector
that
looks
a
lot
like
the
vector
for
Paris
now
there's
an
implementation
of
this
word.
B
Now
we
can
incorporate
this
into
an
application.
In
this
case,
I
have
a
very
simple
flask
application,
so
it's
just
a
basic
web
application
and
I
have
some
routes
and
I
basically
just
interact
with
the
spark
context
and
this
model
training
code
at
while
I'm
processing
requests
to
each
of
these
routes.
B
B
If
my,
if
my
notebook
doesn't
crash,
I
have
I'm
going
to
create
a
model
and
I'm
going
to
call
the
model
Austin
and
the
training
corpus
is
going
to
be
the
Project
Gutenberg
text
of
all
of
the
Jane
Austen
novels
that
are
on
party
gutenberg,
which
is
on
my
own
website,
because
project
gutenberg
rate
limited
me
earlier,
while
I
was
developing
this
application,
so
I'm
going
to
click
the
trainer
model
button
and
it's
gonna
fire
off
a
spark
job.
In
the
background,
this
is
going
to
go
for
probably
about
fifteen
or
twenty
seconds
and.
B
B
So
here
we're
query
against
the
model
Austin
and
it's
gonna
give
us
some
randomly
selected
words
to
say:
do
you
want
to
find
synonyms
for
some
of
those
I'll
take
a
word
that
I
know
is
in
Pride
and
Prejudice
and
find
synonyms
for
that,
and
if
you
find
synonyms
for
Darcy,
who
is
a
character
in
Pride
and
Prejudice,
you
get
a
list
of
other
characters
in
Pride
and
Prejudice.
So
it's
sort
of
a
sort
of
a
cool
application
of
taking
a
bunch
of
text,
training
a
model
on
it
and
learning
something
interesting
about
that
text.
A
But
now
now
you've
got
me
with
visions.
Thank
you
so
much
with
all
playing
it
on
my
favourite
data
sources,
the
public
government
open
source
or
open
gov
sites
for
financial
reporting
and
public
reporting,
so
that
we
can
do
that
vector
of
criminals
versus
Forbes
list
people
on
the
corporate
reporting
filing
lots
of
stuff.
To
think
about
it.
Big
data
is
one
of
those
areas
where
the
tools
have
just
exponentially
grown
and
made.
A
They
made
life
much
more
interesting,
so
maybe
we'll
catch
the
next
month
of
Stewart,
even
move
it
back
on
some
corporate
reporting
from
the
FDIC
or
the
FDIC
that
the
SEC
yeah
I'm,
totally
psyched.
After
this,
this
has
been
defined.
So
there
were
not
a
lot
of
questions.
I
was
incredibly
happy
to
see
you
using
the
jupiter
notebook
and
the
docker
image.
It
looks
like
no
one
had
any
real
trouble
downloading
the
actual
image
of
that.
That
was
absolutely
awesome.
A
A
So
the
question
you're
welcome
to
raise
your
hand
on
Diane
sedima
and
thank
you
if
you're
back
from
Colorado
and
you're
back
in
Victoria
area,
we
need
to
talk
because
of
the
little
people
in
bankers
in
the
Vancouver
area
that
I'd
love
to
get
you
to
talk
about
I'm,
going
to
work
at
you're,
doing
around
big
data
as
well.
Okay,.
A
C
B
So
that's
that
that
is
a
good
question,
so
we
don't
have
like
an
exhaustive
performance
evaluation.
Yet
that's
actually
something
we're
working
towards.
We
are
using.
We
are
using
spark
on
OpenShift
internally
and
we're
not
noticing
I,
guess
I
would
say
we're
not
noticing,
obviously
worst
performance
than
we.
Then
we
saw
running
under
Apache,
maysa
or
running
standalone.
C
A
C
C
Mean
yeah
many
of
our
customers
are
using
it
and
we're.
You
know:
I'm
heading
the
the
past
and
big
beta
practice
here
systems
and
we're
combining
you
know
the
the
architectures
that
you
know
the
Big
Data
folks
put
together
and,
and
we
have
fused
open
ship
out
for
quite
some
while
on
the
on
our
you
know,
a
traditional
cloud
systems
and
what
we're
facing
is
basically
with
very
large
data
sets.
You
know
the
the
cloud
really
you
know
comes
pretty
fast
to
its
limits.
C
So
this
is
what
I
were
basically
trying
to
put
openshift
on
a
on
a
bare
metal
and
then
should
not
be
an
issue
to
develop
a
spark
or
you
know.
Are
there
any
other
big
data
applications
once
you
hit
a
certain
threshold
and
you
don't
need
to
have
you
know
you
don't
need
to
change
your
your
platform
underneath?
So
that's
the
that's
the
beauty
that
we
that
we
see
there
and
probably
in
about
six
weeks
times,
we
should
be
ready
to
announce
that
at
architecture.
A
Awesome
it's
on
in
so
what
I
would
love
to
do
is
have
you
kind
of
talk
about
that
as
a
reference
architecture
in
one
of
the
upcoming
sig
meetings?
Sure
people
would
like
that
to
hear
what
you're
doing
there
and
we
can
touch
base
if
you're
grading,
to
do
that
with
six
weeks
or
if
you
want
help
on
that
door.
A
A
All
right,
I'm
not
hearing
any
other
questions,
which
is
quite
okay,
because
that
was
a
lot
to
chew
on
what
I'll
probably
do
is
drop
this
video
on
the
lapse
of
the
web
web
folks
and
post
it
as
a
blog
post,
with
all
the
links
that
will
gave
at
to
the
image,
and
that
should
be
up
by
probably
Friday
on
Monday
at
the
latest
and
they'll
schedule.
The
next
big
data
stick
probably
into
the
three
weeks.
Time
depends
on
holidays
as
well,
so
with
that
will
it
is.
B
Well,
it
is,
it
is
sort
of
hard
to
really
give
an
overview
of
a
library
like
spark
in
in
a
short
time.
So
I
would
say
you
know
if
you
try
this
stuff
out
and
have
any
questions
feel
free
to,
let
me
know
or
or
reach
out
to
the
community,
and
we
can.
We
can
keep
the
discussion
going
thanks
for
the
opportunity
to
give
the
talk.
Everyone
and
thanks
for
attending.
A
Well,
stay
tuned
for
more
big
data,
sig
topics
and
email
me
with
anything
that
you'd
like
to
hear
and
you
can
always
find
us
in
Tomsk,
OpenShift,
org
and
all
the
schedules
or
generic
talks
on
briefings
and
specific
ones
on
singh
will
be
listed
in
the
calendar
there
and
one
scene.
One
thing:
thank
you
very
much
for
attending
and
we
all
do
both
that
works.
Avec,
app,
I'm,
going
to
go,
buy
it
out
now
take
care
all.