►
Description
In this introductory Big Data session, Will be reprising his Red Hat Summit presentation and give us an overview into Big Data architecture and concepts to level help the playing field, help us figure out what a data-intensive application should actually look like on a modern container orchestration platform and help us kick off the OpenShift Common Big Data SIG.
In this session, you’ll learn about the anatomy of data-intensive applications, how they come to life, and what they have to accomplish. We’ll pick a few applications and explore their responsibilities, see how they use data, discuss trade-offs they must negotiate, and point to some example architectures that make sense for realizing data-intensive applications on OpenShift.
William Benton leads a data science team at Red Hat, where he has applied analytic techniques to problems ranging from forecasting cloud infrastructure costs to designing better cycling workouts. His current development focus is contributing to open-source distributed computing projects, but he has also conducted research and development in the areas of static program analysis, managed language runtimes, logic databases, cluster configuration management, and music technology. Benton holds a PhD in computer sciences from the University of Wisconsin.
For the latest information on OpenShift 3.2 and available briefings, please visit http://commons.openshift.org or subscribe to the OpenShift Blog (https://blog.openshift.com).
The OpenShift Commons exists to provide a platform for customers, partners, developers and other open source technology initiatives to collaborate, share and accelerate the pace of innovation and adoption of OpenShift globally.
The OpenShift Commons represents a new open collaborative community model designed to facilitate communication and sharing of best practices, feedback and development across the many open source initiatives that integrate with OpenShift. The best way to get involved is to join the conversation today at http://commons.openshift.org
A
A
A
And
we
will
have
another
part,
two
of
this
series
and
another
a
number
of
other
follow-on
guest
speakers
and
that
sig
to
kick
off
those
conversations
and
find
out
what
the
different
use
cases
are
for
big
data
in
the
context
of
open
shift
and
containers.
And
all
things
cloudy
like
that.
So
without
further
ado
that
all
I'd
like
to
give
well
Benten
the
floor
to
lead
us
through
the
beginnings
of
this
conversation
about
Big
Data,.
B
Now
big
data
is
sort
of
sort
of
a
marketing
term
right,
it's
useful
because
we
all
know
what
it
means,
but
you
know
you,
you
sort
of
get
into
this
question
of
how
much
data
is
big
data
and
I
think
that
it's
sort
of
an
interesting
question,
because
a
lot
of
real-world
data
processing
workloads
aren't
actually
all
that
big
in
an
absolute
sense
or
compared
to
the
kinds
of
hardware
we
have
to
run
them
on.
You
can
buy
a
machine
with
a
terabyte
of
RAM
pretty
easily
these
days.
B
That's
arriving
quickly
from
diverse
sources,
and
so
when
we're
talking
about
Big
Data,
we're
really
talking
more
about
a
set
of
challenges
than
about
a
particular
working
set
size
and
so
I'm
going
to
use
the
word
for
the
rest
of
this
talk,
data-driven
we're
going
to
explore
what
a
contemporary
data-driven
application
looks
like
in
containers.
So
I'll
tell
you
a
little
bit
about
my
background
to
start
out.
I
joined
Red
Hat
almost
eight
years
ago
and
started
working
on
cluster
scheduling
and
cluster
management
products
and
contributing
to
open
source
distributed
competing
projects.
B
For
the
last
few
years,
I've
been
working
primarily
with
big
data
and
data
processing,
and
for
the
last
two
years
in
particular,
my
focus
has
been
leading
a
team
of
data
scientists
and
engineers,
as
we
figure
out
how
to
use
cool
new,
open
source
frameworks
and
tools
to
solve
real-world
analysis
problems
at
real-world
scale.
So
I'm
coming
to
you
today,
just
to
share
my
perspective
as
someone
who's
developed,
these
tools
applied
them
and
put
them
into
production
in
a
contemporary
environment
and
an
open
shift.
B
So
here's
what
we're
going
to
talk
about
today,
I'm
going
to
start
with
some
background
about
what
the
future
of
application
development
is.
Looking
like
from
my
perspective,
we'll
talk
about
what
a
data-driven
application
has
to
do,
and
a
few
architectural
options
for
organizing
these
responsibilities
and
then
we'll
put
everything
together
by
showing
some
of
the
tools
and
frameworks
you
can
use
to
actually
implement
these
architectures
in
OpenShift.
B
So
this
first
point
I
hope
is
not
controversial.
I
think
for
many
people
on
this
call
or
watching
this
talk
later.
Micro
services
are
not
just
the
future
they're.
The
present
right
I
assume
that
many
people
have
developed
micro
service
based
applications
and
have
container
based
micro
service
applications
in
production
now,
but
for
those
people
who
aren't
yet
convinced
I'll,
just
sort
of
level
set
and
go
into
some
brief
definitions
and
talk
about
the
advantages
here.
B
So
if
we
structure
our
application
as
micro
services,
which
I'm
defining
as
primarily
stateless
lightweight
services,
that
communicate
via
well-defined
interfaces,
then
we
have
a
bunch
of
benefits
essentially
for
free
one
is
that
these
stateless
services
are
easy
to
deploy
and
manage,
because
we
can
run
them
in
stateless
containers
and
on
stateless
container
orchestration
platforms
like
OpenShift.
These
services
are
commodities.
B
If
one
of
them
goes
away,
we
can
spin
another
one
up
to
replace
it:
tribulus
sort
of
cattle,
if
you're,
using
the
pets
and
cattle
metaphor
they're
easy
to
scale
out,
because
you
can
run
them
on
a
single
machine
or
you
can
spread
them
out
across
multiple
machines.
If
you
have
one
service
that
turns
out
to
be
a
bottleneck,
you
can
load
balanced
behind
a
proxy
and
so
on.
B
These
services
are
also
easier
to
test
and
debug
than
their
stateful
counterparts.
Instead
of
managing
complicated
test
figure
fixtures
that
setup
a
bunch
of
state
before
each
test
and
sort
of
reproduce,
a
sequence
of
events
that
caused
a
stateful
service
to
break.
You
can
simply
ask
a
question
to
your
stateless
service,
get
back
an
answer
and
know
whether
or
not
you
have
a
bug.
B
So
those
are
all
technical
advantages,
but
there
are
also
really
political
and
social
advantages
to
this
kind
of
application.
Development
and
I
think
those
are
most
interesting.
It's
much
harder
to
organize
a
team
of
developers
to
work
on
a
big
monolithic
application
and
it's
harder
to
onboard
new
developers
onto
a
big
big
monolithic,
codebase,
that's
harder
to
diagnose
performance
or
correctness
problems
in
a
black
box.
B
So
when
I
say
that
data-driven
and
data
intensive
apps
of
the
future
I'm
cheating
a
little
bit
again
all
right,
this
is
sort
of
an
easy
way
to
make
predictions
about
the
future,
to
sort
of
say
something.
That's
already
happening
and
for
a
lot
of
people,
these
things
are
the
present,
but
increasingly
it's
not
enough
to
have
a
great
application
that
attracts
passionate
users.
You
need
to
take
the
data
to
your
users
provide
and
the
data
they
generate
and
use
it
to
understand
your
application
domain
and
the
world.
B
You
can
then
exploit
this
understanding
to
provide
additional
value,
offer
new
functionality
or
achieve
greater
efficiency.
People
are
really
getting
used
to
apps
that
get
better
with
popularity
and
longevity
and
will
increasingly
demand
these
data
driven
apps
in
the
future.
So
we'll
start
by
looking
at
a
few
example,
applications
that
have
been
ahead
of
the
curve
on
using
data
to
provide
a
better
customer
experience,
starting
with
a
particularly
famous
and
ubiquitous
one.
B
So
I'm
not
looking
at
the
chat
window
right
now,
but
I'm
gonna
assume
that
we
don't
do.
We
have
anyone
here
who
has
never
bought
anything
from
at
one
of
Amazon's
retail
websites,
I'm
assuming
not
if
so
I
guess
I
want
to
know
about
it,
but
I'm
mentioning
Amazon
retail,
not
just
because
it's
a
great
example
of
a
data
intensive
and
data-driven
application.
B
But
if
we
even
just
look
at
the
user
experience
on
their
site,
the
sort
of
user
visible
experience
of
Amazon,
we
can
see
some
obvious
shifts
in
how
they've
used
data
so
to
see
those
in
action.
Let's
think
about
what
retail
looked
like
before,
Amazon
will
have
to
go
back
to
the
mid-1990s,
so
some
of
you
may
not
have
been
shopping
in
the
mid
nineteen.
Ninety,
then,
you
may
need
to
recall
that
an
album
is
a
generic
term
for
a
physical
copy
of
an
audio
recording.
B
So
if
we
go
back
to
the
mid
1990s,
if
I
wanted
to
buy
an
album
in
1994
I'd
drive
across
town
or
maybe
across
the
county,
to
go
to
a
record
store
that
had
a
good
selection
and
knowledgeable
taste
making
clerks,
they
knew
me
and
they
knew
what
I
liked,
because
I
spent
a
lot
of
time
hanging
out
in
record
stores
in
the
mid
1990s
buying
albums.
So
they
could
recommend
recordings.
I'd
probably
enjoy
paring
down
their
selection
to
things
that
would
be
more
likely
to
fit
my
tastes.
B
These
clerks
also
knew
enough
about
music
to
be
able
to
answer
questions
about
their
stock
and
about
music
at
large,
even
from
people
they
didn't
know.
You
know,
I
just
saw
this
band
in
concert
which
of
their
albums
is
best
if
I
like
this
band
well,
I,
also
like
that
band
and
so
on,
and
there's
something
really
great
about
having
that
connection
to
a
community
of
other
fans
and
getting
guidance
from
experts.
B
So,
let's
fast
forward
a
few
years
to
the
late
1990s
at
this
point,
amazon.com
has
been
in
business
for
a
few
years
and
has
expanded
beyond
books
to
also
sell
compact
discs
and
movies.
So
if
you
want
an
album,
that's
in
print,
even
if
it's
a
rare
one,
Amazon
probably
has
it
at
a
decent
price
and
can
get
it
to
you
within
a
week
or
so
now.
A
lot
of
people
thought
at
the
time
that
having
an
efficient
shopping
website
was
the
real
innovation
of
Amazon.
B
But
even
then,
it
was
pretty
clear
that
a
lot
of
their
value
was
in
the
data
they
had
collected
instead
of
just
giving
you
boring,
publisher,
supplied
metadata,
but
now
more
press
releases
Amazon
would
give
you
a
bunch
of
scored
narrative
from
other
users.
So
you
could
use
this
information
to
some
extent.
To
answer
the
kind
of
taste
questions
you'd
have
needed.
B
An
experience
record
store
clerk
for
just
five
years
before
the
next
step
for
Amazon,
which
they
had
full
swing
by
the
middle
of
the
last
decade,
was
to
use
aggregated,
reviewing
and
purchasing
data
to
make
product
recommendations.
So
now,
instead
of
reading
every
review,
you
might
just
let
Amazon
show
you
only
the
reviews
that
other
users
had
marked
as
most
helpful.
If
you
searched
for
a
product
and
decided
not
to
buy
it,
you
wouldn't
have
to
look
at
every
possible
alternative.
B
Instead,
Amazon
would
tell
you
what
things
people
who
looked
at
the
same
product
ultimately
bought
Amazon
also
started
proactively,
suggesting
products.
They
had
reason
to
believe
you'd
like
after
you
bought
something
when
you
visited
their
site
or
over
email.
Now.
Everyone
has
an
example
of
this
backfiring
right.
You
buy
some
durable
good
like
a
television
or
infant
carseat,
and
immediately
get
recommendations
for
other
items
in
the
same
product
category.
You
know
like
people
who
bought
this
vacuum
also
bought
other
vacuums.
B
So
these
emails
weren't
perfect,
but
they
were
often
very
good
and
I
actually
had
to
turn
them
off
because
they
started
to
get
too
expensive.
Now
Amazon
uses
data
for
all
sorts
of
other
things
as
well.
Optimizing
logistics
setting
prices
even
deciding
what
sorts
of
original
content
commissioned
for
their
digital
media
offerings,
and
that's
just
on
the
retail
side
to
say
nothing
of
what
they
do
with
with
Web
Services
and
so
on,
but
even
just
the
most
obvious
ways
that
Amazon's
retail
operation
has
used.
B
Nuber
has
been
an
enormous
ly,
successful
example
of
a
sharing
economy
company
by
serving
as
a
matchmaker
between
short-term
contractors
and
clients.
They
have
a
great
app
and
identified
a
market
where
they
could
have
an
impact,
but
as
we'll
see,
the
quality
of
their
product
depends
a
lot
on
their
use
of
data.
B
So,
for
a
little
bit
of
background,
the
conventional
taxi
industry
is
heavily
regulated
and
has
a
high
barrier
to
entry.
Taxis
typically
must
be
licensed
by
the
municipality
in
which
they
operate,
and
there
are
often
a
fixed
number
of
licenses
per
city,
so
the
licenses
may
be
far
more
expensive
than
the
annual
salary
a
driver
might
expect.
B
Furthermore,
rates
for
licensed
cabs
are
fixed
by
the
municipality
in
which
they
operate,
so
there's
no
opportunity
for
a
real
market
for
cab
services.
Prices
stay
the
same
regardless
of
supply
and
demand,
meaning
that
it
may
be
difficult
or
impossible
to
get
a
taxi
at
busy
times
from
unsafe,
neighborhoods
or
to
distant
destinations.
B
This
is
an
industry
that
was
ripe
for
disruption.
An
uber
was
there
to
do
it.
Uber
provided
a
great
app,
but
that
was
just
the
beginning.
Anyone
can
become
an
uber
driver
without
spending
most
of
a
careers
worth
of
gross
salary
on
a
taxi,
medallion
or
risking
the
uncertainty
of
leasing.
One
goober
efficiently
matches
drivers
to
passengers
and
their
applets
passengers
know
where
the
driver
is
and
when
to
expect
to
pick
up.
Instead
of
using
licensing
as
a
rough
proxy
for
driver
quality,
uber
uses
a
rating
system
to
isolate
bad
drivers
or
bad
passengers.
B
They
also
use
the
information
they
have
about
how
many
drivers
and
passengers
there
are
at
any
given
time
to
create
a
truly
efficient
market
for
passenger
transportation.
If
there
are
four
more
passengers
and
drivers,
prices
will
go
up,
which
incentivizes
more
drivers
to
get
out
on
the
road
and
ultimately
ensures
that
more
passengers
are
served.
Uber
also
uses
the
data
they
have
to
detect.
Fraud.
Predict
passenger
destinations
and
support
a
lower
cost
carpool
service.
B
Interestingly,
one
of
the
taxi
industry's
responses
to
BER
success
has
been
to
create
several
different
apps
for
people
to
hail,
licensed
cabs,
essentially
assuming
that
what
people
want
is
just
a
conventional
taxi
service,
but
smartphone
app.
Now
it's
possible
that
some
people
really
want
this,
but
ignores
that
the
real
power
of
wood
Eber
has
done
comes
from
their
aggressive
use
of
data
to
improve
their
products
and
services,
not
from
their
app
itself.
B
So
this
third
and
final
example
of
a
data-driven
app
is
one
that's
close
to
my
heart.
I'm,
an
avid
cyclist
and
Strava
is
an
activity
tracker
app,
that's
also
a
social
network
for
athletes.
So,
let's
think
about
fitness
tracking.
Before
the
smartphone
era,
only
serious
fitness
enthusiasts
a
decade
ago
had
a
dedicated
sports
GPS
device
for
tracking
their
rides
or
runs.
B
Everyone
else
was
left
with
a
stopwatch
and
the
notebook
after
phones
with
GPS
wide
receivers,
became
widespread,
a
lot
of
companies
developed
activity,
tracking
applications,
but
most
of
these
offered
only
minor
differentiation
from
each
other
Strava
differentiated
itself
by
offering
a
great
user
experience
an
excellent
social
network.
So
you
could
follow
and
encourage
your
friends
and
rivals
and
a
really
addictive
feature.
B
The
really
addictive
feature
was
a
competitive
aspect.
They
allowed
users
to
designate
parts
of
their
activities
as
segments
like,
for
example,
a
local
Hill,
Climb,
a
town
line
sprint
or
a
stretch
of
County
Highway
along
the
lake
and
then
see
leaderboards
over
time
of
how
their
rides
compared
to
the
fastest
people
ever
to
ride
on
those
segments,
some
of
whom
turn
out
to
be
professional
athletes
and
some
of
whom
are
just
enthusiastic
amateurs.
B
B
Like
the
other
example,
apps,
we've
talked
about
straddle
levers,
their
popularity
to
collect
a
lot
of
data,
and
then
they
use
that
data
to
make
their
product
far
better.
The
first
data
driven
feature
I
want
to
call
it
as
routing,
but
let's
motivate
it
a
bit
first
think
of
going
for
a
ride
or
a
run
in
an
unfamiliar
place.
You
probably
won't
just
head
out.
Even
if
you
have
a
map
you
might
get
lost,
you
might
not
find
any
good
roads
and
you
might
even
wind
up
someplace,
unsafe
or
unsuitable
for
cyclists
or
runners.
B
The
routing
application
you'd
use
to
get
turn-by-turn
data
in
a
car,
probably
waits
candidate
routes
based
on
distance
speed,
limit
and
pasa
traffic
or
road
closure
data.
With
the
goal
of
getting
you
from
point
A
to
point
B
as
quickly
as
possible,
because
Strava
had
millions
of
rides
to
analyze,
they
were
able
to
make
a
routing
application
that
weighted
routes
based
on
the
goals
of
endurance,
athletes
like
elevation
profile
safety
and
frequency
of
interruptions.
After
all,
it's
hard
to
get
a
workout
if
you
have
to
stop
for
a
light
at
every
block.
B
Now
the
underlying
map
data
may
not
reveal
which
roads
are
safe
or
which
have
nice
paved
shoulders
or
which
are
free
from
interruptions
or
have
beautiful
views,
but
you
can
be
pretty
sure
that
roads,
like
those
will
see
athletes
spending
more
time
on
them
than
the
alternatives
will,
with
a
large
enough
population
of
athletes,
doing
activities
rode
popularity.
Data
is
a
reasonable
proxy
for
safety
and
suitability
and
Strava
was
able
to
use
this
data
to
make
an
athlete
focused
routing
feature.
B
Finally,
strawberries
used
data
as
a
product
itself,
offering
data
both
to
users
and
to
non
users.
Their
basic
service
is
free
for
all
users,
but
they
offer
a
subscription
service
with
some
additional
features.
The
subscription
service
offers
a
variety
of
analytical
tools
for
activity
data,
but
one
of
the
very
first
features
they
introduced
was
essentially
the
kind
of
roll-up
and
drill
down
functionality.
B
They've
recently
been
selling
aggregated
data
to
Nissa
palette
ease
that
is
non-users
about
which
routes
are
most
popular
for
cyclists
and
runners,
and
their
goal
has
been
to
inform
local
transportation
policy.
Now
on
that
front,
I
don't
know
if
any
cities
have
built
new
bike
lanes
because
of
straw
by
data,
but
some
counties
have
recently
banned
mountain
bikes
from
multi-use
single
track
trails.
After
seeing
how
fast
people
can
get
going
on
them.
B
B
So
we're
going
to
start
now
by
talking
about
what
your
applications
need
to
do
to
process
and
take
advantage
of
more
data
and
how
you'd
actually
build
these
kinds
of
capabilities
in
a
contemporary
environment
like
OpenShift,
we'll
start
by
laying
out
some
general
concerns
that
all
data
intensive
applications
have
to
deal
with
and
then
look
at
some
application.
Architectures
that
have
been
designed
to
address
these
ultimately
will
focus
on
architectures
that
are
especially
suitable
for
contemporary
container
based
application
development.
B
Fundamentally,
a
data-driven
application
does
everything
that
your
conventional
web
application
might
do,
but
it
also
aggregates
and
transforms
data
from
multiple
sources,
trains,
predictive
models
based
on
that
data
and
then
uses
those
models
to
transform
and
make
predictions
from
new
and
existing
data
alike.
It
saves
raw
and
process
data
to
archival
storage.
B
Now
there
are
a
few
logical
components
necessary
to
support
these
responsibilities.
As
you
might
imagine,
a
lot
of
these
components
have
to
do
with
data
sources,
data
processing
or
data
presentation,
and
some
of
them
have
obvious
counterparts
even
in
applications
that
are
not
particularly
data-driven,
we'll
need
to
handle
data
from
multiple
sources
with
different
characteristics,
transactional
and
telemetry
data
among
other
kinds
of
streaming.
Data
could
arrive
a
record
at
a
time
as
events
on
a
messaging
bus.
B
Now,
in
the
past,
people
have
thought
about
analytics
as
a
separate
workload
that
gets
run
in
at
in
a
data
center,
but
in
a
data-driven.
App
analytics
in
compute
is
really
just
another
component
of
an
application,
so
we'll
need
a
computer
to
actually
do
the
work
of
data
processing
and
analytics
the
models
that
a
data-driven
application.
B
These
are
a
lot
smaller
than
the
data
they
were
trained
on
and
are
typically
much
faster
to
evaluate
or
make
predictions
from
than
they
are
to
train
since
models
only
place
relatively
minor
demands
on
compute
and
storage,
but
need
to
be
accessed
quickly.
We
can
keep
them
in
the
same
kind
of
operational
store,
we'd
use
for
session
data
in
a
conventional
application,
a
fast
and
lean
data
base
or
key
value
store.
B
So
the
one
problem
that
every
data-driven
application
has
to
solve
is
collecting
lots
of
data
from
various
sources
manipulating
and
normalizing
it
and
federating
it
behind
a
common
interface.
This
is
often
called
ETL,
which
stands
for
extract,
transform
and
load.
This
is
not
only
a
common
problem,
but
it's
also
a
big
problem
for
many
data
science
projects
and
data-driven
apps
data
wrangling,
that
is,
the
cleaning
transformation
of
normalization
and
so
on
winds.
B
We
can
evaluate
various
solutions
to
the
problem
of
data
Federation
by
asking
several
questions
and
I'm
going
to
have
us
consider
these
as
we
look
at
application
architectures.
The
first
question
is:
what
is
the
source
of
truth?
Is
there
a
canonical
place
for
our
data
and
where
is
it
just
like,
with
scientific
experiments
or
continuous
integration
tests?
B
The
second
question
is
how
well
our
Federation
solution,
scales,
actual
real-world,
analytic
workloads,
typically
aren't
very
large
because
they
operate
on
pre-processed
subsets
of
data,
but
the
raw
data
can
be
enormous,
so
we
need
a
Federation
solution
that
can
scale
up
by
devoting
more
resources
to
it
as
our
needs
grow.
The
third
question
is
an
interface
question:
how
convenient
does
a
Federation
solution
make
to
access
your
data
and
are
there
access
patterns
or
use
cases
that
are
unnecessarily
difficult?
Do
you
need
to
jump
through
hoops
to
get
your
data
out?
B
We'll
start
our
overview
of
application
architectures
by
looking
at
two
that
have
been
around
for
quite
a
while
in
a
traditional
data
warehouse
architecture,
we
federated
Atta
in
a
relational
database
that
is
optimized
for
fast
concurrent
updates,
our
application
processes,
a
stream
of
events
transforms
these
may
be
applying
some
business
rules
and
then
updates
the
database.
Each
of
these
internal
components
can
communicate
results
back
to
the
others.
B
I'm
calling
out
the
UI
here
as
a
separate
component,
because
users
who
interact
with
our
application
manually
also
generate
events,
but
the
output
they
get
is
also
determined
both
by
business
logic
and
by
the
contents
of
the
database.
This
database
here
is
optimized
for
fast
concurrent
updates
and
we
call
this
part
of
the
architecture.
The
transaction
processing
part.
This
transaction
processing
database
is
the
source
of
truth,
since
it
always
has
current
information
about
the
state
of
our
system
to
extend
our
basic
transaction
processing.
B
Application
with
analytic
capabilities
will
periodically
dump
the
contents
of
our
transactional
database
to
a
database
optimized
for
fast
reads
and
complex
queries.
We'll
use
this
database
to
support
various
kinds
of
analytic
processing,
primarily
to
support
reports
that
project
our
multi-dimensional
data
into
an
easy-to-understand
spreadsheet
that
lets
people
quickly
drill
down
roll
up
along
any
dimension,
for
example,
looking
at
quarterly
sales
by
division,
product
region
or
salesperson.
We
can
also
feed
results
from
our
batch
analysis
back
to
the
business
logic
and
allow
analysts
to
do
exploratory
queries
against
the
database.
B
B
The
disadvantages
of
this
approach
are
that
the
transaction
processing
layer
is
stateful.
You
can't
run
this
at
in
a
container
orchestration
platform
very
easily.
Also
relational
databases
have
historically
been
very
difficult
to
scale
out.
Typically,
if
you
need
to
get
high
performance
from
relational
database,
your
approach
has
bet
have
been
to
buy
the
most
powerful
machine.
You
can
possibly
afford
to
run
it
on
because
of
these
things.
These
are
transaction
processing.
B
Layers
are
typically
difficult
to
manage,
certainly
more
difficult
to
manage
than
stateless
containerized
services
and
the
other
problem
is
that
there's
there's
no
raw
data
in
our
source
of
truth.
If
our
source
of
truth
is
the
transaction
processing
database,
we've
already
transformed
things
by
the
time
they
get
there.
B
Again,
these
are
difficult
to
scale
out,
although
since
the
database
is
more
read
only
it
may
be
easier
to
scale
up.
A
really
important
consideration
for
analytic
processing
is
that
you're
really
limited
to
the
capabilities
and
interface
of
a
relational
database,
so
things
that
you
can
express
a
sequel
queries
or
as
extended
sequel,
queries
or
in
a
procedural
extension.
B
So
a
second
approach
to
look
at
is
the
data
Lake
approach,
and
this
is
popularized
by
the
Apache
Hadoop
project
and
this
federates
data
in
a
distributed
scale
out
file
system.
So
your
interface
for
accessing
data
is
the
file.
If
you
need
to
add
more
space,
you
can
add
more
nodes
to
increase
your
storage
capacity,
but
the
assumption
is
that
once
you've
added
storage
nodes,
they'll
stick
around
essentially
forever,
as
new
events
arrive,
they're
appended
on
to
files
in
the
distributed
file
system.
B
Now
the
raw
data
we
collect
from
events
isn't
likely
to
be
in
the
format
we
want,
so
we'll
write
some
components
to
ant,
perform
ETL
on
this
data
and
analyze
it
in
parallel.
The
Hadoop
runtime
will
schedule
these
by
migrating
compute
jobs
that
operate
on
parts
of
our
data
to
the
machines
that
contain
those
parts.
Do
we
get
skaila
compute
atop
our
scale
out
storage,
unlike
classic
data,
warehouses,
a
dupe,
enables
scale
out
storage
and
processing,
and
it
also
provides
a
fairly
easy
way
to
keep
raw
unprocessed
data
in
the
source
of
truth.
B
That
is
the
distributed
file
system,
along
with
multiple
versions
of
post,
processed
and
intermediate
data.
However,
it's
still
not
a
great
fit
for
a
contemporary
container
based
architecture,
because
Hadoop
depends
on
having
compute
jobs
co-located
with
the
storage
they're
operating
on.
You
won't
be
able
to
scale
your
compute
and
storage
independently,
because
Hadoop
expects
that
your
storage
nodes
are
long-lived.
You
essentially
have
to
dedicate
resources
to
both
storage
and
compute.
B
In
practice,
this
means
that
a
lot
of
Hadoop
shops
are
stuck
doing
everything
on
one
cluster
big
enough
to
support
all
of
their
enterprises,
storage
needs
and
managing
that
cluster
for
a
wide
range
of
compute
users
and
applications.
There's
nothing
wrong
with
that,
but
it
perfects
prevents
us
from
enjoying
all
of
the
advantages
of
container
based
architectures.
B
Finally,
there
are
a
couple
of
technical
reasons
why
Hadoop
might
not
be
the
best
choice
for
new
projects.
The
main
programming
model
that
exposes
is
pretty
low
level,
so
most
Hadoop
applications
rely
on
third
party
libraries,
all
of
which
have
different
interfaces
and
different
expectations,
which
means
that
users
have
to
spend
even
more
time
on
ETL
tasks
to
ensure
that
they
can
glue
together
these
libraries.
B
Finally,
because
of
how
Hadoop
orchestrates
computations,
it
isn't
really
suitable
for
doing
live
analysis
of
streaming
data
or
interactive
analysis
of
batch
data,
so
we've
just
reviewed
a
couple
of
legacy
architectures.
These
are
probably
familiar
to
you
and
they
solve
a
lot
of
problems,
but
they
really
assume
a
monolithic
application
architecture
and
an
inelastic
view
of
our
compute
resources
and
our
applications
requirements.
They
certainly
aren't
a
great
fit
for
running
as
micro
services
or
in
containers.
Now
we'll
look
at
some
architectures
for
data
processing
that
work
really
well
with
my
microscope.
B
A
stap
locations,
along
with
some
example,
applications
that
use
these
the
lambda
architecture
is
an
interesting
approach
to
modernizing
the
traditional
data
warehouse.
The
idea
is
that
you
have
a
stream
of
events
arriving
and
you're,
going
to
multiplex
it
to
a
storage
layer
and
a
stream
processing
layer
which
is
going
to
perform
imprecise,
but
low
latency
analyses
on
the
data,
as
it
arrives,
giving
you
a
view
of
your
very
latest
data.
This
stream
processing
component
is
called
the
speed
layer.
B
B
So
there
are
some
major
advantages
to
the
lambda
architecture
over
the
two
legacy.
Architectures
we
looked
at
earlier.
The
first
is
that
it's
suitable
for
micro
service
architectures
and
that,
unlike
the
traditional
data,
warehouse
raw
data
remain
available.
However,
like
the
traditional
data
warehouse,
the
lambda
architecture
requires
that
you
implement
your
analyses
twice
in
two
different
ways.
One
says
streaming
analyses
in
the
speed
layer
and
one
says
batch
analyses
in
the
batch
layer.
These
are
probably
going
to
be
with
different.
B
Api
is
maybe
even
with
different
algorithms,
just
because
of
how
how
you
have
to
implement
streaming
jobs
versus
batch
jobs.
So
that's
a
real,
a
real
development
cost
to
consider.
However,
this
is
this
is
a
suitable
architecture
for
a
lot
of
problems.
As
one
example,
let's
look
at
an
application
that
does
infrastructure
log
processing.
If
we
have
infrastructure
log
events
arriving
in
a
stream,
we
can
multiplex
them
into
a
stream
processing
component
and
to
a
document
database
for
storage.
B
B
The
Kappa
architecture
was
named
in
response
to
the
lambda
architecture,
and
its
design
reflects
that
our
message:
queue
systems
are
stream
processing
frameworks
and
our
understanding
of
how
to
design
streaming
analysis.
Algorithms,
that
is,
algorithms
that
only
look
at
each
record
once
and
update
a
model
in
place,
I've
dramatically
improved
since
the
introduction
of
lambda
architecture.
The
basic
idea
is
that,
as
you
receive
events,
you
put
them
on
a
message.
Bus
and
you
give
them
a
particular
topic.
B
So
the
component
that
would
be
doing
ETL
in
another
architecture
is
simply
listening
for
messages
on
the
bus
with
a
raw
data
topic
and
then
writing
a
message
with
a
pre-processed
data
topic
as
output.
Similarly,
a
job
that
does
analysis
reads:
pre-processed
data,
writes
analysis,
results
and
so
on,
and
then
our
user
interface
components
are
really
just.
Clients
of
this
analysis
results
topic,
so
this
model
assumes
that
our
cues
are
durable
and
infinitely
replayable.
B
It
also
assumes
that
each
topic
can
be
partitioned
to
provide
scale-out
and
that
our
problem
can
actually
be
solved
by
a
streaming
algorithm.
So
those
aren't
crazy
assumptions,
but
they
are
things
to
keep
in
mind
because
they're
not
always
going
to
be
the
case
with
that
said,
the
Capo
architecture
has
a
lot
of
advantages.
B
It
is
suitable
for
micro,
service,
architectures
and
since
the
stream
is
the
source
of
truth,
if
you
can
always
replay
the
stream,
you
can
always
update
some
components
of
your
analysis
and
get
the
new
results.
However,
it
requires
sophisticated
streaming
framework,
potentially
a
lot
of
infrastructure
work
and
streaming
algorithms,
which
can
be
trickier
to
implement
than
batch
algorithms.
B
A
really
cool
example
of
the
cap
architecture
at
work
is
the
infrastructure
for
the
Fedora
project.
So
there
are
a
lot
of
so
fedoras
is
a
Linux
distribution.
There
are
a
lot
of
moving
parts
in
maintaining
a
Linux
distribution
that
need
to
communicate
coupling
them
all
tightly
together.
Wasn't
the
right
solution
for
fedora,
it
was
unsustainable,
so
the
mechanism
that
ties
all
these
components
together
is
a
message
bus.
B
When
someone
updates
a
package
in
a
fedora
git
repository
may
be
going
to
a
new
version
of
the
upstream
source
or
adding
a
patch.
A
message
gets
sent
out
to
the
bus
when
someone
edits
a
wiki
page
or
starts
an
IRC
meeting,
a
message
gets
sent
out
to
the
bus
when
a
package
build
starts,
succeeds
or
fails,
message
gets
sent
out
on
the
bus
and
so
on.
B
Another
flexible
architecture
for
data
processing
involves
moving
the
data
Federation
problem
from
the
storage
layer
to
the
compute
layer.
Instead
of
federating
unstructured
data
by
storing
it
in
a
file
system
and
defining
ad
hoc
ETL
pipelines,
we're
explicitly
structuring
it
upon
import
into
a
database
or
message
queue.
You
can
use
a
computer
flexible
enough
to
give
you
a
uniform
abstraction
over
multiple
data
sources
to
systems
that
do.
B
Again,
this
architecture
is
suitable
for
micro
service
architectures,
it's
typically
flexible
and
interoperable
with
other
systems.
Instead
of
worrying
about
how
to
convert
data
explicitly
from
what
one
system
expects
to
another.
You
have
a
bunch
of
connectors
to
get
data
into
your
computer
and
then
you
can
access
it
under
a
uniform
interface.
B
Finally,
a
really
nice
advantage
of
this
approach
is
that
you
can
use
it
to
support
both
batch
and
streaming
workloads,
depending
on
the
compute
layer.
You've
chosen
as
an
example.
Let's
look
at
an
application
that
models,
infrastructure
costs
for
applications
running
in
a
public
cloud,
and
here
we
have
a
couple
of
different
data
sources
that
we're
going
to
federate
and
use
to
train
a
predictive
model.
So
we
have
billing
data
from
a
service
provider,
that's
in
an
object
store
and
we
have
service
telemetry
from
a
monitoring
tool.
That's
stored
in
a
legacy
relational
database.
B
We
can
incorporate
both
of
these
data
sources
into
Apache
spark
and
use
them
to
train
a
model
saying
based
on
how
we're
using
the
system,
how
much
it's
likely
to
cost
so
based
on
operational
characteristics
and
telemetry
data.
We
can
have
a
prediction
of
how
much
the
system
is
going
to
cost
to
run
in
a
given
month,
based
on
how
much
it's
cost
when
it's
had
similar
usage
in
previous
months.
B
So
let's
go
back
to
our
application,
responsibility,
diagram
and
we're
going
to
start
at
the
bottom
here.
Obviously,
we're
going
to
be
deploying
orchestrating
and
managing
our
applications
on
top
of
OpenShift
for
processing
streaming
events,
you
have
a
lot
of
options,
including
JBoss,
amq
and
Kafka,
that
can
run
in
containers.
B
Jboss
data
virtualization
provides
you
with
the
unified
interface
to
legacy,
databases
and
reliable
distributed
file.
Systems
like
cluster
and
SEF
are
great
option
for
file
and
object
storage,
both
as
a
source
for
input
data
and
as
a
destination
for
archive
data.
My
team
has
had
a
lot
of
success
using
apache
spark
as
a
compute
layer
for
internal
applications
over
the
past
couple
of
years,
and
the
model
of
federating
data
in
the
computer
is
a
really
great
fit
for
micro
service
based
applications
for
an
operational
store,
providing
a
fast
cache
of
models
and
session
data.
B
B
B
Now
I'm,
not
mentioning
all
of
these,
so
that
you'll
go
out
and
and
buy
everything.
Red
Hat
sells
I,
guess
I
wouldn't
discourage
you
from
doing
that
either
I'm,
just
pointing
out
that
a
lot
of
these
broad
products
and
technologies,
you're,
probably
already
using
in
production,
have
a
place
in
the
data
driven
apps.
You
want
to
be
developing
today
and
tomorrow
and
deploying
on
openshift
so
for
next
steps.
I
have
a
link
here
that
I
can
paste
into
the
chat
later
to
see
a
demo
of
a
developer
workflow
for
a
data-driven
app
on
openshift.
B
This
is
my
colleague,
Mike
McCune,
showing
what
it
looks
like
to
take
a
log
processing
application
that
uses
Apache
spark
on
OpenShift
and
update
in
response
to
the
actual
code,
changing
very
slick,
very
slick
demo
I
have
a
container.
You
can
run
on
your
local
machine
to
try
out
Apache
spark
on
a
notebook
interface.
It
looks
like
that.
Link
is
missing
from
the
slide,
but
I
will
also
put
those
instructions
in
the
chat
and
then
I
would
say
the
next
step
from
there
is
really
to
build
a
data-driven
app
on
OpenShift.
B
A
Awesome
we'll
thank
thank
you
very
much
for
that.
That
was
probably
the
best
overview
I've.
Seen
of
all
the
different
approaches
and
models
for
using
data-driven
or
big
data
on
on
any
platform.
So
much
appreciated.
We
are.
We
don't
have
any
questions
right
now
in
the
chat.
I
think
because
of
the
nature
of
this
is
a
lot
of
to
digest
there
and
when
I
would
suggest.
A
We
probably
be
walking
through
that
and
answering
questions
and
deep
dive
on
that,
but
that
should
come
up
in
I
think
two
or
three
weeks,
depending
on
people's
vacation
schedules.
We'll
we'll
do
that
and
if
anyone
is
interested
in
joining
the
mailing
list
for
the
for
the
future
big
data
events
and
talks
such
as
this
series,
I
think
we've
got
a
series
of
four
different
talks
set
up
already
and
a
number
of
other
Commons
members
interested
in
presenting
their
use
cases
as
well,
though,
there's
there's
quite
a
lot
to
talk
about
in
this
space.
A
If
we've
got
a
question
there,
if
someone
is
yes,
we'll
definitely
take
you
up
on
your
offer
to
do
that.
Demo
live
and
find
a
time
for
that
as
well.
So,
if
there's
isn't
any
questions,
what
I'd
also
ask
on
will
is,
if
you
can
send
me
your
slide
deck
or
the
link
to
the
slide
deck,
because
that
there
were
some
excellent
information
in
there
and
we'll
repost
this
recording
and
all
of
your
links
as
an
on
blog
Vulcan
ship
comm,
probably
in
two
days
time.
A
That's
about
how
long
it
takes
to
edit
the
videos
and
they'll.
Definitely
have
you
back
and
do
a
lot
more
talks
on
Big
Data,
because
it's
it's
been
pretty
much
a
hot
topic
at
all
of
the
conference's.
That
I've
been
going
to
lately
and
there's
a
lot
of
big
data
analysis
that
we
do
under
the
hood
and
I've
seen
a
wonderful
presentation.
A
Dyanne
feta
mark
did
at
Apache
con
on
analyzing
open
shift
on
lines,
log
files,
so
great
content
out
there
and
if
there's
other
folks
who
have
things
that
they
want
to
talk
about
around
big
data
feet,
do
join
the
Big
Data
sig
I'll
make
one
more
pitch
for
signing
up,
accept
Commons
that
open
shift
org,
slash,
sig
ownership,
big
data.
You
can
find
that
evening
and
sign
up
and
you'll
get
that
now
and
less
gone
and
find
some
dates
in
the
next
more
parts
of
this
series.