►
From YouTube: OpenShift Commons Big Data SIG #3: Building Cloud Native Apache Spark Applications with OpenShift
Description
Apache Spark based applications are often comprised of many separate, interconnected components that are a good match for an orchestrated containerized platform like Kubernetes. But with the increased flexibility afforded by these technologies comes a new set of challenges for building rich data-centric applications.
In this SIG session, we will discuss techniques for building multi-component Apache Spark based applications that can be easily deployed and managed on the OpenShift platform. Building on experiences learned while developing and deploying cloud native applications, we will explore common issues that arise during the engineering process and demonstrate workflows for easing the maintenance factors associated with complex installations.
Guest Speaker: Michael McCune Senior Software Engineer, Red Hat
Michael is a software developer in Red Hat’s emerging technology group. He is a contributor to, and core reviewer for the Oshinko project, the Sahara project, the OpenStack Security Project, and the OpenStack API Working Group. For the last two years, he has been creating and deploying data-driven applications and infrastructure for the OpenStack and OpenShift platforms.
A
A
Hello
and
welcome
to
the
first
Big
Data
sig
on
open
ship
Commons
on
Diane
Mueller.
This
is
the
first
big
data
set
of
2017
I
should
say
we
had
a
couple
of
them
last
year
and
their
recordings
of
them
are
up
on
our
YouTube
channel.
So
we'll
post
those
notes
in
the
chat
and
you
can
find
the
past
ones
as
well
and
then
Benison
briefings
on
the
big
data.
The
big
data's
been
a
very
very
hot
topic
and
Apache
spark
on
OpenShift
has
been
a
very
big
hot
topic.
A
So
today
we
are
lucky
to
have
with
us
Michael
McCune,
who
is
going
to
talk
about
building
cloud
native
applications
using
pachi
spark
on
openshift
and
all
that
goodness,
and
so
without
any
further
ado
I'm
going
to
let
him
get
started,
you
can
ask
questions
in
the
chat
bill.
The
session
should
take
about
20-30
minutes
of
his
talking,
depending
on
how
long
how
rosy
is
and
then
we'll
have
Q&A
and
then
open
conversation
afterwards
about
this
so
Mike
without
any
further
ado.
Go
ahead
thanks.
B
Dan
yeah
so
I'm,
Michael,
McEwan
and
I'm,
going
to
talk
about
building
cloud
native
applications,
spark
Apache
spark
applications
on
openshift
today.
So
a
little
bit
about
me
just
for
background
here,
I've
been
in
software
development
for
a
couple
decades
now,
I've
worked
just
about
all
ends
of
the
stack
from
embedded
programming.
B
As
our
neck
steps,
so
the
inspiration
for
this
talk
and
kind
of
a
lot
of
the
development
we're
doing
for
me
touches
on
some
larger
themes
and
these
really
fold
well
into
the
open
shift
paradigm,
and
you
know
the
first
one.
Those
is
developer,
empowerment,
so
you're,
giving
the
developer
the
ability
to
deploy
their
applications
into
this
platform
and
make
them
available
for
others.
B
It
is
a
really
strong
theme
for
me
and
and
kind
of
along
with
that
and
with
OpenShift
comes
this
kind
of
improved
collaborative
effort
as
well,
there's
as
we'll
get
into
the
demo
and
what
not
you'll
see
how
we're
able
to
push
code,
changes
and
update.
You
know
directly
from
our
github
repositories
into
code
there
into
deployments,
and
that
really
helps
out
with
the
with
the
way
we
collaborate
with
others
and
then
finally
operational
freedom.
B
B
So
you
need
to
be
able
to
bring
pieces
in
and
take
them
out,
and
you
might
want
to
suite
a
one
day
you
might
be
using
Kafka
in
the
next
day.
You
might
want
to
switch
that
out,
and
you
know
change
for
ActiveMQ,
so
you
need
to
be
able
to
kind
of
work
with
that
and
and
also
they
can
oftentimes
be
really
challenging
to
debug.
So
you
know
this
is
something
that
just
comes
with
working
in
the
cloud.
B
You
know
we're
used
to
working
with
these
applications
where
we
can
maybe
SSH
into
a
server
and
check
out
its
logs
and
do
all
these
other
things
and
as
we
move
to
a
container
based
kind
of
orchestrated
environment,
those
debugging
kind
of
pathways
need
to
change
a
little
bit
and
so
before
before
I
start
engineering,
an
application.
You
know
before
we
start
writing
all
the
code
and
everything
what
I
like
to
do
is
plan
out.
B
What's
going
to
happen
and
in
in
some
respects
this
almost
looks
like
you
know,
like
a
creative
effort
like
you're,
making
a
movie
or
something
like
that,
but
I
like
to
storyboard
out.
What's
gonna
happen,
you
know,
identify
the
moving
pieces
and
in
this
case
you
can
see
an
example
of
of
what
we
might
be
talking
about
here.
B
We
start
to
think
about
what
is
what
what
is
insightful
analytics
and
and
how
will
we
create
a
data
processing
application
and
a
colleague
of
mine
kind
of
suggested,
this
ingest
process
publish
flow
and
I
really
like
that,
so
what
I
want
to
try
to
display
here
is,
as
you
look
at
your
application
and
we
look
at
the
pieces
that
we
just
put
together.
You
know
we
can
kind
of
see.
B
One
part
of
this
is
the
ingestion
part,
then
we're
processing
it
and
finally
we're
publishing
it,
and
this
is
really
how
how
your
applications
could
be
built
and
and
as
we're
doing
that,
we
need
to
think
about
what
what
is
the
data
set
that
we're
operating
on?
You
know
how
will
that
be
brought
in
what
what
does
that
look
like
and
then
how
will
we
process
that
data?
What
operations
will
we
do
on
top
of
that
data?
And,
finally,
where
will
the
results
go?
B
How
will
we
take
whatever
process
and
we've
done
and
bring
them
back
to
an
end
user,
because
it
does
no
good
if
it
never
really
gets
out
of
the
processing
stage.
So
after
we've
kind
of
planned
the
application?
Now
we
start
to
think
about
how
we're
gonna
build
it,
and
what
I
like
to
do
is
is
start
to
decompose
the
application
component
so
really
break
down.
What
are
the
different
pieces
of
the
application?
B
How
can
we
start
to
separate
those
and
this-
and
this
really
helps
to
do
a
take
feed
into
the
collaborative
efforts
that
that
will
naturally
occur,
and
so,
as
we
start
to
go
back
and
look
at
kind
of
the
adjust
process
publish
model,
we
do
it's
to
say:
where
are
the
natural
break
points
of
our
application?
You
know
where
these
different
pieces
that
we
have
in
place.
You
know
how
can
we
break
them
apart
and
then
also
building
for
modularity,
like
like
I
said
before
you
might
want
to
replace
a
component
at
some
point.
B
Stateless,
because
they're
much
easier
to
restart
and
redeploy
there's,
there's
not
a
complicated.
You
know
setup
that
needs
to
occur
whatever
you're
deploying
one
of
these
applications.
You
can't
always
do
it,
but
you
know
it's
what
I
try
to
strive
for
and
then
really,
as
you
start
to
think
about
collaboration
and
how
to
put
together
these
multi-component
applications
focusing
on
the
communication
points
between
the
applications
is
where
you
can
really
start
to
build
strength,
so
you
want
to
coordinate
on
them
on
the
middle
points,
you
know
what
how
do
you?
B
B
You
want
to
build
the
idea
of
network
resiliency,
and
this
goes
back
to
the
idea
of
building
modular
lis,
so
network
resiliency
means
you
know,
can
one
of
these
pieces
drop
out
and
will
the
application
still
kind
of
stay
up
or
will
the
whole
thing
come
down
because,
as
we
can
see
here,
we've
got
you
know
five
different
components,
we've
identified
and
if
one
of
them
disappears,
we
don't
want
the
whole
application
to
go
away.
We
want
to
be
able
to
recover
gracefully
from
that.
I
mean
it's.
B
If
you're
database
disappears,
it's
not
expected,
the
application
will
work
the
way
it
should,
but
it
shouldn't
all
fall
down
and
then
finally,
the
kubernetes
dns
is
very
helpful
in
this
respect,
because
it
provides
the
ability
to
use
regular
names
to
access
different
parts
of
your
of
your
application.
So
if
your
application
is
built
into
multiple
components
that
are
each
different
containers
or
pods,
then
using
kubernetes
dns
gives
us
a
really
repeatable
name
to
refer
to
those
services,
and
we
don't
have
to
worry
about.
B
So
how
do
we
collaborate?
Then?
You
know
if
we're
doing
this,
if
we're,
if
we've
got
all
these
separate
applications
and
they
all
need
to
fit
together
to
form
kind
of
a
greater
a
greater
whole,
you
know
what
do
we
want
to
think
about,
but
want
to
think
about
the
right
tools
for
doing
this,
and
what
I
mean
by
that
is,
you
know,
make
sure
you're
using
version
control,
make
sure
you're
using
a
you
know,
a
continuous
integration
testing
framework
make
sure
that
you've
got
some
way
to
coordinate
between
the
different.
B
The
different
players
involved
in
in
the
application
development
and
then
going
back
to
the
idea
of
building
module
early.
You
want
to
make
sure
that
each
project
can
be
built
in
a
way
where
it
can
be
tested,
separately
and
and
also
flexible
enough-
that
you
know,
like
I,
said
before
you
can
kind
of
drop
pieces
out
and
bring
them
back
in
by
by
really
thinking
about
each
component
of
an
application
as
being
modular.
That
helps
the
collaborative
effort
and
this
this
may
seem.
B
Now
I'm
going
to,
during
this
case,
study
I'm,
going
to
dive
into
some
code.
Samples
and
I
just
want
to
kind
of
lay
out
ahead
of
time
that
my
my
intention
with
the
code
samples
is
not
necessarily
to
show
you.
You
know
the
most
innovative
and
new
code
out
there,
but
what
I
want
to
show
you
is
how
easy
it
is
to
kind
of
get
these
patterns
working
in
your
own
applications.
So
you
know
we'll
see
as
we
get
to
those
two.
What
does
all
for
Clyde
do
at
a
very
high
level.
B
It
creates
word
to
Veck
models
based
on
text.
Corpora
and
word.
Tyvek
is
a
a
natural
language
toolkit
that
allows
you
to
take
text
volumes
and
then
apply
vector.
It
will
determine
vectors
for
each
word
such
that
you
can
use
those
vectors
to
then
find
similar
meanings
and
and
do
kind
of
interesting
mathematics
to
figure
out.
B
You
know:
we've
got
a
browser,
that's
the
user
outside
of
kubernetes
and
they're,
talking
to
a
node.js
application
inside
of
kubernetes
and
that
node.js
application
talks
to
a
Python
rest
server,
which
is
communicating
with
the
database,
and
the
spark
back-end
at
is
bringing
the
text
data
in
and
then
once
all
that's
put
together.
The
user
through
the
browser,
is
able
to
make
similarity
queries
against
the
text
corpora
that
they've
specified.
B
This
is
kind
of
a
laundry
list
of
some
of
the
top
technologies
we
used.
While
building
this
you
know
patchy
spark
weird
to
back
kubernetes
OpenShift
nodejs
flask,
open
API.
These
were
all
technologies
that
helped
us
build.
What
were
what
we've
created-
and
you
know
big
shout
out
to
all
the
open
source
communities
that
are
putting
together
these
projects.
It
certainly
made
our
job
a
lot
easier.
B
So,
let's
get
into
kind
of
a
deep
dive
now
and
the
crunchy
bits
at
the
beginning
of
the
talk
I
was
I
was
talking
about
focusing
on
the
middle
points
and
that's
where
we
actually
started
this
process.
We
we
talked
about
creating
a
front-end
and
a
rest
server
that
could
do
the
work
for
us
and
we
used
open
api,
which
was
formerly
known
as
swagger
to
create
an
API
definition
that
allowed
us
to
kind
of
figure
out.
B
There
is
a
great
amount
of
tooling
out
there
for
anything
from
creating
the
schemas
to
displaying
them
even
to
auto
generating
code
that
will
create
servers
and
clients
based
around
your
scheme
and,
like
I
said
this
was
like
our
central
discussion
point.
This
is
where
we
began
the
design
process.
Once
we
figured
out
what
we
wanted
to
do
and
what
we're
looking
at
here
is
a
piece
of
at
the
top
we're
looking
at
a
piece
of
the
open,
API
definition.
This
is
from
a
llamó
file.
B
So
as
we
built
these
applications.
There's
a
lot
of
there's
a
lot
of
kind
of
configuration,
data
that
needs
to
go
back
and
forth.
I
talked
earlier
about
stateless
versus
stateful
and
even
in
a
stateless
application
you,
you
still
have
to
inject
some
some
information
to
get
it
running
the
way
you'd
like
it
to,
and
so
when
we
think
about
what
do
we
need?
How
do
we
build
these
applications?
You
know
we.
B
We
start
to
figure
out
what
data,
what
configuration
data
does
each
application
need
to
start
and
and
what
you
can
see
from
the
diagram
here
is
our
Python
application
needed
to
know
how
to
get
to
the
database
and
our
node
application
needed
to
know.
How
will
it
speak
to
the
rest
server
and
you
know
what
address
and
pour
does
that
have
and
what
I'll
show
here
is
kind
of
how
we
took
that
information
and
how
we
used
openshift
and
kubernetes
to
inject
that
information
into
our
applications.
B
Variable
is
that
you
know
that
could
be
a
security
breach.
So
you
you
want
to
use
these
secrets
to
do
that
kind
of
work
and
I
want
to
illustrate
this
because
I
think
it's
important.
If
you're
thinking
about
building
these
applications,
you
know
you're,
you
might
need
to
eject.
You
know
credentials
into
your
application,
and
this
is
the
this
is
the
facility
that
kubernetes
provides
for
doing
that.
Duds.
It's
pretty
easy!
B
You
can
see
here,
you
know:
we've
got
like
a
volume
mount
that
will
that
will
create
a
secret
volume
within
your
container,
and
then
we
have
the
container
definition.
That's
kind
of
telling
us
where
we're
going
to
mount
it,
and
so,
if
you
look
at
the
bottom,
that
you
see,
Etsy
secret
is
where
the
the
secret
will
be
mounted
and
then
within
the
container
with
it
within
the
application
that
runs
in
there.
B
B
And
so
you
can
see
that
the
top
couple
lines
are
just
kind
of
some
normal
PI
spark.
Where
we're
setting
up
the
context
for
the
work
we'd
like
to
do,
then
we
check
to
make
sure
we've
got
a
database
connection,
we're
doing
a
little
setup
on
our
output
queue
to
make
sure
it's
ready,
and
then
we
start
processing
and
for
each
job
that
comes
in.
You
know.
B
As
we
drill
down
into
the
train
function.
What
you
see
is
we,
you
know
we
create
a
word
to
vet
context,
and
then
we
want
to
take
the
text.
We
want
to
load
each
one
of
those
URLs,
so
we
could
break
the
text
apart.
So
you
can
see
this
rdd's
equals
reduce
line
where
we're
doing
this
URL
to
our
DD
function,
and
that
happens
for
each
URL
that
we've
been
given.
And
then
you
know
the
code
sample
below
here
is
the
actual
URL
to
our
DD,
and
so
we
open
the
URL.
B
We
read
the
text
out
of
it
and
we're
doing
a
little
formatting
to
take
out
the
new
lines
and
then
finally
we're
we're
passing
the
our
DD
back
to
the
to
the
previous
function
to
the
train
function,
and
then
we
just
do
this
word:
Tyvek
dot
fit
and
that's
going
to
take
each
our
DD
that,
at
this
point
contains
you
know
each
our
DD
will
contain
an
array
of
words
for
each
URL.
That
was
that
was
placed
in
and
it
will.
B
It
will
do
the
work
at
this
point
now,
on
the
other
end,
when
the
user
wants
to
make
a
query
to
find
similarities.
We
have
this
function
here
now
this
at
this
point,
we're
not
using
spark
to
kind
of
do
the
the
synonym
finding,
but
what
we've
done
is:
we've
created
an
internal
mapping
of
words
to
vectors,
and
so,
as
a
query
comes
in,
we
pull
out.
We
pull
out
the
word
that
we're
looking
for.
B
What
we're
looking
at
here
is
an
open
shift,
console
I'm
using
open
shift
3.4,
which
I
believe
will
get
released
relatively
shortly
and
what
I've
already
set
up
into
it.
Is
this
thing
Ocean
Co,
which
is
a
project
that
a
microbe
is
been
working
on
and
it
helps
us
to
deploy
Bart
clusters
into
our
open
shift
projects
and
so
to
start
with,
in
order
to
deploy
Oh
for
glad
I
did
you
saw
from
the
code
samples?
You
know?
B
We
need
a
database
and
we
need
a
spark
cluster
and
so
I'll
start
by
adding
a
database
and
and
we're
looking
at
the
application
catalog
for
open
shift
here
and
so
I'll
pick
a
datastore
and
I'm
just
going
to
use
the
ephemeral
database
for
the
most
part,
I'm
gonna
leave
leave
these
these
entries
alone.
These
forums
alone,
the
database
service
name,
is
fine.
I
will
add
a
sample
or
a
database
for
ofawadh.
So
it
knows
where
to
talk
and
I'm.
B
Just
gonna
give
it
a
simple
admin
password
to
make
to
make
my
life
a
little
easier
here.
If
you
were,
if
you
were
doing
this
for
real,
you
know,
you'd
probably
want
to
use
different
credentials,
so
we'll
create
that,
and
you
can
see
that
with
the
open
shift
template
for
you
know,
it
actually
gives
you
a
random
username
and
password
if
you,
if
you
didn't
supply
one,
so
you
can
actually
get
a
nice.
You
know
randomized
password
set
up
there.
B
B
Now,
at
this
point,
what
I'll
need
to
do
is
start
the
Olfa
cloud
application,
so
I
go
down
here.
I
have
uncatted
Erised
and
you
can
see
over
clyde
has
appeared
here
and,
as
we
talked
about
before
the
parameters
that
I
specified
in
my
template
get
reflected
into
the
openshift
console
here.
So
you
can
see.
I
can
tell
it
where
I
want
to
go
for
a
spark
and
I'll
tell
it
I'd
like
to
use
OPH
7077
and
that's
the
standard,
a
spark
connection
string
and
then
for
the
database.
B
I'll
also
give
it
a
connection
string
here
as
well,
so
say,
admin,
admin
and
MongoDB,
and
you
can
see
that
I'm
using
I'm
using
the
service
names
that
that
I've
applied
to
my
different
pods,
so
that
I
can
access
those.
So
I
don't
need
to
know
the
IP
addresses
the
kubernetes
dns
will
handle
this
off.
For
me,
this
web
route
hostname,
if
I
needed
to
kind
of
modify
how
I
reach
this
from
an
external
route.
I
could
change
this
to
make
it
look
a
little
different,
but
I
can
use
the
default
for
now.
B
Alright,
so
we
could
see
that
our
application
is
now
starting
to
run
and
it's
running,
and
previously
we
had
what
I
had
done
was
I
had
set
up.
A
couple
builds
to
actually
make
the
containers
that
go
into
this
project,
and
so,
if
we
look
in
the
builds,
you
can
see
that
I
had
two
two
pods
or
two
containers
that
go
into
our
into
our
application
and
I
allowed
openshift
to
build
these
for
me
and
I
set
these
up
ahead
of
time
because
they
can
take
a
little
bit
of
time.
B
B
B
A
live
demo,
yeah,
I,
didn't
I,
didn't
make
the
proper
obeisance
as
to
the
demo
gods.
Apparently
there
we
go
alright,
so
it
took
me
what
you
saw
is
it
took
a
couple.
It
took
like
a
minute
for
the
for
the
DNS
routing
to
go
through,
and
everything
and-
and
some
of
this
could
have
been
because
because
my
application
had
reported
ready
to
kubernetes
but
hadn't
quick
off
to
speed,
it
may
also
have
been
because
the
edge
router
hadn't
quite
put
all
the
pieces
together
yet
to
allow
me
to
route
through
well.
B
What
we're
looking
at
is
over
Clyde
and
we've
got
two
main
views
here:
the
trained
models-
and
this
is
you
know,
per
text
corpus
that
we
want
to
analyze,
and
then
we
have
queries
and
we,
you
know
we
haven't,
run
any
queries.
Yet
we
have
made
new
models.
So
the
first
thing
I'll
do
is
I'll
train
a
model
and
kind
of
the
scan.
An
example
we
use
is
the
works
of
Jane
Austen,
and
so
what
we've
done
is
we've
gone
to
Project,
Gutenberg
and
they've
made
available
to
us.
B
The
these
are
the
works
of
Jane
Austen.
All
in
one
nice
text
file
and
so
I'll
just
copy
the
URL
from
that
and
I'll
put
the
URL
here
and
I'll
tell
it
to
train,
but
we
can
see.
We've
now
got
some
things
happening.
We've
got
something
that's
training
and
if
I
go
back
to
open
shift,
I
can
look
into
the
pod
that
has
our
application
and
go
to
the
logs
and
we've
got
our
web
and
our
training
application.
B
If
I
look
in
the
training
application,
what
I'll
hopefully
see
is
no
errors
and
I
don't,
and
you
can
see
here,
it's
actually
started
to
run
a
task
against
spark,
and
so
this
is
the
way
I
can
I
can
confirm
that
it's
actually
doing
something.
You
can
see
that
it's
run
some
tasks
and
we'll
see
if
it's
completed.
B
So
we
go
back
here.
This
page
updates
a
little
slowly,
sometimes
so,
I
just
give
it
okay,
so
our
model
is
ready,
and
now
we
can
write
queries
against
it.
So
I'll
start
a
query
and
I
know
I
know
from
from
prior
example
that
there
are
many
characters
in
Jane,
Austen's
book
and
I'm,
going
to
give
it
the
name
of
one
of
the
characters.
Mr.
Darcy
and
we'll
see
we'll
see
what
it
can
tell
us
about
that
text,
but
run
a
query
boom.
It
brings
us
back
this
query
now.
B
I
can
see
that,
given
the
word
Darcy,
what
are
some
other
words
that
are
similar
to
it
in
vectors,
and
we
have
you
know:
Wickham
Collins,
Bingley,
Denny
Willoughby.
These
are
other
male
characters
in
the
works
of
Jane
Austen
and
work.
Tyvek
has
has
created
associations
based
on
you
know
the
placement
of
the
word
of
its
position
in
the
text
that
that
kind
of
give
it
similarity
to
this
word,
though,
I'm
not
extremely
versed
on
her
works,
but
from
what
I
know
is
that
these
other
characters
kind
of
our
male
protagonists.
B
You
who
sit
in
a
similar
position
as
mr.
Darcy,
and
so
what
you
can
see
is
that
you
know
we
could
use
this.
We
could
use
this
to
kind
of
just
explore
different
types
of
data
that
are
available
to
us
online
and
I'm
gonna
I'm,
going
to
jump
back
to
now
the
the
next
part
to
this
that
that,
unfortunately,
I
don't
have
that
I
don't
have
the
code
loaded,
but
if
we
go
to
that
to
our
github
project
for
this
and
what
I'm
going
to
do
is
go
into
the
web
model.
B
If,
if,
if,
if
I
wanted
to
make
a
change
to
this
project,
what
I
could
do
is
push
a
code
change
into
my
github
repository
and
then
I
would
go
back
into
the
builds
that
I
have
here
and
in
OpenShift.
What
you
can
do
is
you
can
configure
these
builds
to
take
a
web
hook
from
something
like
github
or
bitbucket
or
wherever
your
code
is
stored.
You
can
you
can
set
those
up
so
that
when
they
receive
a
new
commit
that
can
can
hit
back
to
your
OpenShift
project
and
trigger
a
new
build
now
I.
B
Wouldn't
at
this
point
my
my
cluster
is
hidden
behind
them.
You
know
kind
of
some
firewalls
and
one
on
it's
not
publicly
accessible.
But
what
would
happen
is
if
we,
if
we
made
a
change
to
the
to
the
web
application
for,
say
you
know,
I
could
come
in
here
and
manually
started
a
new
build
and,
and
it
would
by
this
configuration,
it
would
look
to
my
upstream
github
project.
B
It
would
pull
the
new
information
and
build
a
new
image
for
me
and
then
and
then
and
then
create
that
now,
I
didn't,
unfortunately
didn't
set
up
the
code
change,
but
what
I'll
try
and
do
is
I'll
just
try
and
run
this
quickly.
So,
let's,
let's
increase
the
text
size
here
and
see
if
I
can
just
do
a
quick,
a
quick
update,
so
I'll
go
into
a
directory
where
the
client
web
application
is
what
I
have
here.
B
B
B
All
right
push
this
up
to
github,
and
now
we
can
go
back
in
just
you
know,
we'll
check
our
project
make
sure
that
all
right,
so
you
can
see
my
my
changes
come
in
here
is
the
commit
and
what
I'll
do
is
I'll
go
back
in
and
we'll
rebuild
the
web
container.
So,
let's
start
a
build.
Thankfully
this
this
doesn't
take
too
long
to
build.
We
can
watch
the
logs
as
it
builds.
You
can
see
it's.
It's
just
pulled
this
change.
That
I
got
from
my
repositories.
B
You
can
see
change
card
to
blue
and
again
the
only
reason
I'm
manually
doing
this
is
because
I
don't
have
I.
Don't
have
my
cluster
isn't
publicly
available,
so
I
can't
let
the
webhooks
hit
my
cluster,
but
in
in
your
installations
you
know
you
could
set
this
up
to
do
it
internally
or
you
could
have
it
set
up.
You
know
pull
from
outside
and
you
can
see
it's
it's
building
the
it's
building
the
container
here
and
then
shortly.
B
What
it
will
start
to
do
is
push
these
layers
into
the
local
image
registry,
and
so
now,
if
we
go
back
to
the
overview,
what
we
will
see
is
that
openshift
has
detected
this
change
and
is
now
starting
to
redeploy
the
application
for
us,
and
so
this
will.
You
know
this
will
take
a
couple
seconds.
It'll
make
sure
it'll
stop
the
other
one.
You
know
it'll
switch
over
the
new
no
and
it
looks
like
well.
It
thought
it
failed,
but
it
actually
brought
it
up.
B
So,
let's
see
if
our
changes
actually
taken
out,
I
can
go
back
here
and
if
I
go
to
queries
and
I
think
I
need
to
reload
this
page.
There
we
go.
We
can
see
that
it's
quite
ugly
and
I've
chosen
a
poor
color,
but
you
can
see
how
the
change
that
I
made
was
really
quick
to
to
go
through
and
was
actually
automatically
redeployed
by
OpenShift
for
me,
and
so
this
really
is
where
the
whole
collaborative
effort
and
the
DevOps
kind
of
said
things
come
into
it.
B
You
know
you
can
imagine
that
okay,
this
is
a
pretty
bad
choice
that
we
made
and
we
picked
the
wrong
color
here,
but
in
in
a
normal
application
you
would
have
some
sort
of
you
know
continuous
integration
and
deployment
going
where
you
would
be
running
tests.
You
would
make
sure
these
things
go
and
then,
finally,
they
would
get
pushed
to
a
repository
that
would
update
your
openshift
there.
That
would
update
the
image
and
openshift
would
automatically
redeploy
this
for
you.
B
So
you
can
really
see
the
power
of
how
this
can
change
just
by
pushing
changes
to
your
to
your
code,
repositories
and
I
could
have
done
the
same
thing
with
the
training
module
if
I
needed
to
alter
the
way
the
models
were
behaving,
so
the
analytics
component
could
be
changed
out
as
well.
It
doesn't
just
have
to
be
the
web
component,
so
let's
go
back
to
the
to
our
slides
now.
Okay,
so
all
that
being
said,
you
know
what
did
we
learn
from
this?
B
Another
thing
that
went
really
smoothly
for
us
was,
you
know
using
docker
files.
You
know
I,
it's
no
secret
at
this
point
that
docker
and
containers
are
extremely
useful
to
the
development
effort
and
and
by
both
of
us
kind
of
creating
docker
files
and
quick
paths
to
create
containers.
It
made
it
really
easy
to
prototype
this
application
and
work
with
someone
else
who
wasn't
necessarily
sitting
right
next
to
me.
I,
you
know,
I
could
grab
their
latest
version.
B
I
could
build
a
new
container
and
test
with
what
they
had
and
then
lastly,
the
kubernetes
templates
as
well.
Much
like
the
docker
files,
the
kubernetes
templates
made
this
really
easy.
So
if
there
was
a
question
about
how
to
deploy
the
application,
we
could
just
share
a
template
between
us
and
that
template
would
effectively
put
all
the
objects
that
you
need
it
into
openshift
and
give
you
kind
of
this
rapid
uptake
so
things
that
kind
of
require
more
coordination.
B
I'd,
you
know,
I
hesitate,
saying
these
things
went
poorly
because
they
didn't
they
went
well,
but
the
things
you
have
to
think
about
API
construction
so
well
open
API
did
help
us
a
lot.
It
also
kind
of
exposed
the
fact
that,
as
you're
building
api's,
you
really
need
domain
experts
to
be
involved
so
that
you're
not
kind
of
going
in
the
wrong
direction.
So
there
were
some
choices
that
I
made
in
the
API
that
later
on,
didn't
make
sense
as
we
learn.
B
You
know,
as
I
learned
more
about
how
the
kind
of
analytics
component
works,
we
needed
to
adjust
that
API
and
so
that
you
know
that
cost
us
some
time
later
on,
because
we
had
to
adjust
the
way
the
API
work,
compute
resources.
Now
for
this
project,
the
compute
limitations
were
pretty
low.
You
know
we're
just
doing
kind
of
some
simple
word
calculations,
but
you
can
imagine
with
a
larger
operation
or
a
larger
application.
You
would
be
much
more
concerned
about
how
you're,
using
your
computer
resources
and
how
they
were
being
used
efficiently
and.
A
B
Got
another
slide
here,
we'll
talk
about
that.
Persistent
storage
is
another
issue
that
comes
up,
especially
when
you're
working
in
a
containerized
environment.
You
know
how
do
we?
Where
do
we
store
our
databases
or
our
files?
You
know
how
do
we
make
sure
that
they're
backed
up
properly-
and
this
is
something
that
is
gonna,
be
different
for
every
application
you
create
and
you'll
just
have
to
kind
of
assess.
You
know
what
works
for
you
and
you
know
whether
you
use
an
object,
store
or
some
sort
of
network
attached
file
storage.
You
know
these
are.
B
These
are
concerns
that
definitely
require
more
thinking,
as
you
build
your
applications
and
then
finally
kind
of
spark
configurations.
What
what
do
you
need
for
your
spark
applications
because,
as
you
deploy
spark
applications
to
a
cluster
oftentimes,
there
are
you
know,
third
party
packages
you
might
need,
or
you
know
just
extra
bits
of
configuration
need
to
go
to
the
spark
server
and-
and
these
are
things
that
you'll
have
to
kind
of
coordinate
out
and
figure
out.
How
will
you
deploy
your
spark
images
such
that
they'll
have
what
you
need
now
on
the
compute
resources?
B
What
I'm,
what
I'm
really
getting
at
here
is
that
as
kubernetes
schedules
pods
for
you,
what
it's
going
to
do
is
it's
gonna
look
out
across
its
its
cluster
and
the
nodes
of
its
cluster
and
it's
gonna
try
to
find
places
where
the
pods
fit
and
it
will
use
its
scheduler
to
try
to
make
them.
You
know
be
put
in
the
right
places
and
and
for
spark
because
it
uses
the
JVM.
B
If
you
don't
specify
kind
of
the
memory
constraints
and
the
CPU
constraints,
then
one
of
these
pods
could
just
try
to
hog
up
all
the
all
the
memory
on
a
server.
You
know,
for
example,
I've
seen
situations
where
I
you
know
I
would
deploy
a
spark
plug
and
then
I
would
look
to
see
what
it's
doing
and
it's
like.
Why
is
this
thing
trying
to
grab
like
ten
cores,
and
you
know
256
gigs
of
ram
it's
like
well,
because
it
doesn't
know,
there's
a
limitation
that
it
should
be.
B
They
should
be
following
and
and
that's
really
more
of
a
coup
Brunetti,
it's
kind
of
scheduler
thing.
You
want
to
make
sure
that
you,
you
mark
up
your
pods
to
let
it
know
you
know
what
the
constraints
on
those
things
would
be.
Another
way
to
do.
This,
too,
is
to
use
label
selectors,
so,
like
labels
can
be
used
to
make
sure
that
pods
get
scheduled
to
specific
nodes,
though,
if
you
have
a
node
that
has
some,
you
know
special
features
or
you
set
it
up
to
do
all
your
work.
B
You
know,
maybe
it
has
really
fast
drives,
or
it
has
some
really
big.
You
know
hard
drives
or
a
lot
of
memory
or
something.
Then
you
could
use
labels
to
help.
Tell
kubernetes
where
you
want
each
one
of
your
pods
to
land,
and
these
are
just
you
know.
These
are
things
you'll
have
to
take
into
consideration
as
you
build
your
applications,
so
where
you
know,
where
can
we
take
this
project
next,
we
could
certainly
add
more
spark
to
the
mix.
You
know
what
you
saw
that
the
queries,
weren't
necessarily
using
spark
to
pull
information
out.
B
B
B
Finally,
these
are
some.
These
are
some
project
links
to
the
different
different
projects
that
we
use.
There
are
some
of
the
top
ones.
You
know
the
Olfa
cloud
application
and
apache
spark
kubernetes
overshift
the
ischenko
project.
There
are
several
pieces
of
the
ischenko
project
within
red
analytics
within
our
rad
analytics
organization
on
github
and
then
finally,
we've
just
put
up
a
developer
portal
that
you
can
explore.
It
gives
a
little
information
about
what
we're
doing
with
the
rad
analytics
efforts.
B
A
This
is
you
went
good.
This
has
been
really
great.
There
actually
haven't
been
too
many
questions,
so
I'm
just
gonna
ask
out
loud
if
people
have
questions
type
them
in
the
chat
or
raise
your
hand
and
I'll
in
the
chat
now
I'll
open
up
the
mics
for
you,
this
it
strikes
me
is
that
we
need
even
more
sample
applications
and
that
this
might
be
a
good
time
to
maybe
do
a
hackathon
on.
B
A
B
A
And
I'm
always
asking
people,
you
know
to
do
interesting
things
with
some
datasets
that
you
know
that
I
worked
on
in
the
past
from
financial
data
sets
and
things
like
that.
So
I
actually
think.
There's
there's
something
too.
If
you
get
a
couple
more
examples
up
there
that
we
ought
to
do
some
sort
of
hackathon
hosted,
maybe
by
Commons
in
the
not-too-distant
future,
because
I
think
that
would
really
help
get
more
eyeballs
on
Apache
spark
and
using
it
on
an
open
ship
because
you've
made
it
look
ridiculously
easy.
I
have
when.
A
The
works
of
Jane
Austen
and
not
complex
XML
data
source,
which
is
what
I've
been
trying
to
hack
on
and
it's
been,
you
know
typically
I
pick
something
that's
tougher
than
then.
Then
it
should
be,
but
yeah
so
I
think
this
is
a
great
example.
You
could
do
lots
of
interesting
analysis
just
with
the
word
Tyvek
work
here
that
might
be
of
interest
to
to
raise
up
visibility
of
this
work.
A
B
B
So
I
see
a
Christians
question
here:
yes,
we
so
we
did.
We
consider
alternatives
to
database.
We
did,
but
you
know
really
for
this
application
because
we
were
trying
to
rapidly
develop
it
and
I
think
it
took
us
about
a
month.
To
put
this
whole
thing
together,
we
just
decided
to
use
because
it
was
real
easy
to
do
to
store
the
whole
data
structures
in
there,
and
we
already
had
a
template
available
to
us
and
openshift.
A
B
Know
recently,
there
has
been
more
work
going
on
bringing
JBoss
data
grid
into
openshift
and
I.
Think
your
what
you're
gonna
see
is
probably
within
the
next
month
or
two
we'll
have
another
example:
application
coming
up,
that's
like
a
a
movie
recommender
type
service
and
that
uses
JBoss
datagrid
on
the
backend
for
its
storage
platform
there
for
its
storage
surface
awesome.
A
B
A
Well,
this
has
been
very
enlightening.
Are
there
any
other
questions
out
there
I'm,
because
we're
definitely
going
to
get
you
back
and
get
a
few
more
of
the
big
data
folks
from
Red
Hat
and
if
folks
out
there
who
are
listening
to
this
either
the
live
one
or
the
recorded
one
if
they
have
an
example
of
using
Apache
spark
or
other
big
data
tools
on
openshift
and
want
to
give
a
presentation
or
show
it
off
or
get
feedback
on
it.
A
Please
reach
out
to
me
at
openshift
Commons,
on
Twitter
or
via
the
openshift
Commons
mailing
list
and
I
will
happily
give
you
a
podium
to
pontificate
on
as
long
as
it's
educational
and
of
interest
to
a
big
data
say
the
other
thing
that
might
mention
dev
Kampf
is
coming
up
Andrew.
No,
it's
a
big
open
source
conference
around
and
read
mostly
around
Red
Hat
technologies
and
technologies
that
up
the
upstream
into
things,
but
that's
coming
up
in
the
end
of
January,
ish
and
scribe
two
weeks
in
about
two
weeks.
A
So
that
would
be
great
if
you
could
join
us
for
that
and
as
always,
there's
the
open
ship,
Commons
mailing
lists
and
if
you,
if
you're,
not
on
them
yet
and
just
go
to
Commons,
OpenShift,
org
and
scroll
down
to
the
interest
sections
and
pick
the
topic
of
your
interest.
And
we
will
add
you
to
that
topics,
mailing
lists
so
again,
Michael.
Thank
you
very
much
for
taking
the
time
today.
Oh
yes
and
there's
one
other
thing:
dye
amphetamine
is
going
to
be
giving
a
talk
in
victoria
bc.
A
So
if
you're
up
in
my
neck
of
the
woods
on
the
23rd
of
february,
there
will
be
another
interesting
cloud
native
big
data
talk
in
Victoria,
the
British
Columbia,
which
is
the
seat
of
our
provincial
government.
So
please
please
join
us
for
that,
and
these
things
will
be
on
the
calendar
as
well
on
the
OpenShift
commons
page.
So,
thanks
again
and
we'll
see
you
all
soon,
thanks.