►
From YouTube: Persistence Arrives on Diego - Ted Young, Pivotal, Nagapramod Mandagere, IBM, & Paul Warren, EMC
Description
Persistent Storage is coming to Diego! IBM, Pivotal, and EMC have partnered to extend the Cloud Foundry runtime to include persistence storage. We discuss the approach we are taking, how distributed filesystems and block storage services fit well with our service broker abstraction. and how we plan to offer these services in the marketplace. An overview of the types of storage, their differences, and the effect they have on scheduling and the application lifecycle will be discussed. Example use cases for each type of storage will be covered. We will also discuss the pros and cons of running databases and other services directly on Cloud Foundry.
Nagapramod Mandagere
Research Group Leader, IBM
Nagapramod Mandagere received his PhD from University of Minnesota in Enterprise data management. He has been a researcher at IBM Almaden Research center since 2008 working on various systems technologies. He has coauthored several conference papers and has several patents in domain of systems management
Paul Warren
Software Engineer, EMC
Ted Young
Pivotal
Ted has built distributed computer systems in a variety of environments: computer animation pipelines for VFX, live event coordination, and elastic compute platforms. In 2015 he received a Pivotal Research Grant to explore approaches to running persistent workloads in a multi-tenant environment. He is a core contributor to Diego, and heads up the Diego Persistence Team (Persi).
A
A
A
year
ago,
I
came
and
spoke
at
this
conference
about
a
proposal
to
add
persistence
to
Cloud,
Foundry
and
I'm,
happy
to
say
today
that
we've
gone
and
done
it
so
I'd
like
to
welcome
onto
stage
my
coworkers
promote
mana
Jerry
from
IBM
research
and
Paul
Warren
from
EMC,
and
we're
going
to
talk
to
you
about
the
persistence
options,
we've
added
to
Cloud
Foundry
today
and
the
ones
that
are
coming
soon.
All
right.
Ok,.
B
Good
good
morning,
everybody,
a
quick
outline
for
today's
talk.
First
we'll
talk
about
why
we
need
persistence
for
CF
applications
and
by
this
I
mean
native
persistence,
we'll
back
it
up
with
a
few
use
cases
that
we
have
come
across
from
multiple
customer
scenarios
and
multiple
application
use
cases
we'll
follow
it
up
with
the
architecture
we
came
up
with
and
then
go
to
a
couple
of
walk
throughs
for
WordPress
and
one
sample
big
data
analytics
application
and
follow
it
up
with
futures
and
comments.
B
B
There
is
a
ephemeral
file
system
available
inside
a
container
which
is
at
the
disposal
of
the
application,
but,
as
the
name
suggests
its
FML
or
there's
no
guarantees
on
this
file
system.
This
is
local
to
the
container.
If
you
have
multiple
instances
of
your
application,
chances
are,
they
are
deployed,
they're
deployed
in
different
cells,
and
this
local
ephemeral
storage
is
not
shareable
across
instances.
So
there
is
no
common
namespace.
B
The
only
way
to
do
state
persistence
in
Cloud
Foundry
today
is
why,
as
a
service
route,
you
have
a
bunch
of
different
types
of
services
available
in
Cloud
Foundry.
Today
there
are
relational
services
like
db2
Oracle.
There
are
no
sequel
engines
like
Cloudant
they're,
also
the
object,
storage
services
likes
swift,
which
you
can
use
for
persistence.
But
the
key
point
to
note
here
is
these
services
themselves,
don't
run
on
Cloud
Foundry
runtime.
They
run
on
other
runtimes
externally
or
directly
on
infrastructures.
B
Eliminating
all
types
of
file
system
dependencies
in
your
code
is
quite
complicated
and
often
you'll
end
up
probably
rewriting
you're,
better
off
rewriting
the
entire
application
code
from
scratch.
But
this
is
not
always
feasible
for
all
application
scenarios
and
when
it
comes
to
12
factor
conformance
achieving
the
last
mile
of
12
file.
Factor
conformance
is
the
most
crucial
or
the
trivial
part
or
non-trivial
part,
and
that
makes
porting
existing
legacy.
Applications
to
cloud
foundry
really
an
arduous
task.
B
B
Other
type
of
applications
are
composite.
Applications
nowadays
are
composed
of
multiple
micro
services,
each
deployed
on
different
runtimes.
You
might
have
some
micro
services
running
on
Cloud
Foundry
and
some
others
running
on
infrastructure
providers
directly
and
in
these
cases,
porting
existing
datasets,
which
are
the
heart
of
all
these
applications,
becomes
an
a
very
arduous
task,
and
this
is
unstructured
data
and
it's
not
always
possible
to
API
5.
This
data
and
file
system
is
the
only
common
way
of
sharing
data
across
them.
B
Further,
it's
not
just
a
matter
of
doing
a
one-time
ETL
at
the
beginning
of
your
deployment
phase
chances
are
that,
since
your
deployment
strides
across
multiple
run
times,
you
need
to
be
in
a
constant
cycle
of
doing
this
ETL
on
a
day
to
day
basis.
So
if
your
data
changes
in
one
run
time,
you
need
to
again
do
an
ETL
from
file
system.
Two
objects
are
no
sequel
engines
and
ingest
it
into
cloud
foundry
and
then
actually
consume
it
in
your
cloud.
Foundry
applications.
B
So
it's
not
a
one-time
ETL
process,
it's
a
very
constant
process
for
a
lot
of
applications,
and
this
is
a
nightmare,
and
such
applications
typically
cannot
be
ported
to
cloud
foundry.
Today,
there's,
however,
one
alternative
to
overcome
these
limitations,
that's
basically
by
using
fuse
enable
file
systems.
So
what
you
do
in
this
case
is
you
use
user
space
file
systems
like
SSH
Affairs
and
actually
mount
remote
file
systems
into
the
container
directly?
B
By
doing
this,
you
can
access
external
remote
storage
in
file
systems
and
your
applications
don't
need
to
eliminate
all
file
system
dependencies
and
they
can
be
readily
ported
to
Cloud
Foundry.
But,
however,
this
comes
with
a
bunch
of
challenges.
First
and
foremost
is
security.
What
typically
happens
is
when
you're
trying
to
do
a
fused
mount
inside
a
container.
Your
container
needs
to
be
running
in
privileged
mode,
otherwise
containers
does
don't
support
mount
commands,
so
this
is
often
a
non-starter
because
running
containers
and
privileged
mode
opens
you
up
to
a
lot
of
security
challenges.
B
You
know
provider
runtime.
The
second
challenge
is
that
of
concurrency
when
you're
doing
fused
mounts
and
if
you
are
mounting
the
same
file
system
across
multiple
containers,
there
is
no
consistency,
guarantees
that
are
achievable
in
such
systems.
So
the
third
important
point
is
that
of
performance.
Since
you're,
using
a
user
space
file
system,
it
cannot
match
the
native
file
system
access
that
you
can
get
from
traditional
file
systems,
so
these
limitations
make
the
fuze
based
systems
non-starter
for
quite
a
few
situations.
B
C
Yeah,
thanks
for
mate,
say
design
girls.
Let's
have
a
look
at
those.
We
knew
right
up
front
that
we
weren't
going
to
build
hooks
for
every
all
of
the
storage
systems
out
there,
and
neither
did
we
want
T.
So
primary
design
goal
was
to
be
open
and
extensible
in
the
solution
that
we
provided.
We
also
decided
for
a
couple
of
reasons
to
surface
this
through
the
standard
service
broker,
semantics
and
model,
two
main
reasons
there
really
was.
C
We
want
it
to
be
familiar
just
like
attaching
any
other
service
to
your
app
and
secondly,
we
know
that
even
within
the
same
category,
all
of
these
storage
systems
have
have
subtleties
to
the
way
that
work.
So
we
wanted
the
app
developer
to
be
cognizant
of
the
of
the
type
of
volume
that
he
was
attaching
to,
and
obviously
we
wanted
to
solve
that
security
problem
that
promote
talked
about.
C
So
we
wanted
to
be
able
to
do
this
so
that
the
containers
didn't
need
to
run
with
any
special
elevated
privileges,
but
they
can
still
mount
volumes
so
implementation.
How
do
we
actually
solve
that
we're?
Looking
in
the
community,
there
was
no
standard
body
out
there
for
volume
mounting,
but
docker
done
quite
a
lot
of
work
in
this
area
and
having
a
look
at
their
stuff,
they
had
a
fairly
nice
volume
API
in
a
nice
ecosystem
of
volume,
drivers
being
written
to
that
API.
C
Clearly,
we
don't
want
it
to
place
an
app
on
a
cell
that
doesn't
have
the
right
volume
drivers
for
the
type
of
volume
that
the
app
needs
to
mount
to,
and
secondly,
we
actually
had
to
obviously
make
diego
do
the
mounting
itself.
In
a
couple
of
respects
needs
to
mount
onto
the
cell,
firstly,
and
then
it
needs
to
mount
take
that
volume
and
mount
it
into
the
container
picture
paints
a
thousand
words.
So
let's
have
a
look
at
that:
here's
a
rather
simplified
version
of
the
Cloud
Foundry
architecture.
C
You
don't
pick
me
up
on
this
and
it's
simplified.
So
let's
see
what
bits
we
added!
First
and
foremost,
obviously
we
got
the
storage
sitting
outside
of
Cloud
Foundry,
some
sort
of
volume
service,
okay,
could
be
EMC
scale.
I/O
for
block
could
be
EMC
Isilon,
one
FS
for
NFS
stuff
could
be
IBM
spectrum
scale.
We
actually
chose
to
implement
a
reference
implementations
there's
something
that
works
out
of
the
box
and
we
chose
set
for
that
because
it
provided
what
we
needed
from
day
one
and
actually
provided
a
little
bit
of
stuff
for
us
moving
forward.
C
As
I
said,
we
surfaced
this
to
be
the
standard
service
broker
semantics,
so
we
front
that,
with
some
sort
of
volume,
specific
service
broker.
Okay,
that,
like
provides
I,
think
in
modern
lingo
that
provides
the
control
plane
for
the
storage
system,
then
added
a
volume
manager
on
to
each
cell,
and
we
call
this
thing
vol
man
and
that
doesn't
actually
really
talk
to
the
storage
systems
right,
open
and
extensible.
So
we
add
volume
drivers,
we
co-locate
volume
drivers
onto
the
cell.
C
At
the
point
you
install
Cloud
Foundry
and
those
things,
as
we've
already
talked
about
expose
exposed
by
the
docker
volume
API.
So
the
vault
man
talks
those
things
using
the
docker
volume
API
calls.
So
those
are
the
pieces
we
add,
how
does
it
really
work
again,
reinforcing
the
point
that
we
just
it's
just
another
service?
Alright,
but
it's
a
little
bit
of
a
special
service.
Ok,
so
your
developer
needs
to
bind
his
app
to
some
volumes.
It
just
issues
the
same
CF
create
call.
C
Obviously,
then,
when
you
actually
want
to
bind
it,
you
just
issue,
a
CF
bind
call
Cloud
Controller
receives
that
force,
that,
on
to
the
volume
specific
service
broker,
the
best
way
to
think
about
this
is
the
in
response.
The
volume
specific
service
broker
issues,
one
or
more
volume
mount
instructions
back
to
cloud
file.
Cloud
Foundry
then
takes
them
and
does
a
couple
of
things.
C
First
thing
it
does
is
BBS
talks
to
volume
manager
and
the
volume
manager
talks
to
the
volume
specific
driver
on
the
cell
to
perform
the
actual
mount
on
to
the
VM
on
to
the
cell
itself,
and
then,
secondly,
whoops
wrong
way
and
then
secondly,
BBS
instructs
garden
to
take
that
volume
out.
We
just
did
at
cell
level
and
mount
it
into
the
container.
C
That's
going
to
host
the
app
that
you're
pushing
and
that's
pretty
much
it.
So
you
can
see
it's
a
fairly
simple
picture,
but
whilst
it's
simple,
we
think
that
this
thing
caters
for
all
of
the
current
use
cases
and
all
of
the
future
use
cases.
We've
got
moving
forward,
I'm
gonna
hand
you
over
to
Ted
who's,
going
to
talk
a
little
bit
about
those
yeah.
A
Thanks
Paul,
so
let's
look
at
a
couple
specific
examples
to
really
drive
home
how
this
works
exactly
the
same
way.
Other
services
work.
So
let's
talk
about
blank
storage.
This
is
the
most
obvious
example.
You
have
an
application.
Let's
say
it's
a
large
wordpress
application.
You
have
a
very
high
traffic
blog
and
you'd
like
to
use
WordPress
on
Cloud
Foundry,
but
you'd
like
to
use
it.
A
The
way
WordPress
was
intended
to
be
used,
which
is
to
have
a
site
administrator,
be
able
to
install
themes
and
plugins
directly
through
the
admin
panel,
rather
than
having
a
developer.
Do
that
for
them.
You'd
also
like
to
dynamically
scale
the
write
load.
In
other
words,
it's
not
enough
just
to
cache
the
responses
from
WordPress.
You
also
have
a
lively,
comics
comment
session,
maybe
a
forum,
something
that
has
a
write
load
that
can
spike
and
so
you'd
like
to
be
able
to
scale
horizontally.
A
So
Cloud
Foundry
allows
you
to
do
that,
provided
you
have
a
distributed
file
system
in
your
service
marketplace.
So
if
we
look
at
pushing
WordPress
and
then
creating
a
cephus
service
that
attaches
to
it
and
then
scaling
WordPress
up,
you
can
see
works
like
any
other
service
and
we
focus
on
the
calls
we're
specifically
making
to
the
service
broker.
You
can
see
we're
calling
create
service
with
a
premium
plan.
There's
some
storage
specific
options
here,
but
these
are
specific
to
this
service.
A
So
it's
it
wants
to
know
which
storage
tier
you'd
like
do
you
want
SSD
or
spinning
platters?
Do
you
want
any
kind
of
outer
band
backup
to
be
happening?
This
particular
service
broker
provides
we
back
up
so
we're
turning
that
on
and
then,
when
you
bind
it
to
your
application,
you're
asking
the
service
broker
to
bind
it
to
a
specific
mount
point
within
your
application.
So
for
WordPress,
that's
the
WP
content
directory,
that's
well
the
themes
and
plugins
go
it.
A
Then
you
start
WordPress
and
then
you
scale
it
and
it
works.
Just
like
usual:
let's
go
through
another
example.
So
this
is
an
example
of
working
with
an
external
data
set.
So
to
set
this
up,
imagine
you
have
satellite
images
and
every
day
new
satellite
images
come
into
your
system
and
you've
been
doing
this
for
years,
and
those
images
are
stored
on
a
variety
of
storage.
Backends,
the
oldest
stuff,
is
still
on
tape.
Drive
you've
got
things
in
EMC
Isilon.
A
You've
got
something
new,
maybe
that
you're
rolling
out
tomorrow,
you're
gonna,
add
to
that
cluster,
and
currently
your
developers
have
to
know
where
these
things
live,
if
they
want
to
access
them
and
you'd
like
to
move
those
data
processing
workloads
on
to
cloud
foundry,
but
you
can't
move
all
of
them
onto
Cloud
Foundry.
Some
of
them
still
need
to
be
running
in
virtual
machines
or
in
some
older
back-end.
So
how
would
you
do
that?
Will
you
do
it
with
a
custom
service
broker?
A
So
if
we're
pushing
an
image,
processor
app
to
Cloud
Foundry
and
then
binding
it
to
our
satellite
image
service
broker,
and
we
look
at
the
options
that
were
talking
about
when
we
buy
into
that
service
broker.
We're
saying
we
want
to
bind
to
the
daily
snapshots.
We're
interested
in
some
image
sets
specifically
we're
interested
in
the
four-band
images
and
the
multispectrum
images
for
this
particular
use
case,
and
then
we
bind
it
to
application,
we're
giving
it
a
top-level
data
path.
A
Then
we
start
our
image
processor
and
we're
off
to
the
races
now
notice.
There
is
nothing
in
this
create
service
or
bind
service
call
where
I
talked
about
what
type
of
service
was
running
back
there
I'm
not
talking
about.
Please
mount
this
thing
from
this
Isilon
cluster.
Please
mount
this
thing
from
the
tape
drive.
Is
the
application
developer?
You
don't
really
care
about
those
details.
You
just
care
about.
A
I
want
the
four
band
daily
snapshots,
and
so
the
service
broker
is
capable
of
isolating
all
that
information
from
the
application
developer,
because,
ultimately
it's
the
thing
that's
in
charge
of
the
volume
mounts
so
I
think
this
shows
that
this
clean
separation
using
service
brokers
really
allows
you
to
continue
operating
up
at
the
domain
level,
rather
than
delving
into
the
implementation
details
of
how
the
service
actually
works
all
right.
So
let's
talk
about
the
future.
A
A
Well,
you
have
a
ephemeral
space
available
to
your
app,
but
that's
running
on
a
layered
file
system
in
kind
of
a
shared
environment,
and
it's
not
necessarily
all
that
performant
for
heavy
readwrite
workloads
and
you
might
be
running
an
environment
where
you
have
better
disable
and
you'd
like
to
make
those
available
specifically
to
the
apps
that
have
these
heavy
readwrite
workloads.
So
we
plan
on
extending
the
docker
volume
plug-in
interface,
to
allow
drivers
to
advertise
local
resources
and
to
have
Diego
be
able
to
take
advantage
of
them.
A
So
in
this
case
every
new
instance
would
get
a
new
volume
whenever
it's
spun
up
and
when
the
application
instance
is
spun
down,
that
volume
would
be
reaped
and
recycled.
So
there
would
be
no
permanent
data,
but
there
would
be
temporary
scratch
space,
and
this
works
nicely
with
our
current
stateless
environment.
Taking
that
a
step
further,
you
have
single
attached
volumes,
so
this
is
same
thing
as
scratch.
A
Space,
but
you're,
saying
I
would
like
when
I
start
the
application
instance
again
for
it
to
reattach
to
the
same
piece
of
data
and
to
have
there
be
a
consistent
relationship
between
my
app
instance
and
the
application
data.
This
is
something
that
we
can
build
from
a
persistence
layer,
but
it's
problematic
on
a
couple
other
layers,
notably
you're,
now
bringing
identity
and
state
into
the
mix
onto
a
platform
that,
from
my
primarily
associated
with
running
stateless
application
logic.
So
in
order
to
do
that,
we
would
need
to
extend
this
scheduler
to
allow
for
consistency.
A
So
a
consistent
identity
can
be
maintained
across
restarts
and
also
you
would
need
to
be
mounting
things
that
could
somehow
take
advantage
of
these
volumes
and
mostly
what
people
think
about
in
those
cases
are
databases,
but
the
problem
with
databases
is
the
ones
that
people
want
to
run
in
production
today
they
don't
really
cleanly
support
a
separation
between
DevOps
the
person
operating
the
database
and
cloud
ops.
The
person
operating
the
machine.
The
database
is
running
on
pushing
CDs
to
that
things
of
that
nature.