►
From YouTube: Operating PCF via Concourse. Or, How to Sleep More and Worry Less - Therese Stowell, Ryan Pei
Description
Operating PCF via Concourse. Or, How to Sleep More and Worry Less - Therese Stowell & Ryan Pei, Pivotal & Yuri Litvinovich, Scotiabank
Cloud Foundry has a lot of knobs and dials. If you’re an operations team, how can you ensure that you have declarative, repeatable processes for installation, upgrade, and disaster recovery? Leveraging published Concourse pipelines turns operations from snowflake-creation to push-button rinse & repeat.
This talk will feature both the folks building CF and the folks operating CF. They’ll discuss and demonstrate using Concourse to install, upgrade, and manage disaster recovery on CF, including DR strategies like active-inflate aimed at reducing recovery time. They’ll give real world examples of how they manage multiple CF installations and can offer CF as a Product to their app developers, with guaranteed SLAs and well defined RTOs. And they’ll discuss reference architectures that accommodate CF, Credhub, and Concourse.
About Therese Stowell
Therese Stowell is Product Manager at Pivotal. She has worked in the software industry for 20+ years as programmer, interface designer, and product manager. She developed the command line environment in Windows, founded a successful social enterprise, and was part of a startup team to win a Nesta Open Data Institute £40,000 prize. She also has an MA in Fine Art. She has extensive speaking experience, including a keynote at SpringOne and sessions at CF Summit Santa Clara and CF Summit Europe.
A
All
right
well,
thank
you
for
coming
to
our
talk,
as
was
mentioned
earlier,
this
is
about
operating
PCF
with
concours
or,
as
we
like
to
say
how
you
can
sleep
more
and
worry
less
as
an
operator
of
the
platform,
so
first
I
have
to
get
this
fire
exit
announcement.
I
was
going
through
slides
earlier
today
and
realized
Concorde.
The
word
concours
is
in
this
slide,
so
this
is
referring
to
a
public
space
outside
a
building,
not
the
CIC
Dee
engine,
so
yeah
just.
C
A
Right
so
a
quick
round
of
introductions,
my
name
is
Ryan.
I
am
a
PM
for
a
lot
of
different
projects
at
pivotal,
I
mostly
go
around
and
tell
people
to
automate
all
their
operations
as
much
as
possible,
and
that's
pretty
much
what
I
do
we
also
Yuri
who's
a
senior
cloud
engineer
at
Scotiabank?
It's
been
working
on
this,
this
new
platform
there,
which
is
part
of
scotia
bank's
digital
transformation
over
there,
that's
been
ongoing
for
a
couple
years
now
and
also
threes
who's,
the
PM
of
platform
recovery
claim
to
fame.
A
There
is,
of
course,
bbr
the
bash
Bosh
backup
and
restore
tool.
Alright,
so
we'll
go
ahead
and
start
so
yeah
I
saw
from
the
number
of
hands
up
earlier.
Most
of
you
are
familiar
with
concourse,
so
we
won't
go
into
detail
of
what
concourse
is
so
we'll
do
just
a
quick
recap
of
that
then
we'll
dive
into
scotia
banks,
PCF
architecture
and
their
usage.
So
talking
about
how
many
foundations
they
have,
how
those
are
laid
out.
A
What
their
failover
scenario
is
what
the
workloads
in
terms
of
apps
look
like
and
then
also
talk
a
little
bit
about
their
concourse
deployment
and
then
all
the
operations
that
they
do
with
concourse,
such
as
backup
restore
upgrades,
install
and
then
the
kind
of
SLA
is
that
they
set
for
their
users
and
then
we'll
also
talk
about
what
in
general
conch
or
what.
What
the
before
and
after
state
was
for
them
when
using
concourse
and
bbr,
and
then
also
a
talk
briefly
about
the
Plato
platform
at
Scotiabank.
A
A
We've
used
it
for
a
variety
of
different
operations,
such
as
software
updates,
provisioning,
new
environments,
managing
your
networks,
such
as
the
firewalls,
and
make
sure
all
the
right
ports
are
open
and
IPS
are
allocated,
and
things
of
that
nature
also
for
monitoring
so
ensuring
that
the
health
of
your
platform
and
also
the
health
of
your
apps,
continues
to
be
good
and
doing
backups
and
restores
so
a
lot
of
different
activities.
That
can
be
automated
with
Concours
and
we've
seen
many
different
users
of
the
tool
really
grow
to
love
it.
A
So
that's
that's
been
it
it's
a
specialty
so
far,
so
a
quick
anatomy
lesson
in
a
pipeline.
In
case
you
need
a
refresh,
so
the
pipeline
is
comprised
of
a
series
of
jobs
which
are
comprised
of
tasks,
and
the
pipeline
is
essentially
running
those
jobs.
It
can
also
run
them
in
parallel,
doesn't
have
to
be
in
series
and
then
what
ties
these
jobs
together
are.
The
resources
and
resources
can
be,
anything
really
could
be.
A
A
file
could
be
a
schedule
which
is
essentially
creating
a
cron
job
out
of
a
out
of
a
concourse
job,
telling
the
job
when
to
run
it
at
what
time,
and
so
here's
an
example
of
a
pipeline
which
is
doing
an
update
of
a
PCF
tile.
How
many
of
you
are
users
of
PCF
by
the
way
in
a
quick
show
of
hands,
okay,
cool
and
how
many
of
you
have
used?
Dvr,
just
okay,
so
we'll
go
over
that
for
sure
alright,
and
with
that
I'll
hand
it
over
to
Yuri
to
tell.
A
E
All
right
good
morning,
ladies
and
gentlemen,
it's
my
pleasure
to
be
here
thanks
Ryan,
as
Ryan
said,
I'm
a
senior
cloud
engineer,
working
for
Scotiabank
and
I'm
gonna,
tell
you
about
our
story
of
using
PCF
using
Concours
and
what
we
built
with
that.
So,
as
I
said,
I
work
for
a
cloud
development
platform
team
and
we
build
a
platform
that
provides
effective,
reliable,
easy
to
use
and
highly
available
platform
for
developers
to
run
their
applications
and
that
helps
them
to
achieve
high
velocity
that
helps
them
to
promote
innovation
and
agility
and
all
those
good
things.
E
E
So
a
CDP
architecture
at
the
core
of
it
is
the
pivotal
cloud
foundry
and
we
currently
have
10
PCF
foundations
running
on
the
fabric
of
two
cloud
providers,
both
Microsoft
Azure
and
Google
cloud
platform.
Foundations
are
organized
in
the
active
passive
pairs
and
it
means
that
passive
foundation
is
the
full
replica
of
an
active
foundation
and
developers
when
they
push
their
code
and
deploy
their
applications.
They
push
them,
they
use
the
pipelines
and
those
pipelines
they
push
the
code
to
both
foundations,
team
Dainius
li,
but
at
first
all
traffic
goes
to
the
active
foundation.
E
So
then
the
question
is:
why
do
we
have
passive
foundation
in
place
and
I'm?
Pretty
sure
that
many
of
you
know
the
answer?
This
is
to
achieve
high
availability
and
we
have
smart
DNS
configured
that
way.
So
if
something
happens
to
active
foundation,
if
it
detects
an
issue,
it
changes,
DNS
records
to
point
to
passive
foundation
and
all
traffic
single
ously
go
to
now
the
same
app
running
in
the
PCF
passive
foundation.
E
Let's
talk
about
the
workloads
right
now,
so
we
currently
have
50
applications
running
in
production
and
120
in
non
production,
and
those
includes
different
kind
of
apps,
starting
from
user
facing
applications
such
as
components
of
websites,
mobile
apps,
encryption,
decryption
applications
and
services.
Apps
responsible
for
authentication,
as
well
as
internal
apps,
not
customer-facing,
such
as
support
and
miscellaneous
services,
and,
as
you
can
tell
those
are
pretty
critical.
Apps
that,
if
ever
something
happens
to
them
would
be
would
be
a
noticeable
customer
impact.
E
So
all
apps
are
stateless.
We
use
both
clouds
and
on-prem
databases
and
there's
also
a
so-called
incremental
readiness
process
of
how
developers
onboard
or
transform
their
apps
to
run
on
our
platform.
So
the
incremental
readiness
means
that
we
encourage
team
to
build
their
application
to
transform
their
application
with
the
operational
stories
and
in
the
development
lifecycle.
E
So,
instead
of
just
going
or
with
checkboxes
and
when
the
delivery
date
dates
of
approach,
we
encourage
them
to
inject
this
transformation
gradually
and
build
the
applications
in
the
edge
our
way,
and
we
are
proud
to
serve
customers
in
many
countries,
of
course,
Canada,
but
also
Chile,
Colombia
Mexico,
and
that
list
is
growing
all
right.
Let's
go
to
the
next
slide
and
talk
more
about
what
helps
Scotiabank
to
operate
our
PCF
environments
and,
of
course,
that's
the
tool
that
was
introduced
earlier,
which
is
concourse
so
at
Scotiabank
we
have.
E
We
have
concourse
a
separate
concourse
built
into
deployed
into
each
PCF
foundation
and
it
has
its
own
web
subsystem.
It
has
its
own
database
and
two
workers
per
each
foundation.
Concourse
is
deployed
by
secondary
boss
as
the
boss
deployment.
What
are
the
benefits
of
this
approach?
That's
a
good
questions
and
we
can
say
that
that
brings
us
environments
being
fully
segregated
and
isolated,
since
we
have
to
concourse
workers,
they
never
get
overloaded
and
able
to
run
without
any
issues
and
it's
easier
to
operate.
E
E
C
Yuri,
so
Bosch,
backup
and
restore
is
a
framework
that
is
designed
to
backup
and
restore
distributed
systems
which
is
kind
of
a
hard
problem,
because
you've
got
data
all
over
the
place.
There's
essentially
two
parts
to
it.
There's
a
CLI
called
bbr
which
sits
outside
the
thing
that
you're
backing
up
and
restoring.
So
it
sits
on
jump
box
or,
as
recommended
it
sits
in
concourse.
It
will
trigger
backup
and
restore
scripts
that
are
inside
of
the
releases
and
they're
packaged
in
their
releases,
so
they're
they're
always
compatible
with
the
release.
C
The
backup
script
will
generate
an
artifact,
a
backup
artifact
whatever
that
needs
to
be,
and
then
the
bbr
script
will
transfer
that
artifact
back
to
concourse
or
the
jump
box,
so
bbr
can
backup
or
restore
any
bosh
deployment
or
bosh
director
that
implements
the
scripts
the
bbr
scripts.
Currently
it
supports
pas
or
ERT.
It
supports
EF
deployment.
It
supports
the
Boche
director,
it
supports
PCF
Redis
and
there
are
a
lot
more
products
that
are
building
support
for
bbr.
C
E
Teresa
and
so
it's
fairly
easy
to
understand
that
we
built
at
Scotiabank,
we
built
the
concourse
backup
pipeline
based
on
VB,
are
now
so,
but
before
we
used
the
tool
called
CF
ops.
How
many
of
you
is
familiar
with
that?
Cf
ops,
good
I,
see
hands
awesome,
but
the
bbr
is
like
next-gen
way
of
backing
up
PCF
and
we
found
out
that
it's
a
pretty
good
and
we
based
we
build
our
pipeline
based
on
pivotal
VB,
our
PI
point,
plus
a
bunch
of
enhancements
like
we
have
backups
can
be
scheduled.
E
So
you
can
see
three
jobs
here
on
the
top.
This
is
the
job
that
backups
elastic
runtime,
then
the
one
that
backups
primary
boss
director
and
the
third
one
exports
ops
manager,
installation
they
are
all
being
triggered
by
the
concourse
resource
of
a
trigger
conkers
resolves
each
one
of
them
can
be
triggered
at
the
different
time
and
it
all
all
of
them
are
being
uploaded
to
cloud
storage
again
through
concourse
resource,
not
directly
through
the
code.
E
Alright
and
now,
let's
dive
in
a
little
bit
of
theory,
how
many
of
you
are
familiar
with?
What
SLA
is
cool?
Almost
everybody?
What
about
SLO
SLI
less
hands?
Okay,
so
you
can
see
them
on
the
slides,
it's
all
related
and
it's
about
setting
expectations
to
our
customers,
so
service
level
agreement.
It's
like
a
contract
with
your
customers
that
defines
the
level
of
service
that
you
provide.
E
Slo
is
the
target
value
or
range
of
values
for
the
service
level.
That
is
measured
by
why
service
level
indicator
and
SLI.
It's
basically
a
some
quantitative
measure
of
some
aspect
of
level
of
service,
for
example,
app
time,
latency,
availability,
durability,
etc,
and
there
are
two
other
definitions
which
is
recovery,
time,
objective
and
recovery
point
objective.
E
So
let's
talk
now
about
what
scotia
banks
platform
offers
as
their
as
a
part
of
their
service
level
agreement.
So
we
have
three
s:
ellos
for
four
FS
ellos
for
4
SL
is
and
the
first
one
is
platform
uptime
or
platform
availability,
and
we
guarantee
to
have
three.
Ninth
and
five
uptime
for
platform
availability.
However,
platform
had
five
ninth
uptime
year
to
date
and,
of
course,
we
use
health
checks
to
monitor
the
uptime
another
metric
that
we
have
in
our
service
level.
Agreement
is
push
availability
uptime
and
we
set
a
slope
of
that
at
99%.
E
E
We
also
use
concourse
for
PCF
installs
and,
as
it's
mentioned
here
in
the
slide
before
PCF
deployment
was
highly
manual
process.
It
involved
lots
of
toil
and
doing
repetitive
tasks
following
the
official
procedures,
but
we
made
it
better
and
we
implemented
concourse
pipelines
to
install
PCF
foundations
and
they
are
based
on
PCF
pipelines
provided
on
peel
net,
and
we
use
different
pipelines
to
install
different
parts
of
PCF
ops
manager.
We
have
pipelines
for
elastic
runtime
for
tiles,
for
IP
ii.
Other
borscht
add-ons,
and
also
for
different
borscht
releases.
E
Same
with
upgrades
here
is
the
table
that
shows
different
upgrades.
Pcf
upgrades
that
we
undertake
undertook
and
from
one
nine
to
one
point
ain.
The
process
of
upgrade
was
highly
manual
it
again.
If
the
wolf
lots
of
toil,
we
are
following
all
the
procedures,
but
it
it
resulted
in.
It
resulted
in
lots
of
time
engineering
time
spent
on
doing
that
and
total
time
for
all
foundations
was
four
weeks.
E
It
was
painful
and
it
led
us
to
think
you
know
that
we
need
something
better
and
we
reached
out
and
decided
to
automate
that
so
from
one
ten
to
one,
PCF
upgrade
was
now
built
on
pipelines,
multiple
conquers
pipelines
and
the
total
time
for
doing
that.
Upgrade
was
about
two
weeks
so
much
better
so
and
the
next
upgrade
we
did
from
111
to
112.
It
was
pretty
much
one-click
concours
pipeline
that
you
can
see
on
the
top
over
there.
E
You
click
it
once
and
you
go
do
your
job
while
concourse
is
running,
doing
the
upgrade
so
and
the
total
time
to
upgrade
all
the
foundations
we
have
was
less
than
one
week.
So
it's
less
than
24
hours
to
upgrade
one
PCF
foundations
with
all
the
applications
running,
and
not
only
that
not
only.
It
was
painless
and
required
almost
no
human
interaction.
There
was
no
downtime
for
customers,
yay.
E
Thank
you.
Thank
you.
So
how
was
life
was
conquers
in
VBR?
For
us
we
found
it
to
be
good.
We
saw
this
automation
set
in
place.
It
definitely
became
better
and
automation
helped
us
to
eliminate
what
so-called
toil
and
allowing
our
engineers
to
spend
their
precious
time
and
talent,
building
something
from
engineering
perspective
and
sewing
engineering
challenges
not
digging
into
this
routine
procedures
so
and
this
magic
unicorn
edge
already.
This
is
how
our
engineers
look
like
and
feel
after
implementing
automation,.
E
So
the
cloud
platform,
the
CDP
cloud
development
platform
that
I
was
talking
about,
is
actually
a
part
of
a
bigger
vision.
We
have
at
Scotiabank
the
division
of
digital
transformation
and
the
platform
we
the
bigger
platform
that
we
have.
We
call
it
play-doh
Scotiabank
laid
a
platform.
So
what
is
play-doh?