►
From YouTube: Transforming a Bank with a Highly-Opinionated Automated Release Pipeline - Reid Levesque
Description
Transforming a Bank with a Highly-Opinionated Automated Release Pipeline - Reid Levesque, Royal Bank of Canada
Can development teams focus on developing business logic while avoiding the bureaucratic jungle to get to production? Microservices are great at speeding up development time and shortening the feedback loop -- unless every team has to build a pipeline from scratch. In an organization used to monthly or quarterly releases, moving to daily or hourly deployments requires more than a change in application architecture.
This session will cover how we’ve automated all the things while still giving developers the freedom to be successful.
About Reid Levesque
Reid is a Principal Software Developer in the Cloud Platform Engineering team at RBC. He has created several microservice applications using Spring Cloud and Cloud Foundry. Reid is actively accelerating applications to production with an opinionated release pipeline. Reid has a strong development background and received his Bachelor of Mathematics in Computer Science from the University of Waterloo. He is based in Toronto, Ontario.
A
Hi
everyone
we're
gonna
get
started,
but
before
we
do,
I
need
to
read
the
fires
announcement.
So
please
note
the
locations
of
the
surrounding
emergency
exits,
locate
the
nearest
exit
nearest,
lit
exit
sign
to
you
in
the
event
of
a
fire
alarm
or
other
emergency.
Please
calmly,
exit
to
the
public
concourse
emergency
exits
stairwells,
leading
to
the
outside
of
this
facility,
are
located
along
the
public
concourse
for
your
safety
in
an
emergency.
Please
follow
the
direction
of
the
public
safety
staff.
Great,
so
hopefully
you're
all
here.
A
A
So
what
are
we
going
to
talk
about
today?
So
I
want
to
kind
of
take
you
through
for
those
of
you
who
haven't
worked
at
a
big
organization,
government
or
bank.
That
sort
of
stuff
I
want
to
talk
to
you
through
kind
of
how
we've
done
it
in
the
past
some
ways
we
tried
to
fix
that
and
then
how
the
problems
that
led
to
and
how
we
solved
those.
A
So
hopefully,
everyone
knows
a
monolith.
Is
a
big
giant
thing.
That's
unruly!
So
that's
not
really
the
interesting
bit
but
more.
What
comes
out
of
having
monoliths,
so
we
end
up
having
these
long
development
cycles.
So,
typically,
waterfall
three
months,
if
you're,
lucky
you'll
get
a
deployment
out
and
each
application
deployment
is
a
special
snowflake.
So
we
need
to
work
with
every
other
team.
You
know
you
need
to
work
with
the
network
team,
the
database
team,
the
five
op
teams
that
also
use
your
database,
etc,
etc.
A
So
that's
a
mess,
so
we
don't
really
want
to
be
doing
that
anymore,
and
the
other
thing
to
note
is
that
we
do
a
lot
of
retail.
So
a
lot
of
the
banks
applications
revolve
around
these
three
steps,
get
data
from
somewhere,
transform
it
into
the
right
format
and
then
put
it
somewhere
else,
and
so
traditionally
that's
been
done.
You
know
with
some
database
and
then
some
process
and
then
no
database.
A
So
our
idea
at
the
bank
was
to
stop
doing
that
which
I
thought
was
a
good
one
and
we're
gonna
do
some
venture
of
an
architecture,
so
vent
ribbon
is
a
new
hotness.
The
idea
is
that
you
take
your
data
and
turn
it
into
events,
and
then
you
can
write
them
in
processors
instead
of
having
batch
jobs.
So
streaming
is
ultimately
better
than
batch
because
you
can
always
Express
a
batch
process
as
a
stream.
A
Reading
a
file
in
is
just
streaming
the
file
right,
so
we
chose
Kafka
as
our
bus
event
stream
and
we
want
to
put
the
data
into
a
lastik
search.
So
we
have
a
lot
of
use
cases
where
there's
search
use
cases.
So
if
someone
calls
up
the
help
desk
or
hey,
can
you
help
me
here
and
we
need
to
look
up
the
users
information?
So
elasticsearch
is
perfect
for
that
and
then
we
didn't
want
to
do
micro
services.
A
Sorry,
we
didn't
wanna
do
monoliths,
so
he
decided
to
do
micro
services
put
a
REST
API
in
the
front
and
then
the
data
is
exposed
to
the
rest
of
the
bank.
So
we
don't
need
to
go
through
this
mess
all
over
again,
so
just
briefly
through
the
application
architecture.
So
the
reason
I
want
to
focus
on
this
is
what
we've
done.
Is
we've
written
an
opinion
framework
for
this
type
of
architecture?
So
we
didn't
write
a
framework
for
every
application.
A
At
the
bank
we
wrote
a
framework
for
event-driven
applications
using
Kafka
and
some
sort
of
datastore,
in
this
case
it's
elasticsearch,
so
the
data
comes
from
somewhere
goes
into
Kafka.
We
have
some
consumer
which
will
do
some
sort
of
transformation,
put
it
in
the
bespoke
format
that
we
need
for
answering
the
question
and
that
question
will
be
answered
by
the
REST
API,
which
ultimately
just
asked
elasticsearch,
and
then
the
client
will
connect
to
the
REST
API
so
fairly,
a
fairly
simple
architecture.
A
So
we
went
from
monoliths
to
microservices
and
now
we
have
a
whole
new
set
of
problems.
So
all
the
processes
at
the
bank
are
set
up
to
deploy
very
infrequently.
So
when
you
do
a
deployment
every
three
months,
it's
really
hard.
So
you
want
to
be
really
careful
and
you
want
to
be
super
diligent.
You
got
to
coordinate
every
one,
so
there's
all
kinds
of
processes
of
the
change
records
getting
everyone
coordinated.
A
But
if
you
want
to
release,
you
know
once
an
hour
once
a
day
once
a
week
that
becomes
quite
onerous
and
you
spend
more
time
doing
the
process
than
you
do
doing
the
doing
the
actual
deployment,
and
so
the
things
that
they
want
to
make
sure
that
are
done
in
the
process
aren't
necessarily
bad
things
like
you
know,
making
sure
testings
done
making
sure
that
you're
not
going
to
break
stuff.
You
know
regression
testing,
etc,
etc,
but
the
idea
is,
we
want
to
automate
those
checks.
We
don't
want
them
to
be
some
some
manual
process.
A
The
other
thing
is,
as
we
get
more
micro
services,
things
get
hairy
in
a
hurry.
So
if
all
our
operations
is
set
up
around
my
monitoring
one
application,
then
we're
gonna
be
in
for
a
world
of
hurt
when
messages
are
in
Kafka
and
we
have
some
eventual
consistency
in
our
API,
so
things
get
a
lot
harder
and
the
other
bit
is
that
every
team
has
to
do
this
themselves.
So
every
team
I
served
monitoring.
Every
team
has
to
set
up
a
pipeline
to
go
to
production.
A
Everything
has
to
test
to
figure
out
how
to
solve
all
these
problems
and
and
that
sucks,
so
we
need
a
pipeline.
We
need
something
that
everyone
can
just
use
and
and
benefit
from
so
the
too
many
pieces
to
do
it
ad
hoc,
you
can't
you
know
before
you
could
get
away
with
some
some
person
typing
on
a
keyboard
and
it'd
be
alright.
But
now,
if
you
have,
you
know
even
five
microservices
you're
just
going
to
be
typing
forever.
We
want
something
repeatable
and
we
want
some
way
to
automate
all
those
processes
away.
A
A
So
we
were
on
the
cloud
team
and
put
a
cloud
team
and
we
were
doing
the
first
of
these
event-driven
architectures,
and
so
we
had
a
very
tight
deadline
is
about
two
months
to
three
months,
which
is
fairly
tight
when
you
have
nothing
going
on
and
it's
incredibly
tight
when
you
work
at
a
bank
and
everything
has
a
three-day
SLA
to
get
access
timing.
So
what
do
we
have?
We
have
Cloud
Foundry,
so
we
had
Cloud
Foundry
and
we
had
our
code
in
github
and
somehow
we
needed
to
get
it
from
one
to
the
other.
A
So
there
was
nothing
we
had
nothing
there
before
there's
no
pipeline.
We
could
leverage.
We
had
a
few
tools,
though
so
we
have
jenkins.
We
have
essential
jenkins,
so
we
we
opted
for
maven
because
we
knew
maven,
no
other
reason
than
that
and
and
the
jenkins
was
already
up
and
running,
so
we
didn't
really
need
to
setup
our
own.
So
that
was
a
small
win.
We
had
to
sell
the
tool
called
urbancode
deploy,
so
a
quick
show
of
hands
who's
ever
heard
of
that
before
Wow.
A
A
So
it
had
a
few
nice
features
to
it,
so
one
it
was
Bank
approved,
so
we'd
enough
go
through
the
process
of
figuring
out.
You
know:
how
do
we
get
whatever
tool
we're
gonna
use
approved?
We
can
just
go
ahead
and
use
this
one
and
for
for
those
who
worked
at
bigger
organizations,
they'll
know
that
that's
a
non-trivial
task.
The
other
thing
is
it.
Let
us
store
secrets,
so
it
was
a
place
that
was
approved
to
store,
usernames,
passwords,
etc.
So
if
we
need
to
connect
to
databases
or
kafka,
then
then
that's
good.
A
The
bad
part
is
that
we
couldn't
store
config
in
source
control.
So,
in
our
case
kit,
which
is
bad
because
any
time
you
make
a
change,
you
make
a
change
to
the
code
and
then,
if
you
need
to
update
the
pipeline
because
we're
still
fairly
new,
then
you
have
to
go
into
this
UI
and
make
the
change
and
that's
bad,
because
as
your
code
changes,
this
doesn't
really
change
with
it.
A
Who
here
has
used
ansible
to
deploy
in
the
cloud
yeah?
So,
okay,
we'll
get
to
that
as
well?
Was
it
was
good.
We
knew
and
we
knew
as
well,
so
that
was
kind
of
want
to
leverage
some
of
the
things
that
we
knew.
We
had
to
tight
deadlines.
We
want
to
go
ahead
and
not
reinvent
everything.
We
could
store
that
config
and
get
it's
good.
We
can
move
along
with
the
branches
and
we
could
have
different
config.
A
So
ansible
has
a
concept
of
group
bars
and
you
can
have
you
know
a
dev
config
and
a
prod
config,
which
is
good
because
you're
going
to
have
different
servers
for
Kafka
elasticsearch
et
cetera,
bad
ansible
is
really
good
at
deploying
say
you
want
to
install
Tomcat
on
10
servers
and
you
just
kind
of
list
the
servers.
Nobody
goes
it's
great,
deploying
to
Cloud
Foundry,
it's
not
so
great,
so
there's
no
plugins
for
it.
So
we
had
to
write
a
bunch
of
that
ourselves,
which
was
in
scripts
like
Python
scripts.
I
think
it
was
so.
A
It
was
a
bit
bit
too
awkward.
We
also
want
to
achieve
idempotency.
So
the
idea
behind
ansible
is,
if
say,
Tomcats
already
installed
on
on
server
a
then
it
doesn't
do
anything
right.
So
we
want
to
get
the
same
idea
from
from
that
for
Cloud
Foundry.
It's
a
part
of
that
was
driven
by
the
reason
that
we
used
mono,
repos
or
a
mono
repo
I
should
say
so.
A
All
our
all,
our
micro
services
were
in
one
one
repo,
and
we
can
talk
about
whether
that's
good
or
bad,
but
it
was
both
good
and
bad
good
and
bad
in
this
case.
So
because
we
every
time
something
changed,
it
was
going
to
rebuild
and
redeploy
everything
we
wanted
to
make
sure
that
only
redeployed
what
actually
changed,
and
so
we
did
some
fun
and
by
fun
I
mean
terrible
ideas
that
you
should
never
do
at
home.
A
We
just
had
to
use
md5
sums
to
compare
whether
things
had
changed,
so
we
we
took
the
md5
sum
of
the
jar
we're
deploying
so
it's
great,
so
we've
got
that
and
was
stored
in
in
an
environment
variable
in
Cloud
Foundry.
We
can
just
compare
the
next
time
right.
So
the
problem
is
how
everyone
knows
is
the
jars
in
their
headers
have
a
timestamp
or
something
that
changes
every
time
you
build
it.
So
if
you
build
the
same,
you
zip
up
the
same
files
twice
you'll
get
a
different
MD
by
some.
So
it's
like
okay!
A
Well,
we
can
work
around
that,
so
we
expanded
the
jar
into
files
and
then
we
did
an
md5
sum
of
each
of
the
files
and
put
that
in
a
text
file
and
we
didn't
md5
sum
of
that
text
file.
So
it
got
complicated
real
quickly,
but
it
did
serve
the
purpose
of
we
only
redeployed.
What
actually
changed
the
other
tricky
bit
was
services,
so
we
have
a
few
years
to
provide
services,
2.2,
caf-co
elasticsearch,
any
other
services.
We
need,
and
it's
very
easy
to
update
a
user
provide
service.
A
I
mean
it
doesn't
really
need
to
be
idempotent.
The
only
problem
is
that,
if
you're
not
deploying
the
application
that
the
service
is
bound
to,
then
you
need
to
restage
that
application.
So
we
had
to
do
some
finagling
to
figure
out
which
applications
were
bound
to
the
service
have
changed
so
it
got.
It
got
way
too
complicated
way
too
quickly.
A
So
we
then
had
a
second
application
and
it
was
very
similar
to
the
first
very
similar
event-driven
architecture.
We
also
type
tight
deadlines,
because
we
did
such
a
good
job
on
the
first
one
that
they
figured.
We
could
do
an
even
better
job
in
the
second
one,
seeing
as
we
solved
all
the
hard
problems.
So
there
are
still
a
few
problems
that
we
want
to
solve.
A
We're
gonna
we're
going
to
use
the
existing
pipeline
because
we
don't
have
time
to
rewrite
it,
but
we
wanted
things
to
go
a
bit
faster,
so
I
think
I
think
for
the
first
application
we
had
five
or
ten
and
we're
somewhere
not
ranged
microservices,
so
deploying
not
every
time
gets
its
real
slow.
So
what
we
wanted
to
do
is
deploy
all
of
those
at
the
same
time,
because
there's
nothing
really
dependent
between
the
applications
they
could.
A
We
could
do
five
CF
pushes
at
the
same
time-
and
all
of
this
again
is
driven
by
the
fact
that
we
had
a
mono
repo.
So
we
had
some
parallelism,
so
our
first
attempt
was
to
have
urbancode
deploy,
call
ansible,
say
five
times
with
different
parameters,
say
deploy
application,
a
application,
be
efficacy.
So
then
we
broke
the
cardinal
rule
of
of
put
nothing
in
urbancode
deploy
because
then
anytime,
something
changed.
We
had
to
go
back
to
to
this
UI
based
system
which
wasn't
in
get
so
that
was
no
good.
A
A
The
other
thing
we
really
wanted
to
do
was
to
do
that
automated
testing
bit
so
the
first
one
was
a
UI.
The
first
application
was
a
UI
and
it
just
didn't
have
time
to
automate.
Automating
UI
testing
is
difficult,
so
this
one
was
all
kind
of
his
back-end
stuff,
so
we
could
easily
send
some
data
in
the
front
and
check
the
time
a
it
at
the
end
in
the
right
way.
A
So
the
problem
we
had,
though,
with
automated
testing,
was
that
we
ended
up
running
on
a
live
instance,
so
we
would
deploy
the
application
and
because
everything
was
so
entangled
we
had
to
have
all
the
services
up,
and
then
we
ran
testing
on
that,
and
so
that's
okay.
But
if
you
find
a
problem,
it's
kind
of
too
late,
you
have
to
roll
everything
back
and
it
takes
time.
So
this
this
was
definitely
a
lot
better,
but
we
needed
to
work
on
a
few
things.
A
So
what
were
our
shortcomings?
Well,
it
took
too
long.
The
whole
thing
took
about
20
minutes.
Even
if
you
didn't
change
anything,
but
more
often
not
you
changed.
Some
central
library
and
you'd
have
to
redeploy
the
whole
thing,
and
so
then
we
had
a
team,
I
think
five
or
seven
developers
and
if
we
all
want
to
do
something
kind
of,
we
all
want
to
do
work
right,
and
so
every
time
we
go
to
deploy,
we'd
have
to
wait.
20
minutes.
On
top
of
that,
our
automated
testing
was
a
bit
flaky.
A
A
Is
we
if
you
haven't
figured
out
yet
we
picked
a
few,
the
wrong
tools,
so
ansible
wasn't
necessarily
the
right
tool
for
deploying
to
the
cloud
bash
is
fine,
but
not
necessarily
that
robust
and
urbancode
deploy
wasn't
really
getting
us
to
much,
and
they
were
on
top
of
that
there
was
still
a
ton
of
manual
steps.
So
if
we
want
a
new
Kafka
topic,
we
want
a
new
elasticsearch
index
or
we
want
a
new
boundary
space.
It
was
all
manual
stuff.
We'd
have
to
do
that
manually.
A
So
welcome
to
the
third
and
ultimate
pipeline.
So
this
time
we
had
a
bit,
we
had
a
bit
more
time,
so
we
decided
to
rewrite
from
scratch.
So
we
threw
away
the
code,
but
we
kept
all
the
lessons
we
learned
and
we
still
had
a
few
things,
so
we
had
github
still
that
worked
well
and
we
had
cloud
foundry
I
want
to
keep
that
and
we
wanted
to
keep
me
even
so.
A
We
like
maven,
we
know
maven,
so
we
kept
that
one
now
one
of
the
things
we
want
to
do
and
you'll
hear
a
lot
of
it.
Other
people
talk
about
pipelines
and
they'll
focus
a
lot
just
on
the
deployment
side,
and
that's
great
that's
great
that
you
can.
You
know,
certainly
solves
one
problem,
but
we
want
to
go
a
bit
further.
We
want
to
provide
not
necessarily
lock
people
in
but
but
give
them
a
head
start
on
building
their
code
as
well.
A
So
when
we're
doing
when
we're
doing
these
applications
are
fairly
standard,
so
everyone's
gonna
have
more
or
less
the
same
spring
dependencies.
You
know,
you'll
depend
on
elasticsearch
web
Kafka,
etc.
So
we
can
at
least
fix
the
versions,
so
we
have
a
bunch
of
bunch
of
defaults
that
we
have
in
the
CI
space.
So
we
have
a
default
parent
palm,
so
you
can
just
inherit
from
that
which
is
great
and
we
have
some
Aven
jobs
or
sorry,
some
Jenkins
jobs
which
will
scan
your
repo
and
just
grab
your
Jenkins
file.
I'm.
Sorry,
ok!
A
Well,
that's
kind
of
pretty
standard
right,
but
then
what
we
did
is
we
wrote
some
functions
which
we
added
into
our
Jenkins.
So
you
could
just
call
those
from
your
Jenkins
files.
You
just
say
you
know,
do
the
CI
or
do
the
CD
and
and
then
you
wouldn't
have
to
worry
about
all
the
steps
so
like
stuff
and
we'll
get
to
what
they
are
but
stuff
that
are
not
necessarily
straightforward.
A
So
we
had
Jenkins,
we
decided
to
do
our
scripting
in
Python.
We
we
have
some
experience
in
Python.
It
has
a
fairly
good
REST,
API,
fairly
good
to
call
REST
API,
which
we're
gonna
use
for
Cloud
Foundry
hit
run
stuff
on
the
shell
as
well.
So
it's
good
and
the
other
thing
we
used
this
docker,
and
this
is
so
key.
So
we
ended
up
running
Jenkins
on
docker
and
then
the
jobs
would
run
inside
docker
containers
on
Jenkins,
and
so
that's
super
important,
because
now
we
don't
need
to
worry
about.
Like
is
CF
CLI
installed.
A
A
So
we
might
as
well
mention
concourse
concourse,
does
very
well
where
you'll
you'll,
you
can
grab
the
subfolder
inside
inside
your
git
repo
and
say
we'll
just
just
build
if
this
sub
folders
changed,
but
we
didn't
really
want
to
go,
learn
concourse.
We
have
enough
to
do
so.
We
stuck
with
Jenkins
Jenkins
works
best
when
you
make
your
changes
at
the
top,
so
the
top
level
changed
and
there
you
go
so
we
went
multi
repo,
so
every
micro
service
has
its
own
repo
and
github.
A
A
So
so
we
have
it
in
Yama
and
it's
just
simple
stuff,
like
you
know,
what's
your
app
called,
how
much
memory
do
you
need
you
know?
What's
your
topic
called?
It's
not
all
the
options,
so
it
wouldn't
be.
It
wouldn't
be
everything
you
put
in
the
manifest,
for
instance,
it
would
just
be
a
subset
because
there's
a
lot
of
stuff
that
you
don't
really
need
to
worry
about,
like
where
the
Kafka
server
is
or
where
elasticsearch
will
take
care
of
that
as
the
pipeline.
A
You
just
need
to
know
you're
going
to
use
elasticsearch
and
you
want
your
index
to
be
called
this
and
here's
the
JSON
and
such
and
such,
and
so
what
we
would
do
is
part
of
the
pipeline
is
we
were
checked
to
make
sure
that's
okay,
so
you
want
to
know
sooner.
You
don't
want
to
go
all
the
way
to
deploy.
It's
like.
Oh
hey.
The
thing
failed
because
you're,
the
syntax
error
so
I
know
that
as
soon
as
possible.
So
we
check
that
here
and
then
what
we
do
is
we
put
the
artifacts
in
s3.
A
Now
we
can
argue
all
day
as
to
the
best
artifact
repository
some
people,
like
Nexus
other
people
like
artifactory.
We
chose
s3,
partly
for
the
reason
that
we
had
more
control
over
that
and
we
didn't
have
to
rely
on
other
teams.
So
it's
always
good.
When
you're
trying
to
get
stuff
done,
the
hurry
is
to
do
stuff.
You
have
control
over
the
other
bit
we
did
was
so
part
of
our
goal
was
that
each
deploy
each
branch
would
get
its
own
pipeline
all
the
way
through.
A
So
if
I'm
working
on
my
branch
Reed
and
my
colleagues
working
on
his
branch
John,
then
you
know
we'll
have
well
but
we'll
have
a
reed
space
and
we'll
have
a
read
Kafka
topic
and
we'll
have
a
read
elasticsearch
index
and
so
on
so
I'll
be
able
to
deploy
everything.
All
the
way
out
and
and
John
will
be
able
to
do
that
as
well,
and
we
don't
conflict
with
each
other.
So
the
nice
thing
of
using
s3
is
we
can
put
the
buckets
named
after
the
branches
or
named
after
whatever
we're
deploying.
A
So
we
have
a
prod
prod,
bucket
or
whatnot,
which
makes
it
really
easy
to
pour
cleanup.
So
whenever
we
delete
the
branch,
then
we
just
delete
the
s3
bucket
and,
and
it
makes
life
a
lot
simpler
and
then
and
then
grand
plans
which
haven't
quite
been
realized.
Yet
I'll
be
honest,
but
the
idea
is
to
have
a
chat
bot
in
between
the
CI
and
the
CD.
So
when
you're
deploying
to
dev
or
maybe
staging
is
it's
great-
you
just
kick
it
off
automatically,
but
especially
at
a
bank.
A
You'll
want
to
hold
off
your
Prada
ployment
to
have
some
sort
of
manual
step
and
that
manual
step
will
be
someone
saying,
slash,
slash,
pipeline
bot
and
deploy
right
so
that
that's
kind
of
how
that
will
work
all
right.
So
that
was
the
CI.
So
let's
have
a
look
at
the
CD.
So
what
we
did
now
is
we
created
the
environment
parameters,
so
these
are
things
like
server
locations,
API,
URLs,
etc,
and
then
we
combine
those
with
the
application
parameters
to
get
our
full
full
picture.
A
So
you
know
we
know
it's
gonna
be
called
reads,
awesome,
app
and-
and
we
know
it
needs
a
large
amount
of
memory,
so
we
give
it
four
gigs
and
so
on
so
forth.
So
we
get
something
that
we
can
actually
go
off
in
deploy
and
then
we'll
go
and
provision
our
Kafka
topics.
If
they're
not
there
again,
idempotency
will
create
any
elastic
search
indexes
if
needed
and
yeah
and
then
we'll
do
the
actual
deployment.
A
So
the
idea
is
we're
gonna
do
a
Bluegreen
deployment,
so
we
want
to
have
zero
downtime
as
much
as
possible,
so
I
can
never
remember
which
is
which,
but
but
so
we'll
do
the
green
first
deploy
that
and
then
know
what
we
did
this
time
as
we
did
automated
testing
on
the
on
the
non
live
route.
So
we
have
our
temporary
pointing
to
our
new
app.
A
So
we'll
do
the
testing
there,
which
is
great
because
now,
if
it,
if
the
testing
fails,
we
can
just
not
switch
the
red
over
and
then,
if
the
testing
does
pass,
then
we
will
switch
the
road
over.
So
the
idea
is
that
this
automated
testing
is
something
that
the
user
will.
The
developers
will
provide,
so
they
their
development
team
will
plug
something
in
there
and
then
it
will
call
whatever
testing
it
needs
to
happen
and
then
we'll
just
check
the
return
code.
A
So
what
so,
then
this
is
a
picture
of
what
the
developer
experience
looks
like
now
that
they
have
this
pipeline.
So
all
they
have
to
do
really
is
inherit
from
the
parent
palm
add
a
few
parameters
to
say:
here's
my
app
name,
here's
the
memory,
if
my
kafka
topic
and
that's
it
and
and
they
commit
and
away
they
go
so
I
mean
this-
was
so
good
that
when
we
were
developing
it,
we
hadn't
kind
of
finished
everything.
A
So
we
got
like
little
bits
done
and
so
we're
working
on
the
bits
that
weren't
in
the
pipeline
yet-
and
it
was
it-
was
so
painful,
so
we're
gonna
in
little
rooms
like
four
of
us
and
we
were
still
payable.
We
all
turn
around
like
oh
I,
want
to
use
the
pipeline.
Is
it
so
good
so
and
that
at
that
point,
I
knew
that
we
have
something
special.
A
A
Yeah
sure
sure,
so
the
question
is
the
question
is:
did
we
have
to
get
approvals
to
store
data
in
s3,
so
the
bit
that's
hidden
in
this
wonderful,
green
box?
Is
that
it's
an
internal
s3?
So
it's
an
s3
API
on
top
of
internal
storage.
So
in
that
case
we
didn't
have
to
get
approvals
to
put
stuff
outside
onto
public
cloud,
so
that
this
whole
thing
runs
internally.
A
Yeah,
so
the
question
is:
how
much
work
was
this
and
was
it
frustrating
I
felt
like
I?
Should
it
should
be
available
somewhere
else?
So
the
answer
is
probably
too
much
work
or
not
enough.
I,
don't
know
it
felt
like
a
lot
of
work
at
the
time
the
benefits
are
certainly
there.
I
wish
there
was
something
that
we
could
use
straight
out
of
the
box,
and
there
are
some
efforts,
showed
it
to
supreme
Cloud
pipelines.
A
I
know,
Marcin
is
doing
a
great
job
with
that
stuff.
It
didn't
quite
fit
our
use
case,
we'd
kind
of
gone
far
enough
down
the
road
that
we
hadn't
weren't
able
to
use
it,
but
it
really
depends
on
like
how
much
control
you
want
to
give
up.
So
so,
if
you
want
to
do
everything
for
your
developers
or
as
much
as
possible,
they
may
end
up
writing
stuff
yourself,
so
nothing
is
going
to
end
up
creating
you
know,
Kafka
an
elastic
search
as
well.
A
So
if
you
want
to
do
just
deploy
to
PCF,
then
that's
okay,
and
so
the
idea
that
that
we
want
to
do
is
on
top
of
this
is
well
add
the
security
checks,
we'll
add
whatever
bits,
they're
currently
manual
processes
in
here
as
automated
checks.
But
if,
if
we
use
something
out
of
the
box
and
is
that
availability
is
that
is
that
feature
available?
So
it's
kind
of
the
trade-off
anyone
else
yeah.
A
A
Yes,
so
probably
about
three
months
from
from
the
time
we'd
finish
the
second
pipeline
to
here
and
a
lot
of
that,
quite
frankly,
was
spent
deciding
which
direction
we
want
to
go
like
making
decisions.
I
stand
up
here
and
say
we're
using
multi
repo
analysis
so
easy,
but
but
again
to
the
point
where
we
decide
that
that's
the
right
thing
we
wanna
do
and
like
oh,
we
want
to
do.
A
We
want
to
do
Jenkins
and
all
we
want
to
use
the
github
org
plug-in
in
in
in
Jenkins,
and
all
we
want
to
use
docker
like
these
sort
of
questions
and
like
how
much
control
do
we
give
to
developers
was
really
where
we
spent
a
lot
of
our
time
decoding
it
up,
wasn't
terribly
important.
It's
terribly
difficult,
I
mean
it
still
took
time,
but
the
hardest
bit
was
kind
of
figuring
out
what
we
wanted
to
do
there
good
yeah.
A
Sorry,
all
your
docker
containers
stuff,
like
that
you
kind
of
get
for
free
with
Jenkins,
and
we
had
a
ton
of
experience,
I
think
between
us.
There
was
something
like
30
years
of
administering
Jenkins,
so
we
had
a
ton
of
familiarity,
so
it
felt
silly
to
throw
that
away
when
we
had
so
many
other
questions
to
answer
and
about
good.
A
So
the
chat
yeah,
the
chaplet,
so
the
chat
pod
isn't
implemented.
So
this
is
all
kind
of
theoretical,
but
I
will
talk
about
how
the
chat
pod
works.
So
the
idea
is,
it
is
a
PCF
service
yeah
that
would
talk
to,
in
this
case
slack
and
and
the
ideas
that
would
say:
hey
your
builds
done
and
then
and
then
halyard
deploys
done,
etc
and
then
for
higher
level
environments.
You
would
ask
it
to
deploy
that
application.
That
would
be
your
interface.