►
Description
Evolution of CICD in CapitalOne - Gokul Prabagaren & Rajesh Srinivasan, CapitalOne
Capital One is the first U.S. bank to exit on-prem legacy data centers and go all in on the cloud. We came very far in our journey of being a 100% Cloud operated company. From initial days of having separate teams to support and deploy development team’s code changes to production,now we are deploying the majority of our production changes as CICD enabled push button changes.This talk focuses on our CICD journey from our humble beginnings to where we are today.It focuses on various phases of CICD development and adoption within our organization. From initial days of custom scripting to matured inner-sourced framework for CICD,we have gone through various issues,challenges and learnt a lot of lessons over the course of our journey. We felt it worth sharing with a broader audience in this forum.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Hey
everyone
hope
all
are
doing
fine
and
enjoying
the
conference
so
far.
This
is
me
goku
prabhu,
and
I
have
my
colleague
rajesh
srinivasan.
We
both
are
from
card
rewards
r
and
we
are
very
excited
to
bring
our
topic
evolution
of
cacd
in
capital
on
you
all.
In
fact,
we
both
have
been
with
the
capron
for
many
years
and
we
have
lived
this
journey
and
this
topic
is
very
near
and
dear
to
us.
For
that
reason.
A
A
Now
we
are
a
tech
company
which
happens
to
be
doing
the
banking
business
and
that's
how
we
operate
and
that's
how
we
think-
and
we
are
a
founder-led
company
which
stays
too
true
to
its
mission
of
change
banking
for
good,
and
that
also
primarily
signal
like.
There
are
a
lot
of
ways
in
which
we
give
back
to
our
communities
and
specifically
when
it
comes
to
the
take.
A
We
have
contributed
more
than
20
plus
open
source
projects
to
the
community
and
to
name
few
critical
stack,
which
is
a
security
focus
kubernetes
offering,
which
is
a
widely
adopted.
One
and
ruby
khan
is
ml
model
tracking
tool
and
we
have
invested
heavily
on
capital
one
quartus
program,
and
this
is
a
program
which
is
sponsored
by
kaplan,
run
across
multiple
middle
schools
and
this
program
mentors
and
motivates
inspires
middle
school
students
to
embark
on
a
journey
and
see
themselves
as
future
technologies.
And
similarly
there
is
another
program
called
capital
encoder.
A
A
A
Most
of
our
idea,
operations
were
outsourced
and
it
was
built
by
vendor
teams
and
typically
dev
team
requests
for
a
capacity
in
on-prem
server
to
deploy
their
application
and
typically
system
team
which
allocates
them
and
once
the
feature
build
is
done.
Devoting
throws
that
build
over
the
wall
to
the
system
team
and
they
deploy
it
and
they
hand
it
off
to
ops
team
and
they
manage
the
things
in
production
and
that's
how
most
of
the
things
were
done.
It's
not
only
specific
to
us.
A
This
will
definitely
resemble
with
all
the
folks
who
have
gone
through
this
kind
of
things.
Then
we're
dealing
with
on-prem
things,
and-
and
this
is
also
not
a
surprise
for
folks
who
build
things
and
on-prem
environment,
where
ops
team
manages
a
lot
of
their
production
things
and
that's
what
we're
calling
it
as
healthy
support
and
whenever
there
is
an
issue.
Ops
team
comes
back
to
the
dev
team,
hey.
This
is
something
reported
by
customer
and
you're.
Seeing
it's
not
working
the
immediate
response,
always
it's
not
a
bug.
It's
a
feature
right.
A
A
A
They
feel
more
empowered
to
make
decisions
for
their
teams,
and
this
kind
of
environment
actually
fosters
a
lot
more
innovative
working
environment
and
because
they
make
the
decision
which
impacts
their
teams
and
be
it
a
technology
or
whatever
the
things
which
impacts
them
and
they
make
the
decisions
and
they
hone
holistically
from
getting
involved
in
business
teams
and
taking
from
start
till
the
end
and
manage
them
right.
That
gives
the
engineering
teams
lot
more
ownership
and
another
main
advantage
of
doing
this
way.
Is
you
don't
need
ops
team
to
reach
out
to
dev
team?
Hey?
A
A
Products
is
what
we
have
used
throughout
our
tech
stack,
but
that
kind
of
environment
doesn't
actually
frost
a
lot
of
things
which
we
want
to
take
it
to
production
as
a
business
context,
and
you
cannot
innovate
that
quicker
and
whatever
the
knowledge
engineers
gains
right.
That
is
bit
more
vendor
specific
knowledge
and
it
stays
with
brands
of
few
people
and
that's
not
what
engineers
actually
want
to
do.
A
That's
where
we
made
a
declaration
to
adopt
open
source
as
our
primary
way
of
doing
software
development,
and
this
kind
of
standardization
actually
helps
to
propel
your
journey
for
long
term,
right,
open
source
being
the
standard
way
of
developing
the
software
actually
imposed
engineers
and
whatever
the
skills
they
acquire.
It's
not
more
center-specific.
A
Let's
say
some
incident
happens,
and
can
you
imagine
the
time
in
which
we
respond
to
that
kind
of
things
right?
Those
kind
of
issues
are
addressing
the
business
products,
all
those
things
in
legacy.
Data
center
perspective
like
it
it
takes
days
right
there.
There
is
a
rough
estimator
in
those
environments
it
actually
took
months
to
deliver
a
business
feature
to
production,
though
these
things
were,
we
were
discussing
within
ourselves
in
our
organization,
but
in
2015
we
made
a
public
declaration
that
we
want
to
go
all
in
cloud
by
2020
and
last
year
in
2020.
A
A
You
don't
need
to
invest
your
resources
for
doing
a
lot
of
things,
which
are
very
much
specific
for
maybe
on-prem
or
having
something
in
the
cloud.
You
cannot
achieve
that
standardization
crossbow.
What
we
are
seeing
from
making
the
declaration
adopting
cloud
as
our
primary
infrastructure
component
is
that
standardization
is
propelling
our
journey
into
the
tech
space
and
we
are
able
to
see
lot
more
negative
things
coming
out
of
this,
because
we
are
not
spending
our
engineering
capacity
and
engineering,
smart
brains
in
doing
a
lot
of
routine
mundane
work
right.
A
A
And
this
is
not
a
real
depiction
of
all
the
things,
but
this
gives
an
idea
that
we
have
really
invested
in
making
developers
life
easy
right.
We
don't
want
to
do
it
anything
which
is
a
old
legacy
way
of
dealing
things,
and
instead
we
have
heavily
invested
in
lot
more
standard
engineering
tools
which
actually
enables
them
to
deliver
things
faster
in
more
innovative
ways,
primarily
whatever
they
deal
with
doing
things
right
they
they
are
not
doing
anything
on
proprietary
vendor
specific
commercial
things.
A
Now
those
just
gone
primarily
on
standard
engineering
tools,
because
whatever
they
do,
it's
a
standard
across
board
right
what
they
are
using
here.
It's
it's
used
widely
across
the
industry,
and
this
is
what
actually
enabled
our
engineers
to
deliver
things
pretty
fast,
and
there
are
a
lot
see
there
may
be
a
lot
of
companies
which
probably
might
have
been
a
part
of
this
journey,
or
maybe
thinking
to
go
through
this
journey
of
primarily
doing
all
these
things
in
an
automated
way.
Cscd
enabled
right.
A
This
is
just
how
we
taught-
and
this
is
just
an
example
of
how
we
have
for
this
journey
right.
There
are
various
foundational
capabilities
the
investor
companies
may
be
doing
one
of
these
two
of
these,
but
all
these
things
actually
have
to
come
together
and
do
things
cohesively
right
adopting
you
build.
It
develops
culture
along
with
agile
methodology
and
doing
things
with
the
right
engineering
tools
on
top
of
open
source
software
in
cloud
right.
A
Now,
let's
see
what
are
the
ways
in
which
we
have
actually
started
doing
things
in
our
initial
days
of
adopting
all
these
foundational
capabilities,
and
we
started
doing
things
in
case
in
a
specific
way
as
any
other
nascent
technology.
And
then
there
is
things
happening
and
you
probably
may
not
completely
get
hey.
What
is
the
full
potential
of
those
things
right?
That's
how
we
were
initially
doing
things
and
when
we
started
our
journey
into
the
cacd
space,
hey
and
jenkins
is
our
primary
ccd
tool
for
doing
all
these
things.
A
The
screenshots-
you
probably
may
be
able
to
recognize
that
and
how
we
have
dealt
is
having
most
of
this
capability.
What
we
need,
those
were
embedded
in
those
specific
jobs
or
pipelines
as
groovy
scripts,
and
you
can
see
the
screens
are
attached
and
we
have
actually
gone
through
so
far
in
our
journey,
and
I
really
couldn't
pick
up
any
of
our
real
groovy
scripts,
which
we
have
used
years
ago
when
we
started
this
journey,
and
these
are
just
the
samples
what
we
have
used.
You
can
see
the
primary
issue
with
that
approach
is
scalability.
A
Let's
say
you
have
defined
some
specific
security
scanning
tool
or
maybe
a
unit
test
or
a
way
in
which
you
build
your
component.
All
these
things.
For
some
reason,
you
want
to
change
right
from
a
to
b,
making
that
kind
of
simple
adoption
is
pretty
hard
with
this
kind
of
custom
scripting,
because
the
impact
is
huge
right.
Those
were
the
initial
things
how
we
dealt
in
years
ago
and
those
were
kind
of
missteps
we
did
and
from
there
we
adopted
a
different
path
and
to
take
us
through
the
journey.
B
Hey
thanks,
google,
that
was
a
good
build
up
of
the
story
to
continue
with
the
story
of
how
capital
one
has
evolved
in
this
city
journey.
As
gokul
mentioned,
the
capital
one
started
developing
what
we
call
it
as
a
pipeline
framework
right
so
just
to
give
you
some
context.
The
pipeline
is
where
you
have
multiple
stages
to
basically
build
your
code
to
do
the
scanning
to
do
some
testing
and
also
to
deploy
them.
B
So,
as
you
can
imagine,
kepler-1
being
such
a
big
company,
we
had
multiple
lines
of
business
and
the
technology
divisions
within
each
of
these
lines
of
business.
They
started
developing
a
pipelines
so
in
in
no
time
we
started
to
have
multiple
pipelines
belonging
to
these
lines
of
business,
but
each
of
them
pretty
much
doing
the
same
thing.
B
While
it
may
not
be
exactly
the
same,
they
had
a
lot
of
common
functionalities
and
there
was
not
much
standardization
that
happened
across
these
pipelines,
and
most
of
these
frameworks
were
also
addressing
use
cases
specific
to
the
either
business.
It
was
not
kind
of
generic
enough,
and
in
order
to
be
able
to
deploy
the
code
all
the
way
into
production,
we
need
that
kind
of
a
stages
to
be
well
defined
to
say,
hey.
B
So
this
is
where
we
started
getting
into
the
the
standardization
of
so
this.
This
is
the
phase
where
it
was
all
about
standardization
and
automation
as
the
company.
It
was
very
clearly
realized
that
hey
we
need
to
have
one
pipeline,
and
that
is
where
we
started
going
towards.
The
directions
came
from
right
from
top
down
to
say:
everyone
needs
to
now
adopt
a
single
framework.
This
this
particular
framework,
was
an
internally
built
framework
on
top
of
the
jenkins
2..
B
The
goal
was
to
make
it
as
easy
as
possible
for
the
developers,
because
this
pipeline
was
not
only
used
to
deploy
the
code.
It
was
also
used
to
deploy
our
infrastructure.
Basically,
you
could
pretty
much
think
of
anything
that
needs
to
be
deployed
whether
it
is
an
infrastructure
or
it
is
a
pop
code.
It
will
go
through
the
pipeline.
B
You
know
we
kind
of
call
it
as
the
one
pipeline,
so
the
effort
started
going
in
towards
standardized
thing:
hey
if,
if
teacher
happened
through
this
pipeline,
what
are
all
the
stages
that
need
to
be
defined?
That
is,
that
is
where
we
had.
We
started
having
like
what
we
called
as
a
pre-approved
release.
If
a
code
needs
to
be
released
through
this
particular
pipeline,
we
had
around
almost
16
stages.
B
It
includes
standardizing
how
we
do
the
scanning,
what
are
all
the
tools
that
needs
to
be
used
for
scanning?
What
are
the
levels
of
vulnerabilities
that
we
need
to
fix
before
a
code
can
be
deployed
abroad
depending
on
the
criticality
of
the
application?
What
kind
of
testing
it
should
happen?
You
know
whether
it
is
a
functional
testing
or
a
performance
testing?
What
kind
of
testing
needs
to
happen?
How
do
we
capture
the
results
and
also
what
kind
of
approvals
need
to
be
built
into
the
process
and,
finally,
how
the
code
can
be
deployed?
B
So
all
these
stages
were
clearly
defined.
It
was
all
standardized
and
we
started
moving
towards
this
one
framework.
While
you
know
this
process,
it
facilitated
the
push
button
deployments.
It
also
ensured
that
the
changes
are
reviewed,
approved
and
are
free
of
vulnerabilities
tested
for
functionality
performance.
So
basically,
we
were
able
to
start
doing
a
fearless
relief
into
production
at
any
point
of
time,
as
we
started
getting
into
this,
you
can
go
to
the
next
side.
Please,
google.
B
Basically,
then,
all
that
engineers
have
to
do
is
to
provide
some
declarations
to
say
this
is
the
name
of
my
lambda.
This
is
what
these
are
all
the
configurations
I
need.
Similarly,
that
goes
into
any
other
aws
interaction
component
that
they
needed
to
build,
although
all
those
things
contributed
towards
you
know
not
reinventing
the
wheel,
and
this
also
gave
us
a
lot
of
time
back
to
our
engineers
to
focus
on
the
business
intent
because
all
the
standardization
it
provided
all
the
other
components
that
are
required
as
part
of
the
pipeline.
B
Also,
as
part
of
this
pipeline,
it
started
supporting
multiple
deployment
strategies,
as
you
can
imagine,
when
we
deploy
the
critical
applications
into
production,
particularly
the
application
that
takes
some
real-time
traffic.
We
need
to
absolutely
ensure
that
the
code
that
is
being
deployed
works
in
production
environment.
B
You
know
we
call
it
as
a
we
talked.
We
talked
about
some
of
the
pillars
plus.
We
also
have
this
a
platform
called
this
one
pipeline.
Now
we
are
building
further
on
top
of
it.
One
of
the
key
things
that
we
are
looking
is
a
shifting
lift
as
we
as
we
are
doing
rapid
releases.
We
want
to
ensure
that
we
identify
any
issues
or
vulnerabilities
as
early
as
possible
in
the
development
process,
so
this
is.
B
This
is
where
we
have
started
using
of
the
tools
for
both
the
open
source
scanning
ethol
electrostatic
scanning,
which
will
identify
vulnerabilities
very
early
in
the
development
process,
and
it
also
integrates
with
the
git.
So
the
engineer
who
is
developing
the
code,
they
will
get
to
know
what
are
all
the
vulnerabilities
they
need
to
fix
as
part
of
the
process
and
not
have
to
wait
until
it
gets
into
qa
or
further
down
the
line.
B
Similarly,
we
are
also
putting
a
huge
focus
on
the
peer
review,
which
is
also
one
of
the
stage
that
goes
into
this
ci3d
process.
We
are
encouraging
the
peer
reviews
to
be
as
effective
as
possible
again
in
the
interest
of
identifying
issues
early
in
the
process
we
are
also
working
on.
This
is
something
that
has
been
already
rolled
out
and
we
are
up
in
the
process
of
having
a
lot
of
applications
to
adopt
this.
B
B
B
This
is
also
give
more
control
for
the
enterprise
to
say
that
hey
these
are
all
the
no
one
can
bypass
any
of
the
stages
that
are
needed
and
they
also
establish
the
security
standards
that
are
needed
as
part
of
the
pipeline
kepler.
One
is
also
in
the
process
of
developing
more
and
more
reusable
gears
that
will
go
into
our
library
again.
B
One
of
the
things
that
we
are
looking
at
part
of
the
future
is
a
canary
deployment,
while
blue
green
deployment
is
great,
it
does
have
own
cons
so
with
canary
deployment.
What
we
are
going
to
be
able
to
accomplish
is
that
in
production
we
will
be
able
to
adjust
the
amount
of
traffic
that
goes
into
the
new
code
and
kind
of
slowly
increase
it.
B
B
The
key
things
are
to
be
able
to
use
the
right
process
and
the
combination
of
right
process
and
the
right
tools.
That
is
how
kaplan
has
been
able
to
come
so
far
and
yeah.
We
are
continuing
to
evolve,
but
we
have
already
started
seeing
the
the
fruits
of
all
these
hard
works
that
has
gone
in
the
combination
of
all
the
engineers,
engineers
in
engineering,
innovation,
plus
all
the
tech
leadership.