►
From YouTube: John Rzeszotarski, KeyBank, at Red Hat Summit 2017: Banking on OpenShift—a DevOps journey
Description
John Rzeszotarski, director of Continuous Delivery and Feedback at KeyBank, shares how his organization tackled the modernization of a complex online banking system consisting of different merged banks with various legacy computer systems. Learn how KeyBank used Red Hat OpenShift and a DevOps approach to successfully update their online banking platform.
Learn more: redhat.com/en/summit/2017/agenda/sessions
A
Please
welcome
teabag,
director
of
continuous
delivery
and
feedback
John
residue
nurse.
Thank
you.
Thank
you.
Yep
I'm
good,
make
sure
you
know
the
handshake
was
going
to
go.
Low,
I
was
going
to
be
there
for
it
because,
if
not
so
funny
now
we're
excited
to
be
here
to
tell
you
our
journey
a
little
bit
about
Key
Bank.
We
are
one
of
the
top
15
largest
banks
in
the
United
States
that
you've
probably
never
heard
of,
but
we're
headquartered
in
Cleveland
Ohio,
so
go
Cavs.
A
Yes,
thank
you.
I
was
waiting
for
somebody
to
clap.
Please
so
know
we're
real
excited.
We
grew
up
through
acquisition,
we're
about
a
hundred
and
ninety
years
old.
So
if
you
think
about
how
many
banks
we've
acquired
in
the
time
of
190
years,
to
become
one
of
the
top
15
largest
banks,
that's
a
lot
of
servers
and
machines
and
applications
all
coming
together
and
that's
going
to
be
a
big
part
of
the
the
story
I'm
about
to
tell
and
every
good
story
starts
with
admitting
that
you
have
a
problem.
A
One
of
our
first
issues
that
we
we
had
a
couple
years
ago
and
it
was
a
very
significant
issue.
It
took
us
down
pretty
much
the
entire
day
and
when
we
tried
to
fix
it,
we
actually
made
it
worse.
I
know
if
anybody's
actually
gone
through
something
similar,
but
it
was
really
because
we
didn't
understand
our
complexity.
I
had
mentioned
that
we
acquired
a
lot
of
banks.
Most
recently
we
acquired
First
Niagara
out
of
Buffalo.
We
executed
that
acquisition
last
year
as
part
of
that
we
got
a
new
unisys,
mainframe
I'm,
pretty
sure
saying.
A
So,
as
so,
we
put
the
we
looked
at
exactly
what
our
online
banking
flow
actually
was
and
when
we
routed
out
every
individual
step
and
when
we
did
that
we
identified
so
many
different
issues.
We
identified
the
fact
that
when
one
user
logged
in
we
actually
had
over
200
Network
hops,
we
were
bouncing
back
and
forth
between
our
two
different
data,
centers
anywhere
from
10
to
30
times,
and
it
was
just.
It
was
just
so
much
so
complex
that
we
were
like
any
time
we
try
to
fail
something
over
it.
A
Just
wasn't
working,
so
we
really
had
to
take
a
step
back
and
all
right.
Let's
go
through.
Let's
do
an
assessment,
let's
realize
where
our
issues
are
and
let's
try
to
correct
those
now
how
we
got
there
was
was
because
of
this
acquisition
because
of
these
problems-
and
you
know
what
I'm
sorry
can
you
go
back?
A
So
we
had
to
fix
all
the
network
issues
we
had
to
fix
all
of
the
system
issues
and
where
we
were
running
systems
of
record
between
two
different
data
centers.
That
was
the
easy
stuff.
The
hard
stuff
was
saying.
How
do
we
prevent
this
from
happening
again?
I
mentioned
the
metrics
as
well
a
big
answer
for
that
was
DevOps.
How
can
we
change
our
culture
to
have
more
of
a
culture
of
engineering?
A
Now
anybody
starting
a
DevOps
journey?
Don't
rework
the
wheel.
There
is
a
ton
of
material
out
there
for
you,
gene
Kim
says
humble
John
Willis.
They
all
give
you
the
answers
that
that
you
need
and
the
way
I
like
to
explain.
Devops
is
actually
talking
through
Farley's
logs,
so
law
number
one
people
believe
before
they
Dow
to
err
is
too
human.
We
all
make
mistakes
and
we're
going
to
continue
to
make
those
mistakes.
Law
number
two
things
are
more
complicated
than
you
think
we
don't
understand
the
complexity.
A
We
actually
hit
that
when
we
hit
our
big
issue,
he
has
a
third
law
as
well,
and
it's
all
stuff
is
interesting
and
I
think
it's
basically
these
first
two
laws
are
so
depressing
that
he
needs
third
law
there
to
make
himself
everybody
feel
better.
But
what
is
he?
He
tells
us,
though,
that
to
combat
this,
you
have
to
combat
these
two
laws
with
experimentation.
A
It's
all
about
we're,
know
we're
going
to
make
mistakes,
and
we
know
that
we
don't
understand
the
complexity,
so
you
know
we're
going
to
fail,
so
we
have
to
fail
fast
and
that's
exactly
what
we
wanted
to
do
and
this
directly
correlates
to
our
methodology.
We
were
a
typical
waterfall
organization,
like
so
many
other
large
enterprises.
We
assess
that
an
opportunity
we
plan
for
it.
We
develop
it.
We
made
sure
we
had
all
the
requirements,
then
we
test
it
and
we
deploy
it
and
it
went
perfectly.
A
So
if
we
look
at
Farley's
laws-
and
we
apply
that,
yes,
we
still
have
to
assess-
we
still
have
to
develop,
but
let's
automate
everything
we
can
in
between
automate
infrastructure
components,
automate
testing,
automate,
release,
automate
validation,
and
that's
that's
that's
where
you
get
these
fast
cycles
and
then
you
accept
to
reject
that
change
because
of
Farley's
laws.
We
know
that
we're
going
to
fail
because
we're
human
and
we
don't
understand
the
complexity.
A
One
of
our
tech
leads
actually
drew
this
on
the
whiteboard.
While
we
were
going
through
one
of
our
implementations-
and
it
makes
so
much
sense
because,
if
you're
a
big
snake
taking
two
big
bites,
it's
kind
of
hard
to
digest
all
that.
But
if
you
take
little
bites,
that's
much
easier,
so
we
went
out
and
a
team
of
folks
on
me.
We
built
a
pitch
for
our
cio
and
once
again
following
the
godfathers,
the
gene
Willis,
the
the
gene
Kim,
the
John
Willis
they'll,
tell
you
right
away.
A
Metrics
driven
approach,
go
tell
your
CIO
what
metrics
are
going
to
affect
for
us.
It
was
mean
time
to
resolution
and
release
frequency.
Those
were
both
going
to
be
two
big
items
we
were
going
to
hit
and
she,
when
we
sold
her
on
it,
we
had
to
narrow
down
scope.
So
here's
the
thing
like
people
that
want
to
say
hey
we're
going
to
build
a
DevOps
practice
or
we're
going
to
do
this.
We
can
do
that.
You
know
you're
not
going
to
change
the
world
overnight.
You
got
to
pick
your.
A
You
got
to
pick
your
battles
for
us
it
was
three
main
areas.
It
was
containers.
It
was
automated
testing
and
by
the
way,
if
anybody
tells
you
a
DevOps
story,
and
it
doesn't
include
automated
testing
they're,
not
telling
you
a
DevOps
story
and
the
last
piece
is
continuous
delivery.
So
you
need
that
to
kind
of
bring
all
the
blue
together.
A
A
We
were
angel
funded,
so
our
CTO
and
our
chief
architect
kind
of
opened
up
their
wallets
and
said
we're
going
to
build
a
little
mini
start
up
inside
our
bank
and
then
the
project
selection
don't
go
after
the
easy
falling
off
the
tree.
Go
after
something
that's
going
to
be
impactful
at
our
time
we
were
actually
redesigning
our
online
banking
platform.
A
We
were
really
excited
about
it,
so
the
way
that
we
put
the
team
together
and
actually
I'm
referencing
here,
something
that
Matthew
Skelton
wrote
a
blog
about
different
ways
to
introduce
DevOps
into
an
organization,
and
this
concept
is
called
DevOps
as
a
service.
So
we
basically
were
a
small
team.
We
sat
between
development
and
operations
and
really
help
to
manage
the
release
of
the
online
banking
application
as
it
was
being
built
now,
I'll
tell
you,
you
don't
want
to
be
a
new
silo.
A
A
So
our
first
scope
item
was
containers
and
everybody
makes
up
words
at
these
conferences
and
stuff,
so
mines
container,
getting
because
I
think
it's
coming
and
it's
going
to
destroy
everyone
and
it's
fantastic.
So
when
we're
building
an
application,
we
start
with
a
product.
We
require
infrastructure
so
and
as
part
of
that
first
piece
of
infrastructure,
I'm.
Sorry
there
we
go
ok.
So
as
part
of
that
we
take
that
infrastructure,
we
virtually
slice
it
in
half
and
we
have.
We
virtualized
it
so
that
we
can
get
more
bang
for
our
buck.
A
A
All
right
there
we
go
so
then
we
install
the
platform
itself
and
then
we
finally
install
the
applications
that
are
up
on
top
of
it
right
now,
we're
done
no.
Now
we
have
to
go
through
and
we
have
to
test
and
validate,
or
we
have
to
make
sure
we
have
the
right
security
model
and
the
right
operational
configuration
so
that
we
have
all
the
right
logging
in
place
and
we
have
all
the
right
alerting
in
place.
Now
we're
done
right.
A
No
now
we
have
to
test
and
we
have
to
validate
each
individual
component
from
a
dependency
perspective.
Now
we're
done
no,
we
got
to
start
all
over
again
when
we
patch
upgrade
and
hotfix
all
of
the
different
environments.
Now
each
of
these
boxes
is
typically
at
a
different
team,
so
imagine
being
the
manager
trying
to
cross
communicate
across
all
of
these
different
boxes
and
lines.
A
Containers
build
once
build
it
in
code,
move
it
around
on
immutable.
Infrastructure,
removes
responsibility
from
our
legacy
infrastructure
teams,
I
like
to
say
we
cheated,
basically
at
our
kicking
off
DevOps.
This
is
shifting
left
and
being
able
to
produce
fast
infrastructure,
that's
more
consistent
and
less
error-prone,
because
there's
no
human
involvement,
fantastic
piece
of
technology,
so
containers
are
great
by
themselves,
but
they
don't
give
me
high
availability.
A
They
don't
give
me
zero
down
deployments,
they
don't
give
me
readiness,
checks
and
all
the
goodness
that
I
need,
because
I
have
to
keep
my
application
available
as
at
the
highest
service
levels.
So
that's
where
kubernetes
comes
in
and
and
we
evaluated
several
of
the
different
platforms
that
were
out
there.
Kubernetes
was
the
winner,
you
know
by
far
and
I'm
looking
at
Raphael's
facility.
He
helped
us
make
the
decision
right
back.
A
So
yes,
so
we
now
we
had
kubernetes,
we
had
docker.
Now
the
rest
of
it
was
really
starting
to
fall
in
line.
We
focused
very
heavily
on
test
automation.
Moving
over
to
cucumber
and
our
business
analyst
started
writing
gherkin
I
mean
it
was
just.
It
was
insane
and
I
think
the
really
important
thing
to
look
at
here
was
all
of
our
project.
Teams
are
now
actually
producing
something
that's
going
into
our
automation
pipeline.
If
it's
gorkon
its
business,
you
know
they're,
writing,
they're,
writing
test
scripts
in
gherkin.
A
Your
infrastructure
is
writing
code
that
goes
into
chef.
That
builds
our
images,
our
application
developers
and
our
test
developers
are
writing
test
code,
and-
and
now
we
have
this
ability
to
produce
these
amazing
dashboards,
to
go
directly
back
to
our
development
and
project
our
project
management
teams
to
actually
tell
them
the
health
versus
somebody
managing
it
in
a
spreadsheet
in
a
PowerPoint
deck
that
says
we're,
yellow
I,
don't
know
why
we're
yellow
but
we're
yellow.
So
our
test,
automation
changes
were
extremely
successful.
A
So
what
we
called
our
test
item
a
before
would
take
1200
minutes
to
run
and
it
typically
found
zero
defects.
When
we
were
when
we
were
in
the
middle
of
doing
the
majority
of
the
development,
are
we
actually
bored
using
selenium
grid
and
we
were
running
it
out
on
open
shift
and
distributing
that
load?
And
it
was
running.
You
know
over
1200
different
tests
in
12
minutes
and
we
were
identifying
defects
ten
times
a
day
on
a
daily
basis.
So
from
test
automation
perspective,
it
was
a
huge
success.
A
This
is
the
part
where
I
like
to
beat
my
chest
a
little
bit
so
I
mentioned
that
we
acquired
First
Niagara
last
year,
so
during
the
busiest
time
of
the
day
and
the
most
logins
per
second.
During
this
acquisition,
we
had
to
make
changes
because
guess
what
we
had
some
user
experience
issues
for
the
first
Niagara
customers
that
needed
to
enroll.
It
was
a
little
complicated
for
them
and
we
needed
to
be
able
to
react
fast.