►
From YouTube: Software Quality in the Age of DevOps - November 2021
Description
With DevOps transforming the way we work with a focus on speed to market, how do we ensure the right level of quality in the system? In this meetup we will be joined by Rosalind Radcliffe, IBM Distinguished Engineer, Chief Architect for DevOps for Enterprise Systems. She will discuss some of the lessons learned and experiences on improving overall quality while achieving speed of delivery. She will use case studies, including existing backend systems running on IBM Z, that she has included in her new book “Enterprise Bug Busting – From testing through CI/CD” to deliver business results. For more information on Rosalind visit her LinkedIn page https://www.linkedin.com/in/rosalind-radcliffe/.
A
A
There
are
what
kinds
of
testing
is
out
there.
There
are
so
many
different
ways
of
testing.
There's
functional
testing,
there's
load,
balance,
testing,
there's
all
kinds
of
different
kinds
of
testing.
So
I,
when
I
met
rosalind,
I
and
I
heard
that
she
had
a
new
book
out.
I
actually
reached
out
to
her
shamelessly
on
linkedin
and
asked
her
to
to
do
a
presentation.
A
Rosalind
is
an
ibm
distinguished
engineer
in
the
area
of
devops,
including
testing.
She
has
a
strong
background
in
testing.
She
recently
published
a
book
called
bug,
busting
from
testing
through
ci
cd
to
deliver
business
results,
which
I
think
is
a
fabulous
title.
So
she
is
going
to
be
enlightening
us
today
with
everything
that
she
has
learned
over
the
course
of
her
career
in
the
topic
of
testing
and
I'm
going
to
add
one
more
person
to
the
discussion.
A
B
Thank
you
yeah.
I
I
want
to
talk
about
testing,
though
saying
all
the
testing
through
my
34
years
in
ibm,
maybe
I'll
skip
some
of
some
of
the
some
of
all
of
what
I've
done
in
testing,
but
I
think
it
is
important
to
think
about
testing
in
today's
world
and
what's
going
on
in
today's
world,
because
with
all
of
the
devops
transformation
and
with
all
of
the
things
that
have
happened
recently,
it's
really
important
to
recognize
that
software
quality
is
definitely
more
than
testing
software
quality.
Really
is
an
enterprise
level
consideration.
B
It's
everything.
It's
part
of
the
entire
enterprise
and
everyone
has
to
play
the
appropriate
part
in
that
we
can't.
You
know
we
can't
just
test.
We
can't
we
have
to
pay
attention
from
software
quality
from
the
very
beginning,
and
I
am
going
to
see
if
this
wonderful
system
wants
to
behave
today
and
let
me
share.
B
I
think
I
think
that
shows,
though
it
probably
shows
slack
in
the
background,
with
a
miserable
set
of
things,
but
that's
okay.
When
we
think
about
software
quality,
we
really
have
to
think
about
this
overall
picture,
and
it
really
is
everyone,
and
I
used
to
use
this
picture
to
talk
about
user
experience
when
I
was
early
in
my
career
talking
about
user
experience
from
the
standpoint
of
what
the
user
experiences
is
not
just
on
the
glass,
but
it's
everything
underneath
and
I
thought
when
thinking
about
it
from
a
quality
perspective.
B
I
think
that's
just
as
true
and
that
we
really
do
have
to
think
about
this
from
the
overall
perspective
and
when
we
think
about
it
and
we
think
about
what
we've
done
in
the
devops
transformation
to
try
and
break
down
the
silos
to
try
and
bring
people
together.
When
we
talk
about
full
stack
engineers,
we
end
up
not
necessarily
focusing
on
the
value
that
each
individual
might
be
able
to
bring
and
focus
on
the
fact
that
it
would
really
be
good
to
have
full
stack
teams.
B
B
B
C
Unmute
yourselves
I'll,
just
ask
the
question
on
your
job
transition,
which
I'm
curious
about.
What
do
you
think
you're
gonna
take
well.
First
of
all,
where
do
you
see
ibm
kind
of
on
the
continuum
of
of
software
quality
maturity
internally
relative
to
external,
or
maybe
it's
not
a
fair
question,
but
I'm
curious,
given
it's
a
it's
a
it's
a
notable
shift
you're
making
from
externally
facing
to
internal.
B
So
I
have
been
responsible
for
building
product
in
ibm
as
well
and
so
building
our
product
capability,
as
well
as
working
with
clients,
and
now
I'm
moving
to
our
internal
processes.
Ibm
is
a
company
as
big
as
any
other
company.
I
think
some
areas
of
ibm's
internal
development
process
are
ahead
of
other
areas
that
I've
seen
in
the
industry
and
some
areas
are
not
that's
the
polite
way
of
putting
it.
I
think
that's
the
best
way
of
putting
it.
B
So
I
I
think
we
have
everything-
and
I
think
ibm
is
representative
of
many
large
enterprises
in
the
fact
we
have
everything
we
have
this
combination
of
things.
We
have
we've
gone
through
all
of
the
phases
that
everyone
else
goes
through.
You
know
you
were
waterfall
and
then
you
went
agile
and
you
dispersed
things
and
you
had
all
the
teams
doing
what
they
wanted
to
do
and
then
well.
Security
matters
a
lot
and
and
actually
in
ibm
security
by
design
security
and
privacy
by
design
has
mattered
a
lot
for
a
long
time.
B
But
how
do
you
enforce
that?
How
do
you
ensure
that?
And
so
I
that's
one
of
the
reasons
we
have
the
the
new
role
for
me
to
come
into
to
help
drive
this
dev
sackops
model
across
all
the
teams
to
help
provide
the
right
support
for
all
the
teams,
give
them
the
right
flexibility
so
that
they
can
deliver
value,
but
also
provide
some
centralized
capability
to
make
it
simpler
for
them
it
is
a.
It
is
a
big
change,
though,.
B
B
One
of
the
things
on
this
chart
is
metrics
and
that's
actually
one
of
the
things
I
like
to
talk
about
when
I
talk
about
software
quality,
and
I
I
like
hearing
stories
from
others,
metrics
drive,
behavior
or
metrics
can
drive
behavior
and
they
drive
whether
or
not
you're
going
to
end
up
producing
quality.
They
can
help
you
produce
higher
quality
or
they
can
help
cause
bad
behavior
and
I've
seen
many
times,
metrics
used
to
help
improve
things
like
I've
got
to
have
a
hundred
percent
code
coverage.
B
B
I
mean
the
dora.
Metrics
are
a
set
of
metrics
that
are
very
valuable,
but
then
there
are
others
that
you
may
want
to
measure
at
a
team
level.
You
might
want
to
help
the
teams
understand
where
they're
going
and
100
code
coverage
is
probably
one
of
the
examples
of
one
that
I
think
has
caused
more
trouble
than
value,
but
improving
code
coverage
has
been
a
positive
in
the
metrics
measurement.
C
Yeah
now
just
this
comment
rosalind
relative
to
what
you
said.
I
think
it's
interesting
metrics,
both
from
a
what
are
you
discovering
and
what's
being
identified
as
a
result
of
testing
activities.
C
Those
metrics
also
extend
externally
to
users
of
an
application
right,
whether
that's
an
internal
or
external
application,
b2b
b2c
and
it's
finding
one
of
the
things
that
we
think
about
at
tesleo
is
is
how
to
strike
the
right
balance
between
those
two
things.
As
we
work
with
our
clients,
is
the
balancing
act
between
the
internal
metrics
of
how
are
you
doing
from
a
qa
process
perspective
with
externally
the
users
of
the
app?
And
how
do
you
take
those
inputs
and
those
data
points
as
well.
B
That
that
feedback
from
a
user
on
what
they
perceive
of
the
system
and
how
they
can
interact
with
the
system,
I
think,
is
one
of
the
most
important
metrics.
Actually
in
this
you
know
I
can
have
zero
defects,
though
okay,
I
can't
actually
have
zero
defects
in
code.
Let's
that's
not
realistic,
but
if
the
users
think
the
experience
is
lousy,
it
just
doesn't
matter.
C
Yeah
yeah,
we
see
that
we
see
that,
where
folks
make
the
false
assumption
that,
if
you
have
internal
metrics
without
a
balancing
act,
some
of
it
is
stakeholder
driven
right,
which
is
certain
parts
of
an
organization,
will
care
more
about
the
external
metrics
versus
the
internal
one.
So
thanks
for
confirming
that.
B
B
Sorry,
there
is
just
no
perfect
code,
but
you
can
write
better
code,
especially
if
you
have
the
right
tests
associated
with
that
to
validate
that
you're
actually
doing
what
you
intend.
So
I
do
think
there
are
some
metrics
about
improving
unit
testing
or
improving
coverage
or
improving
functional
coverage
or
the
areas
that
are
business
critical
coverage,
the
key
transactions.
I
do
think
those
kind
of
metrics
help
you,
but
that
last
top
bit
of
the
pyramid.
That
should
be
the
smallest
amount
of
testing
at
least
early
testing.
A
B
Or
what
their
experience
is
and-
and
it
needs
to
be-
and
it
depends
on
the
system-
some
systems
have
to
be
simple
and
easy
to
use
for
children
to
grandparents
and
some
systems
oops
have
to
be
set
up
so
that
they,
you
know
they're.
I
they're
used
by
a
particular
class
of
user.
There
are
applications
that
are
designed
for
specific
agents
or
specific
environments,
and
they
have
a
different
kind
of
expectation
of
experience
than
the
the
end
user
experience.
B
Other
area
of
manual
testing
that
I
find
absolutely
wonderful,
is
actually
malicious
testing,
I.e,
really
trying
to
get
those
people
who
are
really
really
good
at
finding
problems
to
actually
go
at
it
in
a
free-form
way
and
and
try
and
break
the
system
or
try
and
find
an
exposure
from
a
security
standpoint
or
try
to
see
that
it
just
doesn't
work.
The
way
that
you
need
it
to
that
kind
of
malicious
testing
is
the
other
thing
that
I
think
is
very
helpful
from
a
manual
standpoint,
but
if
possible,
everything
else
is
automated.
A
So
just
a
comment
rosalind.
The
continuous
delivery
foundation
is
reworking
the
cdf
landscape.
It's
a
great
project
because
it's
overdue,
since
the
pipeline
is
changing
and
one
of
the
areas
is
automated
testing.
So
this
I
think
this
pyramid
gives
a
there
were
some
questions
about
how
testing
is
kind
of
segmented
in
the
different
tools
in
the
space.
So
I'm
just
you
know
putting
a
little
bug
in
your
ear
to
say:
hey
if
it's
something
you'd
like
to
participate
in
to
help
build
out
what
that
testing
landscape
looks
like
we
could
use
you.
B
B
So
testing
is
there
to
verify
that
you
have
what
you
expect.
You
have
to
have
built
quality
from
the
beginning.
I
like
unit
early
testing
allowing
the
developer
to
get
as
fast
feedback,
because
it
helps
them
think
it
helps
them
think
about
the
problem.
It
helps
them
focus
on
the
issue
and
so
by
getting
this
early
testing
done,
helping
them
make
sure
they're,
they're
thinking
about
all
the
scenarios
and
reminding
them
it
is
by
design
you're
trying
to
design
in
quality.
Have
you
put
in
the
system
enough
for
reliability?
Characteristics?
B
Have
you
put
in
enough
from
a
logging
standpoint
or
an
understanding
so
that
you
actually
can
find
the
problem
at
2?
Am
I
I
actually
think
one
of
the
biggest
one
of
the
most
fun
changes
in
the
new
product
align
teams
and
getting
rid
of
separate
teams
per
se,
but
having
the
product
teams?
This
idea
of
developers
having
to
answer
the
page
in
the
middle
of
the
night.
B
It
changes
the
way
you
write
code,
no
matter
what
anybody
says,
you
you,
you
don't
write
code,
the
same
way
if
you're
going
to
be
responsible
for
it
because
you
are
going
to
want
to
have
to
you,
don't
want
to
have
to
get
up
at
2am
and
figure
out
what's
wrong,
so
you
put
in
more
logging,
you
pay
attention
to
more
things
and
and
that
that
change
and
thought
process
is
also
really
important
for
application
developers.
So
they
understand
the
impact
of
what
they're
building.
B
And
it's
not
just
accountability.
I
know
teams
have
been
accountable,
but
they
didn't
have
to
get
up
in
the
middle
of
the
night
that
that
that
twist
has
made
well.
I
know.
B
That's
a
different
discussion.
The
other
thing
that
I
think
is
really
important
about
this
pyramid
is
the
system
performance
scalability
testing,
that
when
I
talk
about
testing
and
testing
in
the
enterprise,
one
of
the
things
that
I
think
is
really
important
and
cuts
gets
me
into
trouble
every
time
I
say
it,
so
I'm
gonna
say
it.
Is
you
can't
test
in
production.
B
Okay,
I
haven't
gotten
in
trouble,
yet
you
can't
test
in
production,
I.e,
you
have
to
actually
make
sure
the
code
works
before
you
get
it
into
production,
you're,
not
testing
for
defects
in
production,
you're,
testing,
user
experience
through
a
b
testing
or
scenarios
or
things
you
want
to
know.
If
it
helps
the
user
marketing
campaigns.
Those
kinds
of
things
not.
B
D
B
Okay,
I'm
not
sure
I
moved
it
a
little,
but
okay,
the
the
idea
of
performance
and
scalability
testing
and
in
an
enterprise.
There
are
a
few
comments
that
that
cause
problems,
things
like
fail
fast
and
test
and
production
and
the
the
reason
in
some
cases
that
happens,
especially
I'm
going
to
pick
on
financial
institutions.
B
You
can't
afford
to
be
wrong
and,
more
importantly,
I
don't
want
them
to
be
wrong.
My
bank
account
balance
shouldn't
get
screwed
up.
I
shouldn't
have
two
transactions
two
subtractions
go
through.
I
always
joke.
I
wouldn't
mind
if
you
give
me
my
paycheck
twice,
but
somebody
else
is
gonna
mind.
So
that
can't
happen.
The
point
is
they're
placed
they're
things.
B
You
can't
test
in
production,
they're
external
scenarios
that
you
can
but
performance
and
scalability
testing
is
one
thing
that
many
people
seem
to
try
and
do
in
production
and
that's
what
causes
a
lot
of
failures
and
problems.
So
the
more
you
can
figure
out
how
to
do
that
performance
testing
early
on
in
the
development
effort,
even
long
before
the
code
is
fully
complete.
Even
when
you've
just
started
it
making
sure
you
can
actually
get
some
kind
of
performance
or
scalability
testing,
you
get
an
understanding
how
it's
going
to
perform.
B
So
you
know
what
you
need
to
do
with
the
application.
Also
understanding
what
are
the
qualities
of
service
for
an
application
if
I'm
building
an
application
that
has
to
you
know,
maybe
work
or
it's
acceptable.
If
there's
an
error,
because
it's
you
know
it's
a
temporary
problem,
you
come
back
and
you
know
you
refresh
the
screen
and
everything's
fine
and
there's
no
issue,
then
that's
one
level
of
quality.
B
However,
if
you're
writing
a
system
like
a
banking
system
or
something
where
you
don't
have
that
choice,
then
there's
a
different
criteria
for
quality
that
you
need
to
deliver
and
banking
system
is
just
one
example.
I
I'll
take
it
to
an
extreme
the
pacemakers
in
people's
hearts.
The
software
there
better
be
really
good
or
the
self-driving
cars
that
we're
trying
to
make.
B
So
software
quality
has
different
ramifications
in
in
different
environments,
and
we
need
to
take
that
into
consideration
as
we're
building
as
we're
considering
the
criteria
for
the
software
and
how
we're
going
to
test
the
software.
You
brought
up
the
comment
about
different
kinds
of
testing
and
lots
of
different
test
wear
with
all
of
these
kinds
of
systems.
B
A
You
might
have
to
float
down
and
there
should
be
a
little
kind
of
grayed
out
arrow.
I
noticed
sometimes
in
this
my
keyboard
doesn't
work.
B
B
The
point
on
the
chart
really
is
about
how
development
and
how
the
new
modern,
modern
development
practices
and
I'm
putting
it
in
quotes,
because
this
is
not
so
modern
anymore.
The
idea
of
being
able
to
have
this
independent
development
and
getting
that
fast
feedback
is
actually
one
of
the
things
that
should
be
improving
our
overall
quality.
B
But
this
says
everybody
really
has
to
be
working
together
as
part
of
this
process.
Everybody
has
to
be
working
together
to
allow
this
to
flow
quickly,
because
we
really
really
want
a
change.
That's
coming
in
into
the
system
to
be
able
to
be
deployed
into
that
production
environment
as
quickly
as
possible.
So
I
can't
afford
a
time
delay
of
people
waiting
for
a
cue
and,
and
actually
I
think
it
enterprises
this
change
to
not
have
tickets
to
get
someone
else
to
do
something
for
you
in
in
many
places.
B
B
The
only
what
we
have
and
the
reason
everyone
becomes,
a
developer
is
they're
all
writing
code
of
some
form
or
they're
writing
something.
That's
checked
into
an
scm
that
then
flows
into
an
environment,
and
so,
if
everyone
is
either
writing
business
logic
or
they're,
writing
test
cases
or
writing
infrastructure
code.
If
everybody
working
in
the
same
way,
they
can
collaborate
more
effectively
and
work
together.
A
B
I
I
I
have
to
admit.
I
did
a
panel
recently
on
on
testing
and
made
that
statement
and
we
went
down
a
a
long
discussion
and
but
I
do
think
it
is
important,
and
I
I
think
there
are
a
couple
of
reasons
for
that
importance,
one.
If
we
can
really
have
a
common
way
of
working
and
a
common
way
of
understanding,
then
we
can
deliver
better
value.
If
we
can
work
together
to
collaborate
to
deliver
the
value,
then
we
work
together
better.
B
B
It's
the
developer's
fault
that
something
went
wrong.
It's
the
developer
and
they
would
always
say
it's.
You
know
something
else
or
it's
always
the
network.
You
take
your
choice.
You're,
going
to
blame
the
easiest
person,
really
what
you're
doing
is
you're
blaming
the
person
that's
hardest
for
them
to
prove
it's,
not
their
fault
in
the
new
world.
B
One
of
the
one
of
the
unfortunate
stories
that
I
I
like
to
tell
about
why
automation
really
matters,
and
I
can
tell
the
story
because
it
hit
the
news
press.
Otherwise
I
never
would
be
able
to.
There
was
a
a
financial
institution
and
I
still
it
a
little
so
it
doesn't.
They
don't
get
too
upset
with
me
that
had
a
change
go
into
production
and
it
was
a
simple
code
change
that
they
made
in
production
and
and
pushed
it,
and
they
all
of
a
sudden
couldn't
balance
any
account
balance.
B
B
B
That
was
a
change
in
production.
Obviously
it
wasn't
tested,
but
we're
going
to
ignore
that
problem.
It
was
a
user
who
typed
something
into
a
file
into
production
every
time.
I
think
about
that,
and
that
was
long
enough
ago,
that
you
could
look
it
up,
but
it's
pretty
old
press.
If
you
look
at
that
scenario,
you
think
about
that's
a
really
obvious
reason
why
I
want
to
make
sure
everything's
automated.
B
One
of
the
other
things
that
I
think
drives
software
quality
and
I
can
cause
another
problem-
is
the
actual
code
review
stage
the
thing?
No
one
ever
likes
that
I
have
to
review
someone
else's
code
or
I
have
to
have
someone
else
review
my
code
that
that
phase
of
code
review
or
collaboration
over
the
code,
helping
make
sure
that
you
have
four
eyes
on
the
code,
actually
helps
improve
the
code
and
helps
people
learn
and
helps.
People
evolve
in
understanding
the
code
and
deliver
better
quality.
B
B
A
B
Many
times
the
piece
of
code
you're
reviewing
is
a
small
change
anyway,
so
it
may
not
be
a
big
piece
of
code.
That's
reviewing.
Sometimes
I
can
remember
it
was
very
well
in
the
old
days.
It
was
probably
a
really
large
change,
because
you
do
all
the
coding
first
make
sure
it
all
works,
and
then
you
turn
it
over
to
code
review
right.
You
remember
those
days.
C
Big
big
shifts,
big.
You
know
big
blocks
of
stuff
with
lots
of
stuff
that
could
go
wrong
along
the
way.
B
B
B
Separate
calls
so
designing
the
services
in
the
first
place,
so
they're,
effective
and
efficient
is
another
way
to
help
improve
the
overall
quality
of
the
system
and
truly
ensure
this
ability
to
deploy
independently,
because
one
of
the
big
values
of
microservices
should
be.
I've
got
independent
units.
B
If
we,
if
we
think
about
this
testing
process
and
we
think
about
the
pipeline,
one
of
the
other
key
aspects
in
testing
is
test
data.
And
how
do
you
get
the
right
test
data?
How
do
you
have
the
right
data
available
to
work
in
an
environment
and
testing
at
a
unit
test
level?
I
need
the
right
test
data.
That's
the
error
conditions,
the
boundary
conditions,
yeah,
boundary
conditions.
Why
do
people
forget
to
test
boundary
conditions
all
of
the
that
kind
of
data?
So
I
can
really
test
and
then
having
the
right
data
throughout
the
process.
B
One
of
the
things
I
always
say
is:
production
data
is
lousy,
test
data
because
it's
good
data
by
definition.
Production
data
is
good
because
it
got
into
the
system
it
actually
made
it
through
the
processing
and
therefore
it's
good.
It
might
not
be
really
good,
but
it's
good.
It
exists
it's
in
the
system,
so
we
want
to
look
at
getting
our
data
for
early
testing,
not
from
production
data,
but
through
fabrication
methods
or
through
ways
of
guaranteeing
your
data
that
could
never
get
into
the
system.
B
The
one
thing
I
always
find
a
challenge
with
testing
tools
is
they're
really
good
at
validating
the
data
for
you.
So
if
the
data
field
is
supposed
to
be
alpha,
numeric
you
can
all
you
can
put
in
if
it's
supposed
to
be
numeric,
the
testing
tool
may
stop
you
from
putting
non-numeric
data
into
the
field
to
run
for
your
test.
B
In
my
opinion,
the
testing
tool
needs
to
allow
me
to
do
whatever
I
should
be
able
to
put
in
the
strangest
characters
in
order
to
cause
a
problem
so
that
I
can
ensure
I'm
actually
testing
all
the
error
scenarios
that
could
get
in
to
the
application
and
making
an
assumption
that
somebody
before
you
is
going
to
test
the
data
is
one
of
the
key
critical
areas.
I've
seen
people
cause
problems
with
in
their
testing
because
they
didn't
assume
they
could
get
bad
data
and
all
of
a
sudden.
B
I
was
working
with
a
client
and
and
happened
to
discover
in
their
test
data.
My
name,
my
social
security
number.
My
address
my
real
data
in
the
test
environment.
B
Yeah
it,
I
don't
think
it
would
happen.
Well,
okay.
I
hope
it
wouldn't
happen
again,
but
that
that
is
a
challenge.
I
like
no,
thank
goodness
when
we
think
about
the
other
thing
about
production
data.
That
is
a
problem.
We're
considering
testing
is
production.
Data
is
really
good
for
large-scale
scalability
testing
and
I
actually
think
a
pre-production
environment
with
production
data
in
it
to
run
large-scale
testing
is
a
really
good
idea,
but
in
early
stages,
why
do
we
need
to
run
a
thousand
records
that
have
the
same
result?
B
What
value
in
our
testing?
Doesn't
that
slo
it?
Well?
It
slows
down
the
testing.
It
makes
it
slower
to
get
the
results
to
see
the
the
positive
or
failure
results
for
no
value
reason.
So
we
need
to
make
sure
that
our
data,
the
data
we're
running
through
the
system,
is
going
to
turn
return
value.
We
actually
are
testing
the
right
things.
B
One
of
the
things
that
I
found
in
going
through
environments
is
a
lot
of
organizations
that
have
started
down
the
automated
testing
path.
They
just
create
more
tests
every
time.
I
do
something
I
create
a
new
test
and
I
put
it
in
the
test
bucket
and
so
the
test
bucket
just
gets
larger
and
larger
and
larger
without
cleaning
or
without
managing
the
test
bucket
and
in
reading
gary
gruver's
book
engineering,
the
digital
transformation.
B
B
You're
getting
the
failure
as
soon
as
possible,
now
later
stage
tests
or
are
tests
that
are
less
related
to
this
change
are
still
useful
to
run
because
you
could
have
affected
them,
but
they're
less
likely
to
be
affected,
so
it
it
is
valuable
to
still
run
them.
It's
not
saying
don't
run
them,
it's
just
running
them
later
in
the
test
run
so
that
I
can
get
feedback
along
the
way
and
if
anything
goes
wrong
or
majorly
wrong.
B
In
order
to
do
that,
though,
you
have
to
have
a
well-managed
testing
set
a
well-managed
set
of
tests
and
have
to
have
as
the
as
the
business
developers
are
writing
code.
The
test
developers
are
building
up
their
tasks
and
managing
the
tests
associated
with
the
code
change
so
that
you
have
everything
flowing
together.
B
One
one
question
I
get
and
one
question
I
have
is:
if
you
are
in
a
world
in
which
you
have
to
translate
your
code,
so
if
you're
somewhere
and
you
have
to
transit,
we
in
ibm,
we
have
to
translate
all
our
products.
So
we
have
to
do
language,
translation,
all
the
error
messages.
Everything
have
to
be
translated,
so
we
come
up
with
translation.
Translation
test
is
always
at
the
end.
B
Translation
test
is
always
after
everything
else
and
in
fact
it's
usually
very
well
very
near
the
end
of
the
whole
cycle
that
in
that
means,
when
you're
building
something
you
have
this
cycle
at
the
end.
That
says,
I
can't
deliver
it
until
I've
finished
doing
all
the
translation,
but
I
don't
really
want
to
translate
while
I'm
gonna
make
changes,
so
I
delay
the
release
in
order
to
get
translation
done
for
a
certain
set
of
capability.
B
B
Is
security
scanning
integrated
in
the
very
beginning
in
the
ide,
so
the
developers
can
get
that
feedback
as
they're
building
the
code
to
say
that
it's
going
to
meet
the
security
requirements
and
they
don't
have
any
exposures
or
which
I
think
is
the
right
thing,
because
the
sooner
you
get
that
security
understanding
just
like
the
sooner
I
can
write,
run
my
unit
test
just
sooner
that
I
can
get
the
feedback
the
better
off.
I
am.
B
It
helps
in
the
process
or
scanning
my
dependencies
if
I
know
the
dependency
that
I'm,
including
has
a
problem
the
sooner
in
the
cycle
that
I
can
find
that
and
address
that
the
better
off
I
am
it'll,
be
more
effective
and
more
efficient
I'll
be
able
to
deliver
the
code
in
the
end
rather
than
testing
at
the
end
and
discovering
now
I've
got
a
vulnerability.
I've
got
to
go:
rework
it
in
the
entire
system.
B
Comments,
the
and
the
last
area
that
I
really
want
to
delve
into
is
this
area
of
metrics
and
how
metrics
really
do
drive?
And
I
don't
know
how
many
on
the
phone
remember
six
sigma
on
this
webinar
I'm
seeing
some
head
shakes.
So
some
people
do
six
sigma
applied
to
code
and,
if
you
think
about
six
sigma,
I've
got
to
continuously
improve
and
if,
let's
say
your
defect
rate
is
somewhere
near
two
or
three
in
a
year,
the
the
next
the
improvement
process
would
say:
you'd
get
to
zero
defects.
B
I
haven't
gotten
anybody
raising
their
hand
and
I'm
just
gonna
assume
nobody
wants
to.
One
of
the
things
we
have
to
be
careful
with
is
how
we
measure
and
how
we
look
at
measurements
in
in
that
story.
That's
actually
in
the
book.
It
talks
about
how
we
had
multiple
development
teams
and
they
were
ibm.
Development
teams
and
one
product
team
had
a
defect
rate
that
was
much
worse
than
that,
and
so
they
could
continuously
improve
each
release.
They
could
significantly
decrease
the
number
of
defects.
B
B
We're
not
causing
the
right
behavior
we're
not
causing
the
right
attitudes
and
metrics
are
what
does
this?
In
many
cases,
the
culture
of
the
organization
and
how
we
see
people,
how
we
interact
with
people
and
how
we
work
on
things
so
making
sure
we
pick
metrics
that
that
bring
out
that
the
good
value-
and
I
you
know
I
like
the
happiness
metric.
B
You
know
that
might
sound
a
little
weird,
but
I
actually
think
if
there
are
ways
that
we
can
measure
the
happiness
of
a
team,
because
teams
that
are-
and
I
got
to
be
a
little
little
careful
in
the
just
happiness-
you're
not
going
to
be
happy
all
day.
Every
day
there
are
going
to
be
challenges.
There
are
going
to
be
problems,
but
in
overall
teams
that
are
happier
they're
continuously
learning,
they're
they're,
more
positive
in
nature.
B
Do
better
in
what
they're
building
they
build
higher
quality
code
because
they're
more
energized,
they're
more
excited
this
happiness
is
really
an
important
metric
learning
is
an
important
metric.
If
you
have
a
team,
that's
learning
and
growing
and
evolving
they're
going
to
build
better
quality
code.
Those
two
metrics
don't
directly
measure
quality,
but
they
measure
something.
That's
very
important
that
helps
drive
the
quality
into
the
system
and,
as
was
discussed
earlier,
the
business
value,
the
customer
satisfaction,
the
nps.
B
If
we
think
of
project
to
product,
I
don't
necessarily
think
that's
a
software
quality
book,
but
the
organizational
structure,
the
way
you
set
up
your
teams,
the
way
you're
delivering
value,
having
that
that
team
responsible
actually
improves
value
having
that
that
stream
aligned
team
to
make
sure
that
they
understand
the
impact
of
what
they're
doing
gary's
book
in
digital
transform
engineering,
the
digital
transformation,
it
really
does
help
understand
it.
Enterprises.
B
How
can
we
help
improve
and
the
dora
metrics
are
another
way
that
we
help
understand
how
we're
doing
overall
in
our
process-
and
you
can't
resist
the
unicorn
project-
is
just
the
most
fun.
Well,
okay,
phoenix
project
and
unicorn
project
is
fun
books
to
read,
but
having
and
learning
is
the
way
we
do
better.
So
when
I
think
about
software
quality,
one
of
the
things
that
we
try
to
push
is
people
learning
from
others,
people
having
conversations
people
having
the
discussion.
How
do
you
get
better?
B
B
Yeah
yeah,
I
should
slide
it
over
and
I
put
it
on.
I
I
put
the
unicorn
project
because
it
was
a
newer
one,
but
the
phoenix
project
was
the
first
fun
and
then
the
unicorn
project
was
second
fun
and
I
have
to
say
I
like
reading,
so
that
I'm
probably
a
bad
example,
but
I
really
loved
reading
those
books
and
I
really
hated
reading
those
books
because
I
could
feel
for
certain
people
in
the
book
and
it
was
like.
Oh
that's
just
too
real
at
times,
but
yes,
they're
great
reads:.
A
A
But
I
think
it's
interesting
the
way
they
he
they
break
down,
how
microsoft
was
delivering
code
during
that
time
from
beginning
to
end,
and
I've
always
found
it
to
be
an
interesting
read.
I
picked
it
up
a
few
years
back.
I
read
it
when
it
first
came
out
and
then
I
read
it
again
a
few
years
back,
because
I
think
they
cover
a
lot
of
topics
and
expose
a
lot
of
potential.
A
A
On
on
that
note,
we
do
it
is
at
one
o'clock,
so
we
have
ran
out
of
time
and
I'm
just
in
being
respectful
of
everybody's
time
rozan.
Thank
you
so
very
much.
I
have
to
say
that
I'm
honored
that
you
ended
my
my
my
run
here
as
the
cdf
online
meetup
host.
Thank
you
so
much
for
bringing
up
this
this
conversation.
A
I
believe
that
both
security
and
testing
will
continue
to
be
an
important
topic
in
the
in
the
world
of
continuous
delivery
and
everyone
who
attended.
Thank
you
so
much
I
this
will
be
posted
up
to
the
the
cd
foundation's
meetup
site,
probably
within
the
next
day.
So
if
you
want
to
share
this
with
any
colleagues,
you
can
find
it
there
and
on
that.
A
Thank
you
all
so
much
I'm
going
to
stop
recording
and
we
will
see
you
sometime
in
january
you'll
be
getting
a
notice
from
somebody
from
the
cd
foundation
regarding
the
schedule
for
2022..
Thank
you.