►
Description
Keynote Session: The Future of Testing - Kohsuke Kawaguchi, Launchable Inc.
The last decade has seen a relentless push to deliver software faster. Automated testing has emerged as one of the most important technologies for scaling DevOps. So much so that companies are investing enormous time and effort to build end-to-end software delivery pipelines with automated testing built-in. In this keynote, Kohsuke Kawaguchi, creator of Jenkins and co-founder of Launchable shares insights into emerging trends in the testing space. This session will highlight innovations in testing and future approaches that are emerging for those looking to nurture a continuous quality culture.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
A
A
A
I
have
to
every
change
that
I
create
actually
goes
through
at
least
one
hour
or
like
a
testing
cycle
before
they
they
go
up
for
the
code
review
and
then
whatever
little
change
that
the
people
makes
now.
I
have
to
go
see
another
phone
call
in
cycle,
and
then
I
know
that
no
two
hour
cycle
is
actually
on
the
better
side.
A
In
the
grand
scheme
of
things,
I've
seen
lots
of
teams
that
counting
test
execution
in
days
or
weeks,
and
you
know
if
you're
trying
to
deliver
that
was
okay,
if
you're
delivering
every
once
in
six
months,
but
that's
no
longer
okay,
if
you're
trying
to
deliver
every
week
or
maybe
even
every
day,
so
so
the
I
guess
my
point
is
that
the
testing
can't
remain
the
constant.
It
also
has
to
change.
Just
like
everything
else.
You
know
software
delivery,
and
this
is
can
be
hard
because
the
you
know
the
quality
assurance
testing.
A
A
So
you
know
the
really
a
couple
of
you
know
stories
because
of
this
pandemic
right
in
in
japan,
where
I
grew
up,
they
developed
this
mobile
app.
That
does
the
contact
racing.
So
the
idea
is
like
I
guess
it's
using
bluetooth
or
something
to
see
who
you
got.
You
know
if
you
got
close
by
for
any
extended
period
of
time,
and
if
you
later
discover
that
you
are,
you
know,
copied
positive.
A
So
this
has
been
going
on
for
a
while,
but
much
to
the
their
embarrassment.
They
discovered,
I
think,
sometimes
because
in
early
this
year,
that
the
notification
that
was
supposed
to
go
out
to
people
was
simply
not
going
at.
So
it
turned
out
that
this
app
that
a
lot
of
people
have
installed
has
been
sending
zero
notification
whatsoever
for
quite
a
bit
of
time
right
so
that
that's
just
an
embarrassing
fail.
A
It
sounds
almost
like
a
critical
failure
and
then
so
you
know
that's
an
example
of
where
the
quality
assurance
fading
kind
of
did
destroy
the
whole
value
of
the
the
application
like
this.
And
if
you
go
back
a
little
bit
further,
you
know,
there's
they
also
the
boeing
737
max.
That
was
also
due
to
a
software
problem
to
kill
people
in
more
visible
and
easy
ways.
A
So
you
know,
I
guess
I'm
not
trying
to
kind
of
pinpoint
those
projects,
but
the
idea
is,
you
know,
as
software
is
becoming
more
mission,
critical,
every
part
of
the
society,
and
then
that
comes
with
the
increased
responsibility
to
make
things
work
correctly,
not
just
deliver
faster,
so
it
might
be.
You
know
the
the
things
that
the
face
can
break
and
think
that
the
airframe
controlling
software
can
break
wholly
different
game.
A
So
in
the
face
of
this
engine,
it's
like
what's
happening
in
the
testing
like
in
the
authority
space
and
how
is
that
evolving?
So
what
I
find
kind
of
interesting
is
there's
almost
like
a
two
extreme.
You
know
two
completely
opposite
problems
that
we're
facing
in
this
land.
The
one
is
there's
quite
a
bit
of
you
know:
team
software
development
teams,
where
they
simply
don't
have
enough
tests.
A
A
A
You
know
it
makes
sure
that
the
organizational
interest
and
the
so
like
a
contractual
structure
is
aligned
or
like
a
more
conducive
to
doing
the
testing.
So
I
was
talking
to
you
know.
Divinity
engineers
will
be
sort
of
open
team,
and
you
know
like
imagine
like
if
you,
if
you
receive
this,
if
you're
working
on
this
software
contract,
where
you
know
the
contracts
replace
like
you,
can
deliver
this,
then
that
feature
you
know
in
in
this
many
months
and
then
under
this
you
know
and
end
up.
The
total
price
is
like
x
dollars.
A
When
you
know
you
keep
evolving
that
software,
but
yeah,
if
there's
no
guarantee
that
the
future
part
lands
in
your
company
as
opposed
to
somebody
else
or
if
that's
not
part
of
the
current
contract,
then
you
just
can't
you
can't
justify
spending
time
and
therefore
doing
so
so
that
really,
even
though
the
engineers
like
on
the
front
line
wanted
the
right
test,
the
structure
that
just
doesn't
make
you
create
a
conducive
environment
for
that.
So
I
think
you
know
when
the
development
shifting
has
then
that's
a
sign.
That
organization
is
hunkering
down
for
marathon.
A
You
know
they
are
seeing
software
development,
no
that's
like
a
one-time
thing
and
be
dumbies,
but
it's
ongoing
effort,
and
it
you
take.
It
takes
that
sort
of
setup
like
a
financially
and
culturally
to
be
able
to
invest
in
the
authority
effort
so
that
I
see
as
an
encouraging
sign
and
that's
certainly
happening
and
in
all
the
parts
of
different
parts
of
the
world.
A
Another
thing
that's
happening
is
testability,
is
you
know
being
baked
further
and
further
into
the
software
design
itself,
and
this
has
been
going
on
for
a
while?
Also
okay,
you
know
I
still
remember
back
in
now
that
about
when
the
unit
test
became
a
thing
that
the
developers,
the
you
know,
the
thing
not
only
people
started
writing
the
unit
test,
but
they
actually
changed
the
way
they
designed
the
software
for
the
test
with
the
testability
in
mind.
A
So
you
hear
you
know
people
commenting
things
like
okay,
this
you
know
this
design
of
this
part
of
the
system
is
not
great
because
it's
not
testable.
Now.
Some
of
the
design
pattern
like
or
discards
because
of
that
and
stuff
like
that,
so
and
I
see
more
of
this
happening,
you
know
because
in
some
sense
I
think
the
testing
is
hard,
so
I've
seen,
I
think
the
airbnb
is
engineering
team
has
posted
like
a
long
series
of
blogs
about
how
they
change
the
way
they
develop.
A
The
android
app
for
which
you
know
for
them
that
this
is
a
pretty
critical
user
interaction
surface
so
that
it's
all
it's
all
you
know
it
got
this
like
a
testimony
called
baked
in.
So
you
can
launch
this
up
in
the
right
activity
with
all
the
you
know,
the
data
pre-stage
so
instead
of
talking
to
the
actual
backend,
so
you
can
remove
this
like
a
source
of
thickness
from
different
parts
of
the
mobile
app.
A
I
think
another
key
change
that's
happening,
and
I
expect
more
of
that
to
happen
in
the
future
right
and
then,
of
course,
there's
always
a
fish
for
more
automated
testing,
and
you
know,
testing
automating
testing
is
has
always
been
hard,
it's
still
hard
and
it's
going
to
be
still
more
effort
coming
down
the
road
and
it's
a
no
end
in
sight
here.
So
you
know
one
of
the
things
I
see
a
lot
nowadays
is
you
know
no
code,
less
code
test
scripting.
A
It's
almost.
I
think
this
is
like
a
selenium
id
2.0
or
something
like
that.
So
you
know
you
don't
have
to
write.
Well,
let's
see
you,
you
can
take
less
capable
people,
you
can
democratize
testing.
Maybe
that's
a
better
way
to
do
it
right.
So,
instead
of
writing
this
hardcore
code
to
write
the
tests,
you
can
do
this
in
less
complicated
ways
that
opened
up
a
lot
more
people
to
participate
in
writing
tests
and
that's
great.
A
But
you
know-
and
those
are
like
these
things
are
happening
in
a
very
common
platforms
like
mobile
and
developed.
But
again
as
a
domain
of
software
expands.
There's
a
lot
of
spaces
in
which,
just
you
know,
just
automating
is
pretty
darn
hard.
I
was
talking
to
you
know
accompanying
a
team.
That's
doing
right,
creating
video
games
like
imagine,
testing
video
games
right,
like
those
are
like
what
you
get
using
the
the
graphics
on
the
screen.
How
do
you
test
that?
A
What
made
automating
that
is
like
a
challenge
of
its
own
and
not
to
mention
all
you
know,
set
up
and
tear
them,
that's
necessary,
like
some
of
you,
I
know,
maintain
a
device
form
that
you
can
use
for
testing
you
know
doing.
The
proper
parallelization
still
often
involves
people
wiring
things
up
being
high,
so
there's
a
lot
of
still
challenges
around
automating
tests
and
then
run
it
at
scale
and
that
I
think
work
is
also
going
to
continue
now.
A
Sort
of
another
like
evolution
of
testing,
is
in
some
sense
like
somewhat
paradoxically
trying
to
reduce
the
domain
of
the
testing.
So
back,
then,
you
know
like
when
the
software
only
got
shipped
every
once
in
six
months
or
something
the
software
like
a
testing
was
almost
like
a
gate.
You
know
the
you
either
pass
it
like.
You
know
you
prepare
yourself
up.
A
Like
you
know,
we
you
rip
up
the
software
in
good
shape,
and
you
hope
that
you
pass
this
test
again
and
if
the
gate
gives
you
the
green
light,
then
you're
good
to
go
and
then
ship,
but
in
the
world
of
continuous
quality
like
we
are
moving
more
and
more
into
like
creating
defense
in
there.
So
not
only
there's
one
gate,
but
there's
like
a
tons
of
them
and
eat
small
and
then
so
and
then
they
they
collectively
create
this
like
optimal
risk
profile.
That's
how
I
think
about
this.
A
So,
for
example
like
in
production
right,
so
the
the
thinking
behind
is
like
the
old
gatement
id
is.
If
you
pass
a
gate,
then
you
know
all
the
bits
of
like
a
good
luck
like
it
better
function
and
if
not
oops,
but
in
the
world
of
this,
like
a
defense
in
depth,
then
you
kind
of
accept
that
even
in
production,
there's
always
a
risk
that
things
might
break
and
it
will
blow
up
and
then
so.
Instead,
you
try
to,
instead
of
just
trying
to
prevent
that,
you
also
try
to
mitigate
those
risks
in
productions.
A
So
this
is
there's
a
lot
of
like
increased
practices
here.
The
modern
practices
here
involving
like
future
flags
and
circuit
breakers,
and
you
know
the
production,
traffic,
replays
and
heavy
testing
and
more
sophisticated.
All
that-
and
I
mean
you
name
it
now.
So
all
these
practices
designed
to
you
know
start
with
the
assumption
that
you
know.
Sometimes,
despite
your
own,
you
know
good
efforts
like
a
bad
thing,
still
creep
into
all
the
way
to
the
right.
So
how
can
you
find
that
quickly?
How
can
you
minimize
the
damage?
A
The
qa
and
testing
is
also
like
the
moving
so
that
another
way
to
look
at
that
is
like
those
are
testing
shifted
to
the
right,
sometimes
all
the
way
to
the
right
in
production,
but
there's
also
testing
is
also
moving
to
the
left,
and
there
is
this
idea
that
you
know
finding
the
right
place
of
the
testing
across
this
delivery
phase
and
moving
them
around
is
becoming
more
important
so
that
correctively
the
pipeline
behave.
The
delivery
pipeline
behaves
optimally.
A
So
that's
the
kind
of
war
where,
like
people
who
have
too
many
tests,
are
like
you
know,
facing
and
like
tackling
nowadays
and
that's
what
I
see
like
part
of
the
future
of
the
testing.
So
I
call
that
the
data
driven
levels,
in
other
words
like
if
you
have
enough
testing
and
like
you,
know
that
you're
drowning
in
the
sea
of
automation,
then
it's
kind
of
like
you
have
this.
A
You
know
one
of
the
responsibility
among
other
things,
so
I
still
look
at
the
test
results
and
then
so
look
at
what
hired
and
what
failed
and
then
bring
those
failures
back
to
the
right
team
in
the
right
person.
If
you
you
know
and
then
take
on
the
fixing
so
back,
then
every
test
results
which
you
know
again
happening,
maybe
every
night
was
looked
at
pretty
carefully
and
thoroughly
and
then,
but
you
know,
that's
human
effort,
so
that's
costly.
A
Instead
of
trying
to
make
sense
out
of
what's
happening
like
you,
basically
dumbed
it
down
to
a
binary
flag,
you
know
either
your
pulley
test
passed
the
ci
or
it
failed,
and
then,
if
it
failed,
then
it's
kind
of
up
for
the
developer
to
look
at
the
the
ci
servers
log
and
see
what
failed
and
they
they
kind
of
go
from
there.
So
not
a
lot
of
like
no,
not
a
lot
done.
A
We
see
data
coming
out
from
the
system,
but
now
like
with
the
tests
or
the
increasing
in
volume
and
obviously
like
and
not
every
test
can
be
shifted
all
the
way
to
the
left.
You
know
think
about
the
integration
test
or
that
sort
of
things.
Then
it's
about
scale.
This
idea
of
okay-
I
don't
want
to
look
at
like
what
all
this
fire
hole.
Just
give
me
a
binary
information.
A
That
idea
is
breaking
down,
so
you
know
again
in
the
places
where
people
are
drowning
in
the
sea
of
the
tests,
we're
kind
of
back
to
needing
to
make
sense
of
what
comes
out
of
the
test.
So
that's
the
what
I
call
the
data
driven
devops
and
then
that's
the
that's
effort,
that's
happening
in
some
of
the
more
sophisticated
software
development
shops.
So
one
example
of
this
that
really
close
to
me
is
this
idea
of
the
predictive
test
selections.
A
A
Most
of
it
is
testing
a
different
part
of
the
system
that
I
didn't
touch
it's
kind
of
questionable,
so
you
know,
and
the
old-school
way
of
doing
this
is,
you
know,
called
smoke
tests
right
like
we
have
a
very
manually
created
small
subset
of
tests
that
are
supposed
to
catch.
You
know,
good
representative
problem
runs
very
quickly,
so
you
run
that
first,
if
they
all
passes,
then
then
it's
good
enough
to
run
the
rest
of
the
test
cycle.
So
that's
a
way
of
staging
things.
A
So
the
the
modern
practices
instead
of
humans
manually
creating
those
static
lists
of
smoke
tests.
What
if
we
can
make
the
machines,
create
adaptive
subsets
that
act
as
the
smoke
test
and
then
so
those
are
the
predictive,
what's
called
the
predictive
selections,
and
you
know
facebook
famously
produced
a
paper
on
this
a
couple
of
years
back,
so
you
know
again
in
places
like
that.
This
is
an
emerging
practice.
I
that
the
google
does
something
similar.
A
A
Now,
if
you're
trying
to
inspect
every
single
frame
of
the
movie
to
fill
this
extent,
then
you
simply
can
only
go
up
so
quickly,
but
as
you
and
then
even
if
maybe
you
can
deal
with
24
frames
per
second,
but
if
it
goes
after
you
know,
60
frames
per
second
and
then
yeah.
You
know
it's
sooner
or
later
it
hits
the
ceiling.
A
So
instead,
like
what
needs
to
be
focused
on
is
the
delta
between
two
frames.
You
know
when
you
have
24
frames
per
second,
the
change
between
two
adjacent
frames
are
very
small.
So
if
you
can
exploit
that
similarity
to
like
a
reduce
the
eliminated
redundant
work,
that's
happening,
then
that
gonna
give
you
like
a
great
performance
boost
and
then
that's
fundamentally
more
scalable.
Even
if
you
go
up
to
60
frames
per
second,
the
delta
gets
even
smaller
and
then
so
that's
in
my
mind.
A
The
only
way
to
copy
this
kind
of
progress,
so
you
know
this
problem
is
what's
really
close
in
my
heart,
so
that's
currently
I'm
focusing
on,
but
in
my
company
launchable,
but
really
like.
I
see
this
as
a
like
a
part
of
the
broader
trend
to
utilize
more
data,
the
data
driven
devops
that
I
mentioned
about
it.
So
that's
what
I
wanted
to
cover
in
the
future
testing
today.