►
From YouTube: OCTO Speaker Series Episode 2: John Hughes
Description
Experiences with QuickCheck: Testing the Hard Stuff
If you've ever wondered whether there is a better way to test software than by writing and maintaining thousands of test cases, then this talk is for you. I'll explain "property-based testing", which combines test case generation, test case simplification, and using properties to specify your code. I'll show you what you need to think about to make it work in practice, and tell you tales of some of our more entertaining bugs.
About John Hughes
I've loved functional programming all my career; I wrote "Why Functional Programming Matters", a manifesto for the field, way back in the eighties. I urged lazy evaluation on John Backus (but failed to persuade him). I was a co-designer of Haskell. I moved to Sweden, home of the Lazy ML compiler, in the nineties, and at the turn of the millennium I co-invented property-based testing. I co-founded a startup in 2006 to bring PBT to industry, and I've learned to love (other peoples') buggy software! My motto: "make hard things easy with functional programming".
Twitter: @rjmh
GitHub: rjmh
Web: http://www.cse.chalmers.se/~rjmh/
Learn more about the OCTO Speaker series here: https://octo.github.com/
A
A
Instead
of
merely
preaching
the
merits
of
technical
features
of
functional
languages,
the
paper
clearly
demonstrated
their
practical
benefits
and
it
convinced
many
of
us
to
pursue
a
career
in
programming
languages.
Indeed,
without
john's
inspiration.
I
wouldn't
be
at
github
today,
so
very
much
thanks
for
that.
John
john
continues
to
push
the
boundaries
of
programming
language
technology,
in
particular
as
the
ceo
of
cubic,
which
offers
products
and
services
around
property-based
testing.
C
C
So
let
me
just
do
a
simulated
demo
on
the
slide
and
and
show
you
what
the
api
does.
So
this
is
the
adlang
rapple,
the
24,
the
black
stuff
is
the
prompt
ignore
it.
The
orange
stuff
is
what
I
typed,
and
here
I've
called
the
new
function
to
create
a
queue
with
space
for
four
elements
and
the
green
stuff
is
the
response.
I
get
back
the
address.
C
Okay.
So
what
can
I
do
with
a
key
once
I've
got
one?
Well,
I
can
put
data
into
it.
This
queue
holds
integer.
So,
let's
put
one
let's
put
two,
then
I
can
ask
how
many
items
are
in
the
queue?
Well,
obviously,
two
right,
so
I
call
the
size
function
and
that
returns
two
and
I
can
take
things
out
of
the
queue
again
I'll
get
I
get
the
one
I
get
the
two
I
get
uninitialized
memory,
I
get
some
more
uninitialized
memory.
C
C
C
Now,
maybe
I
don't
want
to
include
the
calls
that
read
uninitialized
memory.
I
was
really
abusing
the
api
there,
but
if
I
take
the
others,
I
can
put
them
into
a
little
function.
That
just
makes
those
calls
and
then
matches
the
result
against
the
expected
value.
If
any
function
returns
a
different
result,
then
the
test
will
fail.
C
The
one
interesting
thing
I
want
to
point
out
about
this
very
simple
test
case
is:
it
covers
100
of
the
code.
That's
great.
I've
got
100
branch
coverage,
which
is
quite
easy
because
there
aren't
any
branches
here,
but
nevertheless
it
covers
100
of
the
code.
So
that's
great
right,
so
I've
got
simple
program
seems
to
work
and
I've
got
excellent
testing
for
it.
So
I'm
done
or
am
I
the
way
I
like
to
test
code
is
not
like
this.
Of
course
it's
using
property
based
tests.
C
So
let
me
show
you
what
happens
when
I
do
that.
I
call
our
property-based
testing
tool
quick
check
to
do
so,
and
I
have
to
give
it
a
property
of
the
code.
I'm
not
going
to
show
you
this
test
code,
I'll,
explain
a
little
bit
what
it
looks
like,
but
let's
just
see
what
happens
when
we
run
these
tests.
C
C
The
slide
is
highlighted
in
yellow
now,
so
this
just
shows
the
sequence
of
steps
that
the
test
case
performed
and
we
can
see
what
went
wrong.
So,
let's
look
at
it.
What
happened?
First
of
all,
I
created
a
queue
with
space
for
one
element.
Okay,
then
I
put
something
into
it:
okay,
that's
fine!
Now
the
queue
is
full
there's
one
thing
in
it,
and
then
I
asked
how
many
items
are
in
the
queue
I
got:
zero.
C
Okay,
that's
clearly
wrong,
and
so
a
post
condition
in
my
property
will
fail.
That's
why
the
test
failed.
So
now
we
have
a
little
example.
We
can
just
work
through
this
example.
Look
at
the
code
and
see
if
we
can
debug
it
okay,
what
happened
first.
First,
I
created
a
queue
of
size,
one
and
what
I
did
with
that
parameter
to
knew.
Was
I
saved
it
in
the
size
field
of
the
data
structure?
C
Again,
you
don't
need
to
read
the
code,
but
just
remember
the
size
field
is
one
and
then,
at
the
end
of
the
test,
I
called
the
size
function
to
see
how
many
items
are
in
the
queue
and
here's
the
code.
For
that,
look,
I'm
taking
the
difference
between
an
input
position
at
an
output
position.
Same
sounds
reasonable,
modulo,
the
size,
but
the
size
is
one.
C
That's
why
this
doesn't
work.
This
size
function
can
only
return
zero
when
the
size
is
one.
So
so,
of
course
it
doesn't
work.
Actually,
the
situation
is
worse
than
that,
because
if
I
look
at
the
code
for
putting
something
in
the
other
part
of
the
test,
we
can
see
that
it
writes
some
data
into
a
buffer
and
that
it
increments
this
input
position
modulo
the
size
modulo
1,
and
that
means
that
that
is
also
going
to
remain
at
zero.
I
have
a
little
animation
to
show
us
what
happened.
C
I've
got
a
buffer
and
I've
got
this
input
and
output
pointer
into
it.
I
put
something
in
increment:
the
input
pointer,
modulo
one
and
now
the
buffer
is
full,
but
those
input
and
output
pointers
are
exactly
where
they
were
at
the
beginning.
So
here's
the
problem.
The
code
doesn't
work
because
a
full
buffer
looks
exactly
like
an
empty
one
and
if
you
think
about
it,
this
isn't
just
a
problem
when
the
size
is
one.
C
C
C
C
C
Let
me
show
you
an
animation
of
this
example.
I
put
something
in
I
took
it
out
and
that
increments,
the
output
pointer.
I
put
something
in
again
and
now
I've
put
two
items
into
a
circular
buffer
of
size,
two,
so
the
input
pointer
wraps
around
and
then
I
called
the
size
function.
What
did
that?
Do?
He
took
the
difference
between
the
input
and
output
position,
modular
the
size,
let's
just
fill
in
the
numbers-
that's
zero,
minus
one
modulo,
two
or
minus
one
modulo
two.
What
is
that
-1
modulo
2
is
plus
1..
C
What
I
have
to
do
is
I
have
to
make
sure
that
that
expression
in
the
size
function
is
never
negative
because
I'll
get
the
wrong
answer
if
it
is
and
of
course,
there's
an
easy
way
to
do
that
here,
let
me
patch
it
I'll.
Just
call
the
absolute
value
function
right
and
then,
instead
of
taking
minus
one
remainder,
two
I'm
taking
plus
one
remainder
two
and
the
value
will
be
plus
one,
and
if
I
make
that
fix
in
the
size
function
like
that
and
repeat
the
last
test,
it
passes.
That's
great.
C
C
People
might
often
think
so.
But,
of
course,
I
don't
trust
a
single
test
I
like
to
generate
them,
so
let's
run
some
more
random
tests.
What
happens
another
failure
and
if
we
look
at
the
shrunk
test
in
this
case
it's
a
little
bigger
and
it
starts
off
by
creating
a
queue
with
space
for
two
elements
instead
of
one.
C
Why
is
that?
It's
because
cues
of
size,
one
work,
now
progress,
but
this
case
of
size
two
doesn't
so
we
create
a
cue
of
size.
Two
under
the
hood.
That's
really
three.
What
does
the
test
case?
Do
it
puts
three
things
in
that's
going
to
make
that
in
pointer,
wrap
around
it
takes
one
thing
out
how
many
things
are
in
the
queue
two:
what
did
the
size
function
return?
C
C
My
fix
was
buggy,
so
my
fixer,
I
had
thought
correctly
that
I
have
to
make
sure
that
first
argument
of
percent
is
never
negative,
but
abs
is
the
wrong
way
to
achieve
that
and
luckily
there's
a
different
way.
What
I
can
do,
instead
is
just
add
on
the
size
and
that's
going
to
be
enough.
It's
three
in
this
case
to
make
sure
that
that
first
argument
is
negative,
negative,
but
adding
size
doesn't
change
the
value
modular
size
and
so
I'll
end
up
with
the
correct
result.
C
We've
seen
that
I
could
find
several
different
bugs
with
the
same
test
and
that's
something
that
happens
all
the
time
with
property-based
testing.
We've
also
seen
that
when
a
bug
is
found,
it's
reported
as
a
minimal
test
case
that
provokes
the
bug
thanks
to
shrinking,
and
that
makes
those
bugs
easy
to
understand.
C
But
we
also
saw
that
when
I
made
a
wrong
bug
fix-
and
I
overfitted
by
fix
the
failing
test
that
I
happen
to
have-
I
didn't
get
away
with
it.
I
immediately
found
a
slightly
different
example
that
showed
that
the
underlying
bug
was
still
there
and
I
don't
know
about
you,
but
I've
seen
so
much
weird
code
in
production
systems
that
was
obviously
put
there
by
somebody
trying
to
make
a
test
case
pass
and
they
put
something
that
made
their
case
past,
but
didn't
fix
the
underlying
bug
and
with
property-based
testing.
C
C
C
C
C
C
Well,
we
make
a
model
as
simple
as
we
can
at
the
state
of
the
system
for
the
queue
it
was
just
a
list
of
the
values
that
should
be
in
the
key,
and
then
we
write
a
model
state
transition
function
for
each
kind
of
operation
in
the
test
and
that
lets
quick
check
figure
out
what
the
model
state
will
be
at
each
point.
In
the
test,
we
run
the
test
on
the
real
system
and
we
write
post
conditions
that
compare
the
real
outputs
to
the
model.
C
So
that's
what
a
state
machine
model
looks
like
that's
its
structure
and
we've
found
so
many
bugs
with
this
kind
of
test.
I
want
to
show
you
some
of
my
favorite
bugs
the
first
one
is
a
bug
that
we
found
in
an
ericsson,
firewall
product
and
the
purpose
of
this
product
was
to
connect
multimedia
calls
across
the
firewall.
C
C
C
C
Okay,
so
I
might
talk
to
another
person
and
then
we
can
subtract
a
caller.
But
even
here
you
don't
have
to
stop
now
and
subtract
the
other
one
you
can
keep
going.
You
can
add
a
fourth
quarter
and
it
this
is
what
our
test
case
did,
and
it
turns
out
that
when
you
subtract
the
third,
the
thing
crashes,
okay,
so
I
really
love
this
bug.
C
Why
well
one
reason
is
that
this
was
the
bug
that
persuaded
eriksen
to
invest
in
property-based
testing,
but
I
love
it
also
because
this
is
a
test
case
that
nobody
in
their
right
minds
would
write
in
a
test
suite
look
at
it.
What's
it
doing
it's
adding
and
subtracting
your
caller,
we
try
it
once
it
worked.
We
tried
it
twice.
It
worked
by
induction,
it
must
work
any
number
of
times
only
it
doesn't.
It
fails
the
third
time.
C
C
C
But
of
course
it's
bad
to
have
corrupt
data
in
your
system
and
this
test
case
turned
data
corruption
into
a
crash
making
it
detectable.
So
this
looks
like
a
simple
example.
I
just
want
to
add
that
the
rfc
we
had
to
read
to
write
the
tests
was
212
pages
plus
another
183
about
how
erikson
planned
to
use
it.
So
this
was
actually
quite
a
substantial
testing
effort,
but
we
found
this
lovely
bug
as
a
result,
I
want
to
show
you
a
couple
more
here
is
a
bug
that
we
found
for
volvo
cars.
C
Volvo,
cars
hired
us
to
test
software
from
the
suppliers,
but
it's
the
basic
software
running
in
cars
and
some
of
the
software
in
the
car
manages.
What's
called
the
can
bus?
That's
the
network,
that's
most
widely
used
in
vehicles,
so
we
were
testing
a
protocol
stack
for
the
canvas
and
here's
the
book
that
we
found.
C
C
What
happens?
Well,
we
see
it
being
sent
that's
good.
Now
the
bus
is
busy
and
our
test
case
sends
two
more
messages
with
priorities.
Two
and
three
they
have
to
be
queued
up
in
the
protocol
stack
the
final
step.
The
test
case
is
one,
that's
a
call,
that's
usually
made
by
the
driver
software
for
the
bus
hardware,
and
it
confirms
that
message.
Number
one
has
been
sent
so
now
the
bus
is
free.
The
stack
can
send
another
message:
what
are
the
cued
ones?
C
It
should
choose
the
highest
priority
message
and
highest
priority
means
lowest
number,
so
it
should
send
message
number
two
sent
to
wrong
one
now.
Why
did
that
happen?
It
turns
out
that
it's
because
of
a
weirdness
in
the
can
bus
protocol.
So
these
priorities
they're
also
the
messenger
identifier.
They
identify
the
type
of
message
and
the
first
version
of
the
protocol
allowed
11
bits
for
this
value.
But
that
means
you
could
only
have
2
000
different
message
types
in
the
entire
car.
It's
just
not
enough
in
a
modern
vehicle.
C
C
C
You
want
the
critical
messages
to
have
priority,
so
it
really
matters,
but
secondly,
it's
unlikely
that
you
would
find
this
bug
by
a
handwritten
test
case.
You
might
write
a
test
case
that
looked
like
this,
but
the
bug
depends
on
a
feature:
interact,
interaction
between
priorities
for
the
bus
and
whether
you
have
an
extended
or
a
standard
can
id.
You
probably
wouldn't
write
this
very
test
case
which
is
needed
to
provoke
the
bug,
so
it's
another
kind
of
bug
that
you
can
really
only
find
by
generating
tests.
C
So
this
was
a
great
project.
We
read
3
000
pages
of
pdfs.
We
turned
that
into
twenty
thousand
lines
of
quick
check
code,
six
pages
six
lines
per
page,
that's
not
so
bad.
We
tested
more
than
a
million
lines
of
c
code
from
six
different
suppliers
and
we
found
more
than
two
hundred
in
this
critical
code.
So
so
that
was
great.
C
This
was
a
bug
in
a
database
used
at
corner
clark
were
a
hugely
successful
swedish
startup
in
the
2000s
they're,
a
big
company
now
supporting
payments
for
web
shops,
but
in
the
2000s
their
database
didn't
work
and
here's
a
quote
about
it.
There's
a
lurking
bug:
we've
got
bad
objects
and
premature
enderphile
every
other
month
the
last
year.
C
C
So
by
the
time
I
looked
at
this
problem,
then
the
crashes
had
actually
increased.
They
were
happening
every
week
and
the
guys
working
with
the
database
implementation
had
spent
more
than
six
weeks
at
planner,
trying
to
figure
out
what
was
going
on
and
the
best
hypothesis
at
the
end
of
that
was
it
seems
to
happen
when
the
file
is
about
one
gigabyte.
C
Maybe
it's
something
to
do
with
rehashing,
so
it
was
obviously
some
kind
of
race
condition
and
I
thought
let's
try
generating
parallel
tests
for
this
problem.
So
parallel
tests
are
a
little
trickier
than
the
sequential
tests
I
already
talked
about.
We
generate
tests
that
look
like
this.
First
of
all,
you
do
some
random
stuff
to
get
into
a
random
state.
Then
you
fork
off
parallel
processes
and
they
each
do
some
more
random
stuff.
C
C
Is
there
some
order
in
which
we
could
have
performed
those
calls
that
would
make
the
results
consistent
with
the
sequential
model?
If
so,
the
behavior
was
linearizable
and
we'll
say
that
the
test
passed,
if
not,
it
failed.
So
using
this
idea,
I
can
take
a
sequential
model
and
I
made
one
for
this
part
of
the
database.
It
was
only
100
lines
of
code,
so
quite
simple,
and
by
changing
two
lines
I
could
use
it
for
race
condition.
Testing
instead-
and
here
is
my
favorite
bug
of
all
time-
look
at
it.
C
This
provokes
the
premature
end
of
file
error.
First
of
all,
sequentially
we
open
the
file,
we
close
it.
We
open
it
again
and
then
we
do
three
things
in
parallel.
One
lookup
and
two
insertions
do
all
of
that,
and
occasionally
you
get
a
premature
end
of
file
error.
The
file
is
corrupted
so
when
I
saw
this
test
case,
I
thought
that
something
wrong
is
shrinking.
C
This
cannot
possibly
be
the
smallest
test
that
provokes
this
bug,
but
it
is
look
at
that
open
closed
open.
I
thought
no,
no,
it
must
be
enough
just
to
open
it
once
so.
I
manually
simplified
the
test
case.
I
ran
that
simplified
test,
tens
of
thousands
of
times
passed
every
single
time,
and
indeed
today.
I
know
why,
because
when
we
start
the
test
case,
the
file
doesn't
exist.
We
open
the
file
that
creates
it.
We
close
it.
The
second
open
opens
an
existing
file.
It's
very
slightly
different.
C
So
once
again
ask
yourselves
this:
would
you
write
this
test
in
a
test
suite?
Of
course
not.
Why
pick
that?
And
yet
this
is
the
smallest
test
that
can
possibly
reveal
this
bug
and
it
took
about
10
minutes
of
property-based
testing
to
find
so
hard
work,
but
but
worth
it,
and
I
also
found
the
bad
object
bug
I'm
not
going
to
talk
you
through
it.
You
can
see
it's
about
the
same
size,
not
complicated.
C
C
I
can't
resist
also
showing
you
some
bugs
we
found
in
dropbox,
so
testing
dropbox
is
tricky
because
there
is
the
dropbox
demon
in
the
background
that
can
be
taking
actions.
So
in
our
model
we
don't
know
what
state
we're
in.
We
know
what
stake
we
tried
to
get
to
perhaps,
but
then
we
have
to
model
all
of
the
possible
background
actions
that
dropbox
might
take
and
then
we'll
carry
out
a
step
in
the
test.
C
C
C
So
we
do
that
as
we
run
the
tests
and
if
we
can
find
a
path
that
where
all
the
observations
are
consistent
with
the
model,
we'll
say
the
test
passed
and
if
we
can't
then
we'll
say
the
test
failed
and
in
practice
we
found
that
we
had
to
track
maybe
up
to
a
thousand
possible
states
during
the
test,
but
not
more
than
that.
That
was
enough
to
find
some
really
interesting,
behavior.
C
So,
first
of
all,
on
one
of
the
nodes
we'll
write
to
a
file
we're
only
using
one
file
here,
we'll
write
a
to
it.
That's
fine,
then,
on
the
other,
node
we'll
write
b
to
it
and
we'll
observe
that
we
overwrote
a
file
containing
a
so
when
we
see
that
that
tells
us
that
dropbox
must
have
synchronized
the
file
from
one
node
to
the
other
between
those
two
calls.
C
So
we
know
a
bit
what's
going
on
what
next,
let's
delete
the
file
on
the
first
node
and
when
we
delete
it,
we
see
that
it
we're
deleting
a
file
containing
a
so
that
means
that
dropbox
have
not
synchronized
the
b
back
to
the
original
node.
Yet
and
now
we'll
write
c
to
the
file
and
once
again
we'll
see
that
the
file
didn't
exist,
so
the
b
hasn't
been
synchronized
back
so
now
we've
got
a
conflict
right.
We've
got
we've
written
b
on
one
node
c
on
the
other.
C
C
C
C
If
the
timing
is
right
in
this
test,
no
conflict
file
ever
appears,
and
so
this
is
again,
I
think,
weird,
weird
behavior
and
the
dropbox
developers
agreed.
This
was
a
bug
it
should
have
been
fixed
by
now.
Luckily-
and
it
was
always
rare,
but
we
found
these
rather
cool
behaviors
in
a
distributed
system
like
dropbox
as
well.
C
C
You
know,
maybe
in
this
case
there
are
five
steps
that
cause
the
failure
and
that's
the
bug
we're
looking
for
it's
those
five
steps
we
want
to
shrink
to
that
will
show
us
what
the
problem
was,
but
you'll
notice.
There
are
lots
of
other
steps
here
as
well
often
those
other
actions,
don't
matter
much.
C
So
it's
not
always
true,
of
course,
but
often
if
those
five
red
steps
appear
anywhere
in
your
test
case,
the
test
will
fail.
So
what
does
that
mean?
That
means
when
we
run
one
longer
test
in
a
sense,
we're
testing
every
subsequence
and
how
many
of
those
are
there
we're
getting
exponentially
much
value
out
of
a
linear
amount
of
testing
work.
C
I
think
this
strictly
speaking,
this
isn't
true,
but
it
is
truthy
to
borrow
a
word
from
stephen
colbert,
so
this
is
intuition
for
why
this
approach
of
generating
tests
and
shrinking
them
works
so
extremely
well,
of
course,
it
doesn't
always
work
nicely.
Let
me
just
mention
a
couple
of
things
that
can
go
wrong.
C
This
is
one
trap
you
can
fall
into
when
you
use
a
model
as
a
reference
for
testing
a
real
system.
You
end
up
replicating
some
of
the
code
from
the
system
in
the
model.
That's
not
good!
For
a
start,
it's
expensive!
It
feels
as
though
you're
doing
the
same
work
twice
expensive
is
not
good,
but
even
worse.
C
So
how
do
we
test
this?
How
do
we
predict
what
the
resulting
collection
should
be?
Well,
of
course
we
could
re-implement
insert
in
the
test
code,
but
there's
no
point
in
doing
that.
Right,
we'll
be
falling
into
the
trap
that
we
don't
want
to
fall
into,
but
here's
a
different
way
of
thinking
about
how
to
test
this
operation.
C
You
can
ask
yourself
how
would
changing
the
input
of
insert
change
its
result,
changing
the
input,
what
I
mean
well,
for
example,
suppose
we
insert
something
else
before
we
do
the
insertion
we
want
to
test
now.
You
might
say,
instead
of
having
one
unknown
collection
on
my
slide,
I've
got
three.
Is
that
better?
C
C
C
C
Of
course
you
may
be
wondering.
Is
this
really
true?
You
may
be
thinking
about
it?
Don't
think
I
like
to
let
quick
check
do
my
thinking
for
me.
So
if
I
want
to
know
if
this
is
really
true
I'll
just
test
it,
and
when
we
do
that,
it's
not
look
at
the
example.
The
interesting
part
is
in
red,
so
these
are
the
two
key
value
pairs
that
make
this
property
false.
C
So
when
you
construct
metamorphic
properties
like
this
very
often
they're,
not
always
true,
there's
some
case
like
this
some
special
case
that
makes
them
false.
That
doesn't
mean
we
have
to
throw
it
away.
It
means
we
have
to
adapt
the
property
a
little
bit.
So
I
can
just
add
a
clause
that
says
if
the
keys
are
equal,
then
what
I
just
said,
the
last
insertion
wins.
Otherwise
you
can
swap
the
assertions
over
and
by
doing
that,
I
get
a
property
that
passes
all
its
tests.
C
C
So
when
you
implement
cryptocurrency,
then
you
need
to
represent
coin
amount.
Let's
have
a
type
for
that.
This
defines
a
new
type
in
haskell.
A
coin
basically
just
contains
an
it,
but
it's
not
the
same
as
an
int,
because
there
is
an
upper
limit
on
the
total
number
of
coins
that
may
ever
exist,
and
I'm
gonna
just
assume
that
it's
one
million
for
the
purposes
of
this
example
in
reality,
it's
much
more
than
that
like
billions,
but
now
there
should
never
be
a
coin
value
larger
than
one
million.
C
So
let's
just
write
a
validity
checking
function.
A
coin
is
valid.
If
the
integer
in
it
is
in
the
correct
range
of
values,
and
then
we
need
to
define
operations
on
coin
values
like,
for
example,
addition
we're
going
to
need
to
be
able
to
combine
coin
quantities,
and
all
of
these
functions
have
to
check
for
overflow
and
if
the
flow
occurs,
then
in
haskell.
Typically,
we
would
return
nothing
instead
of
the
value,
so
nothing
it's
the
hassle
equivalent
of
the
null
pointer,
so
no
results
in
that
case.
C
Okay.
So
this
code
is
all
very
simple,
but
it
still
needs
to
be
tested
right.
So
what
property
could
I
write
for
testing
this?
Well,
I
could
write
an
addition.
Property
that
takes
two
randomly
generated
coins,
as
arguments
adds
them
together
and
then
checks.
We
get
the
right
result.
What
is
the
right
result?
C
C
C
I
just
don't
know-
and
this
is
a
problem
with
property-based
testing-
that,
because
you
don't
see
the
test
cases,
there's
a
risk
but
you're
doing
a
lot
of
trivial
tests,
and
but
you
don't
realize
it
so
we
always
have
to
mitigate
that
risk
by
making
measurements
of
the
actual
tests.
What
we
do
is
we
label
the
test
cases
and
there's
a
function
in
the
quick
check
api
for
doing
that,
so
I'll
just
attach
a
label
to
every
test.
C
That
is
a
string
and
I've
written
a
little
function
summarize
for
computing
that
so,
if
the
total
is
in
the
right
range,
I
return
normal
if
it's
outside
it.
I
return
overflow
and
now,
when
I
run
the
tests,
I'll
get
some
statistics
and
we
can
see
that
oh
yeah,
it's
good.
I've
got
about
50
of
each,
so
that's
good.
Both
kinds
of
tests
are
appearing.
Often,
okay,
now
just
think
about
this.
A
little
bit.
C
C
I
would
say
the
add
function
is
imposing
a
boundary
between
normal
and
overflow.
Is
that
boundary
in
exactly
the
right
place
to
find
out?
I
need
to
run
tests
that
fall
close
to
the
boundary
at
the
moment.
I
don't
know
how
many
tests
did
that,
but
I
can
find
out.
I
can
just
enrich
my
labeling
function
and
I've
chosen
arbitrarily
to
say
if
the
total
is
within
three
of
the
maximum,
then
this
is
a
boundary
case.
C
C
All
you
you
have
to
pay
attention
to
here
is
that
I'm
calling
the
choose
function
to
choose
a
uniform
random
number
between
zero
and
the
maximum
value.
Well,
obviously,
if
I
take
two
of
those
and
add
them
together,
it's
very
unlikely
that
the
result
will
fall
close
to
the
boundary
that
I
care
about.
Now.
At
this
point,
you
might
be
tempted
to
say:
well
why
don't?
C
C
And
then
I'll
choose
between
three
things:
either
just
return
the
small
number
or
return
the
maximum
value
minus
the
small
number
or
choose
uniformly
in
the
entire
range.
So
this
generator
can
still
generate
any
coin
value
that's
valid,
but
it
will
more
often
generate
values
near
the
endpoints
and
if
I
change
this
generator
instead
and
re-test
by
property-
oh
it
fails.
Actually
it
fails
for
the
case
of
zero
of
a
million,
because
when
you
add
them
together,
add
returns.
Nothing
and
the
property
expects
it
to
return
just
one
million.
C
If
I
show
you
the
code
again,
I've
highlighted
the
bits
that
caused
the
bug
and
it's
obvious
what
the
problem
is
in
the
validity
test.
I
said:
a
valid
coin
must
have
a
value
less
than
or
equal
to
the
max
and
in
the
addition
function
I
just
tested.
Is
it
less
than
max?
So
I've
got
an
off
by
one
error
and
it's
impossible
to
say
which
one
is
right.
C
I
can
make
either
choice,
but
I
must
make
it
consistently
and
if
I
change
the
code
so
that
it
is
consistent,
then
the
tests
pass
and
again
see
the
measurements,
and
it
seems
that
about
four
percent
are
boundary
cases.
Now
the
nice
thing
about
changing
the
generator
instead
of
just
writing
some
unit
tests
for
add
is
that,
with
this
new
generator,
every
property
that
needs
some
random
coins
will
get
ones
that
provoke
values
close
to
the
boundary
when
they're
added
together.
C
C
Here's
one
that
sums
to
one
million
zero
and
a
million
here's
one
that
sums
for
a
million
of
one
zero
and
a
billion
and
one
oh,
oh
no.
You
can't
do
that
of
course,
because
a
million
one
would
not
be
a
valid
argument.
No
here
you
need
one
of
the
million
and
one,
so
you
see
quick
check,
finds
a
way
of
hitting
each
of
these
things
in
a
valid
way.
C
C
C
That
one
can
use
when
you're
writing
property-based
tests.
Of
course
you
have
to
write
a
property,
but
you
should
still
think
about
what
are
the
important
kinds
of
case
you
want
to
test,
but
don't
write
unit
tests
write
labeling
code
measure.
How
often
each
kind
of
important
test
is
generated
and
if
you
don't
like
the
result,
tune
the
generation
and
keep
doing
it
until
every
kind
of
important
test
is
tested
sufficiently
often.
C
Okay,
so
what
have
we
seen?
Well,
I've
shown
you
that
property-based
testing
has
helped
us
find
bugs
in
real
systems
that
you
would
never
find
with
a
handwritten
test
case.
It's
particularly
good
at
finding
feature
interactions,
which
is
just
impractical
to
write
tests
for
all
of
those,
but
property-based
testing
can
find
them.
I
hope
I've
persuaded
you
that
the
result
of
shrinking
those
shrunk
and
failed
tests
are
really
easy
to
debug
a
lot
of
the
test
case.
Simplification
that
you
would
do
manually
is
done
for
you.
C
C
B
A
Thanks
very
much
john,
oh
fascinating
tool,
so
everyone
on
the
here
on
the
webinar.
If
you
have
questions,
please
submit
them
via
the
go
to
webinar
our
questions
facility,
a
couple
of
them
I
have
come
in
already,
so,
if
that's
all
right
so
I'll,
be
the
the
possibility
who
passes
them
on
so
john,
you
talked
about
how
you
generate
are
quite
large
are
failing
failing
inputs,
and
then
you
shrink
them
down.
A
C
C
C
C
So
we
can
see
here
that
the
probably
probability
of
a
test
revealing
the
bug
rises
dramatically
as
the
size
of
the
test
case
increases
and
the
intuition
is
the
one
that
I
gave
during
the
talk
right.
It's
because
what
matters
is
that,
somewhere
in
this
larger
test,
the
bad
sequence
that
provokes
the
bug
happens,
but
it
doesn't
matter
so
much
exactly
where
it
appears.
Now
you
might
object
and
you
might
say
well,
yes,
longer
tests
may
be
more
likely
to
hit
the
bug,
but
they're
also
more
expensive
to
run.
C
C
And
if
we
look
at
that
graph
here
it
is
on
the
x-axis.
You
still
have
the
total
size
of
each
test
case,
but
what
this
shows
is
how
much
each
step
contributes
to
the
probability
of
failure.
So
if
you
generate
test
cases
of
length
four,
as
we
saw
it's
zero,
if
you
generate
test
cases
of
length,
30
we're
around
the
peak
and
for
each
step
then
you've
got.
You
know
1.6
chance
of
fighting
the
bug,
and
this
graph
really
shows
how
the
bang
for
the
testing
buck
varies
as
the
size
of
the
test
case
grows.
C
Longer,
tests
are
quite
simply
far
more
effective
per
unit
of
effort
of
provoking
faults,
and
this
is
an
effect
that
we
have
observed
many
many
times
with
property-based
tests
and
other
people
working
in
random
testing
have
also
observed
something
similar.
So
it
seems
to
be
a
very
general
phenomenon
and
you
want
to
know
about
it
and
take
advantage
of
it.
The
way
you
take
advantage
of
it
is
by
generating
larger
tests,
but
to
make
debugging
the
results
feasible.
C
A
C
What
order
different
bits
the
model
will
get
evaluated
in,
and
people
who
try
to
use
side
effects
there
get
very
confused.
That
said,
there
are
re-implementations
of
the
property-based
testing
idea
in
virtually
every
major
programming
language,
including
the
ones
that
are
not
functional,
but
I
would
say
when
you
use
those
implementations,
you
should
use
them.