►
From YouTube: Reliable Async Systems with Coyote - Part 1
Description
Coyote is a set of .NET libraries and tools designed to help detect potential problems in your asynchronous code.
In the episode, Rich chat with Chris Lovett about the issues developers face with writing asynchronous code and how Coyote can with discovering those hard to find issues.
[00:44] - What is Coyote
[02:22] - What’s hard with testing async code?
[07:45] - Visualizing async code flow
[10:57] - Coyote TPL demo
[17:20] - Detecting deadlock demo
[26:42] - Next steps with Coyote
Coyote Homepage
https://microsoft.github.io/coyote/
Coyote GitHub
https://github.com/microsoft/coyote/
.NET Threading
https://docs.microsoft.com/dotnet/standard/threading/?WT.mc_id=ondotnet-c9-cephilli
A
A
Hey
welcome
to
another
episode
of
the
on
dinette
show
today
we're
going
to
be
talking
with
Chris
Lovett
about
how
to
write
reliable,
async
code
with
a
new
project
he's
been
working
on
called
coyote,
so
Chris
thanks
for
coming
on
the
show-
and
you
know
maybe
you
could
introduce
yourself
and
then
we'll
get
into
the
topic.
Yeah.
B
Hi,
rich
thanks,
yeah
coyote
is
a
new
project
and
I'm
all
excited
to
share
this.
With,
with
folks
get
the
word
out.
There,
we've
been
working
on
how
to
improve
the
state
of
the
art
for
testing
and
debugging
asynchronous
code.
We
have
customers
and
Azure
that
are
pulling
their
hair
out.
You
know
staying
up
all
night
trying
to
find
that
really
pesky
little
bug
to
do
with.
You
know
parallel
code
concurrent
code,
async
code,
and
you
can
get
yourself
into
some
some
really
difficult
situations.
So
coyote
is
a
project
that
came
out
of
Microsoft
Research.
B
It's
been
around
for
a
while
under
a
different
name
called
P
sharp,
which
some
people
may
be
familiar
with,
and
we're
sort
of
going
through
a
bit
of
a
product,
ization
phase
right
now,
making
this
more
than
a
research
project
with
putting
devs
on
it,
we're
putting
PMS
on
it
and
we're
really.
You
know,
making
the
customers
happy
and
not
not
just
pure
research,
and
so
that's
why
it's
rebranded
under
this.
Under
this
new
name,
it.
A
A
B
A
So,
let's
actually
back
up
a
step
which
is
you
know,
you
just
said
that
there's
lots
of
problems
like
people
can
get
into
so
like
is
async
writing
async
code
like
fundamentally
problematic,
and
should
people
not
do
it
like
I
say
you
know,
people
often
talk
about
how
cord
multi-threaded
code
is
hard
and
dangerous
is
async
like
that.
Well,.
B
I
mean
it
can
be
so
you
know,
I,
think
that
see
shop
has
done
some
really
amazing
things
with
async/await.
You
know
the
whole
task.
Parallel
library,
that's
built
into
done,
that
is
fantastic.
It
makes
writing
asynchronous
code
super
easy,
but
it
also
it's
it's
a
little
bit
like
giving
you
enough
rope
to
hang
yourself
as
well.
B
So
if
you
are
not
careful
when
you're
using
lots
and
lots
of
tasks
and
everything's
running
in
parallel,
it
can
get
tricky,
and
so
we're
coyote
I
think
is
sort
of
filling
a
gap
is
is
to
help
you
test
that
stuff
help
you
reason
about
it,
design
it
understand
what
it's
really
going
to
do
in
run
time
in
production
and
and
then
be
able
to
test
it
systematically
and
not
just
depend
on.
You
know.
Random
number
generation
not
just
depend
on
stress
testing,
which
is
like.
B
Oh
surely,
if
we
just
stress
test
this
enough,
we'll
find
that
weird
interleaving
between
those
two
tasks
that
causes
a
bug
right.
Well,
that's
not
guaranteed,
and
so
you
know
stress,
testing,
long-haul
testing,
all
that
kind
of
stuff
that
people
do
today
is
isn't
really
enough
and
that's
where
coyotes
coming
in
saying:
okay!
Well,
let's,
let's
see
if
there
is
a
better
way
to
grapple
with
this
stuff,
so
that
so
that
it
you
can
go
crazy
with
with
async/await
and
not
have
to
worry
about
it.
So
much.
B
I
can
show
you
I
can
show
you
some
sequence
diagrams
from
some
of
the
customers
in
Asia.
That
will
make
your
hair
call
it's.
It's
really
astonishing
the
amount
of
asynchronous
stuff,
because
in
the
micro
service
world
or
this
cloud
world
in
order
to
reduce
cost,
you
gotta
maximize
throughput
right.
So
there's
this
tension.
You've
got
a
in
order
to
maximize
throughput.
You
get
to
maximize
the
parallelism
and
so
they're
really
pushing
the
envelope
on
what's
possible.
A
One
last
question:
before
we
kind
of
get
into
Kairi
specifics,
which
is
you
know,
you
talked
about
dotnet
and
TPL
and
async
await.
How
would
you
come
more
general
programming
landscape
with
respect
to
this
problem?
Is
this
problem
you
know
unique
to
done
that?
Is
it
worse
or
better
with
donna?
Can
you
talk
to
that.
B
Well,
yeah
I
can
I
can
you
know
I
don't
have
experience
with
every
programming
language,
mostly
C++,
Python
JavaScript
and
in
all
the
dotnet
languages,
but
in
Java.
But
you
know,
Java
has
some
built-in
primitives,
like
synchronized
methods
and
so
on,
and
you
know
each
language
tends
to
have
its
own
way
of
dealing
with
it.
You
know
Python
basically
says
just
don't
do
it
for
the
most
part
and
JavaScript,
you
know,
which
is
a
solution
right
I
mean
a
lot
of
the
asp.net
is
a
good
example
of
where
they
create
an
environment.
A
B
Sort
of
thing
so
creating
these
higher-level
platforms
that
sort
of
create
this-
you
know
nice
little
world
that
you
can
live
in
and
ignore
concurrency
is
is
really
you
know,
probably
the
best
long-term
sustainable
solution.
Humans
have
a
really
hard
time,
thinking
about
and
just
visualizing
concurrency.
For
some
reason
we
have
a
hard
time
writing
concurrent
code.
That's
correct
and
I've
seen
team
after
team,
you
know
jump
into
it
with
Gaston
and
all
excited.
B
Let's
go
write
some
parallel
code,
and
then
you
know
it's
a
very
humbling
experience
for
that
team,
and
you
know
six
months
later,
they're
still
debugging
it
right.
So
I
think
that
you
know
c-sharp
has
done
some
fantastic
things
with
async/await.
You
know
I
always
try
to
explain
to
people
that
async/await
is
actually
building
an
async
state
machine
for
you
under
the
covers
it's
actually
a
piece
of
magic
and,
as
in
a
super
powerful
people
should
totally
use
it
task.
Dot
run,
on
the
other
hand,
is
where
you
get
yourself
in
trouble
right.
B
A
B
If
you
take
a
look
at
the
coyote
website,
I'll
put
a
link
here.
We
have
a
whole
section
on
on
documentation
here.
That
explains
what
Cody
is
about
and
what
it's
for,
but
I,
think
the
the
really
quick
way
to
give
you
a
sort
of
a
drill
down
on
what
it's
doing
is
to
look
at
this
example
program.
This
is
actually
a
fault-tolerant
cluster
of
five
servers
that
are
acting
in
a
fault-tolerant
cluster.
B
So
it's
it's
implementing
a
protocol
called
raft
and
you
can
actually
use
search
online
and
find
some
really
great
explanations
of
raft
with
some
nice
little
animations.
Now
what
this
really
does,
if
I
run
this
again
I'm
going
to
show
you
what
this
will
really
do
when
I
run
it
in
in
parallel,
so
this
is
running
sequentially,
but
if,
if
I
run
it
like
this.
B
B
Every
time
this
is
just
one
iteration
of
a
test
using
coyote
and
it's
checking
and
and
the
really
cool
thing
about
running
a
test.
This
way
is
that,
if
it
finds
about
it
can
actually
reproduce
it
a
hundred
percent
right
without
any
timing
dependencies.
It's
not
dependent
on
on
the
operating
system,
scheduling
anymore,
so
this
test
actually
found
a
bug.
It
found
a
bug
where
two
of
the
cluster
servers
elected
themselves
as
a
leader
which
it
breaks
the
protocol.
You
can
imagine
if
you're
developing
a
fault,
tolerant
cluster.
You
really
can't
have
bugs
like
this.
B
You
know
it's
super
embarrassing,
so
you
want
to
make
sure
it's
thoroughly
tested
and
that's
where
we're
coyote
can
help.
Now
that
was
just
one
test.
Iteration
now
imagine
running
millions
of
these
test
iterations,
because
you
know
the
state
space
of
all
of
those
tasks
and
interleavings
of
those
tasks
is
huge
right.
It's
an
exponential
explosion
problem.
B
So
what
coyote
has
done
is
it's
built
in
some
in
intelligence,
scheduling,
algorithms
that
were
developed
by
Microsoft
Research
to
help
search
this
state
space?
So
you
can
think
of
this
as
a
search
problem,
massive
amount
of
possibilities
that
you're
searching
and
you
have
to
find
the
bug-
that's
hiding
in
there
somewhere
and
that's
what
coyote
is
really
good
at.
B
B
Just
does
a
small
delay
and
sets
a
local
variable
to
the
string
that
comes
in
here
and
I'm
going
to
pass
it
different
strings,
hello,
world
and
good
morning,
and
when
I
run
this
in
parallel,
one
of
those
tasks
is
going
to
run
last
right.
It
has
to
one
of
them
has
to
be
scheduled
at
the
end
and
whoever
runs
last
is
the
one
that's
going
to
actually
be
the
final
setter
of
this
value
and
down
at
the
bottom
I'm
going
to
assert
that
that
value
is
equal
to
hello
world.
B
B
Right
exactly
right,
so
this
is
a
classic
example
of
non-determinism.
Obviously
it's
a
very
trivial
example,
but
non-determinism
happens
in
our
programs
all
the
time
when
we're
doing
parallel,
programming
and
so
coyote
then
can
actually
come
along
and
test
this
thing
and
what
I
can
do
is
I'll
use
the
coyote
tool.
I
can
actually
install
it
with
net
tool,
install
there's
a
new.
You
know
done
that
tools
are
really
cool,
so
we've
got
one
called
Microsoft
coyote
dot,
CLI
command
line
interface,
I'll
just
make
it
global.
For
now.
B
And
pretty
soon,
I'll
have
the
coyote
command-line
tool,
which
has
some
command-line
arguments.
What
this
tool
can
do
is
it
can
actually
test
this
program.
But
let
me
explain
one
thing:
first,
which
is
in
order
to
test
this
program.
What
coyote
wants
to
do
is
systematically
test
every
possible
interleaving
one.
That
means
is,
you
know,
task
1,
then
3,
then
2,
then
4,
then
5
in
one
iteration,
the
next
test.
Iteration
that
might
test
this.
You
know
ordering
of
tasks
and
and
so
on
and
so
forth.
B
You
can
imagine
this
is
a
long
list
of
possibilities
and
that's
just
one
set
of
possibilities.
There's
actually
multiple
scheduling
points
in
here
because
of
the
task
delay,
so
it'll
actually
test
every
interleaving
of
asynchronous
operations
in
this
program.
In
order
to
do
that,
what
we've
done
is:
we've
created
a
drop-in
replacement
for
the
net
task,
called
coyote
tasks,
Microsoft
upgraded
up
tasks,
so
notice
I'm,
not
using
system
decoding
the
tasks
here
instead
I'm
using
this
one.
B
B
We've
got
right
there
and
I'll
give
it
a
hundred
iterations.
Maybe
that
will
be
enough
time
for
to
discover
the
interleaving
that
results
in
the
SS
so
to
fail
right
and
this,
this
coyote
test
tool
is
only
useful
if
it
can
reliably
reproduce
that
boat,
as
opposed
to
totally
depending
on
random
operating
system.
Scheduling.
B
Okay,
there
you
go,
it
did
not
reproduce
the
bug.
Let's
give
it
more
iterations,
which
you
know
it's
obviously
one
thing
you
can
do:
okay,
after
iteration
60
this
time
they
had
found
the
bug,
and
you
get
a
little
report
here.
It
gives
you
the
call
stack
and
then
it
gives
you
the
assertion,
failure
that
happened
here
now.
What's
really
cool
about.
This
is
not
that
I
found
the
bug,
but
the
really
cool
thing
is
that
you
can
now
replay
this
book.
B
So
what
we
do
is
we
don't
we
produce
that
log
file,
but
we
also
produce
a
schedule
of
all
of
the
choices
that
were
made
in
that
in
that
iteration
that
failed,
so
iteration,
60
I,
guess
it
was
that
failed,
we'll
have
a
schedule
associated
with
it,
and
now
I
can
replay
that
bug
exactly
and
I
can
replay
this
in
the
debugger
and
I
can
step
through
this
program
doesn't
matter
how
many
breakpoints
I
hit.
It
will
always
guarantee
to
hit
the
button,
which
is
super
cool.
B
You
can
imagine
how
that
would
be
a
lifesaver
in
situations
I
mean
we've
literally
had
customers
that
use
coyote.
That
they've
told
us
that
they've
been
trying
to
find
a
bug
for
months.
There
was
that
tricky,
and
then
they
got
it
all
up
and
running
in
psycho
Eddy
and
they
found
it
in
in
minutes
and
were
able
to
debug
it.
So
this
can
save
you
a
lot
of
time.
So.
B
B
So
hopefully
that
makes
sense
now
what
we
did
is
you
know,
let's,
let's
show
a
slightly
more
complicated
example.
So
there
was
this
extreme
programming
challenge
that
was
put
out
there
by
a
guy
named
Tom
Cargill,
and
in
that
challenge
he
said
things
like
you
know.
Concurrent
programs
are
hard
to
test
because
the
combat
combinatorial
explosion
of
the
state
space
that
must
be
covered
yeah
absolutely
this
is
this
is
like
he
was
writing
this
intro
paragraph.
B
But
if
somebody
then
does
a
put,
then
the
notify
will
wake
up
that
that
guy
and
he
said
that
these
kinds
of
programs,
this
program
actually
has
a
really
nasty
deadlock
and
he
was
asking
the
people
hate,
find
the
deadlock
for
me,
figure
out
a
test.
You
know
an
extreme
testing
methodology
that
will
guarantee
that
people
that
are
moving
in
a
really
agile
environment.
Don't
ship
bugs
like
this,
and
so
we
thought.
Okay,
that's
a
that's
a
fantastic
example
for
for
coyote.
B
So
what
we
did
is
we
ported
that
program
to
c-sharp,
because
it
was
originally
written
in
Java.
C-Sharp
doesn't
have
synchronous
methods,
but
it
has
something
very
similar.
Actually,
it's
more
powerful,
which
is
a
synchronizing
object
that
you
can
use
in
a
lock
statement.
Everything
else
is
available,
so
the
system
threading
monitor,
has
a
weight
method.
So
if
the
buffers
full
we
can
wait
and
pulse
is
how
you
notify
the
other
threads
that
they
can
continue
and
sure
enough.
B
When
you
run
this
program,
it
looks,
you
know
all
looks
fine,
here's,
our
buffer
and
so
on.
It's
a
straight
straight
port
and
sure
enough.
If
you
run
this
program
in
c-sharp
it
deadlocks
sometimes-
and
you
know
how
often
the
deadlocks
is-
is
unknown,
and
how
do
we
produce
that
deadlock
is
really
hard?
You
know
it
absolutely
does
not
reproduce
a
deadlock
in
the
debugger.
If
you
start
putting
logging
statements
in
here
to
understand
what's
happening,
it
stops
repoing,
it's
one
of
those.
B
You
know,
pull
your
hair
out,
stay
up
all
night
kinds
of
bugs,
and
so
what
we've
done
is
we've
used
coyote
to
see
if
we
can
find
this
bug.
So
what
we?
What
we
do
with
coyote
then,
is
we
have
a
drop-in
replacement
for
system
monitor.
So,
instead
of
using
the
system
that
monitor,
we
actually
used
a
coyote
type
called
synchronized
block
and
since,
as
the
c
chap
lock
statement
doesn't
know
about
it,
we
have
to
use
it
this
way.
So
it's
so
using
synchronized
block.
Everything
else
looks
the
same.
B
So
there's
a
wait,
there's
a
pulse
and
so
forth.
So
it's
pretty
much
the
exact
same
code,
except
that
it's
a
type
that
coyote
can
control
in
a
systematic
testing
kind
of
way.
So,
but
now
the
question
that
was
posed
on
the
on
extreme
programming
challenge
is:
how
do
you
even
test
this
thing?
How
do
you?
How
do
you
test
to
even
know
that
the
bug
is
there?
It
was
interesting
if
you
look
at
the
discussion
online,
a
lot
of
people
were
we're
trying
to
test
it
and
concluding
hang
on.
B
So,
what's
the
minimal
repro
that
can
demonstrate
the
bug
in
the
code,
so
what
we
did
with
coyote
is
we
use
coyote
random
number
generator
to
generate
a
random
buffer
size,
a
man,
a
number
of
reader
threads,
a
random
number
of
writer
threads
and
random
number
iterations,
and
each
one
of
those
threads
and
so
on.
So
when
you
use
the
coyote
random
number
generator,
it's
something
that
the
coyote
test
tool
can
play
with
those
random
choices
and
explore
that
space.
B
Then
we're
able
to
create
a
bounded
buffer
with
that
given
size,
we're
able
to
add
all
of
the
reader
tasks.
All
of
the
writer
tasks
where
reader
and
writer
is
just
done
here
so
reader
is
just
something
that
does
a
certain
number
of
reads
and
writer
is
a
certain
number
of
puts
and
takes
right.
So
that's
that's
the
best.
This
is
now
going
to
explore
lots
of
different
ways
of
testing
this
thing,
and
we
can
run
that
like
this.
A
B
A
buffer
DLL
now
this
guy
has
some
different
test
methods.
So
it's
going
to
ask
us
to
choose
one
and
I
will
choose
the
test
method
named,
find
deadlock
and
lets.
You
know
the
cool
thing
about
this
exploration
is
I
can
really
now
just
go
to
town
and
say
you
know,
run
a
million
iterations
and
until
you
find
a
bug
and
the
other
thing
you
can
do
is
you
can
tell
it
to
keep
running
and
find
all
the
different
test
configurations.
B
What
this
will
do
it
found
it
found
a
configuration
that
fails
and
that
what
it
does
is
it
prints
out
what
that
was,
so
it
prints
out
that
the
buffer
size
was.
We
had
five
threads
that
were
reading
three
threads,
that
were
writing
and
so
on
and
so
forth.
Now,
that's
great!
That's
nine!
My
five
plus
three
is
eight
tasks,
not
too
bad,
but
what's
the
minimal
tests
and
what
we
can
do
is
we
can
say
explore
so
instead
of
stopping
on
the
first
bug,
just
keep
running,
and
so
we
did
this
and
ran
it.
B
You
know
you
can
actually
run
it
overnight
if
you
want,
but
you
can
run
it
long
enough
that
you
could
get
a
whole
bunch
of
different
test
configurations
and
it's
finding
a
lot
of
bugs
here
right,
which
is
great.
So
we
did
that
and
we
found
the
minimal
test,
which
is
this
one,
which
we
can
then
code
up
the
minimal
test
once
you
know
what
it
is
and
it's
a
buffer
size
of
one,
it
is
to
read
threads
and
one
writer
thread.
Okay,
that's
pretty
cool,
and
let's
see,
if
that
one
fails.
I.
B
Will
just
do
this
again
to
get
the
name
of
that
test
method,
M,
minimal
deadlock
and
with
this
one
I
can,
you
know
just
run
100
iterations
should
be
enough
sure
enough,
like
iteration
1,
it
already
hit
the
bug
all
right,
so
cool.
Now
we
have
a
minimal
testing
and,
as
I
said
before,
you've
got
a
replayable
schedule.
Now
the
the
really
cool
thing
about
that-
and
we
have
it
listed
up
here-
is
that.
B
Once
you've
got
the
minimal
repro,
you
can
actually
go
through
and
instrument
it
with
all
kinds
of
logging,
because
you've
got
that
scheduled
file
right.
So
now,
I
can
log
every
detail
and
that's
what
I
did
here
in
in
this
program-
and
this
is
a
little
blog
here.
That
explains
what
we
did
and
I'm
just
looking
for
the
graph
that
shows
the
bug
so
I
was
I
was
able
to
inject
logging
inside
the
they
take
and
the
put
methods
every
section
of
those
methods
where
it
you
entered.
B
Where
you
waited
where
you,
the
wait,
was
complete,
where
it
was
actually
doing
something.
The
pulse,
the
exit
of
the
method
and
both
methods
that
the
take
in
the
put
were
instrumented
the
same
way
and
then
I
run
their
schedule.
Right
I
run
that
replayable
schedule
to
see
what
happened,
and
we
can
see
here.
This
explains
you
know
why
it
was
deadlocking,
so
you
got
to
reader
threads
that
are
waiting.
The
take.
The
put
thread
goes
through
and
succeeds.
It
sends
a
pulse
over
to
here
now.
B
What's
interesting,
is
it's
it's
trying
to
wake
up
a
reader,
but
what
I
didn't
know
was
possible
with
system
threading
monitor
is
that
this
blue
thread
was
able
to
continue
running
and
get
back
into
and
re-enter
that
lock
hit.
The
wait
state
before
the
reader
thread
woke
up,
so
there's
a
little
delay
in
time
there
that
happens.
B
When
you
do
a
pulse,
it's
not
a
it's,
not
predictable,
it's
not
guaranteed
and
that's
why
it's
so
sensitive
to
debugging
and
logging,
and
so
on
that
that
timing
condition
has
to
be
just
right
for
it
to
hit
that
deadlock,
and
this
woke
up
the
reader
thread
which
was
able
to
receive
that
item.
It
then
sent
a
pulse
and
here's.
The
real
problem
is
that
the
pulse
woke
up
the
wrong
task
it.
B
It
woke
up
this
reader
that
was
waiting
and,
of
course,
they
buffers
now
empty
because
the
buffer
size
was
1,
and
so
the
this
reader
thread
just
has
to
go
straight
back
to
sleep,
and
then
the
orange
thread
goes
back
around
the
loop
of
iterations
and
re-enters
the
lock,
and
it's
now
waiting.
So
we've
got
all
three
tasks
waiting
and
you
have
a
deadlock.
So
this
is
an
absolutely
crystal-clear
explanation.
Now
why
this
thing
is
dead,
locking
so
I
think
you
can
now
see.
B
A
A
A
B
That
is,
you
know,
helping
it
to
intercept
these
types
at
test
time
at
runtime
in
production,
the
Micra
coyote
namespace
is
actually
a
very,
very
thin
wrapper
on
top
of
system
that
threading
the
tasks.
So
it's
it's
just
delegating
directly.
Until
you
run
inside
a
coyote
test,
then
you're
you're
using
the
coyote
systematic
test
engine
so
yep
there
is
a
dependency
there
on
Microsoft,
a
coyote.
If
that
doesn't
work
for
your
team,
then
you're
out
of
luck.
B
We
are
investigating
other
ways
of
doing
this,
where
it
does
not
require
a
runtime
dependency,
but
for
now
this
is.
This
is
how
it
is.
We
have
teams
in
Azure
that
have
been
using
this
for
quite
a
while
in
production,
so
we're
pretty
confident
that
this
is
pretty
rock-solid
code
goes
through
full.
You
know
Azure
DevOps
pipelines
of
continuous
integration
testing.
B
We
have
huge
test
code
base
for
this
thing,
we're
pretty
confident
that
that's
it's
good
to
take
a
dependency
on
it,
but
you're
correct
that
some
people
may
not
be
able
to
in
terms
of
getting
started.
Then
yeah,
all
you
need
to
do
is
add
a
reference
to
that
NuGet
package
and
start
coding.
You
you
write
the
you
know
you
drop
in
the
system,
coyote
tasks
types
and
then
you
use
that
command-line
tool
that
I
showed
you
to
systematically
test
your
program.
B
So
one
thing
that
I
didn't
I
glossed
over
a
little
bit
before
is
how
does
Kody
know
what
to
test
and
that's
done
with
a
test
attribute.
This
is
a
little
bit
like
what
you
would
see
with
X
unit
or
in
unit
or
something
where
you
have
a
test
attribute
on
a
method,
and
that
tells
coyote.
This
is
the
entry
point.
Coyote
has
some
other
things
to
help
with
logging
and
random
number
generation,
and
so
on.