►
Description
Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/debugging-the-ethereum-merge-with-parallel-universes/
The Ethereum merge is a high-stakes event for the entire industry, and requires several large, stateful, distributed software systems to behave flawlessly in order for it to succeed. Unfortunately, it’s very difficult to develop distributed stateful systems like a blockchain with high confidence. Many of the worst bugs in these systems don’t show up in the happy case, but require particular network, hardware, or timing conditions in order to manifest.
Speaker(s): David Searle, Will Wilson
Skill level: Beginner
Track: Layer 1 Protocol
Keywords: Merge,testing,fuzzing
Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/
Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/
A
A
If
you're
in
the
wrong
room,
this
room
is
talking
about
testing
the
merge
with
parallel
universes.
My
name
is
David
Searle
and
I'm,
the
head
of
Amir
for
a
small
startup.
That's
been
working
with
the
ethereum
foundation
for
just
over
a
year
now
who
likes
Bugs.
Okay,
I
brought
some
bugs
along
with
me
today,
so
you're
going
to
have
to
be
like
who
would
like
a
bug,
because
I've
got
some
bugs
that
actually
I'd
like
to
give
to
various
members
of
the
audience
anybody
you'd
like
one
okay.
A
A
I've
got
three
more
to
go
there
we
go
see
so
the
it
talks
about
testing
ethereum
and
is
is
basically
looking
at
using
a
pretty
sophisticated
and
unique
testing
methodology,
which
is
based
from
a
company
called
antithesis
which
I
belong
to
the
the
company
is
involved
in
using
deterministic
simulation
techniques
to
allow
us
to
basically
explore
a
whole
host
of
iterations
of
of
complex
distributive
technology,
that's
running
in
a
simulated
environment.
So
let
me
just
get
my
my
little
Clicker
I
brought
a
few
Stars
along
to
the
show.
A
Obviously
I've
just
explained
who
I
am
little?
Did
you
know
that
doctor
strange
and
Cher
are
also
going
to
participate
in
this?
This
presentation
I've
got
about
25
years
worth
of
experiences
in
the
tech
industry.
For
the
last
two
and
a
bit
years
now,
I've
been
working
for
this
business
antithesis
and
they've.
A
You
know
basically
made
me
into
somebody
of
that.
There's
a
bug
Hunter
buyer
by
trade,
I,
never
thought
testing
could
be
fun,
but
there's
nothing
like
hitting
a
seg
fault
and
actually
getting
a
real
kick
out
of
it
and
that's
what
I
do
on
a
day-to-day
basis.
This
is
my
house
in
the
UK.
You
can
tell
from
my
my
accent
and
as
you
can
see,
I've
mostly
got
a
few
gpus
I've
got
to
sell.
A
So
if
you're
interested-
please
let
me
know
no
joke,
but
some
the
the
day
of
the
merge
came
round
and
we've
been
doing
a
lot
of
work
with
with
all
the
various
clients,
all
that
hard,
arduous
testing,
making
sure
that
we
call
every
last
bug
what
was
going
to
happen.
Was
it
really
going
to
be
smooth
sailing?
Yeah
I
think
we
saw
this
morning,
which
obviously
is
great
news,
but
yeah
I
was
wondering
if
we
were
going
to
see
this.
You
have
to
press
play
on
the
the
video.
A
Yeah,
it's
pretty
bad
eh,
not
my
fun
Garden
again,
so
yeah
testing
is
really
important,
and
this
is
where
you
know
we
talk
about
it,
but
the
merge
was
a
significant
event.
That's
why
we
had
so
so
much
celebration
across
the
world
about
kind
of
what
was
really
going
on
when
when
the
modes
took
place.
So
congratulations
to
anybody
that
was
involved
in
the
merge.
You
know
phenomenal
job
and
yeah
I
think
we
can,
you
know,
hold
our
heads
up
high
there.
A
So
I'll
say
testing
is
hard
and
you
only
got
to
use
Cura
to.
Basically,
you
know
ask
a
question:
you
know
what
you
know.
What
makes
software
testing
difficult
yeah
it's
a
bit
of
a
high
level
question.
You
know!
Apologies
for
that,
but
you
know
these
two
really
kind
of
sort
of
struck
me
here
like
the
first
one.
You
know
this
is
the
basically
fundamentally.
This
is
an
impossible
requirement,
because
absence
of
evidence
is
not
the
evidence
of
absence,
and
so
often
we
run
these
simulations
in
parallel.
B
A
And
you're
like
well
is
that
good?
Who
knows
like
you
know
we,
we
can
only
really
understand
how
much
coverage
we've
got
and
it
may
be
maybe
good,
maybe
bad.
Who
knows
so?
That's
that's.
A
phenomenal
kind
of
you
know.
Kind
of
you
know
requirement
the
the
second
one
is,
then
about
kind
of
understanding
like
testers,
have
to
think
about
all
those
possible
scenarios
where
issues
may
arise
and
ensure
that
they
are
handled
by
the
code,
and
you
know
that's
just
not
it's
just
like
how
do
you?
How
do
you
approach
that
with
a
distributed?
A
You
know
kind
of
architecture
like
we
have
with
ethereum.
We've
got
a
whole
host
of
different
clients
using
different
architectures.
We've
got
the
execution
layout,
we've
got
the
consensus
layer,
it's
just
it's
amazing
to
see
it
actually
working
and,
and
but
the
complexity
of
that
and
the
is
just,
is
just
phenomenal.
A
If
we
then
look
at
kind
of
what
ethereum
has
done
to
approach
this
task-
and
this
is
you
know,
this
is
not
just
looking
specific
antithesis.
We
are
a
part
of
the
equation
and
I.
Think
we're
very
much
of
a
complementary
is.
Is
that
ethereum
is
using
unit
testing
test
Nets
we've
got
a
list
of
Shadow
forks
and
test
net
mergers
occurred,
some
technologies
that
are
being
used
called
Hive
and
ketosis
that
do
similar
heaters
to
the
equation.
A
We
then
have
antithesis
that
is
then
doing
this
deterministic
piece
which
allows
us
to
to
Really.
You
know
get
into
the
detail
of
of
all
the
different
iterations
that
are
possible,
and
then
we
we
have
a
host
of
different
fuzzing
technologies
that
the
ethereum
are
using
to
to
really
try
and
isolate
and
kind
of
hammer
areas
that
are
of
concern.
A
So
you
know
throughout
the
entire
year
you
know
a
huge
amount
of
testing
has
been
done
and
on
top
of
that,
they've
also
used
static
analysis
to
just
you
know,
help
with
doing
code
Audits
and
making
sure
that
we
catch
things
before
the
merge
was
to
take
place,
I'd
step
back
a
little
bit
in
terms
of
kind
of
the
testing
as
hard
statement.
You
know
it
is
a
phenomenal
kind
of
like
you
know,
obstacle
to
try
and
get
over
like
how.
A
How
do
we
find
every
iteration
and
how
do
we
find
those
hard
to
reach
bugs
that
you
know,
may
not
be
common
ground,
but
if
they
do
occur
will
be
catastrophic,
and
so,
if
we
just
look
at
the
right
hand
side,
this
is
a
representation
of
a
distribution
Network.
So
on
the
right,
we're
going
to
we're
going
to
call
this
node
we're
going
to
call
this
node,
you
know
using
Lighthouse
and
Geth,
so
the
lighthouse
and
gether
running
on
this
node
they're
running
on
an
operating
system.
A
A
On
top
of
that,
the
the
complexity
of
all
those
different
communication
channels
operating
just
means
that
the
search
space
for
Hidden,
Away,
Little,
Gems
I,
call
them
gems,
but
you
know
they're
bugs
basically
those
those
they
can
hide
away
for
a
long
time,
and
you
may
never
find
that
search
when
you're.
Looking
at
testing,
you
may
never
find
that
that
particular
condition
where
the
network
was
slow
and
there's
some
there's
a
bit
of
thread.
A
We
have
the
ability
to
kind
of
run
the
entire
collection
of
consensus,
clients
and
execution
clients
in
in
a
simulation,
run
it
and
and
understand
exactly
kind
of
every
way
in
which
we
can
actually
kind
of
see
outcomes.
That's
pretty
intense,
you
know.
How
do
you
do
that
and
fundamentally,
how
do
you
do
that
in
a
deterministic
fashion?
If
we
find
a
certain
situation
where
Lighthouse
crashes
well,
can
we
represent
that
again?
Can
we
replay
it?
A
Can
we
yes,
the
first
thing
you
do
right:
can
we
reproduce
it
and
often
bugs
are
not
reproducible?
They
often,
you
know
they
might
try
something
different.
They
might
move
around.
What
like,
how
do
we?
How
do
we
debug
this,
and-
and
this
is
again
where
we
step
in
so
what
do
we
do?
We
leveraged
the
ability
to
Auto,
generate
networks
inside
the
simulation
environment
and
stand
up
all
the
containers
that
were
necessary
to
bring
up
ethereum.
A
We
then
were
able
to
then
use
Network
faults
and
other
types
of
faults
to
just
inject
chaos
into
that
running
environment.
Obviously,
it's
great
to
see
the
software
handle
that
it's
designed
to
handle
these
types
of
things.
So
you
know
under
duress
we
see
things
going
wrong
and
we
we
basically
use
fuzzing
technology
to
to
basically
hit
the
entire
system,
not
just
individual
pieces
of
software,
but
the
entire
system
is
fuzzed
and
that
allows
us
so
basically
kind
of
deterministically
replay
the
complete
orchestration
of
a
situation,
not
just
one
particular
application.
A
But
we
know
that
in
this
combination,
different
clients
under
these
Network
conditions,
something
goes
wrong,
and
so
we
use
strategies
inside
the
system
to
allow
us
to
seek
rare
events.
And
so
when
you,
when
I
start
looking
at
the
numbers
and
kind
of
what
we've
been
doing
over
the
last
year,
you
know
we
hear
a
huge
amount
of
code.
Edges,
like
you
know,
a
huge
amount,
and
so
how
do
we?
A
A
A
We
continue
to
promote
that
and
say
keep
keep
going
with,
that
we
have
the
test,
Nets
and
Shadow
Forks
again,
I
think
you
know
we
can
all
say
that
they've
been
tremendously
successful
and
useful,
and
then
we
have
our
parallel
universes.
We
have
the
ability
to
run
not
just
one
simulation
but
we're
running
simulations
literally
every
day
that
are
generating
and
using
you
know,
exploring
the
various
search
spaces
that
exist
across
any
number
of
different
branches
inside
the
the
repositories.
So
what
does
this
mean
for
us?
A
One
so
so
for
us
winning
is
finding
these
edge
cases
right
now,
your
winning
isn't
like.
Oh
all,
tests
passed
yay.
We
want
to
find
those
those
really
intricate
kind
of
piece.
You
know
combinations
and
iterations,
where
we've
covered
14
million
different
scenarios
and
we've
got
you
know,
maybe
not
more
than
one,
but
we
at
least
have
one.
You
know
we
can
actually
hold
our
hat
on.
So
that's
that's
an
example
of
what
we're
doing
and
we
think
that
kind
of
represents.
You
know
pretty
pretty
good
sort
of
analogy
if
we
move
forward.
A
A
The
branches
are
kind
of
how
we
get
decision
points
we
we
get
like
an
if
then
else
on
a
kind
of
statement
inside
the
code
very
simple
example,
but
just
just
take
it.
We
can
basically
Branch
off
at
that
point
and
exhaust
both
Avenues
and
actually
see
what
happens
in
both
situations.
Underneath
those-
and
you
know,
under
those
test
conditions,
that's
an
example
of
a
branch
and
obviously
the
code
The
Edge
is
seen.
A
Is
we've
got
the
instrumentation
happening,
so
we
can
see
what
kind
of
functions
and
and
pieces
of
code
are
actually
being
executed.
Pretty
pretty
insane.
You
know,
that's
one
run
I've
got
180
000
edges
that
we're
seeing
across
across
the
entire
network.
It's
a
busy
Network
and
obviously,
with
all
the
different
clients
happening.
That's
there's
a
lot
going
on,
so
we
see
ourselves
as
a
complimentary
piece.
A
We
you
know,
we
we've
we've
been
looking
at
this
as
a
you
know,
a
great
example
of
a
project
where
we
can
basically
work
alongside
all
the
other
pieces
that
the
EF
are
kind
of
using
and
different
client
teams
are
using
so
we're
just
another
layer
in
in
the
growing
list
of
of
of
strategies
being
being
used
and
I
think
that's
sort
of
Testament
to
the
approach
that
EF
have
used
to
to
Really
hold
on
to
this
resilience.
You
know
making
sure
that
we
take
advantage
of
every
way
we
can.
A
We
can
make
sure
that
you
know
nothing
untoward
happens
in
the
future.
You
know
more
more
and
more
upgrades
coming
I'm
sure.
So
you
know,
we've
only
got
4844
around
the
corner,
I'll
say
around
the
corner,
we'll
see,
but
it's
certainly
something
to
look
at
in
terms
of
you
know,
making
sure
that
we're
involved
at
every
step
of
the
way.
A
So
this
is
a
very
simple
sort
of
illustration
of
what
we've
been
doing.
We've
been
building
all
of
the
clients
giving
them
in
in
place
and
actually
establishing
Genesis.
So
we
actually
have
been
using
this
at
Genesis
block
and
then
moving
it
forward
towards
the
merge
in
our
in
our
world.
We
would
have
the
merge
kind
of
positioned.
A
A
That's
a
great
one,
obviously,
to
encourage
forking
of
the
chains.
We've
then
also
got
you
know:
delays
happening
drops
of
packets,
the
nodes
themselves.
We
can
stop.
You
know
we
do.
We
do
we
put
this
thing
through
its
faces.
We
stop
things.
We
pause
things,
we
kill
things,
we
bring
them
back
up,
I
suppose
in
some
respects,
it's
quite
realistic.
You
know
things
do
get
rebooted
and
violators
come
back
up
again.
A
We
we
have
all
that
kind
of
like
happening
inside
the
simulation
and
then
from
that
we
then
have
you
know
any
number
of
threads
being
paused
released.
There's
there's
yeah
it's
it's
busy.
Every
type
of
this
sort
of
configuration
of
faults
that
we
put
into
a
simulation
is
complete
again
completely
deterministic.
So
if
we
know
that
this
ended
up
with
a
segful,
we
know
we
can
literally
put
exactly
the
same
into
the
into
the
environment.
A
It's
basically
creating
a
pure
function
of
the
entire
system,
and
we
know
we
can
get
the
exact
same
output,
which
I
think
is
a
tremendous
piece
of
technology
which
we
have
under
the
hood.
What
the
numbers
like
well
yeah,
you
know.
In
one
year
we've
we've
conducted
31
years
worth
of
24
7
non-stop
testing
that
that
has
processed
and
and
explored
over
50
million
edges
of
code.
You
know
out
of
that,
and
you
can
see
there.
You
know
out
of
that.
A
A
It's
been
great
to
see
our
efforts
put
to
put
to
the
test,
as
he
wants
an
example
or
I'll.
Give
you
I'll
give
you
some
so
here's
here's
an
output
of
our
of
our
sort
of
one
of
the
runs.
This
would
come
through
from
an
email,
and
hopefully
you
can
see
it's
Biven.
Our
chart
you've
seen
the
side
of
the
select
the
stuff
on
the
right
hand
side,
but
down
here
you
can
see
a
whole
host
of
assertions
that
we've
got
running
across
across
the
entire
network.
A
This
this
is
actually
flagged
up
by
fail
and
they've
got
a
fail
on
a
seg
fault.
This
is
this.
Is
you
know
something
that
is
is
absolutely
real:
zigzag,
V,
illegal
storage,
access
attempt
to
read
from
nil
question,
mark
I,
guess
I'm
really
interested
ones
you
get
like
you.
Please
report
this.
Oh,
please
repeat,
please
fix
this,
so
this
is
just
an
example
here
and
we're
like
What.
Do
we
do
now?
How
do
we?
How
do
we
take
it
to
the
next
step?
Because
it's
great
saying
well,
you've
got
a
problem.
A
This
you
can
see
here
is
actually
for
the
node,
that's
running
Nimbus
and
Geth.
So
we
know
there's
an
issue
there.
So,
okay,
fine,
have
you
got
any
other
occurrences
on
any
of
the
combinations
we
do
so?
Okay,
that's
cool!
Well,
let's,
let's
look
at
an
example
of
an
actual
kind
of
log
entry.
That's
coming
through.
So
this
is.
This
is
a
unified
log,
the
serialized
activity
that's
occurring
across
the
entire
simulated
environment
bit
of
mouthful,
so
you
see
you
can
see
here
on
the
third
column,
the
blue
column.
A
Here
again,
this
is
just
showing
us
all
the
different
combinations,
so
we've
got
Cloud
we've
got
prism,
client
Aragon,
we've
got,
there's
nimbers,
never
mind
nimbuspessu
all
kind
of
serialized,
and
you
can
see
that
we've
got
a
concept
of
time
running
through.
So
this
blown
up
piece
is
basically
the
stack
trace
of
showing
us
kind
of
what
is
happening
and
Nimbus
is
having
some
issue.
It's
receiving
some
Json
trying
to
figure
out.
A
A
Again
I,
don't
love
sort
of
seeing
panics,
but,
like
you
know
it
does,
it
does
show
that
there's
something
like
of
value
here
that
we're
really
stressing
the
the
entire
environment.
So
again,
invalid
memory
address
yeah,
okay,
cool.
What
do
we
do
now?
You
know:
do
we
descend
it
to
the
client
team?
Well,
yeah
you
can
and
they
might
go.
I
I
wrote
that
code
I
can
see
what's
going
on
here.
A
Well,
she
had
to
do
it.
Didn't
she
yeah
if
she
could
turn
back
time.
Well,
the
great
news
about
deterministic
environments
is,
you
can
turn
back
time
right,
so
we're
going
to
basically
jump
into
the
actual
environment
and
actually
say,
okay
right.
How?
When
does
this
manifest?
Like?
Is
it
okay?
It
crashes
was
it
a
second
ago?
Was
it
five
seconds
ago
what
happened
in
the
execution
pass
of
all
the
different
Opera
operating
nodes
with
all
the
fall
injection
going
on
or
or
was
it
yeah?
What?
Where,
where
do
we?
A
Where
do
we
see
this
like
actual
kind
of
thing
manifest,
and
so
we
we
want
to
look
back
so
many
seconds.
We
may
want
to
turn
on
car
packet
capture
pack
capture
isn't
by
default
just
because
the
amount
of
data,
but
it
isn't
turned
on
by
default.
We
want
to
be
quite
selective
about
kind
of
where
we
see
packet
capture
happening
and
and
then
we
can
look
at
the
data
see.
If
again,
we
can
rerun
it
again.
Obviously,
determinism
you're
stressing
a
little
bit
of
kind
of
will.
A
It
still
be
deterministic
if
I
suddenly
turn
on
packaged
at
a
moment
in
time.
Let's
have
to
look
at
that,
but
but
what
we've
seen
is
actually
yeah
you
generally,
if
you've
seen
something
you
know
manifest,
then
we
can.
We
can
actively
kind
of
kind
of
turn
on
things
to
help
kind
of
debug
and
work
out.
What's
going
on,
there's
some
really
interesting
stuff
coming
through
next
year,
which
I
can't
share
but
really
cool,
so
this
is
what
we
can
do
today.
A
This
is
what
we've
done
with
the
with
the
is
basically
look
at
this
mountain
of
data.
Like
you
think
of
all
the
stuff,
that's
going
on
all
the
different
scenarios,
branches,
Paths
of
execution
that
have
happened.
We
can
we
have
all
these
available
in
a
big
massive
data
set.
So
we
can.
We
can
look
at
this
kind
of
analysis
and
look
at
the
x-axis,
which
is
your
time
versus
the
probability
of
the
bug
occurring
on
the
y-axis.
Now
the
top
right
is
where
the
bug
is
seen.
A
We
can
then
look
at
the
common
routes
that
are
happening
across
all
the
data
and
try
and
then
bring
it
together
to
see
what
is
happening.
You
know
where,
where
does
the
execution
occur?
Where
suddenly
the
probability
of
this
actually
becoming
a
bug,
yeah
happen,
and
and
lo
and
behold
we
have
a
huge
jump
here
from
literally
0.05
of
kind
of
the
the
kind
of
bug
occurring
to
over
50
this
jump
here
in
the
middle.
We
can
start
to
go
okay
right.
We
know
how
many
seconds
to
go
back
now.
A
A
So
what's
next
well,
merger's
great
love,
it
we're
doing
some
Pokemon
testing
on
some
stable
branches
which
isn't
you
know,
we've
got.
You
know
things
in
a
fairly
good
order
there,
but
we
know
ethereum's
not
standing
still.
We've
got
obviously
EIP
4844
just
around
the
corner,
I
say
again,
thanks:
sharding
withdrawals,
you
know
a
whole
host
of
new
capability,
which
is
you
know,
brand
new
products,
brand
new
code,
brand
new
testing,
there's
other
pieces
in
there
around
you
know.
Actually
the
one
that's
interesting
is
using
things
like
malicious
clients.
A
A
Does
it
operate
in
a
different
manner,
so
other
pieces
here
you
merge
clothes
clean
up,
you
know,
there's
changes
to
you,
know
kind
of
established
pieces
of
code
that
are
in
in
place
now
all
those
changes
introduce
you
know
probably
good,
and
it
could
appear.
You
know
Downstream,
so
pretty
cool
we're
working
actively
with
the
clients
and
open
to
really
you
know
kind
of
broadening
that
relationship
further.
So
that's
the
end
of
that
q.
A
any
questions
yeah.
D
A
That's
a
good
question,
so
the
just
because
the
amounts
of
data
we've
got
available.
We
we
know
every
kind
of
path,
execution,
that's
occurring,
that
the
outcome
doesn't
occur
with
a
bug
and
we
have
then
obviously
all
the
different
outcomes
that
do
translate
to
the
bug
happening
and
so,
where
the
ability
to
you
know
do
some
simple
calculation.
We
can
see
that
you
know
the
path,
the
path,
that's
sort
of
trodden
and
worn-
and
you
know
this,
but
this.
If
this
is,
if
we
see
this
happening,
then
we
can
calculate
the
bug
occurring.
A
A
That's
a
good
question,
and
so
we
we
see
a
huge
amount
of
bugs
that
do
manifest
and
show
us,
like
you
know,
duplication
like
you
know,
like
the
one
you
saw
with
nimbuson
and
Geth,
that
we
can.
We
can
run
that
literary
day
in
there
and
then
it
will.
You
know
with
the
right
commit
code
which
is
in
history,
but
we
would.
We
would
be
able
to
see
that
and
we'd
see
we.
We
have
counts
on
that
occurrence,
so
we
can
see
very
quickly.
This
isn't
just
like
an
edge
case.