►
Description
Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/quest-for-the-best-tests-a-retrospective-on-testingthemerge/
A retrospective on #TestingTheMerge and all the various testing avenues we explored. I'd also give a brief summary of what went right and wrong in our approach.
Speaker(s): Parithosh Jayanthi
Skill level: Intermediate
Track: Layer 1 Protocol
Keywords: Testing,Merge,DevOps
Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/
Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/
A
A
About
the
quests
for
the
best
tests
today,
we're
going
to
look
at
all
the
different
testing
tools
that
we
used
and
I'm
going
to
be
talking
about
the
bugs
that
we
didn't
find
on
mainnet.
If
you
were
here
for
two
talks
before
mine,
Mario
spoke
about
the
bugs
that
we
did
find
so
I'm
gonna
do
the
contract
the
first
part.
Let's
talk
about
what
we're
actually
doing
with
testing.
Why
is
the
merge
complicated
to
begin
with?
Well,
we
have
roughly
20
different
client
combinations
and
regressions
sneak
in
very
easily.
A
You
might
see
a
regression
in
one
of
the
client
combinations,
but
none
of
the
other
ones
and
that's
very
tricky
to
pinpoint
the
specs
for
an
active
development.
There
were
quite
a
few
of
the
early
test
Nets
where
we
kind
of
didn't,
pin
the
spec
version,
so
someone
was
implementing
something
on
a
different
commit.
A
If
I
have
to
wait
for
the
Australians
to
wake
up
for
anything,
it
does
take
quite
a
bit
of
effort
and
quite
a
bit
of
planning
on
our
side,
we
had
to
kind
of
schedule
around
people
picking
up
their
kids
from
school
where
to
schedule
around
people
waking
up
in
Australia.
We
have
to
schedule
around
Americans
a
lot
of
different
things
and
figuring
out
how
to
do
all
of
this
in
a
reliable
Manner
and
on
a
timeline
was
crazy.
A
The
last
one
was
debug
knowledge.
We
were
really
surprised.
The
type
of
debugging
you
need
to
do
for
CLS
and
years
are
totally
different.
We
had
to
see
how
to
bring
all
of
that
competence
in
one
place
and
how
to
actually
figure
out
when
something
goes
wrong
and
they
did
go
wrong.
A
lot
in
the
beginning,
and
the
nice
part
is,
if
you
figure
this
out
once
we
figure
it
out
for
future
test,
Nets
so
happy
that
that
worked
out
what
could
possibly
go
wrong.
A
So
the
merge
has
two
parts,
the
consensus
layer
and
the
execution
layer,
and
they
communicate
via
this
thing
called
the
engine
API.
So
if
you
mess
up
the
consensus
layer,
you're
going
to
have
a
network
that
can't
agree
on
anything,
if
you
mess
up
the
execution
layer,
you
have
a
network
that
can't
do
anything.
So
those
are
the
two
high
level
problems
that
we're
in
trying
to
ensure
that
never
happen
just
to
enumerate
a
bit
from
the
regular
testing
world.
What
sort
of
tests
even
exist?
A
So
we
have
unit
tests
and
it
is
in
our
decentralized
world
with
all
the
client
teams.
The
client
teams
take
care
of
this
themselves.
We
don't
have
to
do
anything,
no
coordination
work
from
our
side.
This
makes
sure
that
there's
no
regression,
but
these
are
localized
regressions,
something
that
they
might
have
seen
on
their
client.
Based
on
how
they've
built
their
client,
we
have
integration
tests.
A
A
Then
we
have
system
tests,
and
this
is
where
external
coordinators
come
in,
so
I'm
going
to
say
the
EF
was
part
of
the
external
coordinator
for
the
testing
efforts.
So
these
tests
end-to-end
functionality,
we
spin
up
a
test
net,
we
run
transactions,
we
withdraw,
we
set
up
faucets,
we
look
at
explorers,
work
and
so
on,
and
then
we
have
production
tests.
A
This
is
something
you
guys
might
have
seen
with
shadowfox,
so
production
tests,
sort
of
work
on
a
prod
like
environment
and
I'll
go
into
what
shadow
Fox
are
later
on,
but
on
a
high
level,
we
inherit
all
the
complexity
of
mainnet,
and
this
also
includes
public
test
Nets.
So
if
you
guys
remember
Kiln
or
kinsuki
that
falls
under
this
bucket,
so
you
have
everyone
all
over
the
place,
testing
their
things.
Layer
tools
are
deploying
over
there.
We
have
random
D5
protocols
deploying
over
there,
and
these
find
issues
that
only
happen
on
real
world
workloads.
A
You
probably
can't
simulate
them
any
other
way,
so
the
second
part
I'm
going
to
talk
about
what
different
tools
we
had
and
just
give
you
a
brief
overview
and
just
to
tell
you
how
this
is
going
to
go
once
you
have
an
overview.
What
sort
of
testing
tools
we
have
I'm
going
to
talk
about
what
we
actually
did
with
them
and
what
we
didn't
find
so,
the
first
one
starting
at
a
high
level.
We
have
spec
tests,
it's
the
great
thing
about
the
consensus
layer.
A
They
have
an
executable
spec,
they
have
specifications
with
a
ton
of
tests.
Client
teams
can
then
import
these
specs
and
make
sure
that
they
can
test
them
in
their
local
CI.
This
means,
whenever
they're,
making
a
release.
At
least
you
know
that
they're
coherent
to
the
spec.
This
is
largely
a
sanity
check.
It's
not
meant
to
find
any
massive
bug,
but
it
ensures
that
there's
no
regressions
happening.
A
We
currently
have
the
spec
tests,
I
think
running
every
night
on
on
a
new
CI
machine.
The
second
one
is
hype
tests.
You
might
have
seen
this
being
referenced
a
couple
of
times,
so
Hive
tests
run
with
the
simulator
and
they
essentially
start
up
the
clients
and
then
run
the
tests
against
a
predefined
interface.
These
are
a
couple
hundred
tests.
These
take
anywhere
between
I,
think,
a
day
or
two
days
to
run
everything
and
a
brief
example
of
how
this
is.
Is
it
sets
up
a
tiny
instance
of
another
mine
node.
A
It
sends
it
two
terminal
blocks
and
asserts
how
the
transition
happens.
This
is
a
lot
of
awesome
work
by
Mario.
He
should
be
somewhere
in
the
crowd
shout
out
to
him
and
follow
him
on
Twitter.
We
found
a
lot
of
edge
cases
in
this
and
once
we
do
find
an
edge
case,
it's
always
in
this,
so
that
we
make
sure
in
in
future
updates
whatever
it
doesn't
happen.
If
people
check
the
website,
of
course,
then
we
have
this
thing
called
kurtosis.
A
It's
an
external
tool
that
we're
working
with
and
kurtosis
obfuscates
all
the
complexity
with
setting
up
a
test
net.
You
don't
have
to
worry
about
how
Genesis
works.
You
don't
have
to
worry
what
format
that
never
mind
needs
its
Genesis
file
in
nothing.
You
just
Define
it
in
a
yaml
file.
You
say:
I
want
a
five
node
network
with
this
Docker
image
and
that's
what
happens
for
you.
A
We
time
this
actually
nightly
and
they
think
we've
gone
through
the
merge
at
least
a
couple
hundred
times
this
year,
a
lot
of
them
with
issues,
but
a
lot
of
them
without
which
is
great
to
see.
We
use
this
also
to
rapidly
iterate
ideas.
So
once
there's
a
new
spec
version,
or
if
you
want
to
try
out
a
new
testing
tool,
we
throw
it
in
there.
A
First,
we
also
had
map
boost
integrated
into
kurtosis,
so
we
could
test
it
out
and
do
a
lot
of
cool
things
there,
a
general
view
of
kurtosis
is
it.
It
checks
the
happy
case,
if
you
can't
figure
out
the
happy
case,
there's
no
point
checking
the
rest
of
it.
So
it
just
starts
up
a
test
net.
Make
sure
everything
is
fine.
If
it's
fine,
it
shows
you
green
on
your
CI.
A
The
next
one
is
sync
tests.
There's
no
point
on
a
network.
If
you
have
nodes
that
can't
sync
up
to
the
network,
so
what
we
do
is
we
spin
up
nodes,
I
think
at
a
week's
notice
right
now
they
sync
up
to
the
head
of
the
chain
and
then
they
assert
whatever
we
specify
over
there.
So
if
you
notice
over
here,
you
can
say
is
execution.
Healthy
is
consensus,
healthy
are
both
synced.
Are
they
both
reaching
head?
A
So
you
can
kind
of
Define
what
side
a
type
of
syncing
you
want
to
do
here
and,
as
you
can
see
on
the
right,
there
are
a
ton
of
different
options.
You
can
do
and
a
ton
of
Networks
and
the
cool
thing
is.
You
can
also
assert
bad
cases.
You
can
say
start
your
El
stop
once
it
reaches
head,
then
start
your
CL
and
then
build
weird
scenarios.
There
shout
out
to
Sam
for
building
this.
He
was
also
running
it
and
sending
a
summary
I
think
he
even
presented
it
once
on
Orca
devs.
A
So
we
could
make
sure
that
at
least
when
we're
making
releases
and
when
we
merge
that
we
can
sync
the
network-
and
this
is
the
meaty
one-
we
had
test
Nets
and
shadowfox.
So
a
shadow
fork
and
testnet
help
us
coordinate
all
client
teams
in
one
place
and
we
check
compatibility
largely.
We
take
whatever
assumptions
we
have
in
the
spec
and
we
assert.
A
If
those
assumptions
are
true
on
a
large
level,
what
we're
doing
is
we're
taking
the
Genesis
configuration
of
any
one
network,
and
then
we
modify
a
couple
of
values
here
and
there
and
what
happens
when
those
modified
values
are
hit?
Is
we
split
away
from
the
main
network,
but
we
continue
staying
sync
connected
on
the
gossip
Network,
so
we're
importing
all
the
transactions
and
we
have
the
old
load,
but
it's
on
our
side.
We
it's
it's
just
a
side
Fork.
A
It
runs
parallel
to
the
main
Network
and
no
one
cares
about
it,
except
for
us,
because
we
can
find
dozens
and
dozens
of
bugs
there.
This
allows
us
to
stress
all
of
our
assumptions
and
we've
done
a
lot
of
them.
I
have
a
summary
on
how
many
we
actually
did
in
the
end
I
sort
of
look
at
this
as
a
release
test.
So
it's
kind
of
one
of
the
last
things
we
would
do
on
any
of
the
future
Forks
that
we
have
it's
kind
of
when
we're
almost
figuring
out.
A
Are
we
ready
to
go
ahead
with
this?
Are
we
ready
to
move
forward
with
with
committing
to
this
Fork?
Is
there
any
unknowns
that
we
don't
know
yet,
and
then
we
have
fuzzles
and
in
external
organization
called
antithesis.
So
antithesis
is
a
deterministic
hypervisor
that
allows
us
to
perform
Network
splits
packet
loss,
all
sorts
of
really
weird
edge
cases.
A
We
don't
necessarily
expect
the
network
to
be
put
in
these
edge
cases,
but
if
they
are,
we
know
that
the
clients
can
handle
it
on
the
right
side.
You
could
you
can't
even
actually
see
it,
but
each
of
those
is
a
256
thread
machine.
We
have
three
of
them
all
running
fuzzles.
It
was
insane
we
had
way
more
I've,
never
seen
that
much
compute.
In
my
life
we
actually
had
the
IBM
data
center
go
out
and
buy
more
CPUs
because
we
bought
up
all
of
them.
A
A
So
to
give
you
a
brief
idea
on
the
testing
life
cycle,
let
me
tell
you
it's
never
that
clean,
but
that's
what
we
want
to
hope
to
achieve.
So
you
have
the
client
releases
happening
and
once
the
client
releases
happen,
they
go
into
the
integration
tests.
So
we
make
sure
that
testnets
can
be
run,
that
integration
tests
work
fine
and
once
that's
done,
we
move
on
to
test
Nets
and
then
we
move
on
to
stress
tests
and
we
push
whatever
we
find
on
these
stress
tests
onto
specifications.
A
We
do
regression
tests
and
then
we
do
fuzzing
and
hopefully,
whatever
we
find
in
the
last
couple
of
stages,
go
back
into
client
releases.
So
that's
a
kind
of
a
nice
way
to
look
at
a
full
life
cycle
of
how
testing
would
work,
we're
hoping
to
adopt
the
same
life
cycle
for
future
test
Nets.
Maybe
we
moved
around
a
tool
here
or
there,
but
that's
the
general
idea
and
the
end
game.
What
did
we
actually
do.
A
It
was
really
hard
to
find
a
graph
that
actually
fit
all
the
test.
Nets.
We
had
that's
how
many
so
we
had.
We
started
off
in
April
of
2021
with
ryanism
and
since
then,
we've
had
four
public
test
Nets
that
anyone
could
permissionlessly
join
into.
We
had
six
devnets
meant
for
all
the
client
teams.
We
had
five
Gully
Shadow
Fox
and
in
the
end,
13
midnight,
Shadow
Fox,
and
after
all
of
this,
we
had
three
test
Netflix
and
only
then
we
hit
mainnet.
A
So
the
whole
number
of
just
testing
hours
in
the
merge
is
insane
I'm,
quite
sure.
If
we
added
it
all
up,
it
would
be
at
least
a
couple
ten
thousand
hours
put
into
this.
So
what
didn't
we
find?
This
is
a
really
interesting
part,
because,
even
though
we
have
all
of
these
Cool
Tools,
there's
still
always
going
to
be
something
we
didn't
find
this
99
participation,
98
participation
or
great
blocks
being
produced
awesome,
but
I
want
to
know
what
we
still
didn't
find.
A
So
first
one
we
had
in-memory
databases
that
were
too
low
to
process
main
net
blocks.
It
just
so
happened
that
we
did
too
good
a
job
of
deciding
which
machines
run
our
test
Nets,
which
means
we
didn't
have
any
resource
constraints
and
we
kind
of
missed
the
people
running
nodes
on
8,
gig,
ram
machines
or
people
running
on
16
gig,
ram
machines.
We
didn't
account
for
that
or
we
didn't
account
for
older
versions
of
ram
being
used
whatever
it
is
another
one
that
we
missed
is
non-optimal
block
production.
A
We
were
super
focused
on
making
sure
that
we
didn't
see
any
zero
transaction
blocks
on
the
network.
We
didn't
compare
that
to
what
the
optimal
blocks
on
the
network
could
be.
So
it
is
an
optimization
problem,
of
course,
and
it
is
an
awful
that
on
mainnet
we
have
some
reduced
load,
but
it
is
something
we
missed
and
something
we
probably
should
look
into
in
the
future.
A
The
next
one
there
was
a
really
specific
way
in
which
the
terminal
blocks
could
arrive,
that
broke
Netherland
and
it
broke
Netherland
by
causing
missing
receipts,
and
we
only
noticed
that
issue
when
there
are
deposits
being
made.
So
we
completely
missed
this
and
for
what
we
still
yet
to
figure
out.
Why?
But
this
is
mainly
an
issue
for
lodestar,
never
mind
and
no
other
combination,
or
at
least
we
didn't
get
the
Memo
from
other
breaking
combinations.
A
So
we
want
accurately
simulating
that
so
potentially
on
future
shadowfox,
we're
gonna
have
to
add
a
bunch
of
syncing
nodes
that
start
up
later
on
to
see
what
could
happen
there
and
the
last
one
that
we
didn't
find
was
a
failover
beacon,
note
scenario:
a
lot
of
people
are
really
obsessed
with
making
sure
they
don't
miss
a
single
attestation,
which
means
they
have
multiple
Beacon
notes
set
up
and
that's
something
we
just
never
did
so.
A
lot
of
the
requests
were
just
being
sent
to
the
primary
Beacon
node
and
not
the
backup.
A
So
when
the
failover
happens,
the
backup
wasn't
ready.
I
think
this
has
also
been
patched
right
now,
but
it
was
a
really
tricky
thing
and
we
should
try
more
failover
backup
scenarios
in
the
future.
I
guess
one
cool
thing
about
why
all
of
this
wasn't
found.
Most
of
them
are
optimization
bugs,
as
you
can
see,
merge
still
went
fine,
but
we
could
there's
still
some
room
for
improvement
and
it's
a
really
hard
trade-off
to
make
between.
Do.
A
We
want
to
spend
more
depth
time
in
fixing
these
optimizations,
or
did
we
want
an
earlier
much
and
that's
kind
of
the
tricky
thing?
Do
we
want
earlier
fox
or
do
we
want
completely
buckless
free
free,
free,
Fox,
no
idea,
but
we're
definitely
going
to
be
adding
resource
constraint
machines
to
the
mix
next
time
we're
thinking
of
just
getting
a
bunch
of
Raspberry
Pi's
and
doing
some
Shadow
Fox
on
there
see
what
happens
there
and
another
reason
some
of
the
bugs
appeared,
but
they
didn't
appear
on
the
shadow.
A
A
So
we
have
a
bunch
of
testing
tools
that
we've
completely
open
sourced.
If
you're
someone
who
wants
to
run
that's
nothing
kubernetes
if
you're
someone
who
wants
to
just
run
kubernetes
nodes.
If
you
want
to
set
up
your
own
networks,
we
have
easy
ways
to
set
up
Genesis.
If
you
want
to
yeah
use
the
same
stuff
we
use
for
shadowfox.
All
of
it
is
completely
open
source.
Quite
a
few
organizations
are
actually
reusing
our
our
stuff,
so
shout
out
to
them
contribute
back.