►
From YouTube: Killing ETH - Finding Consensus Issues on Layer 1
Description
Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/killing-eth-finding-consensus-issues-on-layer-1/
Ethereum's core protocol is highly security critical and thus needs to be tested thoroughly. This talk will discuss the testing methods we use to make sure that all execution layer clients implement the same protocol:
- Regression testing on hive
- EVM-fuzzing
- Devnets
- Shadow forks
We will discuss a bunch of interesting issues that we found with that
Speaker(s): Marius van der Wijden
Skill level: Beginner
Track: Layer 1 Protocol
Keywords: Testing,Fuzzing
Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/
Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/
A
A
So
next
up
we
have
Mario's
Van
Der
viden,
who
is
on
the
Geth
team
and
works
on
fuzzing
and
testing
and
some
other
things.
I
know
Danny
Ryan
likes
to
take
all
the
credit
for
the
merge,
but
these
guys
played
a
crucial
role
as
well
and
he's
going
to
talk
about
some
of
the
bugs
that
they
found
and
maybe
some
of
the
things
that
could
have
gotten
horribly
wrong.
Didn't.
B
B
And
so
we,
over
the
years,
we
developed
a
bunch
of
testing
strategies
for
exactly
these
problems
and
one
the
newest
one
that
we
developed
for
the
merge
are
Shadow
Forks.
Basically,
we
take
a
copy
of
the
blockchain.
We
configure
some
node
some
of
our
nodes
with
the
new
rules
and
at
the
merge
transition.
The
new
the
new
nodes
create
their
own
chain,
which
is
in
parallel
to
the
main,
to
the
main
chain
which
what
is
really
interesting
about
it.
B
B
B
Another
big
part
of
what
we
do
is
differential
testing.
Basically,
because
we
have
these
different
implementations,
it
has
a.
We
have
a.
We
have
a
really
nice.
We
have
this
really
nice
property
that
we
can
verify
them
against
each
other.
So
we
create
an
input.
We
give
it
to
the
different
implementations,
and
then
we
collect
the
outputs
of
these
different
implementations
and
verify
that
everyone
does
exactly
the
same
thing
and
there
are
different
strategies
to
generate
these
inputs.
B
B
Something
new
that
we
also
did
during
merge
testing
is
create
malicious
notes.
Malicious
notes
are
basically
folks
of
a
client
software
that
does
that
changes.
Some
of
the
rules
and
so
malicious
notes
can
insert
bad
transactions,
change,
header
fields
and
really
big
values
or
no
values,
and
can
try
to
break
the
other
nodes
on
the
network.
B
For
example,
go
evm
lab
is
a
toolkit
for
evm
testing
that
we
use
to
create
a
test
case
executed
on
on
a
client
and
collect
the
the
output
from
the
client.
We
have
Hive,
which
is
a
really
nice
continuous
integration,
regression
testing
Suite,
where
we
have.
We
run
these
ethereum
tests,
which
is
a
set
of
48
000
test
cases
every
night
against
the
different
implementations
and
verify
that
nothing
has
gone
wrong
when
they
update
something.
B
And
then
we
have
these
malicious
nodes,
both
both
on
the
CL
and
the
El
node,
the
CL
and
the
El
layer.
And
then
we
have
some
companies
that
we
work
with
kurtosis
and
antithesis
for
fuzzing
and
and
test
Nets.
These
are
like
bigger
test
Nets
that
we
that
we
set
up
run
once
one
time
through
the
transition
and
then
then
verify
that
the
transition
was
correctly
and
then
we
scrapped
the
test
net
and
and
start
from
start
fresh.
B
B
So
the
rest
of
the
talks
is
just
talking
about
some
of
the
interesting
bugs
that
we
found
over
the
years.
This
bug
was
found
by
Martin
right
before
the
merge.
So
whenever
we
have
a
whenever
we
have
a
consensus
upgrade
whenever
we
have
a
fork,
we
actually
we
ramp
up
testing.
We
we
take
out
all
the
all
tools
that
we
have
and
and
try
to
break
it
here.
B
The
Visa
client
had
a
had
a
bug
in
the
gas
calculation,
as
you
can
see
there,
the
gas
goes
down
and
at
some
point,
there's
an
underflow
and
the
gas
goes
extremely
high.
This
is
a
consensus
issue
because
other
clients
will
compute
this
correctly
and
on
networks
where
biso
is
the
only
client.
This
would
actually
be
a
Dos
Vector
on
the
network,
because
the
nodes
will
keep
running
the
transaction
and
never
stop.
B
B
So
we
set
up
the
kinsugi
test
net,
which
was
the
the
real
big
test
net
that
we
had
where
we
also
invited
the
community
to
participate
in
in
testing
and
on
this
Kentucky
test
net.
We
set
up
the
bad
block,
generator
the
the
bad
block
generator
from
from
the
execution
layer
and
it
changed
the
blocks.
So,
for
example,
it
created
a
block
with
invalid
extra
data
and
this
it
actually
I
think
it
actually
set.
B
The
extra
data
and
Nimbus
was
unable
to
sync
because
of
that,
and
then
we
had
the
TX
further
running
the
x-fuzzer
created
a
transaction
that
had
the
revert
op
code
in
it
with
a
and
and
this
triggered
a
consensus,
an
issue
in
ecmjs,
and
so
they
couldn't
couldn't.
There
wasn't
consensus
issue
there,
and
then
we
had
a
the
three-way
consensus
split
which
actually
broke
the
network,
and
this
was
a
split
between
Geth,
the
tiku
guest
combination
and
piso
another
mind
yeah,
so
the
three-way
consensus
split.
B
B
This
check
was
actually
in
this
in
the
spec,
but
some
of
the
clients
didn't
implement
it
correctly.
So
bisu
did
not
have
this
check.
Another
man
actually
had
the
check,
but
it
also
cached
the
payloads
by
block
hash,
and
so
we
looked
up
the
the
block
hash,
the
wrong
block
hash
so
that
it
was
in
the
cache
and
then
the
the
implementation
just
assumed
that
that
the
payload
was
valid.
B
It
created
the
block
where
the
block
number
was
set
to
one
and
in
Geth
we
have
a
cache
to
check
whether
we
need
to
sync-
and
one
issue
is
with
tiku.
They
executed
all
Forks
of
the
blockchain.
There
were
like
30
different
competing
Forks
executed
them
simultaneously,
which
flushed
the
cache,
and
so
we
queried
the
database
by
parent
hash
and
block
number
minus
one
block.
B
So
after
we
had
this
three-way
consensus
split,
we
decided
at
some
point.
Okay,
it's
just
too
hard
we're
going
to
deprecate
deprecate
the
con.
The
kinsuki
test
net
I
already
talked
a
bit
about
Hive
Hive.
We
use
Hive
to
execute
a
bunch
of
tests
and
most
of
them
are
spec
tests.
So
we
for
the
specification
we
create
test
cases
and
I've
found
an
incredible
amount
of
bugs
so
shout
out
to
Mario.
If
he's
I
think
he's
sitting
here,
yeah
and
so,
for
example,
we
had
a
division
by
zero
in
the
exchange
transition.
B
Configuration
call
some
some
rules
around
the
time,
step,
didn't
really
work,
and
so
I've
really
found
a
lot
of
issues
that
we
have
between
guest
and
the
spec.
And
then
we
had
the
testing
the
merge
effort,
which
was
really
nice.
Over
400
people
got
involved
with
testing
the
merge
they
sent
transactions
on
their
test
Nets,
they
set
up
notes,
reported
issues
and
the
most
important
thing
they
create
a
documentation.
So
we
should
be
better
at
this.
We
should
be
better
at
creating
a
documentation,
but
we
are
unfortunately
not
so.
B
We
kind
of
rely
on
the
community
to
educate
other
people
in
the
community
and
they
also
found
some
some
interesting
issues
in
in
go
ethereum
that
were
kind
of
related
on
on
the
usage
of
it
and
I
think
they
also
found
a
bunch
of
other
bugs.
So
the
community
found
a
bunch
of
other
bugs
in
different
implementations,
and
then
we
had
the
shadowfox
the
cool
thing
about
the
shadow
Fox
is.
B
We
are
actually
testing
with
the
real
Network
and
the
real
data,
and
so
one
thing
that
on
the
on
the
first
year
of
work,
we
saw
that
the
gas
limit
was
quickly
dropping
and
the
issue
there
was
the
default
gas
limit
in
gas
is
actually
was
actually
8
million
and
we
never
caught
that
and
we
caught
it
for
the
first
time
doing
this.
This
Shadow
Forks,
because
on
mainnet
the
the
miners
back
then
voted
to
to
increase
the
gas
limit.
B
B
B
So
if
you're
interested
in
getting
a
job
at
the
foundation
and
and
help
us
test,
the
new
updates
contact,
Mario
and
yeah
so
for
when
I
inevitably
have
too
much
time
on
my
hands.
I
have
another
eight
minutes.
I
have
a
bunch
of
backup,
slides
with
more
cool
bugs
and
issues
that
we
found.
You
might
remember
this
one,
the
bug
that
took
down
infuro.
B
Basically,
the
memory
returned
by
the
return
data
copy
of
code
was
shallow
copied,
and
so,
when
we
modified
it
broke
something
and
we
actually
found
the
bug
rather
quickly
and
fixed
it.
But
the
fix
wasn't
announced
we
created
a
new
version
of
get
and
we
kind
of
forgot
about
it,
and
someone
actually
found
that
bug
and
triggered
it
on
mainnet,
because
they
saw
that
only
one
percent
of
the
nodes
on
mainnet
actually
still
have
this.
B
B
This
one
was
also
a
really
interesting
one,
so
we
have
fuzzles
that
run
on
OS
as
fast
continuously
and
at
some
point
we
got
an
email,
hey.
We
have
a
panic
and
whenever
we
get
a
get
an
email
about
a
panic,
that's
usually
pretty
bad,
because
then
someone
could
just
send
a
transaction
to
the
network
and
all
of
the
guest
nodes
would
go
down,
and
that's
not
great.
B
B
Then
we
have
the
some
bug
that
we've
got
via
the
Bounty
program,
so
we
have
the
Bounty
program.
If
you
find
something,
and
instead
of
triggering
on
it
on
mainnet,
send
us
an
email
we're
especially
pissed.
If
we,
if
you
send
us
an
email
and
then
go
trigger
the
background,
mainnet
so
send
us
an
email.
First
talk
to
us
and
you
will
actually
get
a
bounty.
B
The
Bounty
size
was
increased
to
250k
and
we
actually
quadrupled
it
for
the
merge.
So
there's
a
lot
of
money
to
be
had
if
in
finding
in
funding
consensus,
bugs
here
when,
when
we
input
in
256
min
2
to
the
power
of
255
it
would
the
the
implementation
of
nethermine
for
the
for
the
modular
operator
would
actually
negate
the
result-wise
resulting
in
2
instead
of
-2.
B
B
That's
that's
a
network
packet
that
requested
a
missing,
try,
note
and
you
could
crash
a
the
guest
mode
with
it.
These
kind
of
issues
are
not
that
bad
because
you
only
crash
one
node.
If
we
have
crashes
in,
if
we
have
crashes
within
the
evm,
those
are
usually
worse
because
you
can
crash
a
bunch
of
notes
and
actually,
during
investigating
of
this,
we
created
a
fuzzle
for
it
and
the
fuzzle
found
a
second
panic
here.
B
We
have
one
I,
actually
wrote
my
master
thesis
on
fuzzing,
evm,
stuff
and
created
this
fuzzy
VM
program
and
the
fuzzy
VM
found
an
issue
in
the
copy.
Op
codes
called
Data
copy
code,
copy
x,
copy
return,
data
copy,
and
the
issue
was
these
op
codes
consume
three
items
from
the
stack
destination,
source
and
length
of
the
data
to
copy
and
never
mind,
actually
halt
the
execution
if
length
was
zero,
so
you
can
see.
This
is
one
of
the
tools
we
have
where
we
can,
where
we
have
two
different
executions.
B
The
the
110
22
bug
was
actually
one
I
guess
122
was
actually
the
version
that
we
wanted
to
use
for
mainnet
for
the
merge
and
it
contained
a
regression
that
could
corrupt
the
local
state.
Basically,
whenever
we
we
have
so
we
have
these
try
notes.
This
is
where
all
the
stuff
from
the
other
contract,
State
and
the
counts
are
stored,
and
when
we
modify
them,
they
need
to
be
flushed
to
the
disk
in
the
same
order
as
they
are
inserted
into
what
we
call
the
dirty
cache.
B
And
so
you
want
to
you
want
to
insert
if
you
modify
d
b
and
a
you
want
to
insert
D
first
B
second
and
then
a
third
and
what
we
did
was
actually
insert
B
first
and
then
insert
a
and
then
we
saw
that
we
should
insert
D,
but
we
saw
that
b
was
already
inserted,
so
the
code
assumed
that
everything
was
correct
and
we
never
inserted
D.
So
we
ended
up
with
some
dangling
dry
nodes.