►
Description
Curbing Node Size with Bonsai Tries in Hyperledger Besu - Danno Ferrin
The amount of disk storage needed to operate a "full" node in the Ethereum ecosystem has always been a challenge. The needs to balance full current states for new transactions against storage needs for full historical queries is often misunderstood. Measurements of needed disk space vary from 400 GiB to 6TiB or more depending on use cases to run a mainnet node. Bonsai Tries is Hyperledger Besu's next-generation storage layout designed to address some of these concerns. It provides a minimal impact to full sync while allowing full historical transaction execution capabilities with large savings in space. This approach also limits the long term growth of a full node. In this demo the methods used to achieve this savings will be shown as well as live demos of some of the key features.
Session Details & Slides (if available)- https://sched.co/j3fk
Access All Sessions: https://events.linuxfoundation.org/hyperledger-global-forum/program/schedule/
----
Hyperledger – Open Source Blockchain Technologies
Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory
Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter
Follow-us on Twitter @Hyperledger
Learn about Hyperledger Membership: https://www.hyperledger.org/about/join
#Hyperledger #HyperledgerGlobalForum2021 #Blockchain
A
My
name
is
daniel
farin,
I'm
a
protocol
engineer
and
I
am
on
the
hyperledger
basically
project,
I'm
one
of
the
original
maintainers
of
the
project
and
one
of
the
problems
that
we
had
with
hyperledger
bay
su
going
at
mainnet
one
of
the
places
we
had
a
lot
of
our
performance
issues
had
to
do
with
the
size
of
the
nodes
and
anybody
who's
worked
with.
Ethereum
knows
that
the
real
test
of
scalability
is.
Can
your
node
operate
at
mainnet
scales
on
these
are
scales
that
are
typically
larger
than
most
enterprise
scales
right
now.
A
A
There
is
a
state
group-
and
this
is
a
a
merkle
hash
of
a
patricia
tree,
which
is
a
bunch
of
fancy
computer
science,
words,
meaning
we
take
the
hash
of
a
node.
We
put
16
of
them
together
and
we
roll
them
up
and
we
take
that
hash
for
the
next
level
and
we
go
down
to
each
level
and
we
do
these
branches
based
on
the
address.
A
So
in
this
case,
if
we
have
an
account
100,
this
would
be
the
zero
branch,
and
this
would
be
the
one
branch.
Then
this
would
be
the
the
one
branch
and
this
would
be
the
zero
branch
so
to
get
to
a
particular
account
in
a
tree.
You
would
go
down
to
this
account
100
and
all
of
the
the
children
locations
below
it
are
all
put
into
this
tree
and
all
hatched
up
and
stored
to
come
up
to
a
singular
state
route.
A
So,
when
ethereum
validates
a
block
and
sends
it
to
everyone,
it
doesn't
send
an
entire
copy
of
all
the
storage
tree.
It
stores
one
mathematical
number.
That
is
a
hash
that
is
representation
of
all
of
that.
So
to
keep
that
in
memory,
we
need
to
keep
these
nodes
stored
on
disk.
So
we
take,
you
know
the
state
root
node.
A
We
take
the
first
level
of
the
root
nodes
and,
however
many
there's,
like
you
know,
five
or
six
of
these
until
you
get
down
to
the
particular
account
nodes
and
each
account
also
can
have
state
storage
within
its
account.
This
is
how
a
lot
of
the
smart
contracts
and
the
rc20
tokens
store
their
data
as
they
store
state
underneath
the
account
so
underneath
each
account.
We
similarly
have
a
state
route
and
then
intermediate
branch
nodes,
followed
by
a
final
leaf
node
that
contains
the
actual
value
of
the
node
of
the
data.
A
That's
that's
being
stored,
so
one
particular
state
in
time.
If
we
look
at
just
the
left
one,
we
have
a
collection
of
branches,
a
collection
of
leaves,
and
this
represents
one
state
and
the
the
load
impact
for
just
one
state's
not
too
bad.
It's
it's
at
worst,
double
the
amount
of
actual
data
being
stored.
A
A
The
problem
is
that
storing
these
I
call
this
the
forest
mode.
Storing
this
forest
of
trees
is
actually
has
a
huge
impact
on
the
amount
of
state
stored.
So
here
we
do.
We
just
we
just
do
one
change
on
one
data
file.
We
have
to
update
three
different
nodes
to
to
change
one
value,
and
that
has
a
couple
impacts
and
it
takes
time
to
read
it,
and
it
also
takes
up
more
space
to
write
it.
So
one
data
value
results
in
four
different
rights.
A
A
A
So
to
solve
the
solution,
I
took
inspiration
from
bonsai
trees
and,
if
you
look
at
both
of
these
trays
on
a
dna
level,
these
two
trees
are
exactly
the
same
tree.
It's
a
chinese
elm,
but
because
of
the
way
that
it's
manicured
and
maintained
on
the
left,
it
fits
into
a
nice
small
pot
and
looks
a
lot
more
well
controlled
and
a
lot
more
useful
versus
this.
A
This
chinese
elm
tree,
which
is
great
for
shade
it's
great
for
outdoors,
and
if
you
have
the
space
for
it,
just
use
it
outside,
but
not
everyone
who
runs
ethereum
node
wants
to
have
these
great
huge
bushy
outside
trees
when
they're
maintaining
a
small
when
they
have
a
small
place,
they
want
to
maintain
and
manage
it.
It's
a
lot
easier
to
maintain
and
manage
the
smaller
tree.
So
I
took
inspiration
from
some
of
the
aspects
of
a
bonsai
tree
to
to
basically
help
us
figure
out
how
to
curb
our
node
size.
A
So
the
very
first
principle
that
I
applied
in
in
moving
to
this
new
bonsai
mode
is
to
make
it
leaf
focused
when
you're
querying
a
value.
You
really
only
care
about
the
leaf.
The
branches
are
just
how
you
get
there,
so
we
split
the
leaves
and
we
stored
the
leaves
by
location,
and
we
put
the
branches
in
two
different
table
spaces
and
that
alone
provided
for
a
good
amount
of
database
speed
up
just
because
we
have
a
smaller
set
to
index.
A
We
go
straight
to
it,
but,
most
importantly,
we
don't
have
to
do
any
tree
climbing
to
get
a
value
under
the
tree
mode
untraditionally.
You
would
have
to
go
down
each
level
until
you
find
the
tree
and
get
the
pointer
to
the
content,
because
you
don't
know
where
the
content
is
going
to
be.
But
if
we
focus
on
the
leaves
we
store
just
the
leaves
by
the
location,
we
can
get
there
quickly.
So
we
get
rid
of
tree
climbing
by
having
it
be
really
focused.
A
A
We
store
the
tree
branches
by
the
location
on
the
tree,
not
by
what
the
content
is
and
if
you're
going
to
do
it
in
force
mode,
you
can't
store
it
by
location.
You
have
to
store
it
by
the
content
id,
and
so
when
we
start
by
the
location,
we
get
an
immediate
speed
up
and
it
has
a
huge
impact
on
the
database.
A
Caching,
because
the
root
tree
node
has
the
same
node
and
the
caching
is
easier
to
figure
out
that
this
root
level,
this
you
know
the
block,
zero
and
the
block
a
through
f
and
one
eight.
You
know
in
the
first
256
you
know
first,
two
layers:
they
always
come
in
with
the
same
location,
so
it
can
do
a
much
better
job
of
catching
these,
but
the
second
side
effect
is
also
created
a
little
bit
of
implicit
pruning.
A
If
you
change
a
branch
node
at
a
particular
location,
you
override
it
instead
of
writing
a
new
copy.
So
that's
how
we
keep
it
well,
manicured!
We
don't
let
these
stray
branches
go
out
because
they
keep
getting
replaced
by
the
new
value,
so
that
kind
of
has
a
little
impact
on
some
of
the
other
usability
of
the
trees,
and
we
store
only
by
the
location
is
that
we
can't
go
back
in
time,
and
that
is
you
know.
A
A
Ethereum
nodes
are
notoriously
reorganizing
all
the
time
they
don't
have
finality.
So
you
need
to
go
from
one
block:
that's
going
to
become
an
orphan
and
go
back
to
another
one.
So
to
facilitate
that,
I
keep
what's
called
a
tree
log.
So
every
time
we
transition
from
one
a
log
from
one
block
state
to
another
block
state.
A
I
record
in
that
tree
log,
all
the
values
that
are
red
and
all
the
values
that
are
written
and
I
keep
that
and
it
basically
becomes
kind
of
a
two-way
diff
and
with
that
two-way
diff,
if
you
have
a
two-way
diff
to
a
block
adjacent
to
the
block
that
you're
looking
at,
you
can
use
that
to
roll
back
or
roll
forward
to
a
different
log
state.
So,
by
keeping
this
tree
state,
I
can
go
from
one.
A
So
I
call
these
tree
logs
more
like
a
hearsay
rather
than
a
witness,
because
we
only
have
the
end
result.
We
don't
have
the
proof
to
go
with
it,
but
if
you
roll
back
far
enough,
you
can
generate
that
proof
to
question
how
important
that
proof
is
or
if
you
just
need
to
know
what
the
events
coming
out
of
it
and
what
the
traces
are
of
the
logs.
A
Now,
this
log
rolling
is
very
important.
Here's
for
some
of
the
demo,
some
of
the
graphs
that
I'm
bringing
up
this
came
from
an
early
iteration
of
what
I
was
working
with,
that
we
were
taking
a
particular
where
we're
sinking
gourley,
which
is
one
of
the
smaller
test
nets.
It's
actually
getting
kind
of
big.
A
But
what
that
means
is.
If
you're
listening
to
the
consensus
network,
you
get
a
lot
of
blocks
that
are
going
to
be
orphaned
that
you
need
to
roll
forward
to
so
this
actually
accidentally
provided
an
excellent
torture
test
for
the
roll
forward
and
roll
back
technology
that
I
wrote
into
the
bonsai
trees,
because
in
the
process
of
keeping
in,
for
example,
I
don't
know
if
you
can
see
my
mouse
cursor.
But
you
know
there's
a
couple
blocks
where
we
go.
A
A
couple
of
white
blocks
deep
before
we
go
back
to
the
gray
blocks
and
the
gray
blocks
represent
the
truth
and
the
white
blocks
represent
the
orphans.
So
each
one
of
these
white
blocks
that
the
tree,
the
bonsai
tree
node,
actually
saw
and
actually
process
and
had
to
roll
back.
And
if
it's
two
blocks
deep
to
get
to
the
to
the
canonical
chain,
it
had
to
roll
back
at
least
two
blocks
and
then
continue
rolling
forward
to
get
to
the
block
that
it
needed
to
see.
A
And
this
you
know
set
it
up
for
days
weeks
at
a
time
and
it
it
works
great.
It
works
really
fast
and
actually
was
quite
surprised
by
how
well
it
worked
so
from
all
this
information,
some
actual
hard
numbers
to
go
with
it,
and
these
are
numbers
from
a
mainnet
node.
Once
we
got
it
working
on
gorilla,
we
started
working
on
mainnet
and
these
are
some
some
of
the
more
impressive
numbers
that
I've
seen
from
a
project.
A
I've
been
doing
a
lot
of
optimization
on
vasu
and
I
was
actually
quite
surprised
and
happy
to
see
numbers
of
this
quality
coming
from
when
kareem
actually
got
me
the
numbers
for
these.
But
this
is
a
full
mainnet
block
and
we
did
something
called
a
full
sync,
which
is
where
we
take
the
blockchain
from
the
beginning
of
time,
and
we
play
it
forward
all
four
and
a
half
five
years
every
single
transaction
gets
played
and
we
keep
all
of
the
various
all
the
various
intermediate
states.
A
A
We
go
from
about
794
gigabytes
needed
to
store
the
entire
main
net
history
that
you
could
replay
any
mainnet
block
from
zero
up
to
12
million
387
125
that
took
794
gigabytes
versus
the
forest
mode,
where
we
need
to
track
every
intermediate
branch
and
store
that
12
terabytes
is
about
with
the
estimate
for
for
bonsai
was
we
got
tired
of
of
watching
our
aws
still
go
up
and
and
kareem
just
didn't
finish
it
because
we
got
the
numbers
that
we
needed.
It
was
about
on
par.
A
You
know,
within
within,
within
a
multiple
of
what
geth
was
going.
The
numbers
we
were
seeing
in
the
estimate
is
that
would
come
up
to
about
12
terabytes,
so
the
storage
there
is,
on
the
order
of
you,
know:
10x
15x
improvement
between
the
two
of
them,
so
just
the
storage
of
running
it
on
a
long-term
note,
but
I
think
the
more
relevant
and
the
more
impressive
number
comes
from
the
fastest
notes
and
we
had
a
node
that
I
started
up
in
february.
A
A
Basis
is
consistent
with
it,
and
so
those
are,
you
know
a
couple
of
numbers
that
are
very
encouraging
for
when,
when
bonsai
trees
becomes
a
shipped
feature
in
besu,
hopefully
later
this
year,
so
those
are
the
you
know
we
got
to
make
sure
it
burns
in
got
to
make
sure
it's
a
good
substitute.
It's
not
shipping.
Today
the
goal
is
to
get
it
to
ship
by
the
end
of
the
year
and
that's
assuming
that
we
don't
get
distracted
by
all
the
stuff
happening
in
e2.
A
That's
actually
a
little
bit
of
a
higher
priority,
so
yeah,
so
those
are
some
of
the
hard
numbers.
Those
are
my
demo
yep
cool
and
it
turns
out
my
time
is
up.
So
thank
you
all
for
coming
to
hyperledger.
Thank
you.
That's
my
demo.
Thank
you
for
coming
and
giving
us
a
great
presentation
dinner
thanks,
bye.