►
Description
Mir-BFT: A Scalable High-Throughput BFT for Blockchains - Marko Vukolic, IBM
In this talk we present Mir-BFT (or, simply, Mir), a robust and scalable Byzantine fault-tolerant (BFT) total order broadcast protocol aimed at maximizing throughput on wide-area networks (WANs), targeting permissioned and Proof-of-Stake permissionless blockchains.
We show that Mir achieves unprecedented throughput on WANs without sacrificing latency, robustness to malicious behavior, or even performance in clusters. Our evaluation shows that Mir orders more than 60000 signed Bitcoin-sized and over 16000 signed Fabric-sized transactions per second on a widely distributed 100 nodes, 1 Gbps WAN setup, while preventing a range of attacks including request duplication performance attacks.
To achieve this, Mir relies on a novel protocol mechanism that allows a set of leaders to propose request batches independently, in parallel, while rotating the assignment of a partitioned request hash space to leaders. Several optimizations boost Mir throughput even further, including partial replication through a novel abstraction we call light total order (LTO) broadcast.
Perhaps most importantly, Mir relies on proven BFT protocol constructs, which simplifies reasoning about Mir correctness. Specifically, Mir is a generalization of the celebrated and scrutinized PBFT protocol. In a nutshell, Mir follows PBFT ``safety-wise'', with changes needed to accommodate novel features restricted to PBFT liveness.
----
Hyperledger – Open Source Blockchain Technologies
Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory
Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter
Follow-us on Twitter @Hyperledger
Learn about Hyperledger Membership: https://www.hyperledger.org/about/join
#Hyperledger #HyperledgerGlobalForum2020 #Blockchain
A
Okay,
I
guess
I
guess
we
can
start
that
because
it's
4:15
I
hope
so
more
people
join.
So
my
name
is
Marco
Vukovich
I'm,
one
of
the
original
fabric,
architects
and
I
work
lead
basically
consensus
development
in
research
and
development
in
IBM
Research.
So
this
is
our
latest
work.
That's
going
to
be
finely
fabric
ordering
service,
but
not
only
that
is
going
to
be.
It's
meant
to
be
a
standalone
library
that
other
block
sheets
can
integrate
with.
A
So
I
was
hoping
actually
for
more
audience,
given
the
given
the
topic,
but
so,
as
you
will
see,
mere
BFT
has
quite
unique
features.
It's
the
best
throughput
on
wide
area
networks.
Vfd
concerns
the
protocol
to
date
may
be
a
foreseeable
future
for
the
for
the
features
that
you
will
see
so
just
about
consensus
in
blockchain
a
bit.
So
when
I
say
consensus
in
this
talk,
this
is
going
to
be
basically
the
mechanism
to
replicate
the
this
blockchain.
A
If
we
see
it
as
a
totally
totally
order,
the
list
of
transactions
like
grouped
into
the
blocks
blocks
are
totally
order
across
nodes
and
within
a
block.
We
have
a
essentially
list
of
transactions.
So
that's
establishes
the
global
total
order,
so
the
network
is
untrusted.
We
are
going
to
focus
on
permission
block
chains,
but
whatever
I'm
going
to
say
is
basically
applicable
to
proof
of
stake.
Permissionless
block
choice
as
well,
so
consensus
protocols
are
door.
Those
protocols
that
keep
the
blockchain
data
structure
like
identical
across
the
nodes
so
in
fabric
speak.
A
This
is
the
main
job
of
the
ordering
service
and
consensus
for
block
chains
has
has
an
interesting
history
like
the
problem
on
its
own,
like
predates
Bitcoin.
So
concerns
was
seen.
Untrusted
networks
is
a
problem
that
was
a
that
was
introduced
in
80s,
but
of
course
the
popularity
gets
with
Bitcoin
and
we
know
the
major
drawbacks
of
bitcoins
proof-of-work
right.
So
he
has
higher
energy
consumption
because
it's
based
on
essentially
very
lightweight
mechanisms,
so
mine,
independently
mind
blocks
which
can
lead
to
forks.
A
Okay.
So
this
work
in
one
slide
is
a
robust
I'm
going
to
talk
more
about
what
I
mean
by
robust,
but
it's
a
robust
bft
total
order,
consensus
protocol
with
highest
throughput
on
wide
area
networks.
We
measured
up
to
100,
because
this
is
like
the
setting,
which
is
the
most
relevant
for
permission,
block
Chiefs.
A
You
will
rarely
see
any
use
case
where,
for
example,
the
number
of
orders
in
fabric
network
or
or
total
number
of
peers
even
with
go,
would
go
over
100,
but
like
it's
probably
more
because
you
will
see
how
plot
scale
and
it's
suitable
for
permission
and
proof
of
state
permissionless
blockchain,
so
just
a
bit
of
formalism.
So
this
gives
the
model
I'm
not
going
to
bother
you
too
much
with
with
the
model,
but
this
gives
basically
the
model
that
we
are
working
on.
A
If
you
have
questions
about
membership
reconfiguration,
we
can
postpone
them
to
the
end
of
the
talk,
because
I'm
going
to
assume
a
fixed
membership
of
orders
for
simplicity
like
any
of
them
and
work
to
one
third
of
them
can
be
malicious.
Any
number
of
clients
can
be
malicious,
so
this
is
a
pretty
standard
model.
Then
there
are
different.
A
Protocols
differ
in
synchrony
assumptions
on
the
network,
so
you
have
protocols
like
Al
Gore
on
which
would
assume
you're
more
synchrony,
then
I'm,
assuming
here
you
have
protocols
like
honeybadger,
which
assume
less
synchrony
than
I'm
assuming
here,
but
this
is
the
model
assume,
for
example,
by
tender
min
concerns
who
said
NP,
BST
consensus,
which
is
eventually
synchronous
network,
which
means
that,
up
to
certain
time,
called
global
stabilization
time
here,
I
give
an
acronym
GST.
The
network
can
be
completely
isomers.
A
This
time
is
unknown
to
note,
so,
basically
that
the
protocol
needs
to
tolerate
arbitrarily
long
periods
of
facing
cream,
but
it's
only
required
to
progress
once
the
network
is
synchronous,
which
essentially
means
that
the
safety
properties
that
I'm
just
going
to
list
need
to
be
maintained,
even
if
the
network
is
purely
NC,
asynchronous
and
liveness
for
progress
on
the
network
is
guaranteed
when
the
network
is
interested.
So
eventually
these
properties
are
summarized
the
follow
so
for
total
or
the
broadcast.
We
have
agreement
so
essentially,
roughly
speaking
right.
A
So
all
correct
nodes
deliver
transactions
in
the
same
order
validity.
They
only
deliver
transactions
proposed
by
some
clients.
So
will
not
get
your
garbage
on
your
blockchain
right,
so
there
will
not
be
the
ability
of
malicious
nodes
to
produce
something
that
just
appears
in
the
block.
G.
No
duplications
transactions
are
not
duplicated.
There
are
different
mechanisms
how
how
this
can
be
sold
and
Linus.
A
So
let's
say
that
before
I
give
you
a
protocol
that
that
aims
to
be
highly
scalable
and
to
have
extremely
high
throughput.
Let's
understand
in
classical
leader
based
protocols
such
as
pbft
such
as
tender
mean
what
are
the
bottlenecks.
So,
for
example,
we
have
the
here
I'm
working
with
four
nodes
out
of
each
one,
can
be
Byzantine
I'm
not
going
to
even
spare
this
Byzantine
fault,
but
this
is
just
how
the
protocol
works.
So
there
are
two
approaches.
A
A
Some
concerns
with
protocols
such
as
hot
stuff.
They
really
aim
at
reducing
this
n
square
communic
number
of
communication
steps,
but,
as
we
will
see,
this
is
not
very,
very
important
for
some
reason
they
need
to
do
it
again,
like
I
will
need
half
an
hour
to
explain
why,
but
with
three
F
plus
1
nodes.
They
cannot
really
stop
after
one
step
before
all
communication.
They
need
to
do
it
once
more,
but
not
more
than
that.
You
can
actually
skip
the
third
step.
If
you
hear
five
F
plus
one
knows,
but
that's
a
separate
story.
A
So
now,
if
2/3
majority
of
qualified
majority
agrees
on
the
same
block,
then
basically
correct
notes
can
add
this
block
to
the
block
chain.
So
what's
the
bottleneck
here
as
n
grows?
So
now
we
want
scalability
and
growth
here.
I
have
N
equals
four
and
grows
which
step
of
the
protocol
is
the
bottle
with
any
idea.
A
Commit
late:
let's
assume
there
is
no
execution,
so,
let's
think
fabric,
so
we
just
add
the
block
to
the
blockchain.
So
there's
a
lot
not
not
much
work
there.
It's
up
to
the
peers
to
execute
or
not.
We
don't
care
too
much.
Let's
focus
on
the
ordering
service,
so
it
seems
like
a
classical
answer
would
be
the
the
mid
two
steps
because
there
have
n
square
communication,
but
one
thing
that
I
want
you
to
notice
is
this
is
load
balanced
across
ten
replicas?
A
So
these
two
steps
to
work
like
the
load
on
a
bottleneck
replica
is
actually
a
well
there's
nothing
to
do
with
on
square.
Then
it
pushes
us
back
to
the
first
step,
especially
if
you
read
disseminate
the
requests
along
with
the
block,
so
the
bottle
making
in
classically
is
the
first
step,
especially
if
you
disseminate
transactions
with
the
block.
Basically,
the
agreement.
There
goes
from
the
hash
of
the
block.
If
you
like
you
like
these
two
and
square
communication,
steps
are
really
really
lightweight
and
the
bottleneck
is
in
the
first
stress
then
and
grows.
A
Essentially,
you
have
the
limited
bandwidth
at
the
primary
at
the
leader.
It
needs
to
disseminate
it.
Needs
to
replicate
is
basically
intact,
so
there
are
optimizations
to
this.
That
I'll
come
back
to
in
which
you
include
only
hashes
of
the
transaction
is
in
the
block,
but
as
you'll
see
this,
this
creates
problems,
let's
assume
for
the
moment
that
basically,
we
disseminate
the
payload
with
inside
the
main
ordering
protocol.
So
as
the
number
of
replicas
grow,
this
is
a
simple
basically
plot.
A
The
the
size
of
the
this
message
grows
linearly
and,
of
course,
your
throughput,
which
is
illustrated
on
the
right
hand.
Side
goes,
inversely
proportional
e
to
the
to
the
number
of
nodes,
and
this
is
the
pattern
that
you
see
all
over
again
in
leader
based
protocol.
Actually,
this
is
the
reason
another
n
square
number
of
messages,
so
approach
is
such
as
hot
stuff,
which
limit
the
number
of
messages
in
the
common
case
to
Oh
N.
You
have
nothing
to
do
with
scalability.
A
So,
what's
robust
BFT,
robust
PFT
that
Mir
so
is
a
family
of
protocol
that
Mir
belongs
to
which
basically
began
with
our
drug
hardening
hard
work
is
the
protocol
researcher
by
Clement,
another
CNN
SDI
2009,
which
is
basically
hardening
or
pbht,
give
you
guaranteed
performance
even
in
the
presence
of
attacks,
so
in
blockchain.
Of
course,
we
get
hope
that
there
are
no
attacks
from
the
network,
but
like
it's
better
that
we
designed
the
protocol,
for
example,
now
now
in
fabric,
we
have
been
waiting
for
a
few
years
for
a
protocol.
A
We
are
not
going
to
redesign
it
once
people
say,
but
they
actually
are
active
attacks
on
the
network.
So
it
see
if
this
BFT
then
is
BFT
right,
so
you
should
perform
well
even
under
attack.
So
long
as
the
network
is
synchronous,
so
is
the
main
part
that
Mir
is
following
right
so
which
is
to
avoid
pitfalls
of
fragile
optimizations
such
as
the
client
relying
on
clients
to
request
to
disseminate
request,
because
there
are
many
protocols
like
pbft.
A
This
should
be
ok
like
from
software
implementation
perspective,
as
n
grows,
requiring
the
client
to
send,
for
example,
to
100
ordering
service
nodes
each
and
every
time
it's
a
visa
transaction.
It's
a
separate
problem,
but
let's
assume
we
allow
it
right
and
then
the
Byzantine
client
comes
and
says
well,
I,
look
I,
don't
have
an
incentive
to
do
this.
I
will
just
send
it
to
the
leader.
A
Is
the
main
issue
right?
So
you
have.
The
client
is
not
following
the
protocol.
It's
submitting
a
transaction
to
the
leader
to
the
primary.
This
looks
exactly
the
same.
Like
this
case.
There
is
no
way
client
leader
distinguishes
in
this
moment.
It
proposes
pre-prepare
message:
it
proposes
the
block
and
the
others
don't
have
the
transaction
so
at
best,
if
client
disappears,
they
can
go
to
the
leader
or
if
they
want
to
kick
out
the
client.
They
need
now
to
agree
that
the
client
should
be
kicked
out.
So
this
is
an
incomplete
product.
A
So
do
easiest
fix
to.
This
is
basically
that
nodes
require
the
primary
to
give
them.
Finally,
this
payload-
and
we
don't
want
this
like
we-
want
a
simple
protocol
unit
and
more
robust,
so
how
near
bfpo
walls
so
I'm
going
to
give
you
the
on
this
slide.
Basically,
the
idea
how
it
works,
but
essentially
there
is
another
so
that
there
is
a
throughput
bottleneck
on
as
we
scale
the
number
of
nodes
as
we
grow
the
number
of
nodes,
the
the
leader
needs
to
disseminate
the
whole
block
to
all
of
them.
A
A
So
it's
a
nice
trick
that
we
are
going
to
use,
but
on
its
own
it
doesn't
solve
us
the
problem
so
I'm
referring
to
the
topmost
picture.
It
doesn't
solve
us.
The
problem
of
the
leader
bottom
line
for
fairness,
like
this
was
introduced
by
spinning
in
literature
in
tournament
in
blockchain
practice.
There
is
this
rotation
of
the
leader
after
every
block,
which
is
nice
for
for
fairness.
If
you
want
to
give
a
reward
to
the
to
the
block
producers,
this
is
nice,
but
basically
doesn't
solve
any
of
our
problems
to
him.
A
Spinning
or
entender
mean
you
have
this
latency.
That
dictates
how
many
blocks
can
you
have
in
flight?
For
example,
if
these
three
message
lays
half
a
second,
we
have
two
blocks
per
second
throughput
and
then
depends
how
much
how
many
transactions
you
can
stuff
in
the
in
a
block.
But
it's
a
nice
idea
that
that
we'll
be
using.
It
would
be
just
nicer
that
we
use
this
bandwidth
that
first
and
the
second
leader
have
roughly
the
same
time.
So
this
is
the
main
idea
which
led
us
to
our
interim
designs
from
here.
A
Doesn't
look
like
this,
but
at
some
point
two
years
ago
it
looked
like
the
third
picture,
where
we
essentially
wait
just
from
the
next
leader
to
get
the
block
which
is
from
the
first
leader
and
then
it
disseminates.
So
basically
we
get.
We
cut
this
time
short
between
the
blocks
and
we
try
to
use
the
bandwidth
in
a
more
efficient
way,
but
then
we
realized,
of
course,
there
is
a
better
way
to
do
this,
which
is
to
do
this,
and
this
is
essentially
how
mere
DFT
works.
A
A
For
certain
modules,
for
example,
like
first
node,
proposes
box
one
five,
nine
and
thirteen,
those
that
give
basically
modulus
one
division
of
number
of
rep
of
leaders
and
so
on
and
I'm
showing
here
just
just
the
prepare
messages.
So
there
are
several
instances
of
the
protocol
and
those
like
it
wouldn't
be
visible,
like
nothing
would
be
visible
in
the
picture
if
we
would
be
showing
all
these
n
square
messages
right,
but
they
are
there,
they
are
sent
all
the
time.
So
this
is
the
main
idea
and
it's
not
rocket
science.
A
And
there
is
a
main
issue
which
is
called
request,
duplication,
so
I'm
coming
back
to
that,
but
here
I'm
over
viewing
main
principles
on
which
Mir
BFT
is
built.
So
I
would
like
you
to
focus
first
on
the
third
bullet
with
this
incrementally
built
on
pbft.
So
this
three-phase
pattern
that
I
showed
is
repeated
all
over
again
all
over
again.
It's
just
that.
We
have
now
the
first
bullet,
multiple
leaders
and
essentially
we
are
multiplexing.
It's
a
single
total
order.
A
You
will
see
what
I
mean
by
properly
also
signature,
verification,
sharding
and
I
will
mention
something
about
our
highly
paralyzed
networking
an
implementation,
and
there
are
some
tricks
we
are
using
there
to
boost
the
throughput
when
the
number
of
nodes
is
more
okay,
so
request
application.
This
is
the
main
challenge.
Of
course,
if
you
can
do
this,
multiplexing
and
the
thing
that
can
happen
is
the
following.
So
there
I
am
again
back
to
four
nodes.
Client
sends
the
same
transaction
to
two
nodes
and,
of
course,
if
they
start
their
pbft
instances
independently
of
each
other.
A
What
can
happen
if
you
just
add
this
transaction
to
block
number
one
and
block
number
two
in
parallel?
They
propose
it
and
basically,
at
the
end
of
the
story,
what
we
get
is
we
order
blocks
one
and
two
fine,
but
there
is
this
great
transaction
in
both
blocks,
so
the
client
does
this
to
all
of
the
node,
so
it
sends
a
ten
times
now.
A
Basically,
if
my
transaction
appears
and
times
I
really
didn't
improve
the
throughput
right,
if
you
think
about
the
terms
of
good
put
yes,
my
transaction,
like
I,
have
very
many
transaction
flying
on,
but
like
one
is
repeating,
one
is
repeated
n
times
and
we
have
good
put
is
basically
going
to
scale
like
one
over
and
just
like
he
did
with
leader.
So
you
you
cannot
do
this
right
and
is
the
main
challenge.
So
there
are
certain
proposals
that
go
into
this
direction,
but
they
don't
solve
this
problem
they.
A
A
So
what
can
client
do?
It
can
go
to
the
second
order
order
and
submit
a
transaction
there
and
voila
there.
You
have
the
problem
on
yet.
So
it's
not
like
easy
to
blame
the
client
if
the
transactions
appeared
more
than
once,
it's
actually
impossible,
and
even
if
you
put
because
in
fabric
we
work
for
fabric
ordering
service,
we
work
without
currency
right
without
cryptocurrency,
even
adding
like
payment
for
transactions
here
doesn't
solve.
The
problem
can
discuss
it
in
Q&A
session.
If
you
want.
So
how
do
we
solve
this?
A
Well,
we
are
going
to
use
sharding
of
transactions
by
their
hash.
So
again,
there
is
no
rocket
science,
but
there
is
a
nice
combination
of
all
these
concepts
into
a
single
product,
so
essentially
we're
going
to
partition
the
heart
space
of
the
transactions
into
four
called
buckets.
In
this
example,
I
will
have
as
many
buckets
as
the
leader.
A
Normally,
we
have
many
more
buckets
than
the
leaders,
so
we
can
kind
of
load
balanced,
better
well
with
four
leaders
is
actually
very
nice
to
divide
the
heart
space
in
four
bucket
for
the
purpose
of
the
example
and
each
leader
has
a
different
active
bucket,
which
means
that
essentially
the
leader
one
to
propose
blocks
one
five,
nine,
etc
can
only
get
put
there
two
instructions
from
the
1/4
of
the
heart
space
and
so
on
for
other
leaders.
As
we
will
see,
this
is
not
sufficient
because
of
this.
A
A
So
this
is
the
main
Ovaltine
in
maybe
ft
as
a
protocol,
so
graphically
there
below.
You
have
your
heart
space
for
transactions,
and
there
I
have
4
nodes
with
full
line.
I'm
denoting
the
I,
essentially
I'm,
denoting
the
active
packet.
So
this
node
no
zero
proposes
only
from
the
blue
bucket
and
know
number
one
proposes.
Only
from
the
red
bucket
know.
Number
two
from
the
gray
and
know
number
three
from
this
turquoise
back
and
this
basically
corresponds
to
the
quarter
of
the
heart
space
in
this
example.
A
So
there
is
a
transaction
coming
in
it
crashes
to
the
blue
part
of
the
heart
space.
This
is
this
is
a
collision
free
cryptographic,
hash
function,
they're,
not
cheese,
so
basically
sha-256
in
our
in
our
implementation,
it
hashes
to
the
blue
portion,
and
now
basically
it
means
that
it
can
be
only
proposed
by
the
known
number
zero.
A
We
are
just
rotating
the
assignment
in
a
way
that
now
the
node
number
three
is
responsible
for
this
bucket,
so
it's
going
to
propose,
given
that
it
provided
that
it
has
the
transaction
right.
So
in
limit
we
still
require
the
client
as
any
BFT
protocol
model,
to
submit,
in
the
worst
case,
to
F
plus
1
nodes,
because
whenever
it's
a
miss
to
F
for
less,
you
have
this
strange
attack
of
censorship
where
F
malicious
nodes
just
drop
the
transaction
to
the
floor.
A
So
the
number
of
nodes
I
will
show
you
some
plots
in
which
the
number
of
leaders
is
100,
and
you
can
say
this
is
very
optimistic.
So
what
we
do
is
when
the
faults
happen.
We
shrink
the
num
like
pushing
the
leader
set.
Basically,
we
still
have
at
this
stage.
We
have
the
primary,
which
is
round-robin
like
in
pbft
and
primary
decides.
What
is
his
leader
said?
It's
like
you,
have
the
Prime
Minister
and
decides
his
government.
Basically,
there
are
certain
rules
which
in
which,
if
we
had
faults
happening.
A
The
leader
set
must
shrink
and
if
false
didn't
happen,
the
leader
set
must
grow
or
or
should
grow
right,
because
it
can
stay
the
same
to
avoid
the
churn
between
like
if
some
process
is
really
faulty,
to
include
it
back
again
in
the
leader
setting
and
exploding.
You
don't
want
to
do
this
all
the
time,
so
the
details
are
in
the
paper,
but
this
is
not
the
most
richest
part.
So
here
I
am
giving
you
the
policy.
A
We
are
growing
and
shrinking
leader
set
by
one
leader,
but
this
is
really
the
what
we
are
going
to
make.
This
is
a
tunable
part
of
the
protocol.
You
could
be
aggressive,
you
could
cut
the
leader
size
in
half,
you
can
grow
it
faster
than
plus
one,
so
there
is
basically
the
two
nobility
to
the
protocol.
The
main
principle
is:
if
we
detected
that,
basically,
we
didn't
make
progress
with
that
number
of
leaders.
A
So
then
we
have.
This
is
vanilla,
Mir,
how
it
works
and
we
have
to
optimizations.
One
is
when
the
network
is
basically
the
bottleneck,
and
it's
if
you
want
it's
disseminating
the
hashes
in
a
proper
way.
So
what
this
means
in
a
proper
way?
Basically,
we
are
going
to
deliver
the
transaction
as
opposed
to
delivering
the
payload
of
the
transaction
to
every
correct
node.
A
We
are
going
to
deliver
the
hash
of
the
transaction
to
every
correct
node
and
we
are
going
to
deliver
the
payload
to
at
least
one
correct
node,
which
is
interesting
for
fabric
because,
as
you
know,
in
fabric
the
peers
pull
transactions
from
certain
orders.
So,
strictly
speaking,
you
don't
have
to
guarantee
that
all
orders
have
all
the
transactions.
You
just
need
to
make
sure
that
the
peer
can
find
these
transactions
in
all
places.
A
So
how
this
works
is
basically
that
F,
plus
one
of
the
nodes
get
the
full
payload
which
is
disseminated
by
the
leader
and
others
can
get
the
hash
like
the
block
with
hashes,
so
called
the
light
block
or
the
light
batch,
and
then
the
agreement
basically
proceeds
as
normal.
It's
just
that
you
kind
of
shard
it
this
payload
to
F,
plus
one
knows
so.
The
peer
is
guaranteed
to
find
it.
Of
course,
this
will
be
tunable
and
you
can
switch
it
off.
If
you
don't
like
it,
it's
too
complex.
A
You
want
your
ordering
service
nodes
to
all.
Have
the
payload
you
switch
it
off.
I
will
show
the
impact
on
the
throughput
signature,
verification
sharding.
This
is
for
client
authentication
each
of
course.
I
didn't
show
this
so
far,
but
each
client
transaction
is
signed
by
the
by
the
client
signature
and
essentially,
what
we
have
in
the
protocol
so
far
is
I
didn't
show
these
signatures
right,
so
they
are
verified
by
each
and
every
node
in
the
system.
A
Of
course,
we
can
apply
the
similar
principles
for
light
autoloader
broadcast
that
I
showed
and
basically
have
only
F,
plus
1
nodes
validate
this
signature.
So
it
gives
you
constant
increase
in
throughput
when,
basically,
you
have
a
very
fast
Network
and
Terkel
is
computationally
bound
by
signature.
Verification
but
like
it
gives
you
basically
3x
throughput
on
paper,
which
is
which
basically
translates
in
our
experiments
to
2x
or
something.
So
these
are
two
optimizations
that
we
use
in
the
protocol.
A
So
this
is
the
most
interesting
part,
which
is
the
performance
evaluation
so
for
details
we
can
discuss
it
or
I
can
point
you
to
the
paper.
But
basically,
then
we
compared
Mir
to
several
baselines.
One
is
the
chain
replication,
so
this
is
not
really
used.
So
it's
really
the
best
performing
total
order
broadcast
protocol
for
clusters.
A
It's
not
very
popular
for
block
chains,
because
how
it
replicates
is
very
optimistically
so
it's
like
if
we
would
illuminate
all
of
us
in
the
room
and
do
the
replication
by
just
miss
ending
the
message
to
block
there
and
then
the
block
wrap,
gets
replicated
in
a
chain
pattern.
So
it's
fragile
to
false,
because
if
you
have
a
any
node
failing,
basically
we
have
the
chain
cut,
but
it's
really
nice
as
a
baseline,
because,
apart
from
latency,
it's
really
throughput
efficient,
I'm,
going
to
use
it
just
for
the
baseline
to
show
how
actually
Mir
performs.
A
Then
we
compared
to
Libra
hot
stuff
to
our
version
of
pbft,
so
the
single
we
basically
specialized
our
code
to
do
a
pbft
single
either
version
of
it.
So
we
can
compare
to
single
either
protocols
and
we
did
this
on
local
area
network,
which
is
less,
although
most
of
the
deployments
currently
run
a
single
cloud.
We
are
focusing
on
wide
area
networks,
so
we
went
to
100
nodes
up
to
100
data
centers
on
IBM
cloud.
These
are
the
specs
and
when
we
got
the
VM,
we
got
to
one
gigabit,
full
duplex
bandwidth.
A
We
tested
with
two
transaction
sizes:
one
is
like
what
most
of
block
chains
have,
and
one
is
what
this
very
popular
block
chain
has,
because
it
carries
the
x.509
certificates,
but
we
wanted
a
useful
number
for
fabric
right.
So
I
would
like
to
see
this
disappear
and
actually
go
here,
but
is
this
totally
separate
discussion
but,
like
we
tested
with
these
two
transaction
sizes?
Okay,
so
on
right
there
and
networks.
The
performance
looks
like
this,
so
this
is
Mir.
This
is
chained
PFT,
and
this
is
your
n.
A
If
you
will,
this
is
your
almost
approximating
your
any
leader
based
protocol?
The
absolute
numbers
are,
like
you
see
this
bump
in
scalability,
just
because,
for
example,
if
you
do
so,
this
is
both
optimization.
So
this
is
with
signature.
Verification
optimization
for
Bitcoin
size
is
actually
only
slinging
signature.
Verification
optimization
for
for
nodes.
You
have
half
of
your
like
F,
plus
one
is
two:
it's
not
one-third.
We
have
half
of
the
nodes,
verifying
the
signature,
whereas
for
sixteen
nodes
this
is
actually
one-third.
A
So
there
is
this
bump
in
throughput,
which
is
actually
better
than
if
you
go
with
four
nodes,
but
then
it
stays
flat
and
it's
going
to
drop
like
when
natural
becomes
the
bottleneck.
So
here
we
are
compute
band
and
the
bound
bound
is
actually
the
signature.
How
many
signatures?
Can
you
verify
with
this?
A
A
This
is
hot
stuff,
so
that's
hot
stuff
and
that's
mere
okay.
Now
hot
stuff
is
the
version
that
sends
hashes
only
so
we
just
took
the
code
hot
stuff
cents
crushes
only.
Although
the
protocol
specification
says
that
it
should,
the
leader
should
send
the
payload,
so
it
will
realize
what
I
showed
it
relies
on
the
client
to
disseminate
the
request
and
it
doesn't
have
client
authentication.
So
basically,
nobody
checks
these
client
signatures,
so
I'm
really
not
advocating
this
I'm,
definitely
not
advocating
this,
but
then
we
tune
Mir
to
do
exactly
the
same.
A
A
So
if
you
read
the
geo
replicated
papers,
they
would
typically
give
you
the
bandwidth
between
the
two
sites
like,
but
we
really
believe
that
between,
like
Brazil
Amazon,
Web,
Services
or
IBM
data
center
and
that
in
Europe
you
have
50
megabit
throughput,
you
don't,
but
is
the
limit
on
a
single
connection,
just
go
and
open
another
one,
and
this
is
basically
the
effect.
So
we
open
several
connections
in
parallel
which
basically
go
around
the
bandwidth
delay,
product
limitations
of
the
TCP
and
basically
you
get
better
flow.
A
Now
hot
stuff
doesn't
do
it
and,
as
you
grow
the
number
of
nodes
now,
the
leader
can
disseminate
the
request
to
actually
more
and
more
nodes
because
of
this
effect
right,
but
it
stops
at
some
point
at
some
point,
which
is
actually
with
60
nodes
you
reach,
even
if
the
leader
disseminates
just
the
hashes,
you
saturate
the
network
and
from
then
on
it
just
behaves
how
it
should
be,
which
is
inversely
proportional
to
number
of
nodes,
whereas
mere
skills
much
better.
The
absolute
numbers
are
higher,
but
this
is
because
we
are
now
not
really
again.
A
We
are
not
verifying
validating
client
signatures,
so
absolute
numbers
should
be
taken
here
as
a
grain
of
salt.
These
are
not
the
numbers
that
should
be
expected
from
actual
robust
implementation,
but
it
just
shows
the
scalability
thread
thread.
So
again,
it's
not
important
how
many
messages
your
consensus
sense
like
oh
and
verses,
are
in
square,
but
basically
there
is
this
idea
of
multiple
leaders
and
request
duplication,
elimination
that
that
makes
sense.
Of
course
you
can
make
a
mirror
version
with
hot
stuff.
If
you're
happy.
We
are
just
not
doing
that
at
the
moment.
A
This
is
fabric.
The
red
line
is
fabric
validation,
bottleneck.
Why?
If
somebody
asked
me
like
pika
protocol
and
implement
it
put
it
in
fabric,
I
was
saying
we
actually
need.
Something
else
is
hopefully
obvious
from
this
picture.
There
is
nothing
to
pick
because
like
as
soon
as
you
cross
their
nodes,
anything
that
you
pick
is
a
bottle,
and
since
we
are
not
going
to
do
this
turn
like
n
times.
This
is
why
we
took
time
and
actually
went
and
designed.
A
So
this
is
this-
is
mere
performance
I'm
not
happy
with
this
either
I
would
like
to
see
this
flat,
but
this
is
mostly
because
we
attribute
this
to
heterogeneity
of
not
at
least
on
IBM
cloud.
So
you
get
one
gigabit.
If
you
take
Brazil
datacenter
is
not
really
transforming
to
one
gigabit,
even
with
multiple
connections
etc.
But
basically
we
have
a
certain
drug.
A
Basically,
what
this
plot
shows
is
that
for
any
deployment
of
fabric
up
to
100
nodes,
the
ordering
DFT
ordering
service
is
not
going
to
be
the
bottleneck,
which
was
the
main
inspiration
for
this
work.
So
this
was
the
chief
now
impact
of
optimizations
like
how
lucky
you
told
us
about
this
LTO
and
signature
verification,
but
these
are
really
tricks.
I,
don't
want
to
use
them.
If
you
don't
want
to
use
them.
This
is
the
Bitcoin
size.
So
now
the
the
this
is
the
snapshot
for
16
nodes
any
sixteen
across
sixteen
data
centers
when
IBM
Cloud.
A
This
is
vanilla
Mir,
and
this
is
the
60,000
transactions
per
second
I
reported.
So
if
you
don't
like
the
optimizations,
you
are
going
to
end
up
with
this,
which
is
pretty
impressive,
still
so
you're
going
to
lose
some
30%
of
the
performance,
but
it
will
still
be
at
40,000
transactions
per
second.
This
is
the
impact
of
rotation
right,
so
bucket
rotation.
We
need
it
for
basically
to
eliminate
censoring
by
Byzantine
replicas,
and
it
turns
out
that
there
is
not
much
impact
rate.
So
there
is
a
10%
impact.
You
need
to
do
it.
A
A
Here
we
got
the
numbers
for
throughput
right
flow
when
we
get
the
throughput
to
the
saturation,
so
we
are
increasing
the
number
of
clients
until
basically
throughput
grows.
As
we
increase
the
number
of
clients,
the
roll
load
increases
and
the
latency.
Basically,
there
is
queuing
in
the
system.
Latency
grows
up.
This
is
why
it
shoots
here
so
latest
you
can
read
here
typical
latencies.
You
read
here
right
when
the
system
is
not
under
saturation,
and
these
are
basically
about.
A
This
is
the
duplication,
prevention
impact,
but
I
would
rather
do
we
have
time
like
what
is
the
time
lucky.
There
is
still
some
time
actually
I
have
five
minutes
right,
so
I
would
like
actually
to
have
questions,
but
is
it
the
impact
of
duplication
prevention?
So
this
is
if
we
like,
eliminate
the
duplication,
prevention
and
I,
let
like
other
protocols
I.
Let
me
repose
without
this
sharding
of
requests,
right
I,
let
them
propose
anything
I,
let
the
leaders
propose
anything
that
they
get
and
basically,
if
the
client
sends
to
F
plus
one.
A
This
is
the
blue
plot
and
if
the
client
sends
to
three
F
plus
one,
there
is
a
lot
more
duplicates.
So
there
is,
there
is
the
the
purple
plot
and
essentially
because
this
is
measuring
would
put
because
the
transaction
is
duplicated,
F,
plus
one
or
three
F,
plus
one
times
the
good
put
is
actually
much
lower.
A
This
is
the
performance
with
faults.
You
can
observe
some
like
if
we
have
sixteen
leaders
and
we
actually
crash
the
leader,
the
the
protocol
reconfigures
and
goes
back
to
fifteen
meters,
so
you
do
kind
of
have
the
throughput
dropping
to
zero.
But
basically,
then
you
have
the
shoot
up
and
then
we
were
delivering
in
a
burst
right.
A
So
this
should
be
actually
I
should
put
on
this
pod,
the
average
throughput,
which
would
go
like
something
like
this,
and
then
you
have
the
example
where
two
leaders
fail
and
then
we
go
back
to
14
leaders,
and
then
we
include
the
15,
feeders,
etc.
But
basically
you
see
that
the
average
throughput
is
not
in
this
example
would
be
probably
around
30,000
transactions
per
second
or
so
so.
The
protocol
does
actually
eliminate
leaders
and
come
back
to
the
to
the
good
performance.
So
we
stressed
it
a
bit.
A
There
are
more
stressful
experiments
in
the
paper,
so
invite
you
to
read
it.
It's
currently
fighting
its
way
through,
basically
to
get
published.
There
is
a
lot
of
competition,
like
a
lot
of
people
seem
to
be
liking.
This
idea,
but
it'll
eventually
go
through.
The
main
news
is
basically
this,
so
it's
going
to
in
now
we
are
discussing
how
like
which
features.
I
would
like
that.
All
features
that
I
discussed
here
with
you
are
present
in
the
production:
implementation
of
mihrab
EFT.
A
There
are
many
variables
there
it's
implemented,
but
there
is
the
implementation
which
is
kind
of
IBM
proprietary,
which
I
would
like
to
get
open
source
and
like
I
hope,
regardless
of
it
just
for
the
repeatability
of
the
paper,
is
going
to
probably
do
be
dead,
but
then
we
are
essentially,
we
started
reimplementation
based
on
this
code
of
what's
going
to
become
an
open
source
implementation
yeah.
So
it's
on
go
it's
finished
and
it's
ongoing
at
the
same.
A
It's
a
yes,
it's
a
pluggable
thing,
but
this
is
independent
library.
So
much
like
raft
is
completely
dependent
and
then
we're
is
the
same
concept
which
is
also
nice
for
other
blockchains.
I
would
say
if
you,
if
you
want
to
use
it
so
I
will
stop
here.
I
have
actually
many
different
experiments,
but
there
don't
give
us
too
much
of
a
new
insight
like
actually
we
did
really
really
thorough
experiments.
They
are
all
listed
in
the
paper,
but
I
would
rather
actually
take
few
more
questions
in
this
to
three
minutes
that
we
have
left.
A
A
A
So
so
here,
I
took
the
took
the
problem
apart.
Much
like
fabric
does
in
a
sense
that
the
execution,
of
course,
like
all
these
numbers,
if
you're,
going
to
execute
if
you're,
if
you're
treatment
of
transactions
here
we're
just
ordering
and
validation,
is
already
doing
something
in
fabric.
So
if
you,
if
you
put
an
application,
this
Pro,
these
numbers
are
going
to
most
probably
drop
down,
but
this
is
me
this
is
our
goal
precisely
so
we
don't
want
ordering
service
to
become
a
bottleneck
that
that's
our
job.
A
Quite
a
lot
quite
a
lot
from
from
the
pattern
which,
like
if
root
of
f
is
like
I,
said
we'd
f5,
F,
plus
1.
We
are
not
even
talking
about
three-phase
protocol.
We
are
talking
about
two-phase
protocol,
all
the
time
actually
probably
correct,
and
there
are
other
optimizations.
Now
you
can
start
gossiping
the
payload
much
easier,
because
here
the
gossip
really
alike.
We
talked
about
client
dissemination
and
leader
dissemination.
You
have
something
in
between
which
is
gossip
and
gossip.
A
A
Honestly,
so
I
didn't
give
you
the
with
one
gigabit
this.
This
point
is
like
500
600
Meg's
per
mega
bits
per
second,
so
it's
like
the
so
what
I
expect
is
better
latency
to
certain
extent
better
throughput,
but
not
much,
but
it
doesn't
really
even
matter
at
this
moment.
So
this
moment,
like
I,
would
say
we
need
to
implement
this
really
properly.
It
needs
to
be
robust.
A
It
cannot
be
just
just
reporting
the
with
the
best
case
numbers
and
then
we
are
back
to
the
main
story
of
this
forum,
which
are
applications,
but
at
least
we
get
this
problem
or
for
permission,
blockchains,
I'm,
I'm,
arguing
the
diskens
huge
impact
on
permissionless.
But
of
course,
let's
take
this
for
some
other
talk.