►
Description
Hyperledger Caliper: A Benchmark Tool For Multiple Technologies - Nick Lincoln, IBM
Hyperledger Caliper is a performance benchmarking tool for blockchain technologies.It uses adapters to interact with different technologies and consequently is capable of benchmarking any blockchain technology that provides an adapter.
This presentation will provide information on current and future features of Caliper, as well as providing information on configuring, running, and generating output of a Caliper benchmark for an example blockchain technology.
----
Hyperledger – Open Source Blockchain Technologies
Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory
Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter
Follow-us on Twitter @Hyperledger
Learn about Hyperledger Membership: https://www.hyperledger.org/about/join
#Hyperledger #HyperledgerGlobalForum2020 #Blockchain
A
Yeah,
okay,
great
I'm!
Well,
thank
you
for
attending
and
thank
you
also
for
tearing
yourself
away
from
from
lunch.
My
name
is
Nick.
I
am
a
software
engineer
for
IBM
I'm
here
today,
as
my
capacity
as
a
maintainer
for
hyper
ledger,
caliper
I'm
going
to
be
a
bit
strange
now
and
I'm
going
to
close
off
the
slides
at
the
very
first
point
now
so
I'm
going
to
disappear.
A
A
So
what
we're
going
to
talk
about
so
well
I'm
going
to
be
talking
a
lot
about
caliper
I'm,
going
to
give
an
overview,
then
I'm,
going
to
move
on
to
the
supported
technologies,
I'm
going
to
touch
on
the
concepts
that
caliper
uses
speak
on
the
operational
modes
of
caliper,
then
I'm
going
to
move
on
to
the
other
end,
but
right
off
the
the
end
result
of
caliper.
That
is
it's
the
report
generation
and
it's
all
about
performance
tests
and
results.
A
If
it's
possible
quick
show
of
hands,
has
anybody
used
caliper
before
one
two
yep,
three?
Okay,
as
people
heard
of
caliper
anyone?
This
is
completely
fresh.
What's
what's
possible
here?
Okay,
so
a
good
mix,
great
ok!
So
well,
Hydra
caliper!
It
is
a
block
chain
performance
testing
framework
and
that
allows
you
to
test
different
blockchain
solutions
with
predefined
use
cases
we're
not
bound
to
a
single
technology.
A
The
idea
is
that
you
can
transfer
across
the
same
scenario
tests
to
different
VLT
technologies,
and
you
can
run
the
same
predefined
use
case
and
at
the
end
of
that,
you'll
get
a
generated
performance
reports
for
the
test
case
that
you
have
run
and
in
that
report
you'll
have
details
on
the
transaction.
Success
of
fail
counts,
the
transaction
throughputs,
that's
a
TPS
or
transactions
per
second
transaction
latency.
How
long
one
of
these
transactions
take
circuit
two
completes
and
it
will
also
detailing
resource
consumption,
so
how
much
CPU
is
being
consumed?
A
How
much
memory
you
using
the
network
IO
disk
IO,
so
you
can
see
all
different
points
of
constraints
for
the
system
you're
looking
at
and
that's
probably
the
highest
level
takeaway
for
this
presentation.
So
caliper
is
about
testing
multiple
frameworks,
generating
reports
on
transaction
statistics
and
resource
consumption.
So
why
would
you
want
this?
Why
would
you
use
it?
A
What
could
you
use
it
for
so
you
can
use
it
or
view
its
for
use
on
different
planes
so
for
an
application
developer,
you
may
want
to
see
performance
test
your
smart
contract
so
you've
been
for
the
development
cycle.
You
have
a
solution.
How
performance
is
it?
You
may
be
a
systematic
architect.
You
may
be
looking
at
where
your
resources
are
being
used
where
you
want
to
spend
the
money
so
you'll
be
looking
more
the
resource
constraints
during
a
test
load
or
hopefully,
you're
interested
in
performance
regression
testing.
So
back
up
the
chain.
A
Perhaps,
and
so
you've
got
your
solution
and
you're
iterating
your
solution
and
you're
making
sure
that
you
only
getting
better
transaction
statistics
you're
only
going
faster,
you
have
troopers
greater
your
latencies
are
lower
or
flipping
that
around
you
might
be
a
fabric
developer,
a
quarter
developer
one
or
quarter
yes,
spoilers
say
aetherium,
that's
the
sawtooth
and
you
might
be
changing
the
platform
code.
If
you're
working
in
fabric,
you
might
be
changing
the
core
code
inside
fabric.
If
you're
an
SDK
developer,
you
might
be
changing
the
way
that
eventing
works.
A
A
A
A
We
will
build
that
in
and
publish
at
an
unstable
tag,
so
you
can
gather
instantly
and
when
we
feel
we've
sufficiently
broken
the
original
implementation.
All
we
are
sufficiently
feature
complete,
then
we'll
tag,
an
official
release
level
on
NPM
we're
around
about
two
and
a
half
thousand
downloads
and
docker
hub.
We've
got
about
a
thousand
pools,
so
we
are
getting
there
and
when
you're
using
caliper,
it's
mainly
driven
via
command
line
interface.
A
So
licking
our
supported
technologies,
we're
adapter
based,
there's,
no
real
limit
on
the
technologies
we
can
consume
or
map
to.
At
the
moment
we
are
able
to
run
a
performance
tests
on
best
suit,
borough
fabric,
aroer
sort've,
there's
the
hyper
ledger
items
and
we're
also
supporting
if
cerium
and
Fiske
OBC
RS.
A
A
A
So
this
is
the
definition
of
our
target
blockchain
technology
and
it's
any
one
of
those
supported
platforms,
so
we're
defining
our
network
and
then
we'll
have
a
series
of
smart
contracts.
This
museum
is
the
logic
that
you
have
developed
as
a
developer:
it
governs
the
life
cycle
of
the
assets
being
modeled
within
this
target
block
team
technology.
These
are
generally
bound
together.
A
This
is
not
a
strictly
necessary
part
because
we
all
need
this.
If
we
want
to
stand
the
network
up
itself,
so
we'll
be
standing,
a
network
up,
deploying
your
small
contracts,
that's
an
optional
stage.
The
less
optional
stage
is
all
their
concepts
is
a
test
definition,
so
we
defined
the
way
methods
are
available
in
small
contracts.
So
if
you're
thinking
in
fabric
you're
going
to
be
submitting
specific
transactions
which
target
a
method
you
have
defined
so
well,
caliper
needs
to
know
how
to
deal
with
this.
A
So
we'll
have
a
test
definition
file,
be
called
a
benchmark
and
patna
benchmark
file,
and
it's
really
a
sequence
list
of
events,
so
you
can
consider
it
as
the
flow
for
the
test
being
run,
and
this
will
be
to
say
it's
considered
we're
testing
creation
of
assets.
You
may
have
a
series
of
tests.
You
want
to
run
where
you
change
the
size
of
the
asset
being
created
or
the
content
of
the
asset.
A
Being
created
and
you're
going
to
be
saying
how
you
would
like
that
to
be
controlled,
so
what
rate
do
you
want
to
be
driving
the
test
at
how
long
you
want
the
test
to
be
run
for?
Are
you
running
this
for
a
series
of
trans
and
discrete
transactions
or
you're
running
this
for
a
time?
Duration,
so
you
can
see.
Is
the
entire
control
flow
for
the
output
benchmark
you're
running,
and
this
still
calls
into
tests
libraries?
A
Now
we
give
you
absolute
control
over
this
aspect,
so
you
could
probably
think
of
caliper
or
consider
caliper
as
a
bit
of
a
dummy
item
that
just
sends
transactions
and
it's
up
to
you,
the
user,
to
tell
caliper
is
the
content
of
that
transaction.
So
again,
if
we
consider
creating
an
assets
scenario,
it
may
be
the
inside
your
user
module.
You
have
the
logic.
It's
required
to
populate
the
information
as
going
to
that
asset,
so
isn't
ie
up
to
you.
A
It's
very
easy
to
configure
I
would
say
it's
quite
easy
to
configure,
because
I'm
stood
here
and
I've
used
caliper
for
quite
a
while,
but
essentially
you
define
the
transaction
content
going
through
now.
What
does
caliper
do
so?
Caliper
takes
these
items,
it
takes
the
network,
it
takes
the
libraries
and
your
test
definitions
and
it
will
then
run
through
test
definition,
targeting
these
user
modules
and
once
it's
completed,
this
is
going
to
generate
a
result.
A
So
because
we
are
running
is
transaction,
we're
monitoring
what
goes
on
and
we're
going
to
be
taking
transaction
statistics.
So
a
point
of
note
is
that
the
transaction
statistics
themselves-
these
are
taken
from
the
reference
points
of
the
driving
SDK.
So
if
you're
considering
the
transaction,
latency
is
from
the
point
of
which
the
SDK
is
submitted,
a
transaction,
it's
when
it
comes
all
the
way
back
round-
and
it
is
complete
again
and
it's
same
for
the
throughput,
so
we're
not
measuring
on
the
DLT
itself.
A
It's
as
observed
from
the
client
side
now
I've
been
talking
all
about
these
generated
results,
so
we
collect
those
but
we're
also
collecting
resource
statistics,
and
these
can
be
done
through
the
most
simplistic
way,
which
is
docker
stats.
That's
our
default
option
or
you
can
go
a
little
bit
further.
A
You
can
use
Prometheus
with
custom
queries,
so
we
integrate
with
Prometheus
and
as
soon
as
you
do
that
you
have
availability
of
quite
a
few
scrape
targets,
and
then
you
have
the
other
options
of
looking
at
things
in
real
time
with
graph
fauna,
so
operational
modes,
v4
option
for
caliper
would
be
a
say
user,
the
CLI
command,
and
it's
essentially
you
run
the
test.
But
if
you
want,
you
can
break
this
down
in
the
phases,
so
caliper
would
normally
start
up.
It
would
initialize.
A
So
that
would
be
the
standing
of
the
network
deploying
a
smart
contracts
and
then
we
run
a
test
and
then
you
do
an
end,
which
is
the
teardown
situation.
Now
it
maybe
wants
to
do
a
skip
or
an
only
on
any
one
of
these
options.
Seam
I
want
to
only
starts
only
initialize,
and
then
you
could
do
a
sequence
of
only
test,
so
that
maintains
the
state
of
the
DLT
and
you
can
spin
these
tests
over
and
over
and
over
or
it
might
be
that
you
don't
want
caliper
to
stand
up
your
network.
A
You
have
your
own
processes
that
do
this.
You
may
have
a
whole
load
of
scripting,
you
might
be
using
playbooks,
you
might
have
a
system,
that's
already
been
configured
for
you,
and
all
you
want
to
do
is
test
it.
That's
fine!
You
can
grab
the
credentials
from
the
system,
pass
them
to
caliper
and
do
an
only
test
kind
of
operation.
A
A
So
the
local
Modi's
is
exactly
that
everything
is
running
on
the
system.
We've
installed
caliper,
all
the
containers
will
be.
There
were
sworn
processes
inside
that
same
system
to
target
the
DLT
and
we
use
the
locally
based
resource
monitoring
tools.
So
everything
is
very
much
contained
distributed.
Is
the
absolutely
opposite
of
that?
It
would
be
where
you
have
a
remotes
DLT
have
remote
caliper
clients
driving
a
workload,
you'll
have
a
manager
style.
A
So
yes,
so
a
manager
style
caliper
operation,
which
is
coordinating
the
workers,
collecting
the
results
and
assembling
a
report,
and
then
you've
got
the
hybrid
style
there,
and
perhaps
you
don't
want
one
aspect
of
the
local
system:
that's
fine!
So
you
can
delegate
off
the
resource
monitoring
to
Prometheus
and
scrape
targets.
You
can
then
move
out
your
workers
to
be
distributed
as
well.
I
think
that
kind
of
follows
relatively
nicely.
A
But
this
is
how
all
caliper
looks
if
you're
working
and
not
a
non
distributed
mode,
and
if
we
start
on
the
left
top
corner,
we
have
our
configuration
mentioned
it
before
our
network
definition
or
smart
contracts,
the
test
definition
and
a
test
modules.
These
are
your
parts
that
you
need
to
feed
caliper
and
when
that
comes
through
into
the
either
master,
we
have
an
optional
admin
worker
and
that
works
through
an
adapter
and
to
configure
the
system
under
test,
possibly
stand
it
up
as
well,
and
then
we
have
a
worker
Orchestrator
that
spawns
multiple
workers.
A
Now
each
of
these
workers
are
rate
controlled
through
an
adapter
and
that's
producing
the
workload
as
the
workers
operates.
They're
publishing
their
transaction
statistics
to
a
performance
analyzer.
At
the
same
time,
there
is
a
local
resource,
monitor
that
is
running
and
when
the
test
cycle
is
complete,
it
passes
all
this
information
through
a
report
generator
and
you
get
your
test
report.
A
So
if
you
run
that
locally,
so
if
I
was
to
run
that
on
my
match,
the
results
are
quite
low.
Quite
obviously
because
it's
on
one
system,
CPU
constrained-
and
you
know
at
some
points
you
can
go-
not
many
people
run
their
production
system
on
a
local
PC.
So
yeah
you
want
to
start
moving
towards
this
distributed
realm.
A
So,
let's
look
at
the
hybrid.
We
still
have
our
configuration
and
the
top
left
and
we
still
have
the
caliper
master,
which
has
an
admin
worker
and
a
work
Orchestrator
spawning
local
clients.
The
difference
now
is
that
we've
got
Prometheus
in
the
mix
so
Prometheus.
If
you're
not
familiar
with
it,
is
a
very
useful
tool
for
collecting.
A
Well
information
really,
and
so
we
can
publish
worker
statistics
through
to
Prometheus
and
we
can
use
external
resource
monitors.
They
see
advisor
node
exporter,
so
you
collect
information
from
the
system
under
test.
That's
all
coming
through
to
Prometheus,
which
is
being
held
inside
the
Prometheus
and
containers
and
on
its
own
database
store,
and
then
we
retrieve
all
that
information
from
Prometheus,
so
we're
still
using
the
local
drive
and
driving
clients,
but
we've
delegated
resource
collection
elsewhere,
and
that
then
comes
through
our
report
generator
and
we
get
our
test
report
out
the
other
side.
A
So
next
step
would
be
to
move
the
workers
outside
and
we
can
do
that.
So
we
still
have
our
net
word
definition:
smart
contracts,
the
test
definitions
and
the
test
modules.
There's
the
option
for
the
admin
worker
inside
our
master
process
to
configure
the
system
under
test,
but
we
have
our
workers
external
now.
So
we
have
these
work,
the
workers
good.
Well,
they
can
be
on
any
system.
A
They
are
will
be
there's
docker
containers,
and
so
you
can
scale
these
up
to
your
heart's
content,
but
they
still
work
in
exactly
the
same
way:
they're
still
rate
controlled
through
an
adapter
targeting
a
specific
system
under
test
and
they're
still
publishing
to
Prometheus
and
prometheus
still
taking
additional
details
from
any
resource
monitors,
you've
configured
and
then
this
is
run
through.
Our
same
repo
generator
to
gather
a
report
so
there's
a
kind
of
a
very
blocky
kind
of
description.
A
A
A
Prometheus
is
also
targeting
as
a
adviser,
node,
mitre,
scraped
targets.
It's
pulling
information
from
the
peers
themselves,
because
fabric
is
able
to
export
metrics
and
to
the
caliper
master,
as
well
as
communicating
Philo
workers
which
help
us
orchestrate
the
entire
test
flow,
and
they
can
then
drag
in
any
information.
It
needs
from
Prometheus
to
generate
our
test
reports.
A
So
that
would
be
a
very
concrete
kind
of
view
of
how
you
could
configure
caliper
to
work,
and
you
can
imagine
this
could
work
in
kubernetes.
So
if
you're
in
a
docker,
swarm
kind
of
scenario,
so
yeah
it's
getting,
we
can
move
from
a
local
laptop
running
on
your
Mac
to
something
suitable
for
what
enterprise
level
testing.
So
what
do
we
do?
Caliper
exports
certain
items
to
Prometheus,
so
we
can
read
so
that
we
can
retrieve
them
later.
We
send
out
TPS
the
latency
and
our
success
in
fail
counts.
A
So
this
is
per
worker
and
we
can
then
run
queries
on
these
to
aggregate
results
to
put
into
the
reports.
What
we
do
is
we
use
the
worker
Orchestrator
to
essentially
walk
through
your
test
definition,
so
the
benchmark
configuration
file
you
have
will
step
through
that
will
make
sure
that
each
worker
is
kept
in
stay
in
place
in
sequence
or
they're
all
operating
at
the
same
time
and
between
each
round
the
worker
Orchestrator
will
issue
time,
bounded
queries
for
the
Cupra
statistics,
so
these
are
what
you
get
by
default.
A
So,
okay,
that's
quite
handy
I,
believe
so
you're
always
going
to
get
your
TPS
your
latencies
success
and
fail.
But
you
know
you've
been
configuring,
we
have
the
option
to
configure
additional
resource
matters,
for
instance
see
advisor
and
what
that
then
gives
us.
The
ability
to
do
is
say:
okay,
yeah
sure,
you're
going
to
get
a
certain
number
of
things
feel
free.
If
you
want
something
else
inside
your
report
tell
us
what
it
is.
A
So
you
tell
us
that,
and
essentially,
if
you
can
run
a
Prometheus
query
and
retrieve
the
information,
then
just
put
it
in
as
you
would
well
so
here
basically
tell
caliper
the
query:
you
want
to
run
what
you're
calling
it
and,
in
the
instance
here
I've
run
a
query
saying
you
know:
I'm
interested
in
the
maximum
memory.
Consumption
seems
during
each
test
within
the
entire
test,
as
definition
by
I
want
megabytes.
A
So
we've
got
this
notion
of
a
multiplier
because
I
know
they're
coming
out
and
bias.
So
there
is
a
multiplier.
You
can
add
in
I'd
direct
you
to
the
documentation
for
this.
If
you
want
to
start
using
custom
queries
because
there
are
well,
there
is
a
fixed
definition
with
fixed
options,
but
it
is
highly
flexible.
So
if
there's
any
information
that
you're
storing
in
Prometheus,
you
want
to
pull
out,
we
can
do
that
for
you.
A
A
A
Number
of
iterations
or
number
of
transactions
sent
okay
yeah.
So
on
that
we
cheat.
So
we
say:
when
does
the
time?
When
does
the
test
start
so
that
the
workers
would
be
like
hey,
yeah,
I'm,
ready,
I
finished,
then
there's
an
optional
initialization
phase
and
there's
okay,
I've
initialized
everything,
and
once
all
those
workers
confirm
they're
ready
to
go,
the
master
will
say
and
go
and
we
take
a
time
index
when
all
the
workers
have
completed
and
come
back,
say
we're
done.
A
B
A
I'll
show
you
in
a
minute:
okay,
so
fabric
I
hinted
to
earlier.
Actually
I
mentioned
it's
so
fabric
peon
order.
Metrics
are
exported
for
consumption
by
Prometheus
and
Cystic.
There's
a
reference
point
there.
If
you
want
to
follow
it,
but
that
means
that
we
can
start
pulling
out
exceptionally
customs
queries
so
for
you
and
I'm
only
interested
in
fabric
things,
but
perhaps
I'm
interested
in
well.
This
query
here,
which
is
pulling
out
the
end
you'll
need
the
endorse.
A
A
proposal,
duration,
there's
quite
a
nice
item
to
want
to
pull
out
I-
might
actually
be
interested
in
time.
Spending
CouchDB
I
might
be
interested
in
the
time
taken
to
commit.
So
all
of
those
metrics
are
available.
All
those
can
be
pulled
out.
You
can
create
a
custom
query
for
that
now,
I'm
going
to
jump
straight
across
to
the.
How
would
I
see
how
it
trends
in
time?
So,
if
you're
using
Prometheus,
you
can
also
use
graph
honor
and
grow
finer
exposes
everything
that
stored
inside
Prometheus,
so
you
can
visualize
the
exported
metrics.
A
You
can
do
that
in
real
time,
but
you
can
also
go
back
in
history
and
see
what
has
happening
between
a
specific
run
or
at
a
known
time,
points.
The
item
on
the
right
is
one
of
those
did.
You
know
there
is
a
graph
on
a
dashboard
for
fabric
there.
It
is
so
here's
the
dashboard,
Oh
10
7,
1
6.
It
just
rolls
off
it's
easy
to
get,
and
so
that's
one
fabric,
one
for
X,
but
there
are
lots
of
dashboards
so
on
the
bottom
left.
That
is
a
docker
stats
dashboard.
A
You
ain't
that
just
ties
straight
in
and
you
can
get
nav
a
deep
dive
and
view
metrics.
You
can
also
create
your
own
custom
queries
improve
inside
graph
honor.
Let's
have
a
look
at
something
that
you
want
to
see
and
yeah
definitely
recommend
having
a
look
at
medias.
If
you
never
used
it
and
hooking
graph
Hana
is
exceptionally
useful.
A
We
are
intending
to
put
together
a
graph
on
a
dashboard
for
caliper.
Unfortunately,
we
just
haven't
quite
got
there
yet
so
it
isn't
plan,
so
you've
got
to
jump
through
tabs
and
you'll
be
able
to
either
oh
yeah.
Well,
the
very
least
you'll
be
seeing
the
the
TPS,
the
calipers
producing
or
well.
The
caliper
is
observing
and
the
latencies,
so
there
will
be
something
is
inbound.
A
Okay,
so
I've
spoken
quite
a
bit
about
that
this
so
far,
the
the
way
we
are
being
able
to
set
up
a
distributed
system
and
you
can
run
your
test
and
you
get
some
results.
But
what
are
these
results?
Look
like?
Well,
you
get
a
report
generated
freeze
test
each
test
definition
run.
This
is
our
new
style
of
report
slightly
shinier
than
it
was
before.
If
you
see
the
earlier
ones
and
there's
only
one
test
run
here.
A
So
if
you
had
a
test
definition
which
was
running
multiple
tests,
you
would
see
a
highlight
on
the
very
top
indicating
the
number
of
success
fails,
the
send
rate.
So
that's
what
caliper
was
sending
the
observed,
actual
TPS
and
also
the
latencies,
and
then
for
each
of
these
rounds.
You'll
get
a
slightly
deeper
dive
and
you
can
see
the
bottom.
We've
got
the
results
of
certain
queries
that
were
being
run
through
for
me
here
and
the
results
of
those
queries.
A
If
you
don't
like
looking
at
just
numbers,
I,
just
guess
a
bit
dry
and
sure
okay,
we've
added
the
option
for
graphical
output,
so
you
can
have
a
quick
glimpse
view
of
where
the
expensive
components
are.
So
we
do.
These
radar
charts
I
should
quite
nice
for
visualizing,
but
we
also
do
the
bar
charts,
if
you're
not
a
fan
of
radar
charts,
so
that
that
is
kind
of
the
outputs
of
caliper.
A
And
how
can
you
use
this
and
where
we'll
be
using
this,
so
I'll
move
on
to
focus
about
performance
testing?
So
maybe
not
everyone
here
is
involved
in
performance,
testing
or
testing
at
all.
So
my
view
is
have
a
personal
view
on
performance
testing
is
that
they
should
be
directed.
They
should
be
focused
on
a
specific
aspect
of
the
technology
or
attributes
of
that
aspect.
A
More
importantly,
they
need
to
be
repeatable.
So
if
you're
publishing
some
results,
then
they
need
to
be
repeatable
by
somebody
else.
Okay,
so
you
need
to
list
the
the
resources
used
to
perform
the
tests,
details
of
the
machines,
the
configurations.
So
if
you're
saying
hey
guys,
I
ran
this
test.
This
is
why
I
saw
okay.
Well,
did
you
run
it
on,
and
what
did
you
run
so?
Okay,
we're
talking
here
about
making
available
your
test
definition
with
the
test,
libraries
and
the
systems
you're
running
on,
as
well
as
the
network
definition.
A
A
Okay,
so
this
is
our
benchmarks,
repository
if
you've
been
playing
with
caliper
before
you've
probably
come
across
this,
but
in
here
we
have
provided
benchmarks.
So
these
are
test
definitions.
We
have
the
network's,
so
say
pre
care
network.
You
can
stand
up
and
run
a
test
against
and
we
had
the
source
code
for
the
smart
contracts
to
deploy
and
we
also
have
a
website
which
is
showing
these
results.
So
if
we
look
into
fabric.
B
A
So
again,
this
is
a
another
another
view
of
how
callipers
being
run
so
we're
running,
pushing
our
test
files
through
into
the
benchmark
engine
running
that's
against
tonight's
work,
and
these
are
the
details
of
the
smart
contract.
This
particular
test
was
not
a
scenario
test.
We
are
focusing
in
on
specific
aspects
of
what
you
can
do
inside
fabric,
so
we
might
actually
be
in
doing
anything.
So
you
have
an
empty
contract,
just
returns
instantly
as
soon
as
in
the
vote.
It
comes
straight
back,
maybe
creating
a
single
asset,
so
you're
creating
something.
A
So
that's
a
put
state
operation.
You
might
be
creating
multiple
things
inside
a
single
transaction,
so
you're
doing
a
batch
kind
of
operation.
You
might
be
deleting
something
with
leasing
in
a
batch
performing
a
get
performing
a
multiple
series
of
gets
inside
a
transaction
or
you
might
be
doing
some
kind
of
query.
A
Maybe
it's
a
range
query
or
a
rich
query
and
I've
got
some
more
definitions
here,
but
most
people,
probably
straight
away
through
onto
the
results,
lock
results
here
for
one
for
zero,
most
people
that
hey
you
know
what's
going
on
in
v2
at
the
moment.
For
v2,
the
node
SDK
still
in
beta,
so
essentially
these
results
are
still
in
beta,
but
here's
our
configuration.
A
So
this
is
the
machine
configuration
that
was
used,
we're
running
these
tests
on
rel
system
and
we're
using
the
v2
images
for
fabric
and
chain
code,
but
we're
on
a
snapshotted
version
of
the
fabric
SDK.
As
soon
as
the
official
node
SDK
is
released,
we
will
be
repeating
all
of
these
deaths
and
updating
the
documentation.
A
So
we
have
those
for
running
and
in
value
or
a
submit
style
query.
Essentially,
it's
a
an
ordered
or
unordered
style
operation,
so
yeah
I'd
recommend
checking
those
out
worth
bearing
in
mind.
You
can't
really
compare
between
v2
and
v1
for
on
here,
because
it's
using
a
different
version
of
caliper,
it's
using
a
different
configuration,
so
different
machines
and
it's
using
the
distributed
setup,
rivaling
everything
being
local.
A
We
are
interested
in
doing
a
staged.
Compare
so
the
same
caliper
system
for
all
these
different
versions,
mainly
between
the
major
releases
yeah
we
do
have.
These
results
are
absolutely
welcome
you
to
have
a
look
at
them
and
we'd
also
say:
hey.
You
know
if
you,
if
you
really
want
to
confirm
these
results,
run
them
yourself,
so
we
have
the
benchmarks.
So
it
was
an
API
test
with
a
fabric,
and
these
are
the
configuration
file.
So
is
the
the
test
modules
that
are
running,
and
this
is
the
definite
definition
for
each
test.
B
A
So
we
yep,
so
the
question
was
caliper
records,
TPS
latency,
success
and
failure.
What
is
this
failure
status?
So
if
a
transaction
you
submit
fails,
perhaps
there's
a
timeout
or
perhaps
there
is
some
other
error
within
the
transaction
itself.
It
may
be
say
in
the
case
of
fabric
there's
an
MVC
conflict,
then
that
transaction
is
registered
as
a
fail
and
it
doesn't
complete
successfully,
so
that
would
be
recorded
by
calipers
a
failed
transaction.
A
So
yeah
I
will
repeat
the
question.
The
question
was:
would
you
get
the
reason
for
the
error
inside
the
logs
so
inside
the
report?
The
answer
is
no,
but
you
would
have
the
reason
coming
out
in
any
logs
going
on
inside
caliper,
so
caliper
is
able
to
during
the
process.
It
will
be
logging
either
info
warning
error.
Logging.
You
could
also
look
inside
them
any
of
the
other
information,
so
you
got
other
logging
tools
available,
which
you
would
see
the
error.
A
Okay,
so
the
question
is:
can
you
specify
a
ramp-up
period
for
workers?
The
answer
is
yes,
so
there
is
a
modular
controller
we
have
so
everything
in
caliper
is
based
on
modules.
One
of
those
modules
is
a
race
controller,
so
you
can
have
a
fixed
TPS.
You
can
have
a
increasing
TPS.
You
can
have
a
TPS
which
is
based
on
maintain
and
a
specific
backlog
of
transactions
or
as
a
modular
controller.
A
B
A
Yes,
so
the
question
is:
how
would
you
cause
a
transaction
to
fail
so
yep?
The
answer
there
is
we
give
you
complete
control
inside
the
the
the
code
you
invoke,
so
you
are
in
complete
control
of
the
body
of
the
transaction
and
if
I
was
to
do
it,
I
would
keep
my
logic
there.
So
I
would
be
purposefully.
Sending
through
a
set
number
of
fav
transactions.
I
know
will
fail.
A
A
B
A
A
A
Is
that
even
so
the
question
is
anyone
using
a
caliper
in
production,
I,
don't
know
as
the
honest
answer.
I
know,
people
use
it
in
proof
of
concept
stage
and
I
know.
People
have
targeted
the
large-scale
systems,
but
I,
don't
believe
they've
targeted
something
that's
actually
in
production.
At
the
same
time,
he'd
normally
be
a
prior,
a
pre-production
operation.