►
Description
Node.js Community Benchmarking Efforts - Michael Dawson, IBM
Benchmarks and the information they provide are important to ensure that changes going into Node.js don't regress key attributes like startup speed, memory footprint and throughput. Come and hear about some of the fundamentals of benchmarking, how to go about narrowing down the cause of a regression between versions of node along with the efforts underway in the community benchmarking workgroup (https://github.com/nodejs/benchmarking) to run/capture/report and act on benchmark information.
About Michael Dawson
Michael Dawson is an active contributor to Node.js as a CTC member, the facilitator for the benchmarking working group, participant in the lts, build, api and port-mortem working groups and has contributed many of the changes to enable support for Linux on Power, LinuxOne and AIX. He leads the Node.js team within IBM’s runtime technologies team driving IBM’s Node.js runtime deliveries and the runtime team’s contribution to Node.js and v8 within the Node and google communities. He’s been working in the industry for over 25 years with the last 11 focused on building runtimes including Node.js and Java. Further back, he held leadership roles in teams that developed e-commerce applications and delivered them as services, including EDI communication services, credit card processing and electronic invoicing.
A
So
they
said
today
I'm
going
to
talk
to
you
about
the
node.js
communities,
benchmarking
efforts
to
start
out
with
a
little
bit
about
myself
by
Michael
Dawson,
a
senior
software
developer
with
IBM
I
lead
the
team
with
an
IBM
that
builds
no
gs4
for
bundling
into
our
own
products
as
well
as
used
by
our
customers
through
our
products.
Cloud
products
like
bluemix
we've
done
a
lot
of
interesting
things
like
port
v8
to
power,
pc
and
the
s390
architectures,
and
contributing
the
the
work
needed
to
make
sure
that
IBM.
A
That
nodejs
can
also
run
on
those
platforms
as
well
as
participating
a
lot
in
the
open
source
community
itself.
One
of
the
things
I'm
personally
involved
in
and
some
of
the
other
people
from
my
team
is
working
in
the
benchmarking
workgroup
now
I'm,
just
the
one
who
has
the
pleasure
to
come
here
and
talk
to
you
about
what
we're
doing
I
just
want
to
make
sure
that
you
know
there's
a
lot
of
other
people
were
also
involved
in
the
work
group
and
we're
contributing
to
this
ongoing
effort.
A
The
first
thing
that's
really
important
to
realize
is
that
performance
testing
is
not
the
same
as
functional
testing.
You
know.
Functional
testing
is
relatively
straightforward
to
write
a
test.
That's
going
to
give
you
the
same
answer
every
time
and
it
either
pass
it
or
it
doesn't
in
terms
of
performance,
there's
a
lot
of
challenges
that
make
it
a
lot
harder.
The
first
thing
is
that
there's
pretty
much
a
guaranteed
fundamental
run
to
run
variance.
A
So,
every
time
you
run
your
performance
benchmark,
you're,
going
to
get
a
slightly
different
answer
and
that
can
really
easily
lead
to
false
positives,
where
you
think
I've
got
a
problem,
but
it
turns
out
that
no
that's
not
the
case,
so
you
actually
have
to
run
a
whole
bunch
of
times
and
often
average.
Those
results
together
to
be
able
to
say
yes
or
no
I've
actually
got
a
regression
or
I've
got
an
improvement,
and
so
it's
very
important
to
document.
A
You
know
what
is
the
expected
variance
so
that
when
you
get
a
result,
you
know
whether
you've
got
a
problem
or
not.
The
second
thing
is
that
you
know,
unlike
functional
testing
again
the
environment,
that
the
tests
are
very
very
if
two
changes
the
environment.
You
know,
for
example,
if
you
run
a
benchmark
on
one
machine
with
two
CPUs
and
you
rerun
on
a
machine
with
four
cpus
you're,
quite
likely
going
to
get
a
totally
different
results.
A
You
have
to
make
sure
that
your
environment
is
very
consistent
even
to
the
point
where
you
know,
if
you
have
a
machine,
that's
connected
to
a
network,
even
if
your
your
benchmark
doesn't
use
the
network
just
having
a
heavily
loaded
network
might
affect
the
overall
results.
You
have
to
be
really
really
careful
about
keeping
that
environment
consistent.
A
I
find
is
avoid
jumping
to
conclusions,
I
think
about
ninety-five
percent
of
the
time
when
our
performance
team
starts
out
and
says.
Well,
we
might
have
a
problem
on
this
particular
release
of
node.
After
further
analysis,
it
turns
out
no,
there
wasn't
any
problem.
That
was
one
of
the
other
issues
that
was
going
on.
A
The
other
thing
is
that
when
you're
making
changes,
you're
really
going
to
do
yourself,
a
big
favor,
if
you
limit
and
entropy
try
and
limit
the
things
that
you
change
down
to
one
dimension
at
a
time,
so
only
change
your
application
code
or
if
it's
got
mpm
modules,
depending
on
only
change
one
of
those
don't
change
the
no
GS
version
at
the
same
time
or
the
Machine
try
and
make
sure
that
you
change
one
thing
at
a
time.
That's
going
to
make
it
a
lot
easier
to
figure
out.
A
What's
going
on
overall
there's
roughly
two
big
categories,
the
benchmarking
will
fall
into
there's
micro
benchmarks
and
whole
system
benchmarks,
micro
benchmarks,
benchmark
a
very
specific
thing,
so,
for
example,
buffer
new.
If
you're
concerned
about
how
fast
you
can
do
that,
you
can
easily
write
a
benchmark.
That
just
does
that
a
lot
of
times
and
you
can
end
up
telling
whether
that's
changed.
A
The
problem
is
that
micro
benchmarks
often
don't
translate
into
real-world
performance,
so
you
can
optimize
for
a
bunch
of
micro
benchmarks
and
you
may
get
really
really
really
good
performance
on
those
benchmarks,
but
it
may
turn
out
that
for
real
life
applications,
that's
actually
the
wrong
thing
to
do
so.
Whole
system
benchmarks
try
and
say:
let's
take
a
benchmark
which
simulates
real
life.
A
So
you're
looking
at
you
know
some
performance
on
ugs
and
you
think
you
found
a
regression.
What
do
you
do?
Well,
the
first
thing
is
make
sure,
as
I
said,
ninety-five
percent
of
the
time.
In
our
case
it
turns
out
to
be
that
there
really
isn't
a
problem
there
so
make
sure
you
revalidate
that
make
sure
your
your
environment
didn't
change
and
you're,
not.
You
know
within
your
expected
variation.
If
you
can,
so
you
convince
yourself
now,
yeah,
okay,
there
is,
there
is
a
problem.
So
what
happened?
A
You
have
to
start
looking
at
what
changed
so
was
it
your
application?
Did
you
change
the
runtime
or
your
overall
environment
so
assuming
you've
now
narrowed
it
down
to
the
runtime
nodejs?
You
have
to
start
looking
at.
Let's
compare
between
the
good
and
bad
cases,
there's
a
bunch
of
tools
we
can
use
to
try
and
give
us
hints
as
to
where
to
look,
and
we
could
also
just
fall
back
to
a
binary
search
where
we
slowly
narrow
the
scope
of
you
know
the
changes
that
went
in
between
the
good
and
bad
case.
A
In
terms
of
where
to
look.
Obviously,
you
know
we
can
look
at
nodejs
itself,
things
like
the
buffer
cluster
modules,
depending
on
what
you
see
is
a
regression
at
my
point
in
one
of
those
directions,
but
the
first
thing
I'd
personally
look
for.
Were
there
any
major
changes
in
the
dependency,
so
things
like
v8
openssl,
lib
UV,
they
can
often
have
a
significant
impact
on
your
performance,
so
I'd
first
rule
out
whether
any
of
those
dependencies
have
changed
before
I
went
on
to
anything
else
in
terms
of
some
of
the
tools
that
we
can
apply.
A
V8
comes
with
a
built-in
profiler
and
NOAA
has
been
modified
so
that
you
can
actually
turn
that
on
using
a
command
line
option.
So
if
you
turn
on
the
v8,
profiler
and
I'll
show
you
those
command
lines
a
bit
later
on.
You
can
turn
on
with
command
line
options,
or
even
you
can
use
some
tools
like
at
metrics
as
a
module
you
can
install
and
it
will
actually
let
you
visualize
profiling
information.
A
You
can
also
use
information
about
the
v8
optimizer,
how
v8
optimizes
your
code
is
going
to
make
a
significant
effect
on
performance,
and
so
what
the
optimizer
does
can
actually,
you
know,
can
make
a
big
difference
and
there's
two
options
trace
all
then
trace
the
op,
which
are
often
interesting
to
turn
on.
So
you
can
see
between
your
good
and
bad
case.
Was
there
a
change
in
the
optimization
profile
and
the
information?
A
So
that's
just
a
little
bit
of
an
introduction
to
benchmarking
at
the
end,
I'll
flip
back
to
an
example
where
we
apply
some
of
those
tools
and
techniques,
but
now
I'm
going
to
tell
you
about
what's
going
on
in
the
benchmarking
work
group
itself.
So
the
key
goals
of
the
benchmarking
work
group
are
to
avoid
regressions,
and
you
know.
Basically,
we
want
node
to
be
faster,
every
release,
not
slower,
but
we
also
want
to
be
able
to
track
and
evangelize
when
we
get
faster,
which
is
a
good
thing.
How
do
we
planning
to
do
this?
A
So
what
have
we
done
so
far
and
what
have
we
accomplished
so
far?
We've
gone
through
we've
defined
the
use
cases,
so
we
can
see
you
can
see,
we've
defined
these
main
use
cases
for
for
nodejs,
and
they
include
things
like
back-end,
API
services,
services,
oriented,
service-oriented
architectures,
microservices,
single
page
applications,
even
agents
and
data
collectors.
A
Because
note
fits
nicely
is
a
small
footprint
agent,
small
scripts,
and
so
the
goal
is
that
you
know
we
want
to
cover
all
of
these
in
our
benchmarking,
if
you
see
use
cases
that
you
think
are
really
important
for
node.
That
aren't
here
just
open
an
issue
on
the
repo,
and
you
know
we're
always
looking
to
make
sure
that
we've
covered
all
of
the
use
cases
appropriately
as
we
discuss
the
use
cases.
A
The
other
thing
that
came
became
apparent
that,
in
addition
to
looking
at
use
cases,
we
also
want
to
look
at
key
attributes
for
note.
There's
some
things
about
the
node
runtime
that
people
just
felt
were
really
important
and
even
outside
of
you
know,
a
specific
use
case.
We
felt
it
was
important
to
maintain
a
sort
of
like
the
micro
benchmark
versus
the
whole
system
type
thing,
so
things
like
the
startup
time,
the
footprint,
the
amount
of
memory
that
you
use
even
the
size
of
the
the
binaries,
both
zip
then
unzipped.
A
All
these
are
you
know
key
attributes
and
things
that
we
want
to
make
sure
that
you
know
we
don't
regress
in
terms
of
nodejs.
So
you
know
those
are
things
will
consider
benchmarking
in
addition
to
the
use
cases
as
well.
So
we
took
those
use
cases.
We
took
those
bent
those
attributes
and
we
built
our
table.
That
says:
okay,
here's
our
use
cases
in
our
key
attributes,
and
now
we
want
to
fill
this
in
with
benchmarks,
and
so
we've
started
the
work
of
going
through
identifying
benchmarks.
A
Acne
air
was
one
of
the
benchmarks
that
was
already
out
there
in
terms
of
a
whole
system
benchmark.
So
it's
one
that
we've
actually
implemented
as
the
first
one.
Some
other
people
in
the
community
contributed
benchmarks
for
things
like
require.
Requiring
mileage
modules
is
a
key
thing
when
you're,
you
know
running
small
scripts,
so
we
want
to
be
able
to
do
that
quickly,
start
up
and
stop
time.
A
There
was
a
benchmark
already
as
part
of
the
core
benchmark,
so
we
reuse
that
and
basically
we
took
those
and
we,
you
know,
started
filling
in
this
table
so
that
we
have
some
partial
coverage
in
terms
of
those
key
elements
and
what
we're
benchmarking
building
on
that
step,
we
then
say:
well,
okay,
we
wanted
those
benchmarks
that
we
have
identified.
We
want
to
run
them
on
a
regular
basis
and
capture
the
results.
A
So
we
actually,
if
you
go
to
the
community
CI,
you
can
see
in
jenkins
that
we
have
a
number
of
jobs
and
we
run
and
capture
those
benchmarks
across
7,
64
and
12
right
now,
and
you
can
see
that
you
know
you
can
go
and
see
that
they're
running,
you
can
actually
inspect
the
results
from
the
particular
run.
But
in
addition
to
that
you
know
we
also
want
ppl
people
to
be
able
to
easily
consume
that
and
see
how
we're
doing
so.
A
If
you
go
to
benchmark
and
OJ's
org,
we
take
that
data
and
we
chart
chart
it
every
night.
So
there's
an
update,
and
this
shows
the
performance
for
acne
air,
for
the
require
benchmark
nightly
for
each
of
those
different
releases.
So
you
can
actually
say,
okay,
how
are
we
doing,
and
you
know
it
was
used
in
the
you
know,
when
six
came
out
to
show
that
acne
or
the
transactions
and
throughputs
throughput
increased,
because
you
know
we
did
we
pulled
in
a
new
version
of
the
eight
and
other
changes
that
have
gone
into
nodejs.
A
A
So
that's
how
that's
how
far
we've
gotten
so
far
the
things
we're
currently
working
on
include
expanding
those
benchmarks.
So,
as
you
can
see,
we
have
in
or
use
cases
we
have
our
our
key
attributes
and
we
have
some
benchmarks,
but
there's
a
lot
of
space
in
that
table.
We
need
help
filling
in
that
table
and
that's
what
we're
working
on
and
if
you
have
time
or
energy
you
know
it's
a
great
thing
to
get
involved
in
as
well.
A
The
other
thing
we
want
to
work
on
is
that
no
node.js
core
includes
a
good
set
of
micro
benchmarks,
but
they
are
often
you
know
somewhat
difficult
to
run
consistently
on
your
own
machine
and
can
actually
take
a
long
time.
I
think
some
of
them
run
in
the
default
mode.
Take
like
six
hours
to
run
so
we're
looking
at
setting
up
some
CI
jobs
so
that
people
can
more
easily
run
those
from
the
CI.
A
When
you
know,
if
you
want
to
check
out
your
commits
and
stuff
like
that,
so
that's
the
other
thing,
a
sort
of
stream
of
activity.
That's
going
on
right
now!
So
that's!
What's
going
on
the
benchmarking
work
group
itself
now
I'm,
going
to
flip
up
back
to
an
example
of
a
problem
that
we
ran
into
and
the
story
of
you
know
how
we
went
through
investigating
it.
A
This
was
actually
done
by
Gareth
Ellis,
who
was
a
member
of
the
team
that
I
work
in
and
does
the
performance
evaluations
that
you
know
we
run
internally
on
node
for
every
release
that
we
do.
We
had
this
micro
benchmark
and
it
you
know,
was
basically
just
checking
the
performance
of
the
the
buffer,
the
new
buffer.
So
basically
it
would
say:
okay,
let's
create
a
buffer
with,
though
you
know,
an
array
that
has
those
values
and
we're
just
going
to
do
it
a
whole
bunch
of
times
as
fast
as
we
can.
A
The
performance
between
you
know
four,
four
and
four
three:
two.
We
could
see
that
there
was
about
a
forty
percent
regression,
so
it's
like
wait
a
second.
You
know
something's,
obviously
not
going
something's
not
going
in
the
right
direction
here,
we're
slowing
down
by
a
significant
amount.
So
some
of
the
tools
I
introduced
in
the
first
part
you
know,
let's
take,
let's
take
a
look
at
how
those
helped
us
figure
out
what
was
going
on.
So
the
first
thing
was
turned
on
dash
dash
prof,
which
is
the
option
that
lets
us
capture
the
profiling
information.
A
When
we're
writing
our
node
application,
that's
going
to
generate
us
a
file.
We
then
need
to
post
process
that
file,
so
we
can
actually
look
at
it
and
understand
you
know,
get
into
a
readable
format
in
node
I
forget
which
version
I
think
it
was
sort
of
a
446
version.
We
added
the
option
to
add
dash
dash
prof
process
which
lets
you
take
the
file
that
was
generated
and
converted
into
the
readable
format,
and
you
can
end
up
with
a
file
that
you
can
then
read.
A
There
are
other
modules
like
the
profiler
that
can
help
you
gather
and
visualize
this
information
as
well
having
processed
it
with
the
tick
prophetic
processor.
You
know
this
was
the
difference
between
the
good
and
the
bad
version,
and
you
can
see
we're
spending
a
lot
of
extra
time.
You
know
in
lazy,
compile
in
the
front
object
options,
so
this
immediately
says
it
looks
like
something
going
on
in
this
from
object
might
be
involved
in
or
what
changed.
A
We
also
said:
okay,
okay,
that
was
using
the
the
beat
the
beat
the
built-in
v8
profiler.
Let's
take
a
look
at
the
native
profiler,
so
we
turned
on
perf
and
here's
the
example
of
you
know
the
command
line.
We
used
the
magic.
This
dash
dash
perf
basic
prof,
is
what
tells
perf
to
use
its
knowledge
about
nodejs.
It
says:
ok,
we're
profiling,
something
that's
node,
so
it
actually
knows
how
to
map
some
of
the
Jedi
Code
back
into
the
JavaScript.
A
The
other
thing
we
thought
we'd
look
at
is
okay.
This
is
looking
like
you
know,
maybe
the
compiling
related
to
compilation.
So,
let's
turn
on
the
trace
off
and
trace
d
op
and
see
what
that's
telling
us.
Is
there
something
different
going
on
in
terms
of
how
v8
is
optimizing?
The
code-
and
we
can
see
in
this
case,
is
that
in
the
bad
case
we
saw
the
message
that
says
the
methods
being
d
optimized,
it's
like
ok!
Well,
that's
not
a
good
thing.
A
Instead
of
running
the
optimized
code,
v8
is
deciding
that,
for
some
reason,
I
now
have
to
diop
tamai.
Is
that
that
method-
and
you
know
that
would
explain
why
we
saw
more-
you
know
more
code
in
the
lazy
optimized
at
that
point.
It's
like
okay,
we're
pretty
sure
we
know
roughly
where
it
is,
and
you
know
we
can
go
in
this
case.
A
We
didn't
need
to
do
it
so
much
because
you
know
we
had
a
pretty
narrow
pointer
to
where
we
were
to
look,
but
in
the
worst
case
you
can
always
fall
back
to
binary
chop
where
you
basically
compare
between
a
good
and
bad
get.
Rev
list
is
a
tool
that
lets
you
basically
say.
Well,
here's
the
list
of
changes
between
two
versions
that
can
be
pretty
help
hopeful.
If
you
want
to
look
at
every
one
of
those
it
could
be
pretty
time-consuming,
so
get
bisect
is
another
option
that
basically
says
you
know.
A
A
We
could
see
that
the
change
was
actually
a
change
from
VAR
to
let
and
it
turns
out
that
a
v8
you
know
the
implementation
for
let's
particularly
in
loops,
is
not
fully
optimized
yet
so
you
know
they're
working
towards
turbofan,
which
is
a
new
newer
version
of
their
their
optimizer.
Once
that's
on
by
default.
Lets
performance
will
be
much
better,
but
until
that
point
let
was
actually
much
slower
than
using
bar
in
this
particular
situation.
So
PR
was
submitted
and
we
reverted
back
to
the
original
code
in
this
case.
A
So
that's
an
example
of
you
know
how
we
applied
those
tools,
investigate
an
issue,
and
you
know
did
that
in
the
in
the
benchmarking
work
group
contributed
to
change.
You
know
and
restored
the
performance,
so
I
think
I'm
almost
out
of
town
out
of
time.
My
last
thing
you
know
I
just
want
to
say,
as
you
can
see,
we
had
a
you
know
a
good
size
matrix.
We
have
a
lot
of
gaps,
there's
a
lot
of
work.
A
B
C
A
Two
things
so
some
of
those
are,
you
know,
I
know,
I,
don't
know
if
those
are
just
an
outlier,
so
I
don't
think
it
was
a
regression
that
got
fixed
because,
generally
those
will
go
up
stay.
They
don't
get
fixed
that
fast,
it's
plotted
every
night.
The
other
thing
to
look
at
is
actually
those
scales
are
compressed,
so
they're,
not
necessarily
absolute
starting
at
zero.
So
the
number
the
variation
may
not
be
as
big
as
it
looks
in
those
charts,
and
you
know
there
were
some
issues
we
had
a
discussions
like.
A
D
A
Are
mostly
in
tests,
so
I
think
that
was
discussed
and
I
think
most
of
the
places
you'll
see
those
PRS
for
its
in
tests
where
we're
not
as
worried
right,
like
obviously
not
in
the
benchmark
tests,
but
in
the
just
a
plain,
functional
test:
it's
not
going
to
have
an
effect
in
terms
of
let
for
sure
there's.
You
know,
we've
got
very
specific
cases
where
yet
don't
do
it
at
all,
and
there
was
a
big
discussion
about
these
changes
from
let's
to
bars
and
I.
Think
the
answer
was
we
directed
the
community
say?
A
C
A
Cember
sember
releases
include
a
new
version
of
the
eight.
The
v8
team
works
hard
to
make
things
get
better
at
in
each
release.
So
when
we
pull
in
a
new
v8,
it's
quite
common
that
we're
going
to
see
a
bit
of
a
bump.
It's
actually
good
that
we
see
you
know
acne
air
going
up,
which
was
our
whole
system,
that
okay,
real
life
performance
is
hopefully
getting
better
and
if
some
of
the
other
ones
there
are
cases
where
some
of
the
other
ones
went
a
little
bit
down.
A
So,
for
example,
we've
just
added
octane
and
actually
octane.
It
doesn't
actually
show
up
on
the
front
page
yet
because
we
haven't
got
but
that
one
you
can
actually
see
in
some
versions
that's
gone
down,
but
it's
more
of
a
micro
benchmarks.
Oh
I,
see
that's
a
good
example
where
you
know
the
micro
benchmarks
aren't
the
full
story.
They
may
go
down,
but
you
get
better
overall
performance
and
even
the
the
VA
team
would
you
know
it's
been
involved
with
us
and
said:
well
know
that
that
doesn't
matter
the
overall
performance
doesn't
matters
but.