►
From YouTube: RobotPerf subproject - meeting#1
Description
This is the weekly meeting of the RobotPerf (https://robotperf.org/) subproject of the ROS 2 Hardware Acceleration Working Group (https://github.com/ros-acceleration). We discussed the group organization, made intros and went through the tools and the overall architecture for benchmarking.
00:00 Welcome
00:16 Introductions
10:00 Discussion
17:50 Walkthrough on a first benchmark example with RobotPerf
RobotPerf is an open reference benchmarking suite that is used to evaluate robotics computing performance fairly with ROS 2 as its common baseline, so that robotic architects can make informed decisions about the hardware and software components of their robotic systems. The group meets weekly at this room and discusses robotics computing architectures and benchmarking across various compute substrates.
RobotPerf is a subproject within the ROS 2 Hardware Acceleration Working Group (or HAWG for short), an open and community-driven robotics group that drives the creation, maintenance and testing of hardware acceleration kernels on top of open standards for optimized ROS 2 and Gazebo interactions over different compute substrates, including FPGAs and GPUs. The group meets monthly and discusses topics and subprojects around hardware acceleration.
Minutes of the meetings are available at https://docs.google.com/document/d/185Cy1xjpAOgJygEOnlf5OCgOQTywmF0qgSpS3GiW16Q.
A
The
resources
available,
then
I'm,
hoping
maybe
BJ
and
Jason,
can
comment
a
bit
also
on
the
Strategic
aspects
and
how
we
want
to
drive
this
forward
based
on
past
discussions
and
then
what
I've
done
is.
If
we
have
time
in
this
first
half
an
hour,
I
have
prepared
a
short
one-on-one
walkthrough
on
how
to
create
a
new
Benchmark.
A
A
Officially,
the
robot
Perth
project
is
a
sub
project
of
the
ross2
hardware.
Acceleration
working
group
we're
doing
this
for
many
reasons,
the
first
being
we're
trying
to
leverage
much
of
the
traction
we
have
generated
in
in
Hardware
acceleration.
Also
to
this
very
connected
and
crucial
topic
for
also
high
performance
systems,
which
is
benchmarking
so
we'll
be
conducting
this
in
the
same
room
in
the
same
resources.
A
So,
with
those
minutes
shared
when
we
guys,
maybe
just
do
a
round
of
intros
I-
think
there's
some
books
in
here
who
have
not
participated
in
the
firm
and
might
have
some
contributions
to
send
in
in
a
nutshell,
and
since
I'm
asking
for
it
I'll
get
started.
My
name
is
Victor
hi
everyone,
so
I'm
I'm,
an
assistant
architect.
A
What
bodices
for
more
than
a
decade
now
very
involved
in
the
Ross
community,
and
very
interested
in
benchmarking
in
a
reproachable
manner,
in
a
real-time,
safe
manner,
in
a
somewhat
consistent
way,
robotic
systems.
That's
why,
together
with
some
of
you
started
this
robot
purse
project
and
I'm
very
excited
to
be
contributing
to
it,
I'll
be
supporting
both
on
the
admin
side,
as
well
as
on
the
technical
side
and
yeah,
hoping
that
all
of
you
guys
in
here
can
at
least
contribute
with
one
Benchmark.
That
would
be
fantastic.
B
B
Research
and
some
robotics
research
and
working
on
several
different
types
of
benchmarks,
so
I'm
really
excited
to
work
specifically
on
Ross
benchmarking.
I
will
be
also
working
as
like
an
admin
and
a
technical
contributor
to
this
project,
hopefully
getting
a
few
benchmarks
in
pretty
soon
so
yeah.
So
it's
working
all
of
you.
D
Thank
you,
I'll
go
next,
so
my
name
is
earlier:
I
I
have
like
around
six
years
of
experience
in
robotics,
especially
working
with
Ross
and
Ross
too,
and
I
was
I.
Had
a
company
called
techno
yatra
service
based
robotics
company
called
technology,
and
we
got
acquired
by
Victor
last
year,
so
now
I'm
part
of
acceleration
robotics
leading
the
the
expansion
in
India
along
with
gaurav
who
is
in
there
today.
D
E
Okay,
maybe
I
can
go
next
hi,
my
name
Isa
any
pronouns,
so
my
background
is
control
systems
and
Robotics
and
I've
been
accompanying
the
a
harder
acceleration
acceleration
group
for
a
while
now,
but
now
I'm
like
excited
to
try
to
contribute
to
it.
So
that's
why
I'm
interested
in
this
little
per
sub
project.
F
Yeah
I
can
go
now:
hi
everyone,
I'm
yishun
I'm
a
fourth
year,
PhD
student
with
Vijay
and
a
little
bit
about
my
background.
I've
been
working
on
autonomous
vehicle
drones
and
now
robots,
so
many
to
enhance
the
resiliency
and
associate
in
such
robotic
application,
I'm
exciting
to
contribute
and
then
we'll
be
helping
the
technical
side
with
the
Harvard
team.
G
C
And
also
still,
I
can
work
from
like
a
closely
wisdom
and
my
girlfriend
is
in
the
robotics
that
hunts
machines
to
complete
architecture,
and
also
that,
like
a
kind
of
a
cheap
design,
so
I'm
very
interested
in
the
benchmarking,
is
the
workload
and
things
are
promising
and
I
hope
to
can
contribute
something
here.
Thanks.
H
I
As
many
of
my
students
and
postdocs
are
here,
my
interest
is
primarily
in
architecture
and
systems,
and
I
tend
to
do
architecture
and
systems
across
variety
of
different
workloads,
robotics
being
one
of
them,
but
there's
extended
reality
aspects
of
neuromorphic
computing
and
so
forth.
So
the
lab
does
a
lot
of
different
things.
I
One
of
my
interests
around
benchmarks
is
that
I
love
building
benchmarks,
especially
from
a
community
sort
of
standpoint
like
getting
the
industry
together
and
so
forth,
and
that's
how
I
kind
of
let
Victor
and
support
so
I
built
benchmarks
and
machine
learning
scientific
applications.
Robotics
has
always
been
sort
of
on
that
Horizon,
but
never
quite
seemed
like
it
was
right
and
I'm
pretty
convinced
personally.
That
I
think
the
timing
is
right
to
do
something
something
big
for
the
industry
right
now.
I
think
whatever
we
do
at
least
the
beacon
in
my
head
is.
I
J
Hi
everyone,
I'm
Michelle
I'm
a
third
year
of
Nevada
studying
Cs
and
astrophysics
I
mainly
done
research
in
my
pediatric
physics
background.
So
this
is
like
my
first
teacher
project
in
robotics,
but
it's
something
that
I've
always
been
very
interested
in,
so
I'm
very
excited
to
be
a
part
of
this
team
gifted.
Thank
you.
K
Hello,
my
name
is
Jorge
I'm,
a
artificial
intelligence
engineer
here
in
a
company
in
Barcelona,
doing
partial
recognition
and
I'm
doing
this
in
my
free
time
and
passionate
about
FPA
is
on
our
robotics
I
want
the
Adaptive
Computing
challenge
shown
by
Mao
I,
don't
have
a
lot
of
time.
I
will
try
to
do
my
best.
So
that's
all
for
me.
K
A
A
Awesome
all
right
so,
just
mindful
of
the
time
We've
almost
consumed
half
of
it.
This
is
a
team.
That's
going
to
set
the
industry
standard,
it's
going
to
grow
over
time,
but
you
guys
are
the
bread
maker
for
now.
So
looking
at
the
minutes
and
again,
anyone
else
feel
free
to
add
yourself
or
to
to
to
it.
Just
real
quick
you've
got
the
resources
of
the
working
group
over
there
and
of
the
robot
perf
project
right
in
here.
A
A
So
yeah
I
just
wanted
to
open
up
for
the
discussion.
If
there's
no
topic
to
be
discussed,
I
can
give
everyone
a
quick
run
on
the
source
code
structure
and
how
to
run
a
first
Benchmark
but
I'll
just
I'll
just
leave
that
to
everyone.
So
anything
worth
discussing
folks
in
this
first
meeting
anything
anyone
would
like
to
race,
as
we
are
starting
with
the
project.
I
Ahead,
please
one
thing
I
was
just
kind
of
thinking
is,
like
you
know,
before
you
dive
into
the
the
depth
right.
Maybe
just
we
get
to
kind
of
walk
through
the
readme
and
kind
of,
like
specifically
focus
on
the
different
categories
that
we
want
to
build
the
benchmarks
and
so
forth.
So
I
think
that,
because
immediately
I
think
people
would
be
like
oh
yeah.
I
can
do
something
in
that
space
right
because
we
already
have
people
in
control,
and
you
know,
navigation
and
so
forth.
A
No,
let's,
let's
do
it
yeah
yeah,
indeed,
so,
let's
dive
into
it,
so
here's
the
minutes-
hopefully
you
guys,
can
see
it
and
I'm
going
to
get
into
just
showing
you
the
flow.
So
this
is
the
website,
and
here
we
access
the
resources
in
git
in
GitHub,
so
there's
an
organization.
Everything
is
disclosed,
open,
source
and
we'd
like
to
keep
up
with
that
mindset.
A
There
is
a
general
readme
that
is
hosted
for
the
at
the
organization
level,
but
really
what's
showing
is
actually
one
of
the
repos,
particularly
the
benchmarks,
one
all
right
and
as
as
BJ
was,
was
highlighting.
We
have
here
a
formal
definition
of
what
we
are
aiming
for,
Mission,
as
well
as
vision
of
the
project
which
I
will
not
touch
on
a
lot
today.
A
Just
for
the
sake
of
timing,
but
what's
important
is
to
make
sure
that
we
stress
that
we
are
trying
to
somehow
pack
benchmarks
into
these
five
initial
categories
and
and
I
say
initial,
because
this
will
grow
over
time
and
we're
already
getting
some
requests
actually
from
industry
players
to
add
new
categories.
What
I'm
doing
is
pushing
back
and
saying
contribute
a
benchmark
and
then
we'll
fit
it
into
a
new
category.
A
So
that's
that's
going
to
be
the
attitude
so
far
just
to
try
to
focus
ourselves
on
it,
but
just
as
an
FYI,
there
will
be
additional
categories
and
examples
will
include
very
possibly
infrastructure,
such
as
the
message
passing
infrastructure
that
was
to
has
itself.
This
is
kind
of
like
meta,
meta
aspects
to
Benchmark
when
it
comes
to
robotics,
especially
again,
the
infrastructure
that
allows
different
mobile
components
to
communicate
with
each
other.
This.
A
This
does
not
necessarily
fall
into
any
of
these
categories,
but
it's
something
that
applies
to
all
of
them.
Other
things
include
other
layers,
such
as
the
operating
system
statistic
networking
stack,
such
as
the
data
link
layer.
All
of
these
things
are
also
being
requested,
So,
eventually
again,
as
we
get
more
and
more
industry
participation,
we'll
we'll
enlarge
this
category.
B
So
we
we
would
choose
a
specific
node
or
a
specific
function
within
a
Ros
package
and
we
will
evaluate
it
on
a
set
of
metrics
that
are
listed
in
one
of
these
specifications.
Victor
has
made
called
rep
2014.,
so
our
vision
here
is
that
across
each
one
of
these
areas
of
the
robotics
pipeline,
we
would
choose
different
several
benchmarks,
so
tips
so
several
different
applications
inside
like
perception,
localization
control
and
so
on,
and
then
we
will
also
provide
data
sets
to
feed
into
these
nodes.
B
And
then
we
will
evaluate
how
well
they
do
on
the
on
those
data
sets
or
Frost
bags,
which
is
like
the
name
of
the
repo
on
the
set
specifications
that
Victor
has
made
and
then
once
these
benchmarks
are
created,
then
members
of
the
community
can
like
easily
run
these
benchmarks
on
different
types
of
hardware
and
they
can
evaluate
how
well
they
do
how
well
they
compare
and
performance
to
like
the
reference
implementation,
whatever
Hardware
we've
evaluated.
This
Benchmark
on
that
was
that's
like
a
little
bit
of
a
broad
overview.
B
A
That's
super
useful,
yeah
and
and
thank
you
and
please
guys,
just
interrupting
because
I'm
so
contaminated
that
I'm,
assuming
so
many
things
that
possibly
here
we
we
just
need
lots
of
questions
and
people
questioning
things
and,
and
indeed
all
of
that
is
described
in
the
corresponding
package.
So
so,
essentially,
this
repository
is
a
is
a
package
of
packages.
I
made
a
package,
everything
is
law
space.
A
A
The
Benchmark
specification,
that's
defined
in
here,
and
that
file
just
contains
metadata
to
essentially
help
out
help
out
automate
the
process
of
adding
new
data
from
new
hardware
devices
or
from
the
same
Hardware
devices
just
in
different
instances
of
time
and
then
just
automating
things
from
a
continuous
integration
purpose.
So
that's
pretty
much
the
objective
to
try
to
achieve
as
much
automation
as
possible
and
that's
what
currently
working
on.
A
If
people
has
a
bit
of
extra
time
to
discuss
things
and
launch
questions
as
well
as
taking
maybe
some
action
items,
does
this
make
sense
folks,
or
should
we
instead
do
something
else,
yep
all
right?
So
then
let
me
get
this
ready,
so
what
I'll
start
is
with
a
demonstrator?
If
that's,
okay,
let
me
fetched
this
real,
quick
and.
A
So
here
is
three
years
one
of
my
development
machines
all
right
and
what
we're
going
to
do
initially
is
just
load
the
computational
graph,
it's
pretty
empty
nothing's
happening.
That's
what's
expected,
and
now
we're
gonna
publish
images
into
the
default
camera
image
for
our
topic,
and
if
we
update
the
graph,
then
we
can
see
this
representation
which
depicts
in
the
nut
cell,
how
we
have
a
super,
simple
graph,
just
oops,
okay,
Mouse
issues
and
again
there's
this
rosberg
player,
which
is
publishing
in
the
corresponding
topic.
A
Nothing
too,
fancy
nothing
too
surprising,
and
now
what
we're
going
to
do
is
launch
one
of
the
benchmarks,
and
this
is
happening
in
a
different
screen,
but
just
for
the
sake
of
clarity,
here's
how
you
launch
it.
So
nothing
too
fancy
General
Ross
to
learn
stuff
we're
just
using
one
of
the
DDS
implementations
which
is
known
to
be
reproducible.
The
default
one
is
unfortunately
not
recommended
for
benchmarking
purposes,
neither
for
performance
purposes,
but
you
can
experience
with
that
yourself
and
experiment
with
that
yourselves.
A
This
node
publishes
the
data
into
an
input
topic
and
right
in
here
we
start
putting
the
trace
points
to
collect
data
and
performs
a
perform.
Sorry,
benchmarking
in
a
systematic
and
reproducible
manner.
We
use
roster
tracing
project
to
do
so,
which
is
already
fully
integrated
into
ros2
and
which
provides
many
things,
but
amongst
them,
capabilities
to
abstract
away
the
complexity
of
leveraging
low
low
overhead
and
real-time
capable
tracers
such
as
lttng.
A
So
in
a
nutshell,
there
is
a
series
of
Trace
points
in
here
and
at
this
topic
level,
technically
speaking,
they
happen
in
the
note,
but
anyhow,
just
for
the
sake
of
the
abstraction
information
gets
traced
from
the
moment.
They
go
into
the
input
topic
and
then
we
collect
the
output
outside
of
this
bounding
box
outside
of
the
Benchmark
namespace,
and
so
of
course,
what
we
are.
Benchmarking
in
this
case
is
pretty
well
bounded.
A
It's
within
the
benchmark,
namespace
you
can
see
in
there,
and
so
we
can
push
in
there
multiple
nodes.
We
can
put
a
complete
perception
or
navigation
subgraph
and
that's
pretty
much
what
we
will
be
doing.
Of
course,
each
Benchmark
should
be
well
defined
and
whatever
happens
within
the
bounding
box
within
the
namespace
or
Benchmark
is
what
should
be
what
we
will
be
reporting
about,
and
this
super
silly
example
again
just
benchmarks,
a
single
rectification
operation.
Actually,
you
can
see
in
here
how
still
it
is
that
we
don't
barely
see
the
difference.
A
We
have
the
information
being
published
in
the
first
box
in
the
first
windows
are
in
here,
and
this
is
what's
com.
What
comes
from
there
was
to
back
Ross
back
file.
The
second
one
is
what
we
get
inside
of
the
input
topic,
so
what
gets
pumped
into
the
Benchmark
itself
and
then
what
we
get
out
of
it
is
depicted
in
here.
A
So
what
we
and
what
I
would
suggest
we
do
is
that,
instead
of
developing
natively
in
vs
code,
we
use
something
called
Dead
spaces
which
allow
us
to
develop
inside
a
Docker
container
and
it's
a
cc
as
you
simply
entering
the
docker
container,
the
same
window
reopens
within
the
docker
container.
Only
that
the
dependencies
and
everything
required
is
pre-installed
and
predefined
by
ourselves
and
I'll.
Show
you
just
right
now
in
a
bit
where
these
files
are
defined.
A
A
K
A
No,
actually
here
we
are
okay,
so
it
was
just
a
glitch
I
guess
so
now
we're
inside
of
the
different
container.
So
everything
is
exactly
the
same.
You
can
benefit
from
your
usual
favorite
vs
code,
depth,
environment,
by
the
way,
the
same
capabilities,
also
in
Sublime
Text
and
in
various
versions
of
this.
So
in
case
you
prefer
not
to
use
vs
code,
that's
totally.
Okay,
there
are
alternatives.
A
A
A
A
This
is
typically
what
you
would
be
doing
while
developing,
because
of
course,
you
want
to
validate
that
whatever
piece
of
source
code
you
have
produced
can
actually
turn
into
an
actual
node
that
can
be
launched
so
leaving
that
aside
I'm
taking
a
quick
look
at
the
code.
Let's
look
at
this
new
example
that
I
produced
this
at
your
rectifier,
so
from
the
bottom
of
it.
As
you
can
see,
it
has
a
typical
structure.
It
has
a
package.xml
and
a
c
make
list
of
txt.
A
In
this
case
it's
a
terribly
simple
washview
package,
one
that
is
depending
on
a
previous
benchmark.
Nothing
defense
in
here
just
average
package.xml.
If
you
know
Ross,
this
should
come
pretty
easy
for
you
and
in
the
case
of
these
simulator
txt
it's
trivial
because
we
are
not
defining
any
new
node
or
component.
We
are
actually
just
simply
deploying
the
launch
files
into
the
workspace
there's,
also
the
corresponding
Benchmark
dot
Gamo
file
that
we
described
before
and
again.
A
This
is
needed
because
we
aim
to
Auto
generate
most
of
the
data
in
the
benchmark
repository,
so
there
should
be
a
continuous
integration
system
integrated
and
whenever
there's
a
pull
request
that
gets
accepted,
that
CI
system
should
review
everything
update
the
tables
update
everything
from
the
metadata,
so
we
will
be
using
this
and
that's
pretty
much
it
the
structure
of
the
metadata
of
the
package.
Now
the
real
juice
is
just
this
thing.
So
again,
super
simple
roster
package,
just
a
launch
file
and
what
this
launch
file
is
doing.
A
Actually,
it's
just
launching
a
trace
instant
so
that
we
collect
the
data
and
again
this
is
going
to
produce
an
lttng
Trace
that
we
can
later
post
process
to
collect
the
corresponding
data
and
then
generate
plots
or
whatever
we
wanted.
This
is
going
to
be
what
we
will
be
using
then
to
draw
conclusions
about
how
good
or
bad
this
went
and
the
computational
graph
is
defined
in
here
in
this
compostable
node
container.
A
A
A
That's
probably
to
my
side
yeah.
Well,
probably
something
I
need
to
to
review,
but
there
should.
The
data
should
be
there.
Typically,
the
data
is
in
that
folder
that
I
just
showed
you
and
let
me
just
go
again:
Ross
tracing
and
typically,
let's
just
get
this
one-
that's
one
example:
I
probably
did
before,
and
so
in
here
often
the
case.
We
just
want
to
use
tools
to
visualize
the
data.
The
most
common
one
is
to
use
bubble
Trace
in
here
and
just
lots
of
stuff.
This
one
is
Also.
A
Yeah
this
one's
good,
so
this
is
the
data
actually
that
gets
locked.
This
is
what
we
should
expect
out
of
a
benchmark.
Of
course,
this
needs
to
be
digested
in
a
programmatic
manner.
Nobody
should
be
looking
at
this
manually,
but
this
gives
us
plenty
of
information.
If
you
see
the
time
the
time
kind
of
like
resolution,
we
get
nanosecond
level
resolution
and
we
can
actually
change
the
clock.
This
is
one
of
the
beauties
of
rostitch
racing
and
lttng.
A
So
that
is
a
natural
apologies
for
this
extra
five
minutes,
but
I
thought
it
was
worth
just
doing
a
an
end-to-end
sort
of
like
walkthrough
I'll
find
some
time
in
the
coming
few
days
and
and
wrap
up
this.
This
A2
example
that
I
had
in
this
Branch
and
make
sure
that
it's
up
to
the
expectations,
but
that's
pretty
much.
A
I
I
A
A
Some
of
you
may
need
to
learn
some
stuff.
Some
of
you
may
know
everything
and
it's
just
gonna,
be
just
super
easy
and
and
straightforward,
but
this
is
this
is
pretty
much
it.
Contributions
are
super
welcome.
Improvements
are
super
welcome
and,
as
we
discussed
in
the
past,
we'll
be
iterating
on
every
single
one
of
these
weeks.
Hopefully
discussing
more
Hands-On
results.
A
Expectations
from
our
site
and
from
my
side
should
be
that
the
following
weeks,
I
think
we
will
wrap
this
first
version
of
the
tools
to
start
producing
benchmarks
faster,
and
we
will
also
hook
it
with
our
continuous
integration
system
to
start
throwing
data
at
least
for
preliminary
benchmarks.
So
then,
at
that
point
for
every
single
Benchmark
that
we
add,
we
will
be
literally
just
automatically
producing
data
across
the
the
boards
that
we
have
in
our
build
Farm.
B
No,
that
was
that
was
great
thanks,
so
much
for
the
walkthrough.
That
was
very,
very
helpful,
so
very
much
appreciated
yeah.
So
next
week
we
could
share
with
everyone,
like
maybe
a
plan
of
what
what
Benchmark
people
would
like
to
tackle.
So
that
way,
we're
not
overlapping
in
our
work,
and
so
we
can
kind
of
like
plan
out
what
we
aim
to
accomplish
in
the
next
upcoming
few
weeks.
A
You
guys
mentioned
data
sets
versus
Raw
specs
I,
actually
like
to
generalize
things
and
I
agree
that
we
should
keep
a
an
attitude
that
goes
beyond
just
the
Ross
community.
So
just
give
it
a
thought,
but
if
you
guys
believe
that
data
set
would
be
a
better
naming,
I've
been
super
happy
to
to
take
an
action
there
and
and
change
it.
It's
better
changed
right
now
before
we
automate
everything,
because
then
nightmares
will
come
afterwards.
A
A
B
G
Yeah
that
ties
in
to
what
I
was
about
to
say,
which
is,
we
should
probably
have
like
just
some
nomenclatures,
that
we're
all
using
the
same
terminology.
You
know
when
someone
says
reference
implementation.
What
do
they
mean
when
someone
says
which
model
they
mean
the
same
as
a
task
yeah?
It's
like
a
data
set,
for
example,
just
so
that
we're
not
like
walking
past
each
other,
but
we're
saying
the
same
thing.
I
Yeah
yeah,
the
thing
I
would
simply
say
this
stuff.
You
know
One
Source.
You
know
ground
truth
right
though
yep
I,
like
that,
the
reference
right
like
just
do
the
pr
stuff.
You
know
that
Victor
started
off
right.
So
then
we
could
find
it
and
then
we're
just
going
to
build
on
whether
you
start
off
so.
A
Yeah
so
feel
free
Matthew.
Today
we
put
together
an
additional
document,
a
markdown
file
that
can
go
at
the
root
of
the
repository
with
my
class
here.
I
think
that
would
be
a
great
contribution
and
I
agree
and
speaking
of
brown
troops,
maybe
one
last
thing
before
we
should
be
speaking
about-
maybe
the
first
research
output
that
we
will
accomplish
together,
which
will
be
a
paper
so
just
to
get
the
excitements
up.
A
A
No
all
right,
then
sorry
for
this
11
minutes
extra.
In
terms
of
administration
actions,
we
will
be
recording
the
meetings.
Unless
someone
disagrees
strongly,
we
will
be
making
this
available
as
well
in
the
channel
of
the
hardware
acceleration
working
group,
so
so
that
we
keep
them
archived,
but
that's
about
it
together
with
the
minutes.
That
should
be
enough,
I
think
for
keeping
up
with
this.