►
From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #12
Description
Twelfth meeting of the ROS 2 Hardware Acceleration Working Group (HAWG).
We discussed a) Updates from ROSCon and IROS on hardware acceleration, b) the Robotic Processing Unit, updates and block diagram, c) Standardization efforts in hardware acceleration updates, REP-2008 and REP-2014, d) 2022 Hardware Acceleration Report in Robotics 9, e) RobotPerf, a consortium of robotics leaders from academia, research labs, and industry whose mission is to build fair and useful robotics benchmarks that provide unbiased evaluations of robotics computing performance for hardware, software, and services—all conducted under prescribed conditions.
For more including source code, check https://github.com/ros-acceleration. Minutes of the meeting available at https://docs.google.com/document/d/185Cy1xjpAOgJygEOnlf5OCgOQTywmF0qgSpS3GiW16Q
00:00 Welcome
01:30 Updates from ROSCon and IROS on hardware acceleration
08:15 Standardization efforts in hardware acceleration updates, REP-2008 and REP-2014
15:34 Robotic Processing Unit (RPU) and Robotics MCU updates
17:19 2022 Hardware Acceleration Report in Robotics
19:45 RobotPerf
19:45 Guest talk: RobotPerf, Benchmarking Robotic Systems
46:50 Q&A
For more including commercial support on hardware acceleration topics in robotics, refer to https://accelerationrobotics.com.
A
The
participation
for
this
event
is
yeah,
we're
already
past
40.
It's
going
to
be
big,
so
we're
not
going
to
record
every
single
person
based
on
past
experiences,
which
was
lots
of
work
instead.
I
just
kindly
point
you
to
the
linking
event
we're
in
essentially,
everyone
should
be
registered
I'm,
going
to
start
by
sharing
my
screen
as
usual,
and
also
I'm
going
to
paste
in
the
chat
the
minutes
of
this
meeting.
A
So
we
had
a
number
of
talks
actually
about
Hardware
acceleration
throughout
Roscoe
and
iros
over
the
last
couple
of
weeks
in
Kyoto
Japan.
For
those
of
you
that
weren't
there
and
aren't
familiar
with
these
two
conferences.
They
happen
every
year
across
the
world,
typically
in
a
rotation
Manner
and
they
pack
together,
hundreds
or
thousands
of
roboticists
who
share
essentially
their
views
on
latest
updates
and
research
across
various
Fields,
including
sorry,
across
various
sub
areas
in
robotics,
including
of
course
Hardware
acceleration.
A
We
had
this
year
at
roscom
talk
named
an
open
architecture
for
Hardware
acceleration
in
robotics
and
ross2,
which
was
partially
featured
in
one
of
the
previous
working
group
meetings.
There's
also
a
blog
post
in
here,
which
in
case
you
couldn't
make
it
to
Kyoto,
summarizes
a
bit
of
the
speech
that
was
given.
So
you
can
go
through
it
and
I
won't
bore
you
with
with
more
details.
You
can.
You
can
read
it
and
then,
of
course,
ask
any
questions
or
send
us
any
comments.
A
A
Unfortunately,
we
couldn't
take
questions
because
of
time
issues,
but
overall
I
think
it's
fair
to
say
that
we
pretty
much
had
quite
a
bit
of
excitement
and
and-
and
there
were
certainly
lots
of
people
that
approached
afterwards
and
asked
lots
of
questions.
So
on
that
end,
thank
you
all
of
those
of
you
who
met
or
meet
me
there
over
the
last
a
couple
of
weeks
and
also
thanks
to
everyone,
sending
good
good
feelings
comments
as
well
as
criticisms.
A
That's
helping
us
improve
this
working
group
and
the
work
we're
doing
so
roscon
very
successfully,
encouraging
everyone
to
consider
joining
next
year.
It
is
certainly
a
community
that's
growing
and
and
maturing
very
rapidly
this
year,
impact
over
700
persons,
very,
very
technical,
but
also
lots
of
business
around
next
year
is
happening
in
the
US,
so
do
consider
joining
if
you're
doing,
Hardware
acceleration
we'll
be
there
I
was
was
definitely
also
quite
interesting.
We
presented
a
talk
over
there.
A
That
was
the
result
of
a
collaboration
between
Harvard
and
acceleration
products
as
well
as
clackenford
University,
and
we
also
spoke
about
Hardware
acceleration
and
open
architectures
on
how
to
enable
this
in
our
vendor
and
a
vendor
neutral
and
Technology
agnostic
manner.
The
recording
is
available
and
you
can
check
it.
A
Would
you
what
you
see
right
now
is
the
results
that
essentially
were
produced
after
processing
and
the
input
received
from
the
survey
that
we
conducted
both
in
the
roster
community
and
the
robotics
communities.
Overall,
when
we
asked
the
same
question,
however,
we
only
offered
three
choices
to
the
audience,
and
you
can
see
that
the
spread
follows
with
real
time
and
speed
or
low
sorry
low,
latency
being
the
ones
that
got
the
most
attention.
A
Now,
when
we
expanded
on
that
during
Roscommon
and
that's
what
you
can
see
on
the
right
side,
we
could
observe
how,
essentially,
as
we
provided
more
options,
particularly
as
we
are
learning
more
and
more
and
as
you
guys,
are
providing
more
and
more
feedback.
It
becomes
obvious
that
there's
more
things
than
than
just
the
very
few
ones
that
we
considered
at
the
beginning
and
how?
A
Systems
were
in
essentially
real-time
and
determinism
is
one
of
the
most
relevant
characteristics,
so
so
that
kind
of,
like
is
very
coherent
with
how
we
bring
up
and
build
robots,
but
certainly
interesting
to
also
see
how
about
a
quarter
cares
a
steal,
very
much
about
low
latency
speed
and
also
help
bandwidth
and
and
power
certainly
are
of
relevance
specifically.
I
must
admit
that
I'm
personally,
my
personal
belief
is
that
power
is
going
to
grow
over
time.
A
I
think
still
only
10
cares
about
you
know
getting
computations
with
you
know
in
a
power
conscious
manner,
I
think
this
is
going
to
increase
over
time
as
people
realize
that
mobile
robots
are
actually
very
sensitive
to
this,
but
we'll
get
there
so
yeah.
These
are
interesting
results
that
I
wanted
to
share.
A
We
will
be
conducting
follow-up
surveys,
possibly
possibly
next
year,
I,
don't
think
we're
going
to
annoy
any
of
you
again
anytime
soon,
but
nevertheless,
once
again,
thank
you
to
all
those
of
you
who
participated
either
in
the
survey
a
few
months
ago
or
now
during
roscon
answering
the
people
that
was
fantastic
and
I'm
also
super
happy
to
see
how
there
seems
to
be
some
conclusions.
We
can
certainly
take
out
of
this
and
we'll
be
continuing
with
this
effort
now
connected
to
this
community.
A
Surveying
effort
is
the
standardization
efforts
that
we've
been
pushing
as
part
of
these
walls
to
Hardware
acceleration
working
group,
which
essentially
boiled
down
to
two
ongoing
Community
standards.
For
those
of
you
not
so
familiar
with
the
Ross
ecosystem
within
the
community,
there
is
essentially
a
community
approach
to
Define
standards
on
how
things
get
done
or
should
be
done
in
robotics.
These
are
called
reps
or
Ross
enhancement
proposals,
and
these
reps
essentially
live
in
ross.org
reps,
and
in
here
you
can
see
the
index
so
there's
a
bunch
of
them
that
have
been
created.
A
Historically,
you
may
not
like
the
format,
but
actually
it
works
fantastically
fine
for
what
concerns
defining
this
it
is.
It
is
pretty
easy
to
extend
and
create
new
ones,
and
so
you
can
see
a
number
of
them.
We've
been
working
now
for
a
few
months
and
trying
to
define
those
that
are
relevant
for
Hardware
acceleration
purposes
and
those
ones
again
live
in
those
two
ones
that
I
Linked,
In
Here.
A
For
a
bit
of
context,
this
was
originally
part
of
rep
2008,
but
then,
after
series
of
feedback,
we
decided
to
Branch
it
out
into
its
own
document,
wherein
we
extended
further
the
principles
why
it's
so
important
and
essentially
how
we
recommend
this
should
be
done.
It's
important
to
highlight
that
this
rep,
as
opposed
to
the
previous
one,
which
is
a
specification
this
right,
2014,
it's
informational.
This
means
it
does
not
aim
to
be
setting
any
conventions.
A
It's
not
a
specification,
it's
just
an
informational
rep,
so
essentially
the
the
impact
it
will
have
will
be
still
relevant,
but
still
you
may
choose
to
do
things
differently,
while
still
compliant
totally.
We
cross
two
principles,
there's
here
motivation
and
rationale
on
why
this
is
so
important.
I
just
want
to
spend
one
or
two
minutes
highlighting
the
fact
that
the
roster
Community
right
now
is
is
very
fragmented,
both
in
terms
of
the
roster
distributions,
but
also
in
terms
of
even
Stacks
itself,
manipulation
and
navigation.
A
It's
it's
often
complicated
to
find
alignment
across
the
use.
The
uses,
sorry
of
the
same
package
across
various
stakeholders,
various
vendors
or
manufacturers
or
robots.
So
this
this
of
course
further
complex
complicates
the
reality
when
it
comes
to
comparing
apples
with
apples,
and
that
is
why
this
rep
tries
to
somehow
shed
some
light
into
how
to
do
so
in
a
vendor,
neutral
and
comparable
manner.
A
Again,
the
value
for
stakeholders
is
described
in
there
and
it
does
propose
to
follow
a
strict
quantitative
approach
by
essentially
integrating
tracing,
particularly
low
overhead,
real-time
capable
Tracer,
it's
leveraging
lttng.
That
illustrates
in
toolkit,
for
those
of
you
that
aren't
familiar
with
this.
It
is
a
open
source
project.
That's
extremely
influential
that
is
used
to
instrument
many
software
packages,
that
is
of
extreme
low
overhead.
A
That
means
that
you
can
instrument
a
piece
of
software
and
that
you
can
enable
or
not
the
trace
points
if
you
disable
them,
the
impact
is
negligible.
If
you
enable
it,
the
impact
is
actually
extremely
small
and
it's
still
can
provide
real-time
responses
and
there
are
papers
that
are
cited
in
the
references
at
the
bottom
of
this
document.
That
gave
you
further
insight
into
that.
A
I
wanted
to
spend
just
a
few
seconds
further,
emphasizing
that
the
approach
that
suggested
in
this
document
helps
us
collect
both
functional,
but
also-
and
that
is
the
focusing
here
non-functional
data
about
running,
robotic
systems
running
raw
2
systems,
that
is
information
about
the
latency.
It's
throughput
memory
power
Etc
in
a
gray
box
manner.
That
means
that
we
don't
need
to
create
an
artificial
test
application
that
then
we
run
to
collect
the
data.
A
No,
we
actually
instrument
the
real
application,
the
ongoing
real
application
that
can
run
on
simulation
or
on
your
target,
physical
robot,
and
while
this
application
is
running,
it
is
generating
data
throughout
each
one
of
these
probes.
This
data
is
landing
into
a
file,
a
physical
file,
that's
in
your
root
file
system-
and
this
is
happening
all
with
a
with
a
tracer.
That's
specifically
designed
to
do
this,
not
C,
plus
plus
or
python
custom
made
or
third-party
libraries
that
often
the
case
include
or
impact
significantly
your
application.
A
Certainly
each
one
of
them
does
add
something
different,
but
nevertheless,
coherent
approach
would
have
certainly
done
something
better
for
the
community
and
that's
hopefully,
what
we
are
trying
to
get
in
here
for
what
concerns
Hardware
acceleration
as
well
as
overall
getting
performance
benchmarking
of
any
robotic
application
in
a
stack
agnostic
manner
across
manipulation,
navigation,
autonomous
Mobility
applications,
Etc.
So
anyhow,
encouraging
you
to
go
through
this
document.
A
Rep
2014
is
definitely
going
to
be
highly
discussed.
It
is
now
a
good
moment
actually
to
dig
in
and
maybe
try
to
make
enough
contributions
so
that
you
can
be
part
of
the
offers
leading
this,
and
also
a
word
of
thanks
to
a
few
folks
that
have
been
given
quite
a
bit
of
review,
particular
particularly
Christoph,
Bedard
and
Ingo.
They
have
been
very
helpful
reviewing
this.
A
So,
thank
you
guys,
if
you're
listening
to
this
now
or
in
the
future
and
yeah
with
that
I
think
we
do
cover
the
standardization
efforts
so
far.
There's
minor
updates
regarding
some
of
the
projects
of
the
working
group,
particularly
the
robotic
processing
unit
and
the
robotics
MCU,
but
I
think
for
the
sake
of
time,
I'm
not
going
to
speak
too
much
about
that
today.
A
I'll
only
say
that
there
is
very
exciting
work,
that's
ongoing
in
the
background
about
trying
to
push
out
a
complete
block
diagram
of
the
robotic,
Processing
Unit,
which
hopefully
might
be
ready
for
the
next
meeting
in
December,
and
also
that
for
what
concerns
the
robotics
MCU,
which
we
discussed
in
previous
meetings.
We
are
right
now
working
with
arrival
to
try
to
come
up
with
farther
and
narrow
down
use
case
to
drive
the
development
of
this
open
source
server
and
Hardware
microcontroller
unit
for
robotics
powered
by
race
5..
A
This
wasn't
originally
part
of
what
we
had
in
mind
when
we
kicked
off
the
project
when
Plan
B
and
asked
acceleration
robotics
kicked
or
kicked
it
off,
but
the
reality
is
that
it
came
on
the
way
it
came.
Actually
out
of
the
working
group
which,
as
you
can
see,
we're
taking
feedback
very
seriously
and
Frankly
Speaking
I
can
only
say
I'm
I'm,
shocked
and
excited
to
be
working
right
now,
with
arrival
to
try
to
bring
this
into
the
project.
A
We
are
very
excited
about
what
this
can
lead
to,
because
if
we
manage
not
only
to
show
a
proof
of
concept
of
robotics
MCU,
that
is
going
to
be
a
first-class
participant
of
the
roster
ecosystem,
but
also
one
that
actually
serves
to
fulfill
a
use
case
need
that
right
now
exists
in
robotics
that
other
microcontrollers
can't
I
think
we'll
will
fulfill
various
objectives
simultaneously.
So
so
yeah
stay
tuned
and
also
hoping
to
bring
news
about
this
very
very
soon
with
that.
A
Allow
me
to
just
bring
you
up
to
date
with
a
few
updates
regarding
new
business
over
the
last
few
weeks.
Since
the
last
meeting
that
happened
in
September
there's
been
some
recent
stuff
that
has
been
published.
The
first
one
is
we
finally
made
available
the
2022
Hardware
acceleration
report
in
robotics.
This
is
something
that's
been
going
on
for
a
while
that
we
have
been
reviewing
that
some
of
you
have
been
actually
acting
as
reviewers.
A
That
does
take
some
conclusions
based
on
the
data
and
results
that
are
generated
so
I'll
leave
it
I'll
leave
that
to
you
to
further
review
it
and
skip
that
to
the
last
section
of
the
of
my
introduction
today
and
and
I
want
to
hand
it
to
BJ
ASAP,
so
I'll
just
say
two,
two
quick
more
things.
The
first
one
is
that
we're
gonna
have
a
session
in
December,
it's
likely
to
happen
around
mid
December.
A
Essentially
a
Cuda
enabled
or
accelerated
implementation
of
a
simulated,
lighter
and
that'll
be
touched
on
next
session.
So
stay
tuned
for
announcements.
Regarding
that
just
wanted
to
give
you
a
heads
up,
because
I
think
that's
that's
going
to
be
exciting
and
and
this
this
person
is
coming
from
Robotech,
so
yeah
a
good
company
to
definitely
follow
as
well
in
the
in
the
domain
of
Hardware
acceleration.
A
Last
topic
I
wanted
to
introduce,
is,
and
that's
going
to
be
further
touched
by
BJ,
but
I
just
wanted
to
give
you
a
heads
up.
We
haven't
yet
announced
much
about
this
in
this
course
or
in
social
media,
but
we
are
very
excited
to
be
essentially
pushing
forward,
something
that
we
call
robot
curve,
which
is
a
benchmarking
Suite
that
is
going
to
be
used
to
validate
robotics
Computing
performance.
That,
of
course
includes
Hardware
acceleration
capabilities.
A
The
robot
perf
also
is
going
to
aim
to
not
only
provide
a
suite
that
you
can
use
to
generate
Baseline
performance
results
on
your
own
research,
but
it's
also
going
to
aim
to
become
a
Consortium,
a
Consortium
of
robot
leaders
coming
from
Academia
research
Labs,
but
also
from
industry,
and
the
ultimate
proof
about
industry.
Relevance
is
the
fact
that
we
at
acceleration
Robotics,
are
very
involved
on
it
and
leading
it
from
the
get-go.
A
The
mission
itself
of
this
Consortium
is
going
to
be
to
build
fair
and
useful,
robotic
benchmarks
that
are
technology,
agnostic
that
are
vendor
neutral
and
that
provide
unbiased
evaluations.
That's
that's
in
a
nutshell,
but
we're
going
to
be
aiming
for
and
yeah
I've
I've
just
pasted
in
there
a
link
to
right
now
the
current
ticket
that
is
tracking
the
definition
of
robot
curve.
So
you
have
a
few
more
details,
rational
and
motivation
over
here,
I
I
want
still
you
more
of
your
time.
You
can.
A
You
can
touch
on
this
on
your
own.
If
you
want
to
the
main
objective
of
this
is
to
provide
a
site
right
now,
where
you
can
express
interest
if
you
want
to
so
you
can
drop
a
comment
in
here.
Of
course
you
need
to
authenticate
yourself,
but
I
think
that's
the
least
we're
expecting
for
people
who
wants
to
participate
and
and
yeah
express
your
interest,
maybe
mention
which
organization
you
come
from
again.
This
is
not
a
project
we're
leading
on
our
own,
but
in
collaboration
with
BJ
and
his
team.
A
So
he'll
tell
you
more
about
the
wise
and
the
house
right
now,
but
just
wanted
to
make
everyone
aware
about
the
fact
we're
taking
this
very
seriously
that
we
are
investing
resources,
that
this
is
going
to
be
a
big
chunk
of
the
world.
We'll
be
doing
from
right
now
and
into
the
following
few
months
so
stay
tuned.
A
So
he
is
extremely
experienced
about
how
to
put
proper
industry
benchmarks
in
place.
He's
been
working
with
both
Academia
and
Industry,
he's
definitely
the
right
player
and
has
been
also
doing
lots
of
work,
benchmarking,
robotic
systems.
So
with
that
BJ,
the
stage
is
yours
and
super
happy
to
have
you
awesome.
Thank.
B
You
know
it's
like
it's
a
word
that
we're
often
familiar
with
that
I
want
to
actually
dig
into
it
and
explain
what
what
benchmarking
is
and
why
it
is
actually
extremely
critical
and
then
I
want
to
use
some
existing
examples
to
help.
You
understand
that
if
we
can
build
a
robust
benchmark,
how
it
will
actually
accelerate
the
field
which
I
think
is
really
what
we
want
and
we
need
that
field
acceleration
if
we
actually
want
to
see
robots
outside
the
door.
B
I
often
like
to
tell
robotics
is
this:
you
guys
have
been
working
on
this
thing:
Forever
And!
Yet
when
I
open
my
door,
I,
don't
even
see
it
simple
little
robot
outside.
Why
is
that
I
think
there
are
many
good
reasons
for
that
and
I.
Think.
One
of
the
big
reasons
is
a
lack
of
systematic
methodologies
to
assess
things
which
systems
people.
People
like
me
have
been
working
on
for
a
very
long
time,
so
so
some
lessons
to
a
bar
to
be
borrowed
across
the
fields.
B
I
want
to
give
a
big
shout
out
for
Vicar
on
this
as
well,
because
Victor
is
sort
of
integral
in
helping
kind
of
think
through.
Why
we
need
this
and
so
forth,
as
you
can
see
from
the
Reps
and
today's
presentation,
I'm
going
to
be
splitting
it
with
my
research
assistant,
Jason
who's
actually
on
the
call
and
Midway
through
he'll,
actually
take
over
to
help
explain
how
we're
doing
this.
B
You
know.
The
DARPA
gave
this
grandiose
challenge.
They
said,
okay,
well,
try
and
get
these.
You
know
autonomous
robots
to
work
in
a
XYZ
way
that
gives
a
very
clear
Focus
to
people
and
people
can
go
out
and
you
know,
obey
the
rules
and
kind
of
understand
how
to
solve
that
problem.
So
that's
fundamentally
what
benchmarking
really
is.
It
is
a
standard
or
a
point
of
reference
against
which
things
can
be
compared
or
assessed.
Why
you
need
this
comparison
or
assessment
is
really
because
you
want
to
have
different
solutions.
B
You
want
to
encourage
the
community
or
the
ecosystem
to
thrive
and
develop
new
new
and
different
types
of
solutions,
but
you
need
to
be
able
to
compare
them.
Otherwise,
it's
like
saying
apples
are
comparing
the
oranges.
I,
don't
even
know
what
the
hell
that
means
right,
the
fruit
sure
that's
like
saying,
yeah,
they're
robotics,
but
that
doesn't
really
help.
You
understand.
You
know
if
one
apple
tastes
better
than
the
Apple
another
apple
right.
So
that's
why
benchmarks
are
sort
of
critical.
B
B
However,
if
you
want
to
do
this,
you
cannot
simply
sit
in
your
room
or
sit
in
my
lab
and
say:
okay,
how
about
we
kind
of
like
think
about
what
benchmarks
are
needed
in
order
to
drive
the
ecosystem
forward?
If
you
actually
want
to
do
this,
we
really
have
to
develop
a
systematic
and
rigorous
methodology.
Now
this
methodology
is
hard
to
kind
of
build
when
you
kind
of
think
about
it
in
isolation.
B
What
you
really
need
is
Community
Driven
support
and
activity
around
this.
You
need
the
collective
wisdom
of
the
entire
Community,
because
people
look
at
things
in
many
different
dimensions.
You
want
to
be
able
to
tap
into
that
in
order
to
build
that
Benchmark,
and
if
you
can
do
that,
then
we
can
actually
standardize,
which
use
cases
are
important.
What
are
the
most
important
things
to
actually
measure?
What
are
the
workloads?
B
For
example,
we
can
say
Ross,
but
Ross
is
a
big
ecosystem
right
that
many
different
bags
and
which
ones
do
you
look
at
and
so
forth.
If
you
have
that
Community
involvement,
you
can
actually
figure
out
what
those
workloads
are
going
to
be.
That
means,
then,
with
those
set
of
workloads,
you
can
then
compare
solutions
that
are
coming
about
in
this
ecosystem
and
that
comparison
will
actually
create
a
competitive
environment
that
actually
fuels
the
actual
field.
B
So
we
can
effectively
do
complex
characterization
of
systems
to
understand
what
things
to
improve.
And,
ultimately,
though,
you
come
to
my
room
and
say,
hi
I've
got
a
system
that
does
really
well
well.
If
you
have
a
good
Benchmark,
you
can
say:
okay
well,
if
you're
able
to
run
that
Benchmark,
it
tells
me
something
about
what
your
capabilities
are
right
gives
you
that
level
of
transparency?
That's
actually
quite
critical.
So
let
me
kind
of
ground
these
principles
in
the
past,
eventually,
nothing
fundamentally
new
to
the
systems
equal
ecosystem.
B
There
are
existing
benchmarks,
so
I
want
to
give
you
a
little
sneak
peek
of
a
better
for
history
to
help
you
show
how
these
benchmarks
can
actually
accelerate
the
field.
If
you
consider
the
processor
that
your
you
know,
your
computer
has
right.
General,
proper
CPUs
things
like
Intel
and
AMD
build
well
this
community.
If
you
dated
way
back,
you
know
the
field
is
about
50
years
old,
but
it's
like
if
you
actually
date
it
back
to
their
inceptionaires.
There
was
a
lot
of
confusion
about
like
hey,
which
processor
is
actually
better.
B
What
aspect
of
the
process
was
actually
giving
you
better
performance?
Is
it
the
software,
that's
actually
being
optimized
better
by
the
compiler?
That's
actually
giving
you
the
hardware
performance
or
is
it
actually?
The
micro
architecture
sounds
very
simple
today,
but
if
you
actually
kind
of
think
about
it,
it's
quite
complicated
right
because
you
have
to
understand
what
are
you
trying
to
measure
the
micro
architecture?
Then
you
have
to
understand.
Okay,
what
code
do
I
run?
What
code
if
I
run,
is
actually
representative
in
terms
of
stressing
the
processor
in
certain
ways,
then?
B
What's
the
role
of
the
compiler?
What
are
the
elements
that
you
can
actually
freeze
right,
so
spec
Benchmark
has
been
around
for
a
few
decades
and
it
actually
helped
Drive
generational
improvements
in
processor,
performance,
okay
and
some
of
the
key
goals
there
have
been.
How
do
you
enforce
performance
replicability?
How
can
I
repeat
results
exactly
you
know
easily.
This
is
not
about
verifying
or
validating
people's
results.
It's
really
about
building.
B
On
the
you
know,
good
work
or
prior
prior
work
right,
and
you
want
to
understand
what
are
some
represented
workloads
that,
if
I
hey,
if
I
study
workload
a
even
though
my
workload
in
my
entity
is
actually
a
little
different,
it
actually
gives
you
a
lot
of
perspective
into
how
the
machine
will
actually
perform.
Next,
you
want
to
encourage
Innovation,
that's
exactly
what
spec
did
it
drove
the
state-of-the-art
CPU
improvements
from
one
generation
to
the
next?
It
also
allows
fair
and
useful
measurement.
B
You
actually
measure
things
that
actually
tell
you
something
interesting
about
what
is
actually
happening
in
the
system
and
you
want
to
keep
the
commercial
and
research
communities
working
together,
because
if
you
want
Innovation,
you
have
to
have
that
research
element
integrated
in
there
and
you
want
to
keep
the
benchmarks
affordable.
All
of
these
were
principles
of
what
spec
did
in
order
to
drive
where
we
are
with
today's
Intel
and
AMD
processors.
B
For
instance,
if
you
look
at
machine
learning
which
has
come
about
more
recently,
not
in
you
know
over
decades,
but
in
like
about
the
last
seven
years
we
really
took
cement.
If
you
kind
of
look
at
that
space
well,
we
needed
a
new
Benchmark
and
I'm
on
purpose
such
a
benchmark.
Where,
for
example,
we
want
to
build
High,
Performance,
Machine
learning
systems
similar
to
what
we're
trying
to
do
here,
build
high
performance,
robotic
systems.
We
have
to
figure
out.
B
B
You
understand
like
how
we're
going
to
go
about
potentially
building
this
you
know,
robotics
robot
per
Benchmark
I
want
to
talk
a
little
bit
about
some
of
the
nuances
around
building
a
benchmark,
so
okay,
Mr
perf,
is
a
suite
eventually
similar
to
what
robot
perfect
is
going
to
be.
But
how
do
you
build
something
like
that?
Well,
you
have
to
deal
with
large
public
data
sets
because
it's
a
Community
Driven
activity,
whatever
we
do
has
to
be
in
the
public
domain.
B
You
take
that
and
then
you
build
specific
benchmark
for
specific
aspects
that
you're
trying
to
measure
in
machine
learning.
You
often
have
to
train
a
model
before
you
can
actually
use
the
model.
So
you
have
the
notion
of
training,
The
Benchmark,
how
hard?
What
do
you
actually
stress
when
you're
trying
to
train
a
network
in
inference
where
you're,
just
using
the
network,
there's
a
different
set
of
metrics?
You
actually
want
to
capture
in
order
to
build
this
pipeline.
You
have
to
identify
what
are
some
key
tasks.
B
For
instance,
you
know,
is
it
perception,
planning
control
localization?
What
is
it
that
we
actually
care
about
right?
There
are
lots
of
things
that
happen
in
the
robotics
pipeline.
So
what
are
those
tasks
we
care
about
the
meaning
to
understand
how
are
the
systems
actually
exercised?
It
was
interesting
to
see
what
Victor
was
talking
about
earlier.
We're
emphasizing
real
time
was
sort
of
a
critical
thing,
so
one
might
say
if
you
talk
to
a
traditional
processor
architect,
like
me,
I
would
say:
oh
just
look
at
you
know
throughput.
B
My
machine
is
able
to
Crunch
x
amount
of
frames
per
second
or
something,
but
that's
actually
not
a
real-time
metric
right
and
that
becomes
critical
by
understanding
where
and
how
these
robots
are
going
to
be
deployed
in
machine
learning.
It's
about
machine
learning
has
actually
been
employed,
then
I'm
defining
a
clean
set
of
rules.
What
do
you
measure
when
does
timing
start?
When
does
timing,
stop
and
effectively
capturing
the
right
set
of
metrics?
B
That
actually
tell
us
something
about
something
interesting
that
we
actually
care
about
at
the
end
of
the
day,
right
and,
of
course,
being
transparent.
We
want
to
make
sure
the
results
and
how
data
is
being
collected
is
all
transparent,
because
that
transparency
is
what
is
going
to
allow
us
to
move
forward
now.
If
we
want
to
build
something
like
this,
you
can't
simply
just
you
know
cook
it
up
in
a
lab
right.
You
really
have
to
bring
the
community
together.
So
in
the
context
of
ml.
B
What
we
did
was
we
brought
the
industry
together
because
there
was
so
much
excitement
around
ml
in
robotics,
too.
There's
a
wicked
amount
of
excitement
right.
The
large
number
of
companies
exactly
to
Victor's,
Point
lost
on
and
so
forth.
I
mean
I
was.
These
are
Big
conferences,
lots
of
people.
The
industry
is
really
cooking
towards
this,
so
similar
to
what
I'm,
showing
down
here,
I
think
there's
an
entire
parallel
mapping
of
the
you
know
the
different
companies
that
are
interested
some
potentially
overlapping
here,
but
we
can
gather
this
community
together.
B
Then
we
can
actually
build
interesting
benchmarks
which
have
certain
key
characteristics.
So,
for
example,
a
benchmark
would
cover
many
different
areas
and
machine
learning,
for
instance,
a
vision,
language
and
so
forth.
You'd
have
different
kinds
of
machine
learning,
tasks
or
robotic
tasks
in
our
instance
would
have
different
specific
scenarios
that
we
actually
want
to
evaluate.
So
we
have
corresponding
data
sets.
B
For
that,
like
you
know,
what's
the
input
feed
we
need
to
have
certain
key
metrics,
it's
not
about
just
performing
in
tasks
right,
it's
a
question
of
how
well
you
perform
the
task
in
machine
learning.
For
instance,
you
know
it's
very
important
to
get
the
accuracy
right,
so
we've
specified
certain
targets
and
then
we
provide
reference
implementations
that
you
can
actually
just
run.
You
do
the
same
thing
for
inference
I'm
not
going
to
get
into
the
details.
B
B
Just
using
that
as
an
example.
Again,
you
know
the
each
of
the
dots
shows
generational
benchmarks
right.
So
every
imagine
every
six
months
you
know
we're
collecting
state-of-the-art
data,
then
what
we
can
see
is
across
different
tasks.
We're
seeing
substantial
Improvement
originally
was
Forex
Improvement,
just
based
on
software
and
Hardware.
Then
it
went
up
to
8X.
Then
it
went
up
all
the
way
to
30X
right.
So
the
point
I'm
making
is,
if
you're
starting
to
measure-
and
you
have
a
core
set
of
tasks
that
are
rock
solid,
then
you
can
actually
systematically
make
progress.
B
Much
of
this
progress
came
not
only
from
Hardware
Innovation.
It
also
came
from
software
animation,
because
people
understood
that
there
was
this
Beacon,
which
is
the
Benchmark
that
they
really
wanted
to
knock
out
of
the
park,
and
so
it's
extremely
useful
to
actually
do
the
benchmarking.
Similar
thing
on
you
know
another
Suite
of
workloads,
and
you
see
radical
improvements
over
multiple
generations
of
The
Benchmark
I'm.
B
Just
trying
to
tell
you
that
it's
a
general
thing
that
you
find
that,
if
you're
actually
measuring
progress,
then
you
can
actually
get
progress
and
once
you
start
building
that
sort
of
a
coherent
mindset,
you
can
actually
create
an
organization
that
supports
these
activities
right.
So
here
for
ML,
it's
like
ml
comments.
You
know
a
bunch
of
different
companies
that
all
care
about
machine
learning
have
come
together.
B
Can
we
do
something
like
this
with
the
robotics
kind
of
space,
where
we
have
both
Academia
and
Industry,
collaborating
together,
I'm
an
academic
I'm,
very
interested
in
this
space?
There
are
plenty
of
academics
who
are
very
interested
in
Hardware
acceleration
for
robotics
right.
So
can
we
create
that
Community
together?
If
we
can
do
that,
we
can
really
build
a
robotics
Consortium
that
aims
to
accelerate,
robotic
Innovation
for
everyone.
Think
of
it
that
way,
right
so
I
gave
you
this
notion
of
how
we
can
do
this
in
the
past.
B
You
know
we've
done
it
for
ML
part,
we've
done
some
spec
and
so
forth.
So
the
key
question
is:
can
we
adopt
the
principles
and
the
success
that
we
have
seen
in
some
of
these
existing
benchmarks
and
bring
those
lessons
over
to
the
robotics
domain
right,
where
we
can
actually
reap
some
of
the
core
principles
easily?
We
don't
have
to
reinvent
the
wheel.
C
C
A
C
Okay
cool,
so
thanks
Vijay,
now
that
we
have
a
good
sense
about
how
machine
learning
has
benefited
from
benchmarks.
Let's
take
a
look
at
the
field
of
Robotics
and
maybe
you
we
can
see
how
a
benchmark
in
the
field
of
Robotics
could
accelerate
this
field
and
fill
in
some
of
the
gaps
that
we
see
in
robotics.
C
C
About
this,
we
came
up
with
five
key
characteristics.
The
first
one
is
listening
to
the
community.
What
does
the
robot
Community
value
the
most?
What
are
the
challenges
that
we're
facing
right
now?
The
second
characteristic
is
evaluating
the
correct
of
the
metrics
picking
the
right
metrics.
What
are
the
metrics
that
are
slowing
down
the
field.
C
C
Should
we
look
at
the
next
characteristic
is
applications.
A
strong
Benchmark
should
look
at
a
wide
variety
of
applications,
because
some
bottlenecks
could
occur
depending
on
the
different
tasks
that
you're
looking
at
and
finally,
a
strong
Benchmark
should
be
automated,
or
what
I
mean
by
this
is
that
it
should
be
reproducible
allowing
different
members
of
the
community
to
contribute
and
improve
the
benchmark.
D
C
So
looking
at
that
second
characteristic,
that
makes
a
benchmark
strong.
We
have
to
pick
the
right
metrics,
the
valuable
metrics,
and
so
the
report
asks
asked
the
community
what
they
care
most
about-
and
here
are
the
results.
Victory
briefly
went
over
this
a
few
minutes
ago,
but
a
great
amount
of
the
community
value
speed
at
48
and
then
at
46.
They,
the
Community
Values,
real-time
determinism,.
F
C
D
C
C
Let's
talk
about
some
of
the
widely
used
tools,
so
Victor
and
his
team
compiled
a
list
of
all
the
companies
that
have
kind
of
explored
Ross
or
have
a
project
related
to
Ross,
and
they
came
up
with
they
found
more
than
100
companies
have,
you
know,
had
some
sort
of
relationship
to
Ross
as
as
I
was
going
through.
The
list
I
started,
seeing
some
really
big
names
like
Boeing,
Intel,
Nissan,
Honda
and
the
the
list
goes
on.
So
it's
clear
to
me
that
Ross
is
commonly
used
within
industry
and
Academia.
C
C
D
C
Component
is
the
Ross
perception
stack.
There
were
other
components
of
the
robotic
pipeline
that
were
also
highly
valued
by
the
community,
which
is
the
navigation
stack
and
the
manipulation
side.
So
let's
look
at
the
robotics
pipeline,
so
we
have
the
force
perception.
Planning,
control
perception
refers
to.
D
E
C
Control
the
robots,
actuation
So
within
each
part
of
this
pipeline.
There
are
specific
kernels
that
are
used
for
specific
tasks,
and
what
would
be
interesting
here
to
consider
is
how
the
interaction
between
these
kernels
cause
bottlenecks.
So
you
can
imagine
how
these
kernels
would
be
running
simultaneously.
Some
of
these
criminals
would
be
running
simultaneously,
so
they
would
compete
over
resources.
C
And,
of
course,
cause
some
sort
of
bottlenecks,
which
would
be
pretty
interesting
to
study.
I
also
was
interested
to
learn
about
what
kind
of
Hardware
acceleration
is
being
used
by
the
community
and
again.
Another
question
was
asked
by
the
report
and
it
showed
that
about
70
percent
of
the
community
is
planning
to
use
or
is
currently
using
a
GPU,
while
22
percent
of
the
community
is
planning
or
currently
using
an
fpga
accelerator.
C
How
how
would
the
bottlenecks
look
between
those
different
Hardwares,
Okay
cool,
so
we've
covered
some
of
the
important
metrics,
some
of
the
important
Tools
in
robotics.
Let's
take
a
look
at
widely
used
applications
again.
Another
question
was
asked:
what
kind
of
robot
are
you
creating,
and
this
is
a
little
hard
to
read,
but
the
top
four
categories
were
industrial
mobile,
robots,
uads,
armed
industrial
arm,
robots
and
bike
robots.
C
So
if
we
go
back
to
our
benchmarking
methodology
that
we're
creating
here,
we
can
take
it
yet
another
step
forward
and
consider
how
bottlenecks
might
change
between
different
tasks
or
use
cases.
Of
course,
each
different
tasks
would
use
a
different
stack
of
kernels,
so
it
would
be
cool
to
study
how
that
changes.
C
All
right,
there's
been
some
great
work
already
in
the
field
of
Robotics
and
benchmarks.
They've
created
RTR
bench,
considered
specific
kernels
in
robotics,
and
it
evaluated
different
bottlenecks
in
or
the
bottlenecks
in
specific
kernels,
but
it
didn't
look
holistically
at
a
robotic
system
and
the
end-to-end
system
of
all
the
kernels
competing
over
resources
and
how
that
could
change
well,
next,
I'll,
not
them
specifically
what
presented
a
benchmarking
suite
for
micro
aerial
vehicles
and
robot
core,
although
it's
not
a
benchmarking
Suite,
it's
moving
the
field
of
Robotics
forward
by
helping
integrate
Hardware.
C
C
It's
hard
to
create
effective
benchmarks,
so
I
highly
encourage
you
to
take
a
look
at
the
standardization
effort
that
Victor
is
creating
and
input
some
some
ideas
there.
All
right,
we've
talked
about
metrics
tools,
applications
and
lastly,
let's
take
a
look
at
the
automation
section.
So
our
long-term
vision
of
this
robot
curve
Benchmark,
is
to
create
a
leaderboard
kind
of
similar
to
the
AI
Benchmark.
The
AI
Benchmark
has
a
continually
updated
leaderboard
that
evaluates
AI
performance
on
different
types
of.
C
E
A
That's
awesome,
thank
you
so
much
PJ
and
Jason
fantastic
presentation.
You
know
in
a
very
very
I
would
say
very
timely
moment
since
we're
the
discussions
around
benchmarking
and
are
getting
heated
in
the
community
with
essentially
first
players
very
interested,
so
I
think
we
have
time
for
Q
a
everyone.
D
B
D
C
Yeah
I
totally
understand
what
you're
saying
and
that's
a
great
question.
So
I
think
the
distinction
there
would
be.
Maybe
it
is
fast
and
like
the
speed,
has
been
increased
but
the
number
of
times
it
it
completes
before
the
deadline
might
change.
So
maybe
there
could
be
some
like
it
would
be
evaluate
it
would.
A
A
But
the
moment
you
reach
that
situation,
you're
asking
us
and
you're
asking
everyone
in
here.
Why
do
I
care
about
speed
about
lower
latency
at
all?
If
I'm
already
like
meeting
my
deadlines
right
and
I,
think
that's
wherein
the
higher
performance
speed
comes
in,
if
you're
actually
able
to
deliver
way
lower
latencies,
you
can
change
your
deadlines.
You
can
have
your
system
actually
become
higher
performant
and
there,
and
then
your
real-time
deadlines
or
your
real-time
requirements
can
actually
change
and
that's
possible
thanks
to
the
fact
that
you
do
have
speed.
D
A
It
is,
it
is
often
troubling
to
you
and
I,
often
find
myself.
Sometimes
confused
determinism
real
time
throughput,
latency,
often
the
case
get
intertwined
and
actually
getting
the
right
definitions
out.
There
is
I
think
going
to
be
a
big
work
or
a
big
part
of
the
work
of
robot
perfect.
So
hopefully
you
guys
can
help
us
getting
that
done.
A
F
So
I
find
that
when
you
do
have
benchmarks
people
designed
to
The
Benchmark
and
it
hampers
originality-
and
you
know,
sort
of
research
to
some
extent,
and
that
was
certainly
the
case
when
spec
first
came
out
that
people
try
to
improve
what
was
being
measured
and
it
put
a
damper
on
on
other
avenues
of
research.
I,
don't
know
if
you
want
to
comment
on
that,
but
nothing's
ever
you
know,
100
good.
Basically,
there's
always
trade-offs.
B
Yeah
I'm
super
excited
that
you
know
someone
brought
it
up.
I
was
wondering
myself
whether
I
should
break
it
up
so
I'm
I'm
excited
only
academics,
so
yeah
benchmarks
you
spot
on
Miriam
benchmarks,
there's
a
good
side
of
them,
there's
a
bad
side
of
them
and
there's
a
really
ugly
side
of
them
right
and
so
I
often
feel,
like
you
know,
there's
a
time
when
we
really
want
benchmarks
because
benchmarks
allow
us
to
align
the
community
together
right.
B
So
I
do
think
that
you
know
if
we
get
caught
up
in
once
we
create
the
box
and
we
tend
to
stay
in
the
box,
and
it's
really
really
bad
I
agree
with
that.
But
there
I
think
I
think
like
right
now,
for
instance,
is
a
really
good
time
to
get
that
Clarity
around
what
are
the
common
tasks?
What
are
the
base
functionalities
that
we
really
just
have
to
knock
out
of
the
park,
so
we
can
actually
focus
more
on
higher
level
cognitive
tasks
if
I
may
say
so
in
the
robotics
I
mean.
B
So
that's
why
I
feel
like
now
is
a
good
time
to
kind
of
do
such
a
benchmark.
But
again,
even
when
you
craft
a
benchmark,
you
do
want
to
leave
room
for
Innovation
right,
so
you
don't
necessarily
want
to
Benchmark
a
specific
kernel.
Now
there
are
good
reasons
to
Benchmark.
A
specific
kernel
right
like
I,
might
say.
Okay,
this
specific
variant
of
Slam
is
something
I
care
about.
A
bio
is
something
I
care
about.
B
However,
you
might
also
just
kind
of
want
to
generalize
and
say:
okay,
well,
I
just
want
to
do
the
slam
task
at
large
and
say
how
you
do
it
as
a
combination
of
hardware
and
software
is
a
solution
right.
You
want
to
leave
that
room
within
the
Benchmark
and
spec
is
different
in
that
sense,
which
I
agree
with
you.
It's
like
you
know
it's
it's
a
set
of
ones
and
zeros
that
are
given
to
you,
and
then
you
run
exactly
that
thing
and
there's
no
room
for
any
extra
curativity.
B
So
it
does
kind
of
inherently
limit
that
creative
video
you're
talking
about,
however,
like,
for
example,
in
the
ml
case,
where
the
field
is
still
evolving,
we
knew
we
couldn't
draw
the
box
too
tight,
and
so
in
that
case
we
allowed
for
the
task
definition.
But
we
say
how
you
do
the
hardware
software
code
is
on.
For
instance,
we
can
leave
open,
so
I,
first
and
foremost,
I
do
want
to
acknowledge
that.
B
Yes,
there
is,
you
know,
benchmarks
can
result
in
Benchmark
Engineering,
but
I
also
think
like
the
good
side
of
it
is
that
we
get
the
clarity
at
peace
at
this
time
and
I
think
at
least
for
the
next
few
years.
Until
we
get
that
Clarity
The
Benchmark
will
serve
its
purpose
and
after
that
don't
worry.
You
and
I
can
jump
all
over
this
Benchmark
and
be
like
you,
industry
people
are
not
thinking
part,
a
foreign.
A
So
that
is,
that
is,
that
is
a
fantastic
View
and
also
a
fantastic
question.
Miriam.
Thank
you
so
much
for
bringing
it
up.
I'll
I'll,
maybe
I,
can
give
a
bit
of
insight
on
why
we
industry
people
wanted
so
much
right
now,
so
so
so,
besides,
what
what
BJ
did
share
about
the
fact
that
there's
definitely
epic
meet
in
in
the
field
of
Robotics
right
now,
there's
currently
I
would
say
interesting,
Dynamics
in
the
robotics
ecosystem,
where
we
are
seeing
how
various
silicon
vendors
are
jumping
in
and
we
had
acceleration.
A
Benchmarking
and
this
effort,
hopefully
supported
by
all
of
you-
is
also
going
to
hopefully
shed
some
light
into
creating
a
fair
competitiveness
environment
for
actually
comparing
Hardware
Solutions,
because
Frankly
Speaking
right
now.
It
does
feel
that
you
know
people
is
just
reporting
on
what
they
want
and
and
Frankly
Speaking.
Some
groups
are
just
reporting
on
throughput,
claiming
that
that
is
all
you
care
about,
wherein
in
many
cases
in
robotics,
we
do
care
about.
A
Very
very
different
things
so
so
setting
some
sort
of
like
Common
Sense,
on
how
various
players
speak
about
performance
in
robotics
and
in
Boston
I
think
is
crucial
and
also
this
is
in
a
way,
also
a
community
tool
to
find
out
this.
These
big
companies,
which
often
the
case,
have
lots
of
resources
for
marketing
and
and
certainly
don't
allow
individual
researchers,
whether
from
industry
or
Academia,
to
actually
showcase
their
results,
and-
and
we
do
believe
this
is
definitely
going
to
open
up
that
door.
A
One
of
the
things
that
we
are
concerned
about
is
that
there
seems
to
be
a
discussion
around
whether
benchmarks
should
be
conducted
in
an
OPAC
manner,
meaning
that
a
source
code
that's
being
tested
should
or
should
not
be
shared
and
disclosed,
and-
and
maybe
actually
this
this
can
open
up.
The
discussion.
I
can
just
leave
it
there.
But
what
do
you
guys
actually
think
about
about
the
need
to
actually
have
to
have
the
actual
source
code
to
compare
apples
with
apples?
B
I
can
jump
in
just
from
experience.
I
can
say
that,
like
you,
don't
necessarily
want
to
have
Source
words
in
some
cases
you
know
the
source
code
is
the
IP
of
that
particular
company.
Now
that
said,
you
don't
want.
You
know
to
Miriam's
point.
You
don't
want
Benchmark
Engineering
happening
under
this
magical
black
box.
That's
custom
tuned
for
robot
curve,
for
example,
and
while
the
real
stack
is
completely
different
and
it's
like
you
know,
we
know
how
much
marketing
up
to
address
that,
for
instance
like
their
mechanisms
where
you
can
say.
B
Okay,
we
require
you
to
have
an
available
version.
That's
this
is
a
park
release,
and
so,
if
anyone
wants
to
do
quote-unquote
an
audit,
it
is
actually
literally
you
would
say.
Okay,
this
is
this
official
release
right
and
then
you
would
go,
get
that
release
and
put
that
package
in
right,
you're.
Still
it's
just
a
binary
but
you're
officially
getting
that
binary.
B
I
can
get
into
a
lot
more
detail
around
this,
because
we
have
dealt
with
this
question
around
machine
learning,
which
is
extremely
susceptible
to
this
exact
problem.
But
there
are
certainly
ways
that
the
industry
you
can
you
can
respect
the
industry
in
their
IP
and
still
be
able
to
get
that
Benchmark,
but
it
doesn't
all
have
to
be
open
source,
and
with
that
said,
Victor,
if
you
don't
mind
7
is
been,
has
had
his
end
up
for
quite
a
while.
I
want
to
make
sure.
E
Yeah
hi,
my
name,
is
actually
nolani.
I
know
it's
kind
of
misleading
there.
I
didn't
see
any
way
in
and
Google
me
to
be
able
to
change
that
I'm.
Actually,
one
of
the
the
UC
analysts
here
at
Edgewater,
personal
care
and
I
know
this
is
kind
of
it's
a
bit
of
a
digression
but
I
I.
Don't
remember
the
name
of
exactly
the
name
of
the
person
that
that
invited
me,
but
I
figured.
There
was
something
in
my
skill
set.
They
thought
I
could
contribute
in
some
way,
so.
A
Okay,
well
welcome
to
the
group.
Definitely
and
yeah
I
mean
it's
an
open
group,
so
anyone
can
can
join,
contribute,
I,
don't
know
if
I,
if
you
manage
to
get
it
but
I'm
gonna
paste
again
the
minutes
in
the
chat.
That's
probably
the
best
way
to
dig
in
just
browse
through
the
issues
we
do
maintain
a
community
repo
and
there's
a
number
of
tickets
do
feel
free
to
open
new
issues.
E
A
No
no
Ross
Ross
is
a
robotics
framework.
It's
essentially
a
series
of
tools,
as
well
as
predefined
libraries
for
building
robotic
applications.
So
so
I
can
point
you
the
best
place
wherein
to
maybe
read
a
bit
more
about
this
is
and
I'm
going
to
share
my
screen.
Real,
quick
again
is
this
one?
So
this
is
the
robot
perf
ticket,
that's
Linked,
In
the
minutes
and
there's
a
section
in
here
which
says
why
browse
too
right,
because
maybe
many
of
you
like,
like
you
yourself,
might
be
asking
why
yeah?
A
Why
are
you
guys
like
even
mentioning
this
thing?
Why
aren't
you
just
you
know,
grabbing
a
robot
and
measuring
it
directly?
Well,
there's
a
number
of
reasons
why
Ross
is
so
relevant
in
the
robotics
community
and
why
we're
using
it
as
well
as
why
we're
using
it
as
the
Baseline
for
conducting
measurements
and
benchmarks
and
part
of
it
is
because
it's
widely
used.
E
A
G
Yeah
sure
hi
this
is
Divya,
so
hey
so
I
wanted
to
ask
you
from
my
experience
of
benchmarking,
some
like
applications,
which
customer
demand
us
to
like
right.
On
top
of
the
middleware,
for
example,
how
I
mean
how
can
I
write
like
a
very
generic
benchmarking
framework
which
can
accommodate
doing
actual
work
on
the
system
which
the
other
otherwise
like
a
camera
pipeline
or
a
lighter
stack,
for
example,
in
an
autonomous
driving
situation
we'll
be
doing
some
running
some
actual
algorithms
under
the
hood
in
the
actual
pipeline?
G
G
A
He
he
knows
everything
about
it
right,
so
my
my
answer
actually
would
be
broadly
tied
to
if
you
want
to
be
General
and
want
to
try
to
generalize
across
use
cases
in
the
automotive
space,
for
example,
which
I
know
is
the
one
you
guys
are
touching
on.
So
so,
of
course,
ideally,
you
want
to
have
your
different
layers
in
your
robotic
systems,
instrumented.
A
So
all
the
way
from
your
data
link
layer
up
until
your
layer,
7
of
The
OSI,
is
stack
so
so,
of
course,
including
the
middleware
and
then
each
one
of
the
rows
two
layers,
the
raw
two
layers.
Unfortunately,
already
instrumented
and
Crystal
took
care
of
that,
but
when
it
comes
to
the
actual
application
itself,
most
application
customers,
at
least
that's
our
experience
as
well-
are
providing
us
and
are
writing.
They
don't
come
instrumental,
they
just
don't
so
the
way
you
actually
would
approach.
A
A
That's
what
we
do
at
least
it's
the
handiest
thing
and
then
literally
digest
each
one
of
those
data
points
and
and
then
produce
whatever
Benchmark
you
want.
Maybe
you
want
to
report
about
throughput
or
you
want
to
report
about.
You
know
latency
or
real-time
compliance.
That
I
mean
the
the
the
non-functional
measurement
you
will
be
reporting
on
or
the
functional
one
is
actually
very
tied
to
what
you
want
to
do.
So
that's
application!
A
Specific
and
again,
that's
that's
down
to
you,
but
you
can
do
it
with
each
one
of
these
data
points
and
and
each
one
of
these
scripts
unfortunately,
is
going
to
be
tied
to
each
one
of
these
customer
use
cases.
At
least
that's
my
experience
so
far,
I,
don't
think
you
can,
when
you
can
reuse.
Of
course,
the
this
this
logic
and
these
scripts,
but
being
but
being
General
us
completely
different
applications,
would
require
you
to
define
a
very
strict
hierarchy
of
Trace
points
in.
G
A
A
G
This
recipe,
that
was
my
follow-up
actually
like
if,
if
it's
it's
processing,
my
door
or
camera,
like
mostly
90,
95
percent,
should
be
using
Hardware
acceleration,
as
of
like
today,
some
somewhat
in
some
sense.
So
how
do
you
kind
of
isolate
that
part
and
do
the
tracing
for
it
or
like
measure
the
performance
for
that?
Just
that
component
running
on
the
other
Hardware,
apart
from
the
CPU.
A
A
So
it
depends
of
the
compute
architecture
and
the
computational
graph
you're
studying
it
depends
on
what's
being
uploaded
in
the
accelerator
worst
case
scenario,
you
measure
things
you
measure
things
in
the
host,
so
in
the
CPUs,
with
ltdng
best
case
of
scenario,
you
actually
integrate
tracing
capabilities
inside
of
the
accelerator,
so
there's
there's
some
accelerators.
For
example,
AMD
gpus
have
actually
tracing
capabilities
and
you
can
integrate
Trace
points
directly
into
that
and
and
that's
something
we're
looking
at
then.
A
There
is,
for
example,
proprietary
solutions
that
you
can
then
leverage
which
aren't
ltdng
friendly,
but
you
can
still
leverage
from
both
AMD
in
their
fpgas,
as
well
as
the
Nvidia
Solutions,
and
also
the
Intel
Solutions.
So
each
one
of
these
have
their
own
way
to
do
tracing
their
own
way
to
collect
data
and
to
give
you
introspection
into
what's
going
on
in
the
accelerator
and
they
typically
report
also
in
in
kind
of
like
a
stat
file
that
gets
dumped
into
the
root
file
system.
A
A
Just
to
be
fair,
you
can
still
do
so,
while
relying
on
on
CPU,
compute
and
tracing
and
putting
Trace
points
at
the
CPU,
let's
say
level,
but
but
again,
if
you
wanna,
if
you
wanna
get
like
really
insights
into
multi-accelerate
acceleration,
kernel
interactions
and
so
on
and
so
forth,
then
you
need
to
look
into
some
of
the
proprietary
tools,
because
that
is
still
not
integrated.
It
might
be
integrated
in
the
future.
Okay,
but
again
that
is,
that
is
to
be
discussed
and
in
future
work.
A
If
you
anyhow
would
like
to
expand
on
this,
I
think
a
good
reference
for
you
is
to
take
a
farther
look
at
the
Roman
Core
paper
that
I
wrote
with
Sabrina,
Brian
and
BJ.
We
we
do
discuss
part
of
this
in
this
section,
so
this
is
actually.
This
is
a
thumbnail
and
a
screenshot
that
shows
one
of
these
proprietary
tools.
This
is
the
vitis
hls
analyzer.
A
See
how
how
in
here
we
have
information
about
the
kernel
compute
time.
So
this
is
the
acceleration
kernel
executing
and
in
this
is
actually
the
overhead
of
the
framework
that's
being
used
to
convey
the
information
from
the
host
CPU
into
the
accelerator
itself,
and
you
can
see,
there's
an
overhead
and
so
on
and
so
forth.
But
but
this
level
of
introspection
may
require
you
to
rely
on
specific
tools
or
interface.
These
tools
with
ldtng.
A
Okay,
I
think
we
went
past
50
minutes
of
the
time
and
still
50
folks
in
here.
So
thank
you
guys
so
much
I
think
we
can
close
it
in
here.
If
anyone
still
has
questions
feel
free
to
use
the
Community,
Resources,
Json
and
BJ
I
think
BJ
left
but
Jason.
Thank
you
so
much
for
a
great
presentation
once
again
very
much
looking
forward
to
work
with
you
guys
on
robot
Perth
and
to
the
rest
of
you.
There
will
be
a
meeting
in
about
a
month
approximately
the
last
one.