►
From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #3
Description
Third meeting of the ROS 2 Hardware Acceleration Working Group (HAWG) where we reviewed in more detail the hardware acceleration conventions (REP 2008 PR) and the progress in porting the current infrastructure to the Ultra96v2 board. We also enjoyed a guest talk from researcher Sabrina M. Neuman, who told us about her research around a design methodology for domain-specific accelerators parameterized by robot morphology (robomorphic computing).
For more including source code, check https://github.com/ros-acceleration.
0:00 Welcome and agenda
00:43 Newcomer's intro
07:20 Hardware acceleration in robotics news
08:05 REP-2008 review
22:36 Recap of ROS 2 Hardware Acceleration WG progress
23:36 Simplifying ROS 2 embedded flows with colcon extensions
24:26 Testing and benchmarking methodology for acceleration kernels
25:05 Ultra96-v2: Progress adding a new community supported board
28:19 Robomorphic computing
39:00 Q&A
For more on REP 2008 PR, check out https://github.com/ros-infrastructure/rep/pull/324. For a discussion about this meeting, refer to https://discourse.ros.org/t/hardware-acceleration-wg-meeting-3/22535.
A
B
B
Applications
on
fpgas
and
I
recently
also
held
an
introductional
talk
into
ross
and
ross
ii
as
a
webinar
for
plc2
some
some
hours
ago
and
yeah,
I'm
I'm
also
in
with
fpgas
and
xilinx,
and
I'm
there
to
to
see
what
is
what
is
going
on
and
to
test
the
the
approaches
from
xilinx,
because
I
have
a
longer
background
in
in
acceleration
of
fpga,
stuff,
yeah
and
I'm
just
interested
and
want
to
follow.
All
of
that
thanks.
C
I
guess
I'll
go
next.
My
name
is
louis.
Mario,
I'm
a
principal
dev
lead
at
microsoft,
been
a
member
of
the
tsc.
Since
the
very
beginning.
We
contributed
to
a
number
of
different
things,
including
ross
on
windows
as
well
as
ross
with
azure,
and
we
are.
We
have
a
set
of
ros
nodes
around
using
the
onyx
model
or
onyx
runtime
for
hardware
accelerated
inferencing,
I'm
actually
joined
by
jerome
shang
jerome
from
microsoft
is
not
shang
jerome
from
microsoft
as
well.
E
Yes,
hi,
I'm
kunal,
I
am
one
of
the
current
maintainers
of
pcl.
My
current
work
essentially
involves
accelerating
specific
parts
of
our
robotic
software
stack
using
different
kinds
of
nodes,
so
we
have
some
nodes
which
are
using
onyx
runtime.
We
have
other
nodes
which
are
essentially
cooler
cores,
and
then
we
have
our
own
main
cpu,
which
is
x86
based.
E
F
I
don't
think
I
introduced
myself
last
time,
I'm
greg
and
I
open
robotics,
I'm
mostly
just
acting
as
a
listening
here
for
this
group
to
see
how
things
are
going.
I'm
also
pretty
interested
in
hardware
acceleration,
as
I
am
working
on
the
open
vision,
computer
project,
which
is
based
off
the
sync
chip
set.
F
I
guess
I
can
also
do
a
quick
intro,
I'm
brian.
I
work
with
sabrina
she's
we're
gonna
be
talking
a
little
more
later
or
she'll
be
doing
more
of
the
talking
so
I'll.
Let
her
introduce
herself
but
we're
we're
at
harvard,
and
we
work
on
acceleration
for
motion
planning
at
the
moment,
but
just
robotics
that
in
general
hardware,
acceleration
gpu
fpga
whatnot.
How
can
we
you
know,
move
past
the
cpu
with
the
robotic
stack
to
get
real-time
performance?
G
G
H
A
Thank
you
so
much
sabrina
and
welcome
as
well
awesome
all
right
guys.
So,
let's,
let's
kick
it
off
for
about
20
more
minutes
until
we
give
the
mic
to
sabrina
who's
going
to
tell
us
very
exciting
stuff.
I
just
pasted
in
the
chat
for
everyone.
The
meeting
notes
off
today.
As
always,
this
meeting
note
will
be
updated
after
the
meeting,
including
the
recording,
the
slides
and
etc
so
feel
free
to
make
comments.
A
Unfortunately,
the
permissions
that
we
have
given
so
far
are
just
a
commenting
permissions,
because
there's
been
a
couple
of
persons
already
who
have
removed
the
content
holy.
I
don't
know
why,
but
we
decided
that
it
was
best
to
essentially
allow
people
to
give
suggestions
and
and
make
edits
in
that
manner,
so
that
we
avoid
people
removing
our
work
for
the
past
few
weeks.
A
So
that's
that's
there.
So,
after
the
welcome
and
intros,
I
wanted
to
throw
a
piece
of
news.
Who
is
the
first
one?
That's
highlighted
here
in
yellow
if
you're
looking
at
the
minutes,
so
sorin
who
introduced
himself
gave
a
fantastic
talk
about
how
to
use
hardware,
acceleration
capabilities
and
technologies
for
ross.
He
gave
a
very
nice
overview
of
ross
overall
and
also
essentially
about
the
projects
they've
been
doing,
and
particularly,
he
shared
very,
very
interesting
results
when
using
it
with
the
perception
stack.
A
A
I'm
going
to
share
my
screen
real
quick
in
here
there
we
go
and
to
have
a
quick
peek
at
rep
2008,
which
is
something
we're
working
and
we've
been
working
for
a
few
weeks
now
and
where
we
encourage
and
invite
people
to
make
contributions.
There's
a
bunch
of
you
who
already
have
commented
and
gave
suggestions.
Thank
you
for
that
and
please
keep
it
up
the
the
right
for
those
of
you
not
so
familiar
red
stands
for
ros
enhancement
proposals.
A
It
essentially
can
be
defined
as
the
conventions
and
definitions
that
essentially
roster
packages
and
the
community
should
follow
when
it
comes
to
building
roster
computational
graphs,
there's
a
bunch
of
them.
You
can
check
them
out
if
you
wish
this
one
particularly
aims
to
somehow
shine,
sunlight
and
somehow
lead
the
way
into
how
we
can,
as
a
community
as
a
group
jointly
build
hardware
acceleration
capabilities
that
are
interoperable,
that
are
technology,
agnostic,
meaning
that
you
can
bring
them
from
fpgas
to
gpus
to
even
other
technologies.
A
So
that's
pretty
much
the
mission
we
are
trying
to
fit
in
here
it's
built
again
in
a
technology
agnostic
manner
and
also
in
a
platform
agnostic
manner,
meaning
that
we
are
not
just
stacking
specifically
deep,
embedded
ports
or
bh.
That
is
nowadays
just
so
called
we're,
also
looking
at
workstations
and
even
data
centers,
which
will
come
down
the
road,
but
still
we
did
design
the
architecture
with
this
in
mind
and
that's
what's
proposed
in
here.
A
So
the
official,
the
official
way
to
get
engaged
with
this
read
more
about
it
and
comment.
Give
your
views
this
in
this
full
request,
which
is
listed
in
the
minutes.
You
can
check
the
files
directly,
I'm
just
giving
you
a
sneak
peek
of
how
it
looks
if,
if
rendered
according
to
the
the
reps
repository
and
in
nutshell
again
in
the
abstract,
we
do
describe
what
I
just
went
through.
A
It's
very
important
to
highlight
real
quick,
the
value
for
stakeholders,
and
this
is
one
of
the
key
aspects
of
how
we
are
trying
to
make
it
technology
agnostics.
We
are
trying
to
leverage
value
and
create
value,
not
just
for
package
maintainers,
which
are
very
important
stakeholders
within
the
ross
2
community,
but
also
for
consumers.
A
So
here
are
also
some
of
the
anti-goals
about
what
we
think
we
definitely
don't
want
to
treat,
and
this
is
open
for
discussion,
and
this
is
again
a
short
definition
of
description
of
the
architectural
pilars,
wherein
we
have
built
this
conventions
and
proposal.
This
was
somehow
also
touched
into
previous
meetings.
So
if
you
wish
to
learn
more,
also
encouraging
you
to
check
that
out,
I
will
essentially
victor.
A
A
A
All
the
way
up
to
the
application
layer
is
ltpng.
Linux
tracing
toolkit
next
generation,
which
again
is
vendor
agnostic.
It
works
fantastically
fine
and
there
is
some
very
interesting
ongoing
work
to
instrument
things
farther
beyond
the
cpu
computation
going
down
to
some
of
the
aspects
within
the
gpu
and
some
of
the
aspects
within
the
fpga.
A
The
reason
why
we
are
pushing
this
so
hard
is
is,
and
hopefully
these
two
topics
will
shed
some
light
into
this,
because
we
are
realizing
there's
a
number
of
things
that
will
create
somehow
misunderstandings,
so
so
we'd
like
to
tackle
them
the
earlier,
the
better,
the
first
one
which
is
connected
to
the
daily
work
that
I
do
is
extending
existing
roster
packages
versus
integrating
that
essentially
hardware
acceleration
into
existing
roster
packages.
And
here's
here
is
an
example.
A
The
defaults
from
nvidia
who
are
with
us
here
today,
forked
image
pipeline
and
did
a
great
work,
I
think,
by
essentially
providing
capabilities
to
to
be
able
to
run
it
on
on
gpus.
There's
some
ongoing
work
on
our
direction
from
silence's
perspective
on
that
as
well.
But
there's
there's
a
big
question
mark
in
here,
which
is:
do
we
fork
that
repository
and
somehow
fragment
the
maintenance
of
this
code,
or
do
we
try
to
integrate
hardware
acceleration
directly
into
the
repository,
and
we
don't
claim?
A
J
Yeah,
I
I
yeah
thanks
for
for
raising
this
victor.
I
think
nvidia's
focus
here
is
really
around
trying
to
provide
an
entire
package
that
developers
can
use
and
they
don't
have
to
deal
with
the
problems
of
of
getting
it
working
and-
and
I
think
our
our
priority
is
making
sure
that
we
can
get
hardware
accelerator
acceleration
into
the
hands
of
developers.
J
So
maybe
the
one
concern
I
have
with
the
the
work
we
do
in
the
working
group
today
is
that
it's
entirely
focused
on
compilation
within
a
single
mode,
and
that,
unfortunately,
is
not
our
bottleneck
in
ross2.
Today,
our
bottleneck
and
ross2
is
building
a
graph
or
pipeline
of
nodes
and
providing
hardware
acceleration
between
nodes,
and
so
that's
part
of
the
work
that
we're
doing
with
the
open
robotics.
J
A
And
that's
and
that's
a
great
input
and
and
for
context
in
here,
essentially
what
what
greg
is
referring
to
us
as
a
limitation
for
what
concerns
computational
graphs
is
that
both
gpus
and
fpgas
essentially
have
a
limitation
with
the
existing
boss,
2
architecture,
because
it's
cpu-centric
and
every
single
time
that
information
flows
from
one
node
to
the
next
one.
It
needs
to
go
through
memory,
and
that
involves
the
cpu.
So
so
what
gordon
is
pointing
out
is
that
they
are
working
on
trying
to
avoid
that
by
essentially
making
some
adjustments.
A
A
For
that
and
one
thing
we
can
try
to
reproduce
and
that's
a
suggestion
for
engaging
with
that
work
is
other
working
groups
are
kind
of
like
focusing
contributions
from
different
vendors,
even
as
it
is
this
case
around
the
same
computational
graph.
They
are
defining
kind
of
like
a
reference
computational
graph
that
different
technologies
try
to
improve,
and
in
this
case
we
could
try
to
do
that.
A
Maybe
returning
back
to
the
question,
I'm
still
curious
about
what
folks
out
there
think
about
forking
roster
packages,
as
opposed
to
contributing
directly
to
to
some
of
the
existing
roster
packages.
There's
a
big
button
here,
which
is
that
if
we
start
forking
things,
then
maintenance
is
going
to
become
a
nightmare
and
also
the
second
bat
is
licensing.
The
fact
that
forks,
sometimes
change
the
license
or
rewrite
things
changing
the
license
becomes
a
a
somewhat
conflicting
issue.
So
I'm
particularly
interested
greg
you
from
open
robotics.
A
F
I'm
not
involved
in
this
particular
project,
so
I
cannot
particularly
comment
from
an
official
standpoint.
I
think
just
from
a
community
perspective
like
there's
always
I
mean
I
think,
you're
like
you're
saying
fragmenting
is
to
go
to
a
minimum,
but
I
mean
for
a
particular
project
right
like
sometimes
it's
not
time
trying
to
make
a
like
a
perfect
solution.
Is
it
can
be
difficult?
I
imagine
so.
You
know
what
I
think
it
comes
down
to
like.
F
What's
the
time
involved
and
that's
the
priority,
I
think
it
like
you
know
it's
it's
up
to
it's
up
to
whoever
wants
to
put
in
the
effort
right
if,
like,
if
the
efforts
and
interests
of
you
know,
if
somebody
wants
to
fork
it,
they
can
fork.
If
they
don't
want
to
fork
it,
they
don't
work.
I
think
it's
like
up
to
the
individual
and,
if
they're
willing
to
maintain
what
they've
what
they've
made,
but
you
know
from
a
community
perspective.
A
Okay,
okay,
now
that
helps
and
let's
keep
the
conversation
open
and
say.
If
anyone
has
thoughts
on
this,
please
make
sure
to
to
send
them
to
me
and
and
if
necessary,
I
will
reopen
this
this
topic.
This
is
something
we
as
a
community
as
a
group
need
to
somehow
get
together
on,
because
otherwise
we'll
suffer
down
the
road,
and
this
is
also
connected
by
the
way
to
the
second
item.
A
We
want
to
release
them
depending
on
certain
environments
and
depending
on
certain
situations
within
the
robot,
we
may
want
to
release
them
from
certain
tasks
so
that
we
want
to
enable
or
disable
hardware
acceleration
in
specific
nodes
again,
even
at
lunch
time
or
at
run
time.
So
there's
a
couple
of
thoughts
in
here
which
I
dropped
into
the
meeting
notes.
It
would
be
awesome
if
anyone
has
any
any
ideas
on
this,
because
again,
we
are
still
at
the
design
process.
Right
now,
and
hopefully
some
of
these
can
land
into
the
right
2088.
A
A
The
third
item
into
the
agenda
is
a
recap
of
the
progress
so
far
and
well
so
far
we're
having
not
so
much
folks
actively
putting
their
time,
it's
essentially
being
pedro
and
myself,
and
if
you
to
follow
up
with
that,
I
encourage
you
to
take
a
look
at
the
so
within
ros
acceleration
community
repository
we
have
and
we
are
maintaining
a
project
board
and-
and
it
was
requested
in
the
past
by
one
of
you.
So
thank
you
for
reminding
me
that,
and
this
project
board
shows
progress.
A
It's
very
important
to
accelerate
individual
nodes,
it's
very
important
to
accelerate
computational
graphs,
that
is,
transition
across
nodes
and
get
a
5x
10x
100x
of
acceleration.
But
if
this
is
only
usable
by
a
very
very
few
experts,
it
becomes
essentially
unmanageable.
So
what
we've
been
trying
to
do
incrementally
is
to
lower
the
entry
point
barrier
by
facilitating
tools
that
are
integrated
as
first
class
participants
in
the
roster
ecosystem,
particularly
as
extensions
of
the
aim
and
field
system
and
the
qualcomm
build
tools.
A
So
you
can
take
a
look
at
this
there's
a
bunch
of
them.
We're
working
on
more
and
feel
free
to
send
in
input,
send
feedback
and
related.
So
that's
one,
the
second
one
and
apologies
for
going
so
fast.
We
are
time
limited
in
here,
but
the
second
one,
which
was,
which
is
still
an
ongoing
work.
I
haven't
closed
this
one.
Hopefully
soon
it
is
pushing
publicly
the
benchmarking
architecture
that
I
described
before,
while
going
through
rep
2008.
A
So
this
is
today
demonstrated
with
a
number
of
the
acceleration
examples
we
have
pushed
publicly.
If
you
want
to
have
a
look,
take
a
look
in
here
and
then
in
the
coming
few
weeks,
possibly
I
will
find
time
to
document
everything
and
provide
tutorials
which
we
will
make
available
for
everyone.
Now,
with
time
left,
I
wanted
to
touch
into
an
ongoing
effort
to
port
the
architecture.
We
have
come
up
as
a
group
to
a
new
board
to
a
new
hardware
platform,
the
ultra
96
b2
pedro.
K
Okay,
trying
to
be
fast
because
we
have
not
so
much
time,
I
identified
the
files
to
need
to
be
modified
to
build
a
ultra
96
board.
There
is
no
so
much
files
for
the
recipients.
The
problem
is
petal
linux.
We
need
to
build
a
complete
processing
system
and
peta.
Linux
needs
a
lot
of
resources
on
machines.
You
need
about
50
gigabytes
of
disk
space.
You
need
about
core
system,
so
it's
not
easy
to
build
a
petal
linux
system.
You
will
also
need
a
very
stable
internet
connection,
because
petalinos
works
like
a
gentle.
K
K
A
A
Awesome
pedo,
thank
you
so
much
all
right,
so
we'll
have
a
q
a
session
at
the
end.
If
any
of
you
want
to
give
more
details
or
ask
more
questions,
but
if
everyone's
fine
with
it,
I
would
like
to
welcome
officially
our
guest
speaker.
I
would
like
to
welcome
sabrina,
m
newman
and
and
sabrina
apologies,
but
I
was
looking
throughout
the
whole
web.
A
Well,
the
whole
web,
a
few
sites
in
the
web
and
even
your
phd
thesis
to
try
to
see
what
was
your
middle
name
to
say
it
properly,
but
I
couldn't
find
it.
I
really
couldn't
find
it
so
sabrina
everyone
meet
sabrina
serena
is
a
postdoctoral,
essentially
researcher
at
computing
innovation.
Fellow
at
harvard
university.
Her
research
interests
are
in
computer
architecture,
designed
informed
by
explicit
environmental
and
domain.
Specific
insights
is
particularly
focused
on
robotics
applications
because
their
heavy
computation
demands
and
potential
to
improve
the
well-being
of
individuals
in
society.
H
H
Today,
I'm
going
to
talk
to
you
about
some
work.
I
did
in
that
space.
We
had
a
paper
at
aspos
2021
that
covers
all
this
material.
So
take
a
look
and
yeah.
This
is
work
I
did
with
my
colleagues
at
harvard
and
mit,
including
brian
plancher,
who
is
also
on
this
phone
call
and
I
hope,
we'll
chime
in
later
during
the
q
a
so
basically,
the
idea
is
that
we
want
agile,
robust
untethered
motion
from
complex
robots
right.
This
would
be
a
desirable
outcome.
H
So,
in
order
to
accomplish
that,
of
course,
we
need
real-time
performance
from
the
computation
and,
as
you
as
you
all
know,
robots
have
to
do
a
lot
of
things
online.
All
many
of
them
are
real-time
rates,
so
in
the
full
computation
pipeline,
a
robot
might
have
to
do
a
bunch
of
perception
where
it
sees
the
world
mapping
and
localization,
where
it
places
itself
within
that
world
and
then
finally,
motion
planning
and
control
where
it
chooses
a
valid
path
through
the
space.
H
Today,
I'm
going
to
focus
on
that
last
stage
of
the
pipeline,
the
motion,
planning
and
control,
and
within
that
stage
yeah
it's.
It's
really
paramount
to
have
real-time
rates
of
performance,
often
at
maybe
kilohertz
rates,
and
there
are
a
lot
of
computations
there
that
scale
directly
with
robot
complexity,
especially
when
you
have
these
very
complex
jointed
robots
like
manipulators,
quadrupeds,
humanoids
or
even
some
sort
of
approximation
of
snake-like
software,
bots
and
so
to
try
to
do
all
this
fast
is,
is
very
difficult.
H
Current
software
support
is
often
too
slow,
leaving
a
performance
gap
so,
for
example,
for
nonlinear
model
predictive
control.
This
gap
can
be
like
an
order
of
magnitude.
You
might
desire
something
like
kilohertz
rates,
but
only
be
able
to
perform
at
hundreds
of
hertz,
and
this
gets
worse
with
more
complex
robots.
H
So
today,
the
the
method
of
doing
this,
I'm
going
to
focus
on,
has
one
key
insight,
which
is
that
for
a
lot
of
the
algorithms
in
motion
planning
and
control,
there
is
a
relationship
between
robot,
morphology
and
computational
features
of
the
algorithms
and
so
well.
This
this
applies
kind
of
broadly
across
a
lot
of
kernels
within
motion
planning.
You
can
imagine
that
the
robot's
motion
is
tied
a
lot
to
the
robot's
physics,
and
this
occurs
in
in
a
lot
of
these
different
sort
of
computations.
H
So
the
proposed
solution
is
what
we
call
robomorphic
computing
and
the
idea
is
that
we
want
to
take
a
robotics
algorithm.
That
has
something
to
do
with
the
robot's
physicality
and
we
want
a
box
that
we
can
input
the
particular
robot
and
use
its
morphology
information
to
create
a
customized
hardware
accelerator
out
the
other
side.
H
So
I'll
take
you
through
the
steps
of
the
flow
that
we
came
up
with
so
step,
one.
We
take
our
robotics
algorithm
and
we
go
through
and
we
identify
the
places
where
there
are
features
of
the
algorithm
that
are
tied
to
the
robot's
physical
structure
and
by
identifying
all
of
these
places,
we
can
create
a
parameterized
hardware
template
to
perform
that
algorithm.
H
H
So
that's
the
overview
to
give
you
some
idea
and
make
this
more
concrete,
I'll
kind
of
break
it
down.
So
the
robot
morphology
you
can
think
of.
As
sort
of
broken
down
into
a
topology
of
robot
limbs
that
are
made
up
of
joints
and
links
for
for
these
sort
of
rigid
articulated
robots
and
the
links
themselves
have
certain
inertial
properties
and
the
joints
have
different
joint
constraints
on
how
they
can
how
they
can
rotate
right
and
so
to
translate
these
into
accelerator
hardware.
H
So,
for
example,
if
you're
looking
at
computations
that
wind
up
iterating
through
the
links
or
limbs
of
the
robot,
these
are
opportunities
in
hardware
for
for
parallelism
across,
say,
processing,
elements
or
data
paths,
and
then,
similarly,
the
the
links
and
the
joint
types
are
tied
directly
to
these.
These
sort
of
small,
sparse
matrices
that
wind
up
popping
up
all
over
the
place
in
dynamics
and
other
computations
so
directly
from
the
inertial
properties
of
the
links.
H
You
can
know
constant
values
that
are
going
to
appear
in
these
matrices
and
from
the
joint
types
and
constraints
you
can
know.
Sparsity
patterns
they're
going
to
appear
in
like
joint
transformation,
matrices,
and
then
you
can
use
these
sparsity
patterns.
These
constants
to
streamline
hardware,
functional
units
that
implement
the
linear,
algebra
operations
right,
so
we've
gone
and
we've
mapped
morphology
features
to
useful
hardware.
Accelerator
features
so
I'll.
Give
you
an
example
of
of
doing
this,
one
time
for
one
robot
manually
and
then
later
I'll
talk
about
how
I
would
rather
automate
this
process.
H
And
so
we
just
went
through
that
process.
We
took
the
limbs
and
links
and
we
turned
those
into
parallelism
and
then
processing
elements,
and
we
took
the
inertial
properties
of
the
link.
So
we
made
constant
values
in
all
the
little
functional
units,
and
then
we
took
lincoln
joint
information
and
used
that
to
determine
the
sparsity
patterns
in
the
matrices
which
we
turned
into
customized
functional
units.
By
sort
of
pruning
a
little
multiplier,
adder
trees,
so
again
morphology
turning
into
accelerator
morphology
and
it
it
paid
off
pretty
nicely.
H
So
here
are
some
highlights
from
our
evaluation.
We
compared
against
gpu
and
cpu
state-of-the-art
baseline
implementations
of
the
ford
dynamics
gradient
and
we
had
our
accelerator
implemented
on
an
fpga
board,
and
you
could
see
that
so
the
top
is
the
single
compute
latency.
So
this
is
just
raw
compute
speed
up
from
the
fpga
and
we
got
a
substantial
increase
over
both
cpu
and
gpu,
and
then
we
also
implemented
an
entire
co-processor
system
where
we
had
the
fpga
board
talking
round
trip
with
the
cpu.
H
That
was,
you
know,
calling
that
function
and
receiving
the
results,
and
even
with
all
the
the
penalty
you
paid
for
sort
of,
I
o
overhead.
We
still
maintained
a
speed
up
over
cpu
and
gpu,
so
future
work
all
right.
So
having
done
that
again
once
manually
for
the
one
robot,
the
directions
to
push
are
to
expand
the
same
approach
to
more
robots.
H
We
think
it's
a
generalizable
methodology,
so
we
are
hoping
to
realize
that
across
all
sorts
of
different
robot
platforms,
more
algorithms,
the
same
compute
structures
that
we're
able
to
take
advantage
of
here
show
up
in
a
ton
of
different
kernels
throughout
motion
planning
and
then
finally,
automation,
we're
hoping
to
have
a
push
button
solution
where,
on
the
one
hand,
you
put
in
a
robot
description
in
like
a
standard,
urdf
format
and
then
with
no
no
other
intervention
from
hardware
design.
Experts
it'd
be
great
to
pop
out
the
other
end.
H
H
So
yeah.
In
summary,
we
set
up
this
this
sort
of
flow
to
go
from
robots
to
accelerators,
and
I
think
that
this
sort
of
push
button
solution
is
an
attractive
way
to
sort
of
tame
the
complexity
of
their
box
design.
Space
to
let
us
build
up
automatically
optimize
processors
for
robotics,
I'm
very
curious
to
hear
what
everybody
thinks
and
take
any
questions
so
half
at
it.
A
So
serena,
thank
you
so
much
for
your
presentation
and
and
for
everyone's
knowledge,
I
think
sabrina
and
her
group,
including
brian
and
a
few
others,
are
doing
outstanding
work
reading.
I
think
essentially
much
of
the
newest
work
that
comes
out
of
the
good
uses
of
hardware
exploration
in
robotics,
so
definitely
worth
checking
and
keeping
an
eye
on
their
publications
and
work.
I
H
H
So
we
were
thinking
parallel
from
the
beginning
right
every
opportunity
we
saw
to
parallelize,
especially
along
the
robot's
links
for
the
dynamics,
gradient
calculation.
There
are
a
lot
of
opportunities
where
there
are
partial
derivatives
taken
at
every
single
robot
link
along
the
way,
but
it
was
a
pretty
conventional.
It
wasn't
like
a
data
flow
architecture.
You
know
exactly
it
was.
H
D
They
ended
up
breaking
down
the
systems
pair
vital
blocks
where
each
and
every
of
those
blocks
have
its
particular
function
like
for
collision
avoidance
or
braking
systems
and
so
forth.
The
way
in
the
automotive
industry
has
solved.
It
is
by
breaking
down
a
big
problem
into
small
problem
localized
in
those
vital
elements.
When
you
break
down
a
robot's
pair
limbs
and
links,
you
came
with
a
different
suggestion
to
break
down
this
acceleration,
or
you
still
have
a
uniquely
centralized
approach
to
the
whole
system.
H
H
D
F
F
I
need
each
individual
computation
to
come
like
to
be
done
completely
before
I
get
a
useful
outcome,
but
we
can
separate,
say
this
dynamics
computation
or,
if
we're
doing
say
trajectory
optimization
for
each
time
step.
I
could
compute
those
independently
on
separate
processors.
I
could
compute
some
of
the
other
things
I
need
to
do
like
perception
and
mapping.
D
H
D
H
Yeah,
I
would
say
that
the
compute
that
we're
looking
at
here
is
a
lot
of
operations
on
these
sort
of
small
and
middlingly,
sparse,
matrices,
so
small,
as
in
their
they're
like
six
by
six
element,
all
of
them,
because
that's
just
fundamental
to
how
you
do
the
the
coordinate
transforms
and
middling
least
sparse,
like
say
40
to
60
sparse.
This
is
very
different
from
the
sort
of
matrices
that
you
deal
with
in
ml
applications,
which
tend
to
be
very,
very
large
and
very,
very
sparse,
so
yeah.
H
I
think
that
you
know
the
sort
of
tensor
processor
units
and
all
the
clever,
sparse,
matrix
row
compression
and
whatever
else
tricks
that
are
being
applied
in
machine
learning,
are
kind
of
a
poor,
fit
they're,
they're,
sort
of
high
overhead
and
overkill
for
these
very
small
matrices.
So
I
think
these
are
complementary
solutions.
Maybe
the
ultimate
processor
has
a
sort
of
tensorflow
gpu-like
computation
for
the
perception
stage
of
the
pipeline,
but
then
for
motion.
Planning
and
control
has
a
hardware
accelerator
that
does
the
sort
of
smaller
stuff.
D
F
So
I
I
think
there
there
is
some
challenge
there,
you,
you
can
still
speed
those
up
with
some
custom
code,
but
a
lot
of
the
like
pre-existing
libraries
that
that
you
can
leverage
say
like
a
coupe
loss
or
something
you
know
it's
just
optimized
for
a
much
larger
matrix
multiplication
and
lots
of
little
small
ones.
There
are
batch
versions
in
some
cases,
but
there's
a
lot
of
it's
an
it's,
a
messy
algorithm,
a
lot
of
these
dynamics,
kernels.
A
A
I
love
this
figure
and
if,
if
you
go
through
the
article
there's,
I
think
there's
a
number
of
paragraphs
where
you
discuss
essentially
that
that
these
results
and
and
why
gpus
suffer,
and
particularly
if
you
cite
something
like
that,
they
suffer
because
there's
high
synchronization
and
focus
on
throughput.
Could
you
guys
maybe
comment
a
bit
more
on
on
this
and
and
how
you
guys
think
this
could
be
further
improved
and
or
what
are
the
limitations,
etc?.
F
Yeah
I've
been
more
on
the
gpu
side.
Sabrina
has
been
more
on
the
on
the
fpga
side,
so
I've
actually
done
a
bunch
of
work.
Recently,
there's
there's
a
there's.
Another
paper,
that's
like
just
got,
submitted,
there's
a
pre-print
to
try
to
improve
the
gpu
side
a
bunch,
and
I
did
get
a
lot
of
improvements.
It
requires
a
lot
of
reordering
of
the
algorithm
and
a
bunch
of
just
sort
of
trying
to
be
really
smart
about.
When
do
you
generate
a
bunch
of
temporaries?
F
When
do
you
you
know,
and
then
we
can
do
all
these
matrix
vectors
in
parallel,
but
but
the
underlying
algorithm
that
we're
trying
to
use
here
are
we're
super
optimized
for
a
cpu
or
or
at
least
they're,
very
serial
and
there's
a
sort
of
overall
operation,
which
just
makes
it
quite
hard
to
try
to
extract
a
lot
of
parallelism
in
general
on
the
gpu.
You
can
do
better
and
I
and
I
can-
and
I
kick
the
pants
out
of
this
gpu
version,
because
that
was
my
first
implementation
and
it
was
not
good.
F
I
thought
it
was
good
at
the
time
turns
out.
It's
not
good,
but
I
still
can't
come
anywhere
close
to
the
fpga
time
on
one
computation.
Where
I
can
win
is
I
can
get
good
scalability
until
I
run
into
shared
memory
issues
where
I
start
to
blow
the
cache
basically
so
that
that
sort
of,
like
one
note
and
then
I
think
the
other
one
is
that
as
all
these
hardwares
evolve,
I
think
there'll
be
different,
interesting,
other
opportunities
to
take
advantage
of
different
other
things
on
the
system.
F
F
F
F
Getting
presented
here,
I
mean
that's
where
this
all
started.
It
all
started
because
because
well
for
sabrina-
and
I
got
into
this
in
slightly
different
directions,
but
like
the
reason
why
I
got
into
this-
was
sort
of
like
I
tried
to
write
a
trajectory,
optimization
algorithm
on
the
gpu
for
robots,
and
I
realized
that
there
was
no
good
dynamics
package
that
I
would
need
to
use,
and
so
I
wrote
from
scratch
this
particular
not
so
great
gpu
implementation
of
for
a
serial
chain
manipulator
and
was
like
this
doesn't
scale
to
me.
F
L
Okay,
more
questions.
Anyone
in
the
audience,
I
have
a
question.
My
name
is
ananya.
I
thought
this
was
an
excellent
talk,
very
promising
work.
I
was
wondering
if
you
have
some
insights
on
what
could
be
the
ideal
ideal
morphology
in
terms
of
the
number
of
links
and
the
freedoms
that
they
have.
I'm
trying
to
see
the
problem
from
the
hardware
side
now,
given
a
particular
hardware
constraint,
is
there
anything
that
can
be
said
about
the
ideal
morphology
that
suits
that
hardware
constraint.
H
That's
a
fantastic
question.
I
think
that
I
I
can't
quantitatively
or
definitively
answer
it
yet,
but
we
are
building
hinted
at
automated
flows
to
to
recreate
these
accelerators
based
on
robot
morphology.
We
can
absolutely
put
that
in
a
loop
right
and
and
and
ask
okay
as
we
change
the
rdf
number
of
links,
number
of
joints,
etc,
etc.
How
does
that
shape
the
compute
that
comes
out
the
other
side,
so
I
think
yeah
you're
you're,
just
one
step
ahead
of
the
curve
and
the
moment
I
have
a
tool.
H
C
H
F
I
I
think,
there's
a
there's
another
layer
on
that
which
makes
this
really
hard
to
answer,
which
is
that
you
then
have
to
optimize
for
tasks
within
or
within,
like
a
robotic
sort
of
benchmarking
framework.
So
it's
both
the
hardware
bench
like
computer
hardware,
benchmarking.
But
it's
also
like
the
task-based
benchmarking,
because
we
can
probably
design
the
robot
that's
easiest
to
compute,
but
maybe
it
like
can't
do
any
of
the
tasks
that
you
want,
and
so
I
think,
there's
a
real
challenge
there
to
figure
out
what
the
right
answer
is.
F
A
F
A
Real
real
quick
comment
in
here
and
this
this
goes
back
to
the
initial
discussion
I
pushed
while
presenting
rep
2008
about
benchmarking.
If
you
remember
quickly,
I
discussed
kind
of
like
the
model
we're
trying
to
follow,
which
is
this
non-functional
gray
box,
not
non-functional
in
the
sense
of
measuring
aspect
like
like
the
system
under
test,
meaning
latency,
throughput
and
related.
So
this
is
one
of
the
bits
ryan
was
mentioned
in
the
other.
A
One
is:
is
measuring
the
actual
system
under
test,
the
capabilities
of
the
nash
tag
or
the
manipulation
algorithm
that
you're
testing.
At
that
particular
moment,
those
two
things
definitely
will
go
hand
by
hand.
The
initial
proposal
is
just
to
focus
on
non-functional
aspects,
because
it
will
allow
us
to
compare
hardware.
A
However,
there's
definitely
ground
for
cooperation
in
here.
So
if
there's
folks
out
there
interested
in
in
in
diving
deeper
into
this,
I'm
trying
to
set
up
a
cooperation
with
with
the
folks
from
harvard
to
to
jointly
push
this
forward.
So
so
retails
reach
out
to
us,
sorry
and
and
there's
definitely
ground
for
for
people
to
dive
in
and
incorporate.
H
Yeah,
I
totally
agree
with
everything.
Victor
just
said.
I
am
I'm
tremendously
interested
in
the
way
that
sort
of
a
hardware
or
computer
architect
point
of
view
here,
with
regular
benchmarking,
etc,
etc.
Can
tame
the
crazy
multivariate
space
of
robotics
where
you've
got
tasks,
you've
got
environments
that
the
robots
perform
in
and
you
have
the
robots
themselves,
which
all
look
very,
very
different
and
the
compute
substrates
that
you
can
afford
to
put
on
them.
So
it's
going
to
take
some
clever
benchmarking
and
then
some
clever
design
space
exploration
tools.
G
I
have
just
a
question
about
you
showed
the
system,
which
is
a
nitrogenous
system
where
I
saw
an
oauth
plus
an
accelerator
which
is
which
was
fpga.
I'm
curious
to
know
what
is
the
this:
the
the
current
distribution
of
of
time
between
accelerator
and
and
host,
and
just
to
understand
what
what
you
are
now
doing
on
the
accelerator?
If
you
are
capable
so
to
do
to
do
some
sensing
or
perception,
or
it
is
just
the
control
part
of
the
algorithm
that
isn't
on
fpga.
F
So
this
originally
came
from
work
looking
at
non-linear
trajectory,
optimization
and
using
that
in
a
model,
predictive
control
type
framework,
which
means
we're
doing
both
planning
and
control
at
the
same
time
online
and
for
those
algorithms,
the
the
gradients
themselves
are
about
30
to
90
of
the
total
computation,
depending
on
the
exact
software
implementation
you're
using
the
exact
robots,
you're
working
with
et
cetera,
and
so
this
is
accelerating
that
30
to
90
chunk
of
the
planning
and
control
stack.
It's
not
integrated
with
perception
and
mapping.
F
F
Thank
you,
and
I
think
one
last
note
is
that
when
we
get
there
there
will
be
a
really
interesting
systems
challenge
of
maybe
we
can
super
accelerate
this
on
a
gpu,
for
example,
but
you
need
the
entire
gpu
for
your
perception
computer
vision.
So
then,
maybe
it's
actually
better
to
use
a
cpu.
If
your
gpu
is
busy,
you
know-
and
things
like
that
I
think
will
come
up
too.
F
J
I
think
brian
that
depends
on
your
expectation
of
gpu
performance
over
time.
You
know
the
the
performance
is
improving,
pretty
radically
every
18
months.
So
if
your
your
assumptions
are
based
on
what
you
see
today,
you're
probably
mispredicting
the
future.
I
hope
that
the
gpu
performance
keeps
going
up.
F
Since
that's
my
main
line
of
research
and
I'd
love
for
it
to
just
keep
getting
better,
I'm
just
saying
that
I
think
they'll
be
interesting.
You
know
systems
implications
eventually,
when
we
can
put
every
kernel
on
a
gpu
or
every
kernel
on
fpga
and
then
trying
to
decide.
What's
the
right
balance
for
the
whole
process,.
J
But
maybe
maybe
a
point
to
add,
because
I
I
think,
if
we're
talking
about
an
autonomous
robot
safety
is
a
relevant
factor
and
so,
typically
speaking,
it's
a
good
thing
to
distribute
algorithms
on
different
accelerators,
whether
those
accelerators
happen
to
be
from
different
ship
vendors
or
even
within
the
same
chip.
The
platforms
that
nvidia
builds
for
robotics
have
not
just
a
gpu.
J
We
have
separate
deep
learning,
accelerators
and
even
small
computational
engines,
which
I
don't
think
you
tried
to
do
this
six
by
six
math,
but
that
we
do
have
to
do
that
kind
of
work
as
well.
So
it
really
depends
on
what
tool
do
you
need
to
do
the
job
I
mean
asking
a
cpu
to
do
a
machine
learning
inference
is
not
probably
the
right
thing
and
similarly
asking
a
gpu
to
be
a
real-time
hardware.
Accelerator
where
it's
designed
to
get
peak
computational
performance
is
a.
A
Thank
you.
Thank
you,
gordon
for
those
bits
and,
and
I'm
gonna
try
to
cut
it
here
guys
because
we
are
overtime
already,
we
can
definitely
follow
up
and,
and
brian
sabrina
will
follow
up
on
those
bits.
Gordon
just
shared
last
few
words
from
my
side
are
about
thanking,
sabrina
and
brian
for
showing
up
for
giving
this
fantastic
presentation.
We
very
much
look
forward
to
keep
seeing
a
fantastic
world
from
your
group
guys
and
for
the
rest
of
you.
A
There
will
be
another
working
group
meeting,
hopefully
very
soon,
and
as
always,
all
this
material
will
be
posted
and
disseminated
in
ros
discourse.
So
without
further
ado,
thank
you
once
again
for
the
presenters.
Thank
you,
everyone
for
attending
and
have
a
fantastic
night.
I'm
gonna
run
to
cook
dinner
now
and
have
it
so
guys.
Thank
you
and
have
a
great
one.
Bye-Bye
thanks.