►
From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #10
Description
Tenth meeting of the ROS 2 Hardware Acceleration Working Group (HAWG).
We discussed a) new reference hardware platform and b) ROS 2 hardware acceleration stack. We also covered a collaboration between researchers from Acceleration Robotics and Harvard University on systems architecture and ROS with “RobotCore: An Open Architecture for Hardware Acceleration in ROS 2 1” (paper). Shortly, RobotCore is a hardware acceleration framework for ROS and ROS 2 that helps build custom compute architectures for robots, or robot cores, which make robots faster, more deterministic and power-efficient. Simply put, it provides a development, build and deployment experience for creating robot hardware and hardware accelerators similar to the standard, non-accelerated ROS development flow.
For more including source code, check https://github.com/ros-acceleration. For more regarding ROBOTCORE, refer to https://accelerationrobotics.com/robotcore.php.
00:00 Welcome
00:29 Intros
05:17 Hardware Acceleration in Robotics Newsletter
06:30 Robotic Processing Unit (RPU)
11:21 New reference hardware platforms
11:40 Kria KR260 Robotics Starter Kit
14:02 Other hardware platform support added
17:52 ROS 2 Hardware Acceleration Stack
23:57 ROBOTCORE, An Open Architecture for Hardware Acceleration in ROS 2
29:51 ROBOTCORE vs ROS default performance
31:39 ROBOTCORE supports ROS and ROS 2
32:43 ROBOTCORE for multi-platform production-grade ROS Yocto support (REP 2000)
34:34 ROBOTCORE for ROS 2 type adaptation (REP 2007)
36:48 ROBOTCORE hardware acceleration architecture and conventions in ROS 2 (REP 2008)
37:51 ROBOTCORE for ROS 2 type negotiation (REP 2009)
40:45 ROBOTCORE Perception
44:46 ROBOTCORE vs NVIDIA Isaac ROS, 3.5x speedup in ROS 2 graph
45:53 ROBOTCORE vs NVIDIA Isaac ROS, 6x performance-per-watt improvement
46:39 ROBOTCORE vs NVIDIA Isaac ROS, 30x speedup in perception Harris filter
47:57 ROBOTCORE vs NVIDIA Isaac ROS, 500x latency improvement in robot perception Node
50:55 Q&A
For a discussion about this meeting, refer to https://discourse.ros.org/t/hardware-acceleration-wg-meeting-10/25852.
A
Welcome
everyone
to
the
10th
ross
2
hardware,
acceleration
working
group
meeting
for
those
of
you
new
to
the
working
group.
This
working
group
particularly
drives
the
creation,
maintenance
and
testing
of
acceleration
kernels
on
top
of
open
standards,
particularly
focusing
on
c
plus
class
and
opencl
for
optimized
ross2
and
gasebo
interactions
over
different
compute
substrates.
A
As
usual,
overpass
meetings
will
open
up
the
mic
for
those
of
you
new
to
the
working
group,
to
say
a
few
words
about
yourself
and
introduce
yourself
if
you
wish.
Typically,
what
we
do
is
we
of
course
say
who
we
are
and
then
how
we
are
related
to
hardware
acceleration
in
which
organization
we
work,
and
that
should
be
it
all
right.
So
anyone
that
would
like
to
share
a
few
words.
B
Hi,
can
you
hear
me
right?
You
can
yeah.
Thank
you
hi.
This
is
jam.
I
am
joining
from
japan,
it's
midnight
here,
so
so
I'm
I
have
been
very
interested
in
enhancing
hardware
acceleration
capabilities
in
ross
for
a
while,
I
tried
to
get
in
touch
with
the
people
in
the
agi
working
group,
but
somehow
through
a
twitter
interaction.
I
came
to
know
of
the
hog
and
this
group
has
been
much
more
active.
I
have
read
through
the
paper
you're
going
to
present
today
and
I
have.
B
I
think
I
can
contribute
to
the
hardware
acceleration.
You
know
working
group
add
more
boards
to
the
mix
and
to
in
general,
expand
the
support
for
hardware
acceleration
in
roster.
So
I
work
for
a
company
called
yanmar.
It's
a
japanese
company
and
I'm
here
representing
myself,
not
my
employer,
but
I
am
a
robotics
engineer
in
my
teacher
yeah.
A
C
My
name
is
yes,
I'm
based
on
in
london,
I'm
working
as
a
principal
software
engineer
for
medtronic,
but
I'm
as
open
source
software
contributor
and
I'm
contributor
and
maintainer
for
meta
tegra
tiger.
I
believe
you
use
it
or
one
of
you
use
this.
I'm
yeah,
you
can
see
me
there
and
yeah,
I'm
in
more
meta
trigger
community.
You
can
see
the
nvidia's
there
anyway.
This
is
me
nice,
it's
nice,
to
see
you
all
and
yeah,
I'm
happy
to
I'm
happy
here.
If
I
can
help
or
I
can
contribute
with
you
guys,
yeah.
Let
me
know.
A
Very
welcome
and
glad
to
have
you
elias.
I
think
it's
fantastic.
You
have
more
like
yokto
folks
we've.
I
personally
have
been
pushing
very
much
for
adopting
more
and
more
yokota
increasingly
to
get
you
know
to
get
towards
production,
great
operating
systems,
because
most
embedded
guys,
I
know-
and
I
know
a
bunch
of
them
tend
to
kind
of
like
have
a
strong
opinion
about
anything
else
than
yocto.
So
I
think
it's
always
super
nice
to
have
you
know
yopto
layer,
maintainers,
so
super
nice.
Thank
you
all
right.
Someone
else.
D
Hi
this
is
divya,
so
I'm
joining
from
apex
ai
in
california
and
yeah.
I
have
my
colleague
already
here,
matthias
he's
he's
in
like
the
munich
german
office,
so
I
recently
got
into
using
apex
os,
which
is
a
fork
of
safety
certified
fork
of
ross
on,
like
the
gpu,
we
have
using
opencv
running
some
perception
based
algorithms
using
hardware
acceleration.
So
I
am
very
interested
in
this
topic
and
I
want
to
look
for
opportunities
to
contribute
to
the
open
source
as
well.
Yeah.
D
A
Awesome
great
so
so,
as
a
first
piece,
indie
attendance
we've
been
doing
our
past
meetings.
A
piece
of
news
is
the
fact
that
we've
been
putting
together
from
the
group
that
I
represent
acceleration
robotics
hardware
acceleration
in
robotics
newsletter,
which
contains
essentially
news
related
to
how
hardware
acceleration
can
be
applied
to
robotics
and
covers
various
silicon.
Vendors
covers
various
companies
making
use
of
hardware
acceleration.
A
So
that's
one
change
we'll
introduce
from
this
meeting
on
and
with
that.
I
think
we
can
jump
into
all
business
and
administration
first.
Regarding
a
topic
that
has
been
covered
in
the
past,
that
has
had
a
bit
of
progress,
the
robotic
processing
unit
and
just
as
a
reminder,
the
rpu
is
a
robot,
specific
processor
that
maps
ros
and
robot
computational
graphs
efficiently
to
underline
computer
resources,
including,
of
course,
various
compute
substrates,
including
gpus
fpgas
cpus,
aiming
to
obtain
best
performance
from
a
ros
perspective,
as
it
is
detailed
in
the
corresponding
ticket.
A
The
objective
of
this
is
not
precisely
to
build
new
hardware
itself,
we're
very
happy
to
take
off
the
shelf
components.
Maybe
pgas,
cpus
and
gpus
put
them
together
and
prototype
what
conv,
what
combinations
of
architectures
actually
do
lead
to
best
performance,
which
was
that
is
the
objective
of
this
project
initially
and
then,
once
we
have
more
intuition,
we
will
dive
into
more
designing
hardware
wise.
The
target
itself
is
to
try
to
tackle
typical,
robotic
flows,
including
sensing
perception,
but
also
mapping,
localization
and
a
few
others.
A
A
After
reviewing
the
feedback
and
after
having
internal
discussions,
we
have
for
now
drafted
a
couple
of
use
cases,
one
for
perception,
one
for
navigation
within
perception
will
probably
be
using
for
benchmarking
and
for
driving
the
rpus
development,
two
sub
tests
within
perception
and
then
for
navigation.
We're
still
in
discussions.
There's
been
good
feedback
on
that
regard,
so
please
keep
sending
it.
A
If
you
have
any
and
hopefully
within
the
coming
weeks
months,
will
wrap
the
the
use
cases
driving
the
development
and
start
splitting
work
so
that
we
start
generating
some
data
and
optimize
the
architectures
on
that
regard.
As
part
of
these
discussions
that
have
been
held
after
some
of
the
feedback,
I
identified
the
fact
that
some
of
you
apparently
feel
very
annoyed
about
the
naming,
which
is
totally
okay
and
some
have
proposed
even
some
alternatives.
A
There
might
be
others.
There
were
a
few
others
which
I'm
not,
including
because
I
think
they
were
a
bit
off
topic.
But
please,
if
you
have
a
strong
opinion
about
this,
use
the
form
which
I
have
enhanced
to
include
yet
another
question
regarding
this
particular
topic.
So
for
those
of
you
that
already
feel
the
form,
please
check
it
again,
because
there
might
be
one
more
question
for
you
to
fill
in
for
those
that
haven't
well.
A
We
won't
be
creating
hardware
on
this
physical
hardware.
On
this
first
stage,
we
will
of
course
likely
be
creating
ip
ib
cores
so
yeah,
that
is
that
and
of
course,
check
out
the
form
if
you'd
like
to
be
involved
and
have
any
thoughts
on
the
topic,
nothing
else
to
report
regarding
all
business,
I
encourage
everyone
to
check
the
backlog
that
we've
been
keeping
within
the
working
group.
There's
lots
of
topics
that
have
been
closed
over
these
past
few
weeks
and
right
now,
we've
got
in
progress.
A
These
items,
which
are
in
line
with
the
initial
objectives
we
set
at
the
beginning
of
the
year
regarding
the
working
group-
and
you
can
see
them
summarized
in
here,
including
the
ones
from
last
year.
So
there
are
a
few
that
are
in
progress
and
hopefully
will
be
wrapped
if
not
in
q3,
certainly
in
q4
of
this
year,
if
we're
doing
good,
which
sounds
like
we
should
be
doing
good,
then
in
principle
I
don't
see
any
reason
why
not
to
be
a
bit
more
ambitious
but
yeah.
This
is.
A
All
right,
progress,
review
and
new
business.
So,
regarding
new
updates,
I
think
it
was
about
time.
We
spoke
about
new
ports
and
no
hardware
enablement
and
there's
been
a
couple
of
initiatives
that
have
been
going
around
in
parallel.
I
would
say:
porting
the
hardware
acceleration
working
group
work
towards
new
hardware
to
enable
it
the
first
one
is
the
kr260
which
has
been
included
in
the
repo
over
here.
So
the
korea
kr260
robotic
starter
kit,
which
is
essentially
a
board
that
includes.
Actually
I've
got
it
with
me
in
here.
A
A
It
has
an
endpoint,
a
switched
endpoint
which
leverages
tsn
and
can
do
essentially
time-sensitive
networking,
and
it
has
also
tons
of
other
I
o,
which
can
be
leveraged
for
other
matters,
pretty
pretty
nice
port,
which
I
personally
have
felt
in
love
with
after
testing
it
for
a
few
months
now,
and
it
features,
of
course,
the
the
chinese
song
that
amd
has
been
producing.
This
is
just
the
red
thing
you
see
here
with
the
heatsink
and
the
fan.
What's
what's
underneath?
A
Is
this
tiny
song
that
you
can
of
course
use
for
your
own
projects
if
you're
privatizing
your
own,
robotic
joints
or
sensors
or
cognition
systems?
So
a
fantastic
thing
that
will
be
using
I'm
glad
to
report
that
the
complete
work
of
enabling
it
with
the
roster
hardware,
acceleration
working
capabilities
is
ready.
The
firmware
layer
is
available
actually
two
days
ago,
and
you
can
see
here
the
capabilities
it
is
featuring
as
a
novelty.
A
Now,
that's
not
the
only
update
regarding
new
hardware.
I'm
also
super
glad
to
report
the
fact
that
the
ultra
96
v2
port
has
been
finalized
and
just
for
those
who
haven't
been
following
this.
This
work
started
in
june
2021
so
more
than
a
year
ago,
and
it
was
officially
the
first
community
driven
board.
So
it
took
a
bit
of
discussions
and
alignment
and
back
and
forth,
and
we
went
through
various
amd
invites
releases,
but
ultimately
pedro
martos,
so
pimartos
in
github
did
a
fantastic
job.
A
He's
essentially
completed
a
complete
port
of
the
firmware
and
made
it
available
for
everyone
in
here,
so
so
yeah.
This
is.
This
is
really
really
nice.
I've
tested
it
out
and
with
a
few
hacks
that
I
had
to
apply
that
I'm
still
digesting
the
report
to
pedro
things,
work
so
yeah.
Those
of
you
that
have
that
board
know
that
it
can
be.
A
It
can
be
now
fully
enabled,
and
I
guess
the
next
step
on
this
regard
would
be
to
essentially
get
more
attention
towards
this
get
more
interest
and
try
to
elevate
it
towards
maybe
a
higher
position
so
that
we
also
integrate
in
our
upcoming
continuous
integration
and
continuous
deployment
testing
against
some
of
the
examples
we've
been
building
and-
and
those
are
particularly
the
ones
that
are
mentioning-
are
the
acceleration
examples.
These
are
the
ones
we
will
be
running
against
for
checking
that
the
infrastructure
works.
A
So
yeah,
that's,
I
guess
the
next
step-
and
this
is
what's
currently
in
the
pipeline
for
the
official
boards
and
hopefully
we
can
add
more
official
parts
to
this
list
very
soon,
also
and
related
to
meta
tegra.
Actually,
yet
so
nano
has
been
enabled
as
well,
and
there
is
a
link
on
this
on
this
community
supported
boards.
There's
currently
nobody
that
I
know
at
least
right
now
offering
public
support
for
this
firmware
layer.
A
But
it's
been
built
using
metadegra
as
well
as
the
meta
ross
project,
so
the
roster
yocto
layers
it
is
available
in
here
it
can
be
tested
and
if
you
have
any
issues
as
always
just
create
a
ticket
or
ping
me
and
I'll
be
happy
to
try
to
reproduce
your
issues
or
help.
If
I
have
bandwidth
so
yeah
kudos
to
the
folks
of
metategra
for
making
it
so
easy
for
us
to
to
use
it,
and
hopefully
we
can
get
more
and
more
activity
also
on
some
of
those
boards.
A
That's
pretty
much
it.
Regarding
hardware,
I've
heard
some
good
comments
in
the
introduction
regarding
possibly
more
boards
being
ported
and
maintained,
so
that
would
be
super
exciting
for
those
of
you
showing
interest
on
starting
the
process
of
porting
it
to
new
boards.
Just
so
that
you
know
the
instructions
are
documented
in
here
in
the
community
repo.
I'm
sorry,
if
I'm
going
very
fast
for
those
of
you
new,
but
you
have
here.
A
Besides
the
reference
platforms
and
the
community
support,
once
you
have
a
recipe
on
how
to
go
ahead
and
port
it
to
a
new
board.
If
you
have,
if
you
struggle
or
have
issues,
you
can
always
create
a
ticket,
as
it
was
done
in
the
past
and
ask
for
support
and
we'll
do
our
best
to
try
to
help
you.
But
in
a
nutshell,
the
instructions
are
in
here
and
you
just
need
to
follow
in
principle.
Rep
2008,
which
describes
the
conventions
and
the
architecture
on
how
to
enable
new
new
hardware.
A
A
The
the
stack
itself
is
represented
in
here.
The
objective
of
it
is
to
present
the
capabilities
we
are
building
within
the
roster
hardware
exploration
working
group
in
a
somewhat
organized
manner,
while
structured
with
official
standards
and
conventions
in
the
ross
world.
Technically
speaking,
this
stack
is
a
series
of
extensions,
of
course
too,
which
allow
you
to
essentially
get
faster,
whilst
you
execution
and
a
timing,
safe,
event-driven
programming,
interface,
and
if
this
last
bit
sounds
a
bit
confusing
this
timing,
safe,
event-driven,
programming,
interface,
rosq,
is
precisely
offering
this
attack.
A
Sorry,
an
event-driven
programming
interface
ross2,
is
based
on
the
pops
up
paradigm
and,
as
such,
presents
essentially
an
event-driven
architecture,
so
we'll
be
touching
on
on
on
this
aspect
in
future
meetings
after
the
summer.
I
believe,
after
we
break
for
the
summer
quite
a
bit,
because
you'll
see
how
important
it
is
to
optimize
the
data
flows
within
these
event-driven
architectures,
but
yeah
the
use
the
way
to
actually
start
optimizing.
A
Your
own
computational
graph
is
start
paying
attention
to
this
to
this
stack
to
these
four
conventions,
providing
when
appropriate,
the
corresponding
implementations.
That
is
in
part
what
we
do
within
this
working
group.
We
contribute
and
maintain
open
source
implementations
of
values
of
these
components
in
the
stack.
A
Some
of
them
are
informative
and
we
just
make
sure
that
the
information
available
is
proper,
some
others
actually
trigger
implementations,
and
for
some
of
these,
we
do
provide
such
implementation.
The
complete
stack,
however,
is
beyond
the
scope
of
of
the
working
group,
especially
with
concerns,
support
and
professional
trade
capabilities,
but
there
are
some
implementations
out
there
of
the
complete
stack
that
are
starting
to
appear.
A
Robot
core
is
one
of
them
and
we'll
touch
on
that
just
in
a
bit,
but
there
are
others.
So
I
encourage
everyone
to
also
check
that
out
and
and
keep
an
eye
on
it.
Yeah.
I
put
together
more
resources
in
here
in
the
notes,
just
for
the
sake
of
those
of
you
willing
to
read
more
so
there's
a
blog
post
describing
a
bit
in
more
detail
the
stack,
particularly
because
I've
been
reached
out
by
a
few
folks
asking
questions
about
what
is
what
regarding
type
adaptation
and
type
negotiation.
A
I
think
some
of
the
diagrams
that
have
been
shared
in
this
course
have
been
a
bit
confusing.
So
hopefully
the
upcoming
talk
will
clarify
a
bit
of
that,
but
also
you
have
here
a
discussion
in
this
course.
So
for
those
of
you
that
have
been
reaching
out-
and
I
couldn't
answer-
please
don't
be
shy
and
feel
free
to
just
jump
in
here
as
many
others,
and
ask
your
question
right
away.
I
think
this
is
the
the
way
to
to
tackle
this
out
and
yeah.
A
A
There
was
a
conversation
to
have
a
working
group
meeting
during
july,
and
there
was
also
a
volunteer
from
a
company
that
wanted
to
speak
about
ai
hardware,
acceleration
for
ai
based
applications,
I'm
going
to
hold
on
that,
possibly
for
after
the
summer,
for
various
reasons,
especially
because
I
think
most
of
you
deserve
a
good
break,
so
we'll
probably
re-engage
after
this
summer,
possibly
starting
in
september.
We'll
have
the
next
meeting
so
yeah.
A
That
is,
I
think,
what
was
left
regarding
administrative
aspects
and
with
that,
let's
dive
into
the
guest
talk.
I'm
sorry
you'll
be
hearing
me
a
lot
today,
but
I'm
super
super
super
excited
to
be
sharing
this.
It's
the
work
of
many
many
many
months.
Actually,
I
should
say
this
started
from
early
last
year
to
be
frank,
and
it
is
actually
robot
core,
an
open
architecture
for
hardware
acceleration
in
boston.
A
This
work
was
performed
in
collaboration
with
hardware
university,
with
the
systems
architecture
department
led
by
vga,
and
also
in
collaboration
with
sabrina
and
brian.
It's
been
a
fantastic
partnership
which
led
to,
I
think,
pretty
interesting
results
and
fantastic
news.
Is
that-
and
I
just
learned
about
this
today-
so
that's
what
I
thought
it
was
important
to
highlight
it.
A
There
is
definitely
a
good
chance
to
speak
about
hardware
acceleration
this
year
in
tokyo,
sorry
in
kyoto,
japan,
all
right
and
regarding
a
self-introduction,
I
guess
so
for
those
of
you
that
don't
know
me,
I'm
a
robotics
architect,
but
also
a
part-time
researcher,
with
the
systems
security
group
university
of
clagenfort
in
austria.
I
kind
of
like
break
my
time
between
my
interests
on
systems,
architecture
for
robotics,
as
well
as
security,
and
with
that
what
I'm
going
to
do
is
show
a
few
slides
about
robocore.
A
A
The
objective
of
robot
core
is
to
facilitate
the
process
of
building
custom
compute
architectures
for
robots,
which
in
a
nutshell,
we
named
robot
course,
and
that's
essentially
how
the
name
appeared
as
engineers.
We
don't
break
our
minds
about
coming
up
with
namings,
so
technically
speaking,
robot
core
essentially
tries
to
propose
this
open
architecture
for
hardware
acceleration
in
a
roster-centric
manner,
by
extending
the
commonwealth
to
apis
and
build
processes,
as
well
as
tools
available.
A
It
all
dives
down
to
the
fact
that,
typically,
you
have
a
robotics
application,
which
is
often
represented
in
the
form
of
a
computational
graph,
which
runs
in
the
cpu
most
often,
and
this
robotics
application
again
has
certain
performance
with
the
default.
Whilst
you,
you
will
run
again
mostly
in
the
cpu,
and
it's
only
if
you
dive
into
the
hardware
acceleration
world
that
you
start
seeing
speed
ups
but
diving
into
that
is
complex.
Often
the
case.
It
is
a
very
vendor-specific.
A
Let's
say
struggle
and
whenever
you
want
to
compare
things
or
further
optimize,
mixing
different
compute
substrates
for
different
tasks,
you
start
diving
into
a
complete
mess
of
system
integration,
which
already
complicates
the
task
of
coming
up
with
optimization.
So
as
opposed
to
this
guacamole,
let's
say
sauce.
A
That
has
been
the
trend
in
robotics
with
ross,
and
some
of
the
forks
of
course
too,
which
are
cpu,
centric
and
guacamole
means
in
the
sense
that,
essentially,
you
come
up
with
a
computational
graph
that
runs
in
cpu
and
that's
optimized
and
fine-tuned
for
that
particular
cpu
and
those
bunch
of
sensors.
But
the
moment
you
change
a
sensor
or
add
a
more
powerful
actuator
that
demands
more
bandwidth
or
more
interaction
with
the
computational
graph
or
even
add
more.
A
I
don't
know
more
sensors,
because
you
have
multiple
cameras,
additional
cameras,
then,
at
that
point
the
guacamole
starts
crushing
up
and
the
flavor
itself
is
not
so
tasty
anymore.
So
that
is
the
current
setup
that
we
often
experience
in
robotics.
With
with
this
cpu-centric
approach
and
hardware,
acceleration
brings
the
opportunity
to
actually
go
towards
this
timing:
safe,
even
driven
programming,
architectures.
A
Robocord
does
precisely
that
opens
the
door
for
precisely
that,
and
it
does
so
by
essentially
facilitating
testing
different
options
in
a
roster-centric
way,
so
that
you
can
essentially
keep
up
with
your
development
experience,
and
you
only
need
to
add
some
logic
into
your
cmake
list
of
txt
files
corresponding
appropriate
accelerators,
and
the
same
applies,
of
course,
to
the
c
plus
class
code.
You
need
to
annotate
it
with
the
corresponding
pragmas,
if
you're,
using
hls
or
using
the
right
flavor
of
language
if
you're,
using
cuda
or
you're
using
rokem
whatever
so
so.
A
That
is
pretty
much
what's
been,
enabled
and
allows
to
jump
between
different
options
to
compare
them
to
change,
architectures
and
all
together.
It
is
enabled
by
having
always
kind
of
like
benchmarking,
so
quantitative,
centric
approach,
and
to
do
that.
We
enabled
this
as
part
of
the
stack
of
the
roster
hardware,
acceleration
stack
and-
and
we
did
it
by
essentially
leveraging
past
work
in
the
community
and
particularly
raw
shoe
tracing,
which
we
heard
actually
in
the
last
guest
talk
at
least
an
extension
of
it,
as
well
as
the
linux
tracing
toolkit,
ltt
ng.
A
So
that
is
a
representation
itself
off
of
robot
core.
Now,
for
those
of
you
concerned
about
whether
robot
core
road
course
aim
is
to
actually
you
know,
program
your
node
once
and
then
run
it
in
literally
every
single
compute
substrate
in
its
most
optimized
version
and
format.
That
is
not
really.
The
objective
I
mean
robot
core
requires
you
to
provide
the
corresponding
flavor
for
each
one
of
the
accelerators,
and
there
is
a
rationale
behind
this
as
well.
A
So
so
that
is
something
that
we
kept
in
mind
while
architecting
this
now
trying
to
depict
it
the
technical
value
of
robot
core.
You
have
your
computational
graph,
which
has
a
certain,
essentially
timing
performance,
so
to
speak,
runtime,
measurements
technically
speaking,
and
that
is
the
speed
up
the
baseline
collected
at
the
cpu
of
one
x
and
then
using
robot
core.
We
can
optimize
the
computational
flows
and
we
can,
and
that
can
lead
to
a
significant
speed
up.
A
So
going
back
to
the
presentation.
This
super
super
example,
which
is
also
represented
on
the
right
side
of
the
slide,
with
its
computational
graph
form,
allows
us
by
purely
leveraging
robot
cores
capabilities
to
go
all
the
way
up
to
more
than
three
times
the
performance
per
watt.
So
that
means
that
it's
essentially
performing
from
a
power
consumption
perspective
more
than
three
times
better
when
compared
to
the
default
raws
run
in
the
arm
course.
A
In
the
same
r
course,
sorry
in
the
same
arm
course
as
opposed
to
robot
core
running
in
both
the
cpu
and,
in
this
case
the
fpga
as
the
accelerator,
so
robot
core
can
also,
by
the
way,
perform
the
same
operation
and
process
not
targeting
precisely
this
hardware,
but
maybe
targeting
a
gpu
enabled
hardware.
But
it's
also
something
that's
been
worked
on
and
that's
presented
in
the
paper.
A
Now
robot
core
also
pays
attention
to
the
fact
that
we
don't
only
want
to
be
a
silicon
vendor
agnostic,
but
also
want
to
make
sure
that
we
enable
different
roboticses
using
different
frameworks.
So
particularly
the
main
driver
of
this
working
group
is
ross2
and
that's
where
most
of
our
time
is
spent.
But
we
are
mindful
about
the
fact
that
many
people
is
using
raws
or
many
people
might
not
even
using
ross2
and
purely
just
relying
on
dds.
A
There
are
lots
of
use
cases
in
industry
when
this
is
the
case,
so
robot
core
definitely
encourages
this
and
enables
this
in
in
essentially
by
ensuring
that
these
primitives
we
added
can
be
also
ported
to
various
build
systems.
So
in
the
case
of
roster,
we
extend
payment
and
qualcomm
payment
being
the
ross
to
build
system
and
qualcomm
the
bill
tools,
but
similarly
it
can
be
done
for
us
even
for
old
versions
by
leveraging,
catkin
or
related.
A
This
gives
us
plenty
of
liberty
to
build
and
cherry-pick
things
for
our
root
file
systems,
which
not
only
leads
us
to
leaner
root
file
systems,
but
also
to
optimize
the
computational
graph.
Also
by
knowing
exactly
what
we
have,
what
we
want
to
run
and
so
on
and
so
forth.
Also
one
aspect,
that's
often
disregarded
is
the
fact
that
yoto
varies
very
nicely
with
commercial
and
open
source
hypervisor
such
as
the
send
hypervisor.
A
So
it's
actually
very
easy
to
partition
your
hardware
and
leverage
these
capabilities
of
hypervisors
by
using
yocto
based
approaches,
as
opposed
to
maybe
using
some
more
general
purpose,
root
file
systems
such
as
ubuntu,
which
still
can
be
enabled
on
top
of
the
same
hypervisor,
but
there's
always
a
leap
in
there
because
often
there's
not
so
much
support
for
it.
So
a
robot
core
so
far
has
been
demonstrated
with
these
four
manufacturers
we're
still
working
with
with
intent
to
enable
others.
A
The
plan
itself,
though,
is
to
focus
mostly
on
best
of
class
single
board
computers
and,
I
would
say,
the
top
and
most
used
boards
to
develop
robots.
So
if
you
have
any
tip
regarding
this,
please
feel
free
to
reach
out,
but
so
far
we
think
we're
being
pretty
representative
regarding
the
capabilities
that
we
have
also
implemented
recently
in
a
robot
core,
and
this.
This
is
one
feature
that
has
been
developed
over
the
past
few
months.
A
It
wasn't
initially
included
in
the
paper
because
we
were
still
reviewing
it,
but
it
is
a
reality.
Today
is
the
fact
that
we
included,
and
especially
examples
and
and
demonstrated
how
we
can
leverage
type
adaptation
type.
Adaptation
is
specified
in
rep
2007
within
the
community,
and
it's
also
right
now
merged
into
rush
to
humble
and
rolling.
A
This
is
how
it
operates
right
now
by
default
in
roster.
So
if
you
are
testing
data
throughout
the
computational
graph,
this
is
what's
going
to
happen
now
wrap
2007
and
having
your
nodes
implemented.
Leveraging.
This
capability
means
that
essentially,
by
defining
custom
data
types
you
avoid
for
intra
process
communications,
and
it's
important
to
note
is
for
intra
process
communications.
You
avoid
having
to
cast
back
and
forth
and
spend
time
copying
data
unnecessarily.
A
Another
capability
that
robot
corp
presents
and
it's
covered
within
the
paper
is
the
fact
that
we
defined
an
architecture
wherein
we
are
extending
the
build
system
and
the
build
tools
so
that
we
can
switch
easily
from
vendor
1
to
vendor
2
or
from
board
1
from
vendor
one
to
board
two
from
vendor
one
because
it
went
end
of
life
or
because
it's
not
available
anymore.
Given
the
circumstances
we
have
in
the
semiconductors
world.
A
So
this
is
important,
and
it's
more
important
now
than
ever,
especially
to
give
the
liberty
of
to
robotics
is
to
pick
what's
best
based
on
their
use
cases
and
not
based
on
whatever
vendor
they've
been
using
by
default,
everything
while
maintaining
a
roster-centric
api
for
both
developer
development
as
well
as
development
flows.
So
that
is
what
robot
core
implements
by
complying
to
rep,
2008
and
and
yeah.
That
is
definitely
one
of
the
goodies
behind
it.
A
One
of
the
last
things
I'll
share
there
are
a
few
more
and
I
encourage
you
to
read
the
whole
paper
about.
It
is
the
fact
that
recently-
and
this
is
also
one
of
one
of
the
two
things
that
we
did
not
include
in
the
paper,
but
we
have
implemented
in
robot
core
as
of
today
type
negotiation,
time
negotiation
essentially
and
by
it
I
mean
referred
to
red
2009
in
the
community.
A
It's
a
series
of
capabilities
that
can
be
implemented
on
top
of
the
existing
abstractions
of
us
to
allow
you
to
negotiate
essentially
how
publishers
and
subscribers
behave
and
the
best
way
to
understand
this
is
to
look
at
the
bottom
of
the
of
the
animation.
So
you
can
see
in
the
bottom
representation
how,
when
it
restarts
at
the
beginning,
robot
in
the
initialization
phase.
A
What's
gonna
do,
is
it's
gonna
introspect
itself,
the
computational
graph?
It's
gonna
note
that
there
are
some
interactions
between
essentially
notes
that
have
a
relationship
to
to
accelerators,
either
in
fpga
or
gpu,
and
it's
going
to
reconfigure
the
behavior
of
the
publishers
and
subscribers
so
that
we
don't
need
to
go
back
and
forth
for
every
single
node.
A
Of
course,
the
nodes
codes
still
going
to
run
in
the
cpu,
but
the
accelerators
are
going
to
be
able
to
leverage
essentially
their
whole
capabilities
and
the
data
flow
is
going
to
be
optimized
significantly
and
now,
if
you
look
at
the
top
of
the
of
the
image,
you'll
see
how
in
the
default
rush
2,
if
you
do
leverage
hardware
acceleration,
which
you
still
need
to
do,
you
will
see
how
it
goes
back
and
forth
cpu
to
accelerator,
and
of
course
this
is.
This
is
a
huge
bottleneck.
We
discussed
this
in
the
past.
A
A
So
the
thing
I
personally
like
the
most
about
robot
core
is
that
it
is
particularly
thought
for
building
a
robot
course
so
essentially
accelerators
that
are
rost2
api
compatible,
and
that
is
what
I
encourage
all
of
you
to
think
about
and
to
consider,
because
that's
how
the
community
is
really
going
to
grow
around
hardware
acceleration
if
we
can
just
facilitate
a
piece
of
ip
that
just
makes
users
out
there
faster
for
what
concerns
their
computational
graphs.
In
this
sense,
I
wanted
to
present
to
you.
A
Essentially
one
of
the
of
this
robot
course
that
we
developed
within
acceleration,
robotics
and
and
we
did
it
in
a
very
use,
case-centric
manner.
Most
of
the
work
we
do
in
acceleration
robotics
besides
contributing
to
the
community
and
leading
the
the
working
group,
focuses
particularly
on
case
studies
and
on
use
cases
for
final
users,
final
roboticists
and
that's
what
drove
us
to
actually
dive
into
perception
quite
a
while
ago.
A
For
those
not
so
familiar
with
perception
in
robotics,
the
data
obtained
from
sensors,
like
cameras
or
lidar,
is
typically
fed
into
the
perception
layer.
Turning
into
something
useful
for
decision-making
and
planning,
typically
perception
in
robotics,
thereby
you
could
say,
helps
sense
the
static
and
dynamic
objects
and
build
a
reliable
representation
of
the
environment
of
the
world,
mostly
using
computer
vision,
but
also
increasingly
machine
learning
and
ai
techniques.
A
The
perception
layer
is
thereby
responsible,
typically
for
object,
detection,
segmentation
and
tracking
and
within
a
typical
perception
pipeline.
You
can
expect
three
things.
Approximately
the
first
one
is
pre-processing.
This
gets
followed
by
region
of
interest
detector
and,
finally,
then
a
classifier.
This
is
how
often
perception
works
and
understanding.
It
is
crucial
to
propose
a
hardware,
accelerated
implementation,
which
is
precisely
what
robot
core
perception
brings
in
a
hardware.
A
Accelerated
perception
stack
that
aims
to
reduce
the
runtime
latency
and
increase
determinism
as
well
as
throughput,
and
the
most
important
aspect
that
we
are
very
proud
of
is
that
it's
api
compatible
with
your
roster
perception
stack
so
that,
frankly
speaking,
you
don't
reinvent
the
wheel
and
you
don't
spend
time
redeveloping
what
already
works,
which
is
the
worst
two
perception
stacks.
So
you
can
keep
up
with
the
roster
perception
stack,
I
think,
and
if
it
any
at
any
point
in
the
future,
you
feel
you
need
a
boost.
A
The
point
cloud
is
large
and
a
huge
amount
of
data,
and
it
needs
to
be
processed
directly
at
the
edge
so
that
the
time
is
reduced
and
thereby
directly
in
the
robot,
and
if
it
does
so,
this
leads
to
the
smooth
behaviors
you
see
reducing
any
latency
from
the
sensors
and
all
the
way
into
the
actuator,
so
keeping
it
all
into
the
edge
is
crucial.
Lowering
the
timing,
but
still
while
providing
determinism
is
critical
in
robotics
and
hardware.
Acceleration
empowers
behaviors
as
as
such
and
hopefully
in
the
future.
A
A
That
allows
us
to
further
get
a
boost
in
terms
of
performance
and
go
up
to
100x
in
terms
of
speed
up
and
we'll
see
now
some
examples
of
that,
for
example,
this
this
one
packs
together
pipeline
a
perception
pipeline
using
two
simple
roster
notes,
rectify
and
precise
note,
and
you
can
see
the
representation
on
the
right
side
of
the
slide
and
the
source
code
of
this
by
the
way
should
be
in
here.
This
is
one
of
the
examples
involving
graphs
and
there's.
A
There's
this
one,
and
there
are
a
few
others
regarding
graphs,
so
you
can
check
them
out.
If
you
have
a
curiosity
on
how
this
was
built,
everything
is
open
source
and
by
leveraging
robot
core
perception
on
this
super
simple
computer
vision,
operations,
rectification
and
our
resizing.
We
can
already
see
how,
from
a
runtime
perspective
from
a
latency
perspective,
we
can
get
3.5
times
improvement
when
compared
to
competitive
solutions
out
there.
A
In
this
case,
the
nvidia
is
a
cross
solution,
and
this
was
tested,
particularly
with
this
hardware,
solutions
which
are
comparable
in
terms
of
the
cpu
side.
But
this
is
not
all
it
also.
It
gets
even
more
interesting
if
we
don't
only
look
at
the
runtime
latency,
but
also
on
the
power
consumption,
and
particularly
if
we
take
a
look
at
the
performance
per
watt,
so
the
hertz
that
we
get
out
of
the
perception
pipeline
per
watt.
We
can
see
how
the
actual
improvement
goes
all
the
way
to
six
times.
A
Six
times
means
six
times
faster
data
rate,
or
in
this
case
not
data
rate,
sorry,
but
data
rate
in
relation
to
the
watts.
So
this
is
significant
already
and
it
has
an
impact,
a
significant
impact
on
even
these
simple
data
flows,
but
things
can
get
much
more
juicy.
Things
can
go
really
crazy
if
we
start
crunching
or
putting
together
more
operations
that
crunch
more
data,
one
of
which
is
the
harris
filter
which
is
demonstrated
in
here
and
again.
A
It
was
benchmarked
against
competitive
solutions
out
there,
and
in
this
case
you
can
see
in
the
graph.
We
are
not
having
a
let's
say,
a
multi-graph
perception
pipeline,
just
a
single
node
performing
a
harris
operation,
so
we
could
also
pipeline
it
with
with
rectify
and
resize
and
and
if
that
is
something
you'd
like
to
experiment
with
by
the
way,
I
have
good
news,
because
that
is
exactly
what's
prototyping
here.
So
the
perception
three
notes
demonstrates
particularly
such
things,
and
I
think
this
is
the
launch
file
that
corresponds
with
it
yeah.
A
Most
of
the
components,
specifically,
the
basic
building
blocks
of
robot
core
that
are
described
in
the
paper,
are
open
source
and
you
can
check
them
out.
You
can
extend
them,
you
can
contribute,
please
contribute
central
request.
Everything
is
available,
or
at
least
should
be
referred
within
the
ros
acceleration
organization.
A
E
A
So
from
the
working
group
perspective,
we
have
been
maintaining
this
fork
of
the
image
pipeline
package
and
you
can
see
in
here
that
there
is
there.
Is
this
roster
fork,
sorry
rost2
branch
that
again
forks
the
official
one
and
has
a
series
of
contributions
to
enable
capabilities,
both
in
fpgas
popular
fpgas
and
gpus,
so
dirk
so
far,
the
contributions
have
focused
on
molecular
camera
examples
and
and
notes,
but
dirk
has
been
working
on
a
pull
request
enabling
stereo
capabilities.
A
I'm
gonna
break
for
summer
soon
dirk,
but
if
this
has
an
advanced
match,
I
definitely
can
try
to
save
some
time
after
summer
and
maybe
try
to
push
you.
My
advice
would
be
dirk,
though.
Have
you
checked?
Have
you
checked
with
this
other
user?
Who
is
coming
from
pl
plc
2,
it's
another
company
focused
on
fpgas
based
in
germany,
and
they
they
did
a
pretty
nice.
I
think
implementation
also
of
some
of
the
matching
components.
Have
you
checked?
Have
you
checked
this
out.
E
A
Okay,
so
maybe
what
I,
what
I
recommend
is:
if,
if
you're
stuck,
why
don't
you
maybe
create
an
issue,
maybe
in
the
community
repo,
if
not
available
in
here,
and
we
try
to
get
you
two
guys
together,
plus
some
of
the
folks
in
the
community
who
wish
to
contribute
or
help?
If
I
understood
you
properly
derek,
what
you
need
is
help
with
gazebo,
as
well
as
getting
things
to
build.
Is
that
correct.
E
A
Okay,
I
think
we
can
take
this
offline,
so
feel
free
to
feel
free
to
open
a
ticket
and
and
we'll
take
it
from
there.
I
think
I
have
a
few
ideas
on
how
to
enable
you
with
the
right
hardware
and
connect
with
the
right
folks
who
can
give
you
hardware.
So
that's
one
thing
we
can
explore,
but
again,
let's
see
what
people
can
offer
and
and
we'll
take
it
from
there.
Thank
you
there
for
bringing
that
up.
B
E
C
E
A
Yeah
I
mean
I
I
in
just
in
the
chat.
I
saw
guys
that
so
soren
commented
that
that
they
got
it
working
at
plc,
2
turks.
So
maybe
you
want
to
you
want
to
rate
seren
for
wrapping
your
current
fpga
implementation
and
then
maybe
just
raise
a
ticket
in
the
community
to
get
together,
elias
and
and
jm,
and
and
let's
try
to
start
working
towards
this,
no
vpi
or
gpu
enabled
direction.
A
E
A
B
Now,
in
this
respect,
I
think,
in
terms
of
gpu
acceleration,
I
think
you
exposed.
I
went
to
the
jetson
nano
repository,
I
think
you're
using
cuda
acceleration,
but
that
is
exposed
through
a
vpi
backend
of
some
sort.
So
are
you
planning
to
use
denser
rt,
because
that
really
does
work
much
much
better
than
cuda?
In
some
cases,.
A
Not
at
the
moment
so
currently
the
firmware
provided
for
the
jetson
nano
was
just
an
enabler
for
validating
a
particular
use
case.
I'm
sure
that
there's
many
optimizations
available,
but
right
now
we
don't
have
bandwidth
so
we'll
push
that
towards
the
community
or
media
to
to
contribute
if
they
think
that
is.
That
is
important.
What
I'll
just
clarify,
though
gms?
Maybe
I
need
to
rephrase
what
I
what
I
said
so
what
I
meant
in
this
slide?
A
You
need
to
do
that
yourself
and,
and
that
needs
to
be
understood,
that's
what
we
wanted
to
clarify.
We,
we
so
robot
core
and
again
many
of
the
components
open
source,
the
robot
core,
give
you
capabilities
to
go
ahead
and
keep
up
with
your
roster
centric
development
flow
while
introducing
your
corresponding
accelerators.
But
it's
up
to
you
to
develop
those
accelerators
because
they
are
very
use
cases
specific
in
most
cases
and
also
because
in
some
cases
you
only
have
a
binary.
A
So
so
that's
what
we
meant
by
that
of
course,
and
now
linking
into
your
question.
Of
course,
the
firmware
layers
need
to,
of
course,
support
those
corresponding
components.
So
if
tensor
rt
is
something
that
you're
going
to
be
leveraging
in
your
accelerator,
of
course
you
need
to
add
it,
and
for
that
you
need
to
go
back
to
metategra
and
that's
where
elias
comes
in.
B
Now
the
reason
I'm
asking
is
because
there
are
the
newer
jetson
boards
have
some
accelerators
like
pva
and
deep
learning
accelerators
that
are
only
that
can
only
be
used
using
tensority.
You
can't
use
cuda,
so
if
we
kind
of
ignore
tensorrt,
then
we
are
kind
of
leaving
a
lot
of
performance
and
a
lot
of
silicon
unused,
especially
on
those
groups.
A
Well,
I
think
then
that's
it.
We've
run
three
minutes
past
the
time.
So
thank
you.
Everyone
we
had
a
fantastic
attendance
today.
I
think
the
highest
ever
that
I've
seen
in
any
working
group,
so
big
success
on
that
end.
For
those
of
you
new
to
the
group,
please
stick
around
start
contributing
and
looking
forward
to
the
next
meeting.
A
As
I
said,
we're
going
to
make
a
break,
probably
we're
going
to
re-engage
in
september
after
the
summer,
but
stay
tuned
lots
of
exciting
stuff
coming
up
hardware,
acceleration-wise
ross2
talks
will
be
published
very
soon.
Hopefully,
we'll
have
lots
of
hardware
acceleration
this
year.
Thank
you.
Everyone
and
have
a
fantastic
rest
of
your
day.
Take
care.