►
From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #4
Description
Fourth meeting of the ROS 2 Hardware Acceleration Working Group (HAWG) where we focus on reviewing the progress the group has made recently around various topics including #9 Testing and benchmarking methodology for acceleration kernels, #10 Survey the community interests on acceleration for ROS 2 and Gazebo or #1 Ultra96-v2: Add a new community support board, among others. We also had a guest talk from Shaoshan Liu, PerceptIn's founder and chairman who shared with us his view on hardware acceleration for robotics applications.
For more including source code, check https://github.com/ros-acceleration.
0:00 Welcome and agenda
00:40 Newcomer's intro
05:36 Working group progress review
06:30 Testing and benchmarking methodology for acceleration kernels
10:45 Adaptive ROS 2 Nodes
18:00 Survey the community interests on acceleration for ROS 2 and Gazebo
23:12 Quick update on Ultra96v2 port
24:40 Guest talk: Shaoshan Liu, FPGA-based robotic computing
44:29 Q&A
For a discussion about this meeting, refer to https://discourse.ros.org/t/hardware-acceleration-wg-meeting-4/22972.
A
So,
as
typically
we've
been
doing.
Also
in
the
past,
I'd
like
to
give
the
chance
to
those
of
you
new
to
the
group
to
introduce
yourselves
and
tell
us
a
bit
more
about
your
work,
your
interests
regarding
hardware
acceleration,
so
that
we
have
the
chance
to
welcome
you.
So
especially
those
of
you
new
to
the
group,
please
feel
free
to
introduce
yourselves.
B
Well,
so
I
guess
I
would
go
first,
my
instruction,
so
I've
been
doing
fpga
work
and
robotics
work
for
a
while,
so
glad
to
talk
to
you
guys
today,
and
hopefully
we
can
establish
some
sort
of
collaboration.
I
do
believe
it's
a
very
promising
field.
I
have
a
few
collaborators
joining
today,
dr
professor
yuhao
ju
he's
a
professor
at
the
university
of
rochester.
You
how
you
wanna
say
hi.
C
C
Hi
everybody.
So
my
name
is
you
how
I'm
assistant
professor
of
computer
science,
at
university
of
rochester
and
my
research
is
mostly
around
hardware
acceleration,
fpga
prototyping
of
autumn's
machines
like
self-driving
cars
and
robotics,
and
things
like
that,
and
so
I've
been
working
with
charles,
shannon
he's
his
team
for
the
past
few
years,
really
accelerating
autonomous
machine
algorithms
and
so
yeah
very
nice
to
meet
everybody
here.
B
B
D
A
E
A
A
Great
to
have
you
as
well
all
right,
so,
let's
let's
get
started
then
with
the
second
item
in
the
agenda.
As
always,
and
I'm
gonna
share
in
the
chat
link,
you
can
see
in
the
common
chat
a
link
to
the
minutes
that
we
are
taking.
If
I
missed
anything
or
you
have
any
comment
or
recommendation,
please
kindly
point
it
out.
I
will
process
it
later
and,
as
always,
this
will
be
shared
back
in
the
discourse
forum
and
possibly
also
in
other
social
media.
A
So
the
second
item
in
the
agenda,
as
you
can
see
today,
is
the
work
that
we
have
been
doing
for
the
last
few
weeks
and
the
progress
that
we
can
report.
As
of
today,
we
have
three
items.
We
will
briefly
touch
on
today.
The
first
one
essentially
is
the
progress
we've
had
on
the
existing
item.
We
are
keeping
within
the
board
in
the
github
organization
related
to
testing
and
benchmarking
methodology
for
acceleration
journals.
A
Can
see
it
yeah?
Yes,
thank
you
thank
you
for
confirming
it.
So
this
is
the
ticket
where
we
are
tracking
the
again.
The
progress
regarding
the
benchmarking
methodology
that
we
hope
the
community
will
be
able
to
use
for
benchmarking
across
different
hardware,
acceleration
solutions,
but
also
across
different
kernels.
A
So
far,
we
have
completed
most
of
the
items
related
to
proposing
a
preliminary
architecture,
and
you
can
read
more
about
it
in
this
link
and
by
the
way
for
those
of
you
new
to
the
group,
you
can
access
this
in
the
ros
acceleration
organization
and
within
here
you
have
different
repositories,
particularly
the
board
is
kept
within
the
community
repo.
We
are
keeping
here,
projects
related
to
the
faces
where
we
are.
A
We
committed
to
a
number
of
items
when
we
created
the
group-
and
this
is
where
we
are
tracking
the
fulfillment
of
those
items
and
possibly
a
few
more
that
maybe
people
can
contribute
with,
and
we
ourselves
are
also
contributing
as
a
group
as
a
community
and
also
as
individual
companies,
of
course.
So
the
topic
that
I
was
discussing
particularly,
is
this
one,
so
I'm
gonna
go
back
to
the
ticket
and
again
so
far,
there's
been
the
methodology
discussed
and
the
architecture
proposed.
A
There's
been
an
integration
of
the
tools
we
are
using
for
doing
benchmarking
already
into
a
russified
manner
and
for
context.
What
we
are
using
is
lte
tnt
the
linux
tracing
toolkit
next
generation,
which
allows
us
to
essentially
instrument
different
pieces
of
our
kernels
from
the
ps,
sorry,
the
cpu
code,
all
the
way
into
that
that
runs
in
the
corresponding
accelerator
and
also
across
the
different
abstraction
layers
within
the
linux
stack,
which
gives
us
essentially
the
capabilities.
We
are
looking
for
to
essentially
track
and
instrument
and
benchmark
appropriately,
our
ros
computational
graphs.
A
So
the
extensions
of
the
trace
tools,
which
is
the
integration
of
lt
tng,
the
default
integration
of
ldt
into
ros2,
has
been
done
already,
and
you
can
read
some
of
the
extensions
in
here.
They
are
very,
very
simple,
essentially
probes,
so
far,
which
can
be
extended
by
any
of
you
in
your
corresponding
projects
and
the
tools
that
are
integrated
within
this
also
simplify
the
process
of
creating
the
traces
and
then
extracting
the
information
from
each
one
of
these
traces
and
for
context.
A
You
can
do
so
directly
from
the
ros
launch
the
roster
launch
system,
which
simplifies
quite
a
bit,
the
testing,
iteration
and
collection
of
benchmarks,
so
definitely
encourage
you
to
to
try
it
out
and
ask
questions.
Ask
me
directly
if
you
have
any,
because
this
simplifies
substantially
the
amount
of
work
you
will
need
to
do
to
do
benchmarks.
You
know
in
a
somewhat
reproducible
manner.
A
Now,
besides
that
and
besides
the
integration,
I
also
have
provided
a
couple
of
examples
on
how
to
instrument
two
very
simple,
roster
notes.
Individual
notes
at
this
point
very
very
simple.
The
instrumentation
and
collection
of
data
of
complete
computational
products
is
a
future
item,
and
this
is
also
something
that's
been
discussed
actively
in
the
roster
community.
You
will
find
a
few
of
these
course
threads
where
this
is
being
discussed
and,
together
with
a
few
other
companies.
A
We
are
collaborating
in
the
background
on
how
to
do
this
systematically
and
propose
essentially
a
somewhat
organized
and
de
facto
manner
so
that
you
can
do
such
benchmarking
just
beyond
one
single
node
in
one
single
computer
or
multiple
nodes
in
a
single
computer,
extended
beyond
multiple
computers
and
multiple
nodes,
that's
kind
of
like
the
objective
and
lt
tng
does
provide
those
capabilities.
We
just
need
to
get
together
on
how
to
do
it
so
that
it's
it's
comparable
across
different
projects.
A
Adaptive
component
is,
in
a
nutshell,
just
adaptive
components.
Sorry,
in
a
nutshell,
it's
a
composable,
stateless
container
for
adaptive,
roster,
node
computations.
It
allows
you
to
at
runtime
select
on
whether
you
want
to
run
your
roster
nodes
purely
on
the
cpu
or,
if
you
want
to
run
it
in
the
fpga
or
in
the
cpu,
sorry
or
in
the
gpu,
as
it
corresponds
it's
not
fully
in
one
or
the
other.
A
Technically
speaking,
it
is
a
rush
to
know
subclass,
that's
programmed
as
a
component.
Everything
we're
trying
to
do
in
this
working
group
is
pretty
much
following
this
new
trend
of
making
components
which
allows
for
composability
at
lunchtime
so
that
in
your
launch
files
within
your
roster
launch
files,
you
can
compose
things
as
desired
and
put
them
put
things
together
in
different
kind
of
like
executors,
as
your
use
case
demands
for
it.
A
So
again,
in
this
case,
it's
just
a
component
that
has,
within
the
class
itself
a
single
threaded
executor,
which
is
used
to
somehow
decide
which
one
of
these
sub-components
sub-nodes
is
actually
giving
compute
cycles.
And,
of
course,
typically,
these
sub-components.
These
sub-nodes
are
the
one
that
corresponds
with
the
cpu
node,
the
one
that
corresponds
with
the
fpga
node
and
the
one
that
corresponds
with
the
gpu
node.
A
What
this
allows
us
is
to
have
essentially
a
single
node
single
node
entity,
so
to
speak,
because
under
the
hood,
there's
complexity
and
multiple
components,
but
a
single
node
entity
which
we
can
very
easily
with
roster
abstractions
in
a
way
command
to
run
in
one
compute
substrate
or
the
other.
This
is
not
just
useful
for
benchmarking
purposes,
but
in
many
use
cases
we
will
find
that
this
is
somewhat
interesting,
because,
typically,
if
you
have
a
mobile
robot,
that's
navigating
in
a
workspace
that
has
also
manipulator
mounted.
On
top.
A
You
may
want
to
use
your
hardware
acceleration
capabilities
for
the
task
at
the
hand.
Typically,
it's
navigating
around,
so
you
would
want
to
use
your
acceleration
capabilities
fully
focused
on
the
navigation
and
then
once
it's
arrived
into
its
destination.
You
typically
would
want
those
hardware
acceleration
capabilities,
either
gpus
or
fpgas
to
be
fully
dedicated
to
be
fully
working
on
the
manipulation
afterwards.
So
this
actually
allows
you
to
somehow
simplify
that
on
and
off
and
massage
and
and
and
compose
the
resources
by
just
simply
sending
information
across
the
roster
computational
graph.
A
So
it
makes
it
computational
graph
centric,
which
again
does
does
simplify
a
few
things.
Here's
a
simple
demonstration
that
I
cook
for
the
sake
of
of
this
meeting
and
to
show
you
also
how
I
say
it
is
to
run
it.
So
in
this
case,
I'm
setting
up
the
environment
with
biobut
different
tabs
and
what
you're
going
to
see
on
the
left
side
of
the
of
the
cli
is
essentially
how
I'm
launching
a
node
by
default.
A
Gonna
increase
significantly
and
if
we
essentially
relaunch
the
hertz
command
with
a
window
of
10
packages,
so
that
it
doesn't
take
into
account
the
whole
history,
we'll
see
how
it
has
improved
to
10
hertz.
So
again,
this
is
very
handy
for
benchmarking,
but
also
for
many
use
cases.
So
if
you
find
it
useful,
let
us
know
if
you
don't
find
it
useful
and
have
feedback
feel
free
to
open
a
ticket.
It's
it's
fully
open
source.
A
A
According
to
rep
2004,
it's
it's
been
assessed
as
quality
level
four,
so
so
it's
essentially
ready
for
you
to
start
testing
it
and
using
it
and
also
there's
some
reference
examples
again,
the
ones
that
you
saw
just
in
the
video,
but
also
a
few
more
complex
by
default.
The
implementation
of
this
adaptive
component
is
stateless,
stateless
in
the
sense
that
the
state
of
the
node
itself
from
when
it
runs
in
the
cpu
to
when
it
transitions
into
the
gpu
or
the
fpga,
is
not
retained.
A
If
you
want
to
retain
that
state,
while
transitioning
from
one
compute
substrate
to
the
other
one,
then
you
need
to
implement
that
yourself
and
the
reason
why
we
made
this
choice
by
default
is
because
every
node
is
completely
different.
Every
node
is
its
own
word
and
it's
it's
non-trivial
for
us
to
generalize.
A
So
just
for
simplification
purposes.
We
made
it
this
way,
but
here
we
are
providing
an
example:
how
to
make
it
stateful
and
the
way
to
do
so
is
very
simple.
You
just
extend
this
class.
This
adaptive
node
class
use
a
classic
in
your
own
node,
and
then
you
implement
directly
whatever
retaining
your
state
means
to
your
node.
A
There
are
some
conventions
in
here
as
well
that
we
are
proposing
and
where
we'd
love,
to
hear
your
input
and
feedback.
But
of
course
this
is
just
a
suggestion
which
we
think
it
just
makes
sense
and
makes
things
just
just
simplified.
One
of
the
ones
that
I
would
highlight
is
that
by
default
the
sub
components,
the
sub
components
of
the
adaptive
node
are
being
launched
with
an
underscore
as
a
prefix
by
default.
A
This
in
the
rows
to
work
means
that
these
nodes
are
hidden
hidden
in
the
sense
that
when
you
use
the
ros2cli
tools,
such
as
rost2
node
list,
for
example,
you
would
not
see
them
unless
you
use
a
special
flags
to
list
all
of
the
hidden
nodes.
This
way,
what
we
achieve
is
a
more
let's
say,
clear,
introspection
of
the
computational
graph,
where
only
if
you
are
actually
using
appropriate
flux,
you
will
see
that
that
is
actually
a
composed
or
a
container
of
nodes.
A
I'll
take
that
as
a
note
and
jump
into
the
second
topic,
so
the
second
topic
of
the
agenda
is
one
of
the
commitments
we
made
early
while
essentially
discussing
our
intention.
The
intention,
from
a
silence
perspective,
to
lead
this
hardware,
acceleration
working
group
and
the
hardware
acceleration
efforts
in
the
warsaw
community.
A
So
that
happened
many
many
months
ago
in
this
first
post-
and
here
we
we
edited
this
a
few
times
and
is
the
result
actually
of
lots
of
community
input,
and
we
created
this
list
which
came
down
to
essentially
these
objectives.
One
of
the
objectives
we
set
for
2022
was
to
survey
the
community
interests
in
terms
of
hardware
acceleration
for
both
roster
and
gazebo.
A
A
I
think
this
one
should
do
all
right.
So
I'm
sharing
this
in
the
chat
as
well,
so
that
you
guys
can
have
it
handy
if
it
doesn't
work.
Please
just
let
me
know,
but
real
quick,
and
since
we
are
in
time
to
modify
this
survey,
please
send
feedback
about
anything,
you're
missing
or
any
adjustments
you'd
like
to
make.
This
is
definitely
the
right
place
wherein
to
discuss
it
or
you
can
do
it
offline.
A
Second,
because
I
I've
been
observing
their
responses
which,
by
the
way
quite
a
few,
have
come
so
far.
So
I
encourage
you
and
your
groups
to
also
contribute
and
maybe
share
it
internally
in
your
corresponding
groups,
because
we're
collecting
interesting
data,
frankly
speaking,
but
it's
interesting
how
many
people
seem
to
be
aware,
yet
very
very
few
have
developed
a
kernel
themselves,
so
I
find
that
interesting
at
the
very
least.
A
So
that's
what
I
wanted
to
capture
in
those
first
questions.
Then
we're
highlighting
here
the
the
group
effort
of
trying
to
put
together
a
series
of
conventions
and
an
architecture,
a
reference
architecture
for
ourselves,
but
other
also
other
silicon
vendors
to
join
and
benefit
from
the
community
and
the
group,
as
well
as
add
value,
of
course.
So
that
is
red
2008,
and
here
we
are
collecting
also
a
few
input
about.
A
So
if
you
have
additional
bits
in
here,
please
say
so,
and
then
we're
also
collecting
a
few
additional
data
such
as
which
which
compute
substrate
do
they
prefer
or
do
they
plan
to
use
and
then
which
solutions
are
they
currently
using
and
in
every
single
question
that
has
options.
We
have
marked
yeah,
most
of
them
the
option
of
saying
others
so
that
people
can
provide
their
own
answers.
I
haven't
said
it
at
the
very
beginning,
but
the
intention
is
to,
of
course,
curate
the
responses
and
disseminate
disseminated
back
with
full
transparency.
A
This
is
typically
the
way
things
work
in
the
ross2
community
and
we
intend
to
comply
with
this
and
yeah.
That's
that's
pretty
much
it.
So
please,
please,
please!
Everyone
in
this
call
fill
this
form
if
you
haven't-
maybe
not
now,
maybe
after
the
call,
but
please
do
so
because
this
is
really
valuable
input
for
the
work
that
I
am
personally
doing,
but
also
my
colleagues
at
silence
and
many
others.
I
mean
some
other
colleagues
in
nvidia
and
some
other
people
out
there.
A
A
It
sounds
like
not
at
this
point,
so
I'm
going
to
move
forward
and
then
the
last
topic
about
the
review
or
the
progress
so
far
is
the
first
port,
the
first
community
port
we're
having
of
our
architecture
to
the
ultra
96
b2
board,
and
this
is
this-
is
a
popular
board.
That's
used
by
many
when
it
comes
to
fpga
development
and
pedro
is
taking
the
lead
on
doing
the
port
and
I'm
actually
very
impressed
by
what
pedro
is
doing
given
the
complexity
and
how
fast
he's
doing
it.
A
So
thank
you
pedro
for
all
the
efforts
and
and
and
doing
that,
it's
awesome
to
to
count
on
you.
If
you
are
out
there
or
listening
later,
the
latest
status
is
that
pedro
is
essentially
getting
up
to
speed.
He's
been
able
to
run
some
examples
and
now
he's
essentially
adequating
some
aspects
for
what
concerns
building
the
kernels
and
then
launching
them.
A
I'm
helping
pedro
on
the
way
and
possibly
I
will
try
to
find
some
time
to
give
him
an
extra
push.
But
hopefully,
before
the
end
of
the
year,
we
can
have
a
first
community
official
board
and
then
from
that
one.
I'm
I'm
very
well
aware
that
there
are
a
few
others
ongoing
that,
hopefully
will
come
out
to
the
light,
and
with
that
we
will
have
this
same
architectural
portal,
not
just
to
purely
fpgas,
but
also
gpus
and
and
and
with
that,
take
take
take
off
from
there
and
gain
popularity.
A
All
right,
so
there
are
no
questions
regarding
the
progress
so
far.
What
I
will
do
is
introduce
formally
our
guest
speaker,
so
so
essentially,
I'd
like
to
introduce
to
everyone,
though
he
already
introduced
himself
but
anyhow,
for
the
sake
of
completeness
saucian
leo,
who
is
percepting
founder
and
chairman
sausan
is
an
experienced
roboticist
who's
been
making
scalable
hardware
and
software
integrated
solutions
for
autonomous,
robotic
systems.
For
quite
a
few
years.
A
Now,
in
the
past,
he
led
the
autonomous
driving
system
team,
as
well
as
the
deep
learning
infrastructure
in
beirut,
usa,
and
before
that
he
was
with
the
gpu
research
team,
at
inria
in
france
and
the
runtime
systems
at
intel
research.
Among
other
experiences.
He
has
over
40
publications,
some
of
which
are
really
really
worth
reading.
Actually
this
is
what
got
me
to
saoshan
one
of
his
publications,
which
I
enjoyed
very
much
and
also
he
has
more
than
100
patents
in
the
distributed
computing
field,
as
well
as
in
autonomous
driving
computing
he'll.
A
B
All
right
can
everybody
see,
we
can
see
it
all
right.
Let's
get
started
I'll
fly
through
the
slides.
If
you
have
questions,
stop
me
anytime,
and
then
we
can
discuss.
It's
not
gonna,
be
a
very
long
but
we'll
just
summarize
what
we
have
learned
so
far
about
robotic
computing
on
fpgas,
so
I'll
go
through.
Why
do
we
do
autonomous
machines,
including
autonomous
vehicles,
robots
and
so
on?
That's
our
field
of
expertise
and
then
particularly
our
experiences
of
building
a
computing
system
for
this
kind
of
autonomous
machines.
It's
not
easy.
B
There
are
some
unique
challenges
over
there
and
then
the
main
topic
using
fpgas
for
this
task
and
then
particularly,
I
wanna
highlight
that
there's
a
key
feature
within
silence.
Fpgas,
that's
been
so
underrated,
that's
partial
reconfiguration,
but
I
do
believe
that
this
feature
is
extremely
important
in
robotics
and
then
we
summarize
at
the
end.
B
So
why?
When
we
choose
to
do
something,
we
gotta
understand
why?
So,
today
the
market?
We
are
a
startup,
so
we
start
from
the
commercial
motivation.
Today
the
market
is
quite
not
quite
small.
I
mean
150
billion
dollars,
it's
not
a
small
market,
but
if
you
look
at
the
projection
by
2030,
it's
22
growth
per
year,
so
by
2030
we
have
a
one
trillion
dollar
market,
much
bigger
than
the
selfie
market
we
enjoy
today.
B
B
In
that
time.
The
whole
market
size
is
about
10
billion
dollars
quite
a
small
market.
And
then,
when
you
look
at
the
feature
phone,
nine
percent
of
the
computing
power
go
into
the
basic
functions,
making
a
call
and
so
on
so
encoding
decoding
that
kind
of
function.
That's
the
the
main
computing
driver.
Only
10
percent
goes
into
end
user
applications.
But
if
you
look
at
the
cell
phone
market
today,
it's
a
800
billion
dollar
market.
B
B
But
if
we
look
at
what
we
see
today,
you
take
any
robots
and
you
look
at
computing
system.
You
break
it
down
into
different
functions,
perception,
localization
planning
and
at
the
end,
application
today
only
10
goes
into
application.
What
we
want
to
do
is
to
build
a
computing
system
that
consumes
only
10
15
of
the
available
computing
power
on
chip.
Then
leave
about
80
of
computing
power
for
the
interesting,
robotic
applications,
and
that's
what
drives
the
growth?
That's
what
we
try
to
achieve.
B
So
we
start
with
what
time
driving
by
the
way
for
in
this
talk,
we
each
slide
may
have
a
reference
to
give
you
more
details.
If
you
would
like
to
check
out
the
reference
it's
at
the
bottom
of
the
page
in
blue
font.
So
if
you
want
to
look
into
more
information,
you
can
check
out
there.
But
here
is
the
computing
stack
for
qualcomm
driving?
That's
what
we
do.
B
So
we
are
a
product
company.
We
try
to
shift
autonomous
vehicles
globally,
but
we
were
forced
to
build
a
computing
system.
We
did
not
want
to
build
computing
systems,
but
there
was
nothing
suitable
available
in
the
market.
That's
why
we
were
forced
to
do
computing
system.
We
summarized
the
business
story
in
this
paper
is
quite
an
interesting
week
if
you're
interested
so
we
had
when
we
tried
to
build
a
product,
we
had
three
options:
option
one
is
to
just
use
mobile
soc,
that's
the
easiest
and
the
best
option.
B
They
are
very
cheap
mobile,
socs,
a
few
hundred
dollars.
You
get
a
chip
and
then
it's
backward
compatible.
They
have
very
mature
apis
and
so
on.
So
we
were
hoping
that
a
mobile
soc
would
be
enough
to
sustain
a
autonomous
machine
option.
Two
is
to
buy
specialized
autonomous
driving
computing
systems
back
then,
if
you
remember
and
media
px2,
it's
over
ten
twenty
thousand
dollars
per
piece,
if
we
were
trying
to
ship
products
that
meant
to
be
profitable,
there's
no
way,
we
can
use
that
kind
of
specialized
machines.
B
On
top
of
that,
they
don't
really
again
they're
black
boxes.
You
don't
know,
what's
going
on
inside,
they
give
you
perception,
outputs,
that's
not
what
we
want,
because
autonomous
vehicles
are
concerned
with
safety
option.
Three.
We
do
not
wanna
go
that
route.
It's
very
expensive!
It's
to
build
your
own
proprietary
or
contracting
computing
system,
but
we
ended
up
in
the
option
d.
I
will
explain
why
so
we
did
try
with
mobile
soc.
B
So
the
first
thing
we
did
is
to
allocate
the
workloads
to
different
kind
of
accelerators.
We
figure
out
that
dsp
is
quite
good
at
the
sensor.
Data
processing
such
as,
if
you
do
a
slam
workload
to
extract
the
the
features
from
a
picture.
Dsp
is
quite
good
in
that
gpu.
As
you
all
know,
even
a
mobile
gpu
is
quite
good
in
deep
learning,
so
we
use
it
for
object,
recognition
for
the
four
cpu
for
hardware
threads.
B
We
have
one
on
chip,
we
dedicated
two
threads
for
localization
and
then
two
threads
for
planning
control,
but
it
didn't.
It
was
not
enough
to
drive
a
car.
It
was
enough
to
drive
a
small
indoor
robot,
so
the
mobile
soc
picture
it's
out
of
picture.
We
could
not
use
that
for
our
project
specialized
computing
system.
Again,
it's
a
black
box.
We
did
not
want
to
play
with
a
black
box
because
we
want
guaranteed
safety
for
our
customers
and
then
it's
extremely
expensive.
B
We
want
to
make
any
profit,
this
kind
of
computing
system,
it's
a
nightmare,
so
we
did
not
go
that
route
at
the
end
we
choose
to
build
our
own
system.
It
took
us
some
time.
We
summarized
the
the
findings
in
this
micro
paper
at
the
bottom.
If
you're
interested,
there
are
a
lot
of
details
over
there,
but
there
are
a
lot
of
design
constraints
such
as
latency
throughput,
how
you
interact
with
the
physical
world,
there's
the
energy
and
thermal
constraints.
B
So
you
cannot
burn
too
much
power,
otherwise
you
cannot
cool
it
down
or
you
operating
time
per
day
is
very
low.
Then
you
will
not
be
profitable.
The
third
constraint
is
safety
and
cost.
You
have
to
make
it
low
cost
enough
such
that
you
can
be
profitable
and
then
safety
is
utmost
importance
in
this
design.
B
So,
basically
that's
our
workflow.
You
have
gps,
imu
cameras,
radars
and
so
on.
The
first
thing
you
need
to
do
when
dealing
with
heterogeneous
sensors
is
to
synchronize
them.
So
you
need
a
sensor
synchronization
hardware
to
deal
with
that.
The
different
sensors
will
go
into
different
routes
in
computing.
Some
go
into
localization
some
go
into
a
scene,
understanding
of
perception
in
general,
then,
once
they
process
data
they
go
to
the
the
backend
for
planning
control.
So
that's
the
basic
software
path,
hardware-wise.
B
What
we
did
at
the
end
for
the
details
here,
you
can
read
the
paper
a
lot
of
detail
over
there,
but
what
we
did
at
the
end
is
that
we
use
the
fpga
board
for
sensors
or
sensing
processing.
What
I
meant
by
that
is,
first
of
all,
the
benefit
of
fpga
is
that
they
are.
I
o
rich.
So
you
have
a
lot
of
different
io
ports.
You
can
connect
to.
You
can
connect
to
heterogeneous
sensors,
and
then
you
can
use
the
hardware
logic
to
do
sensor.
Synchronization,
that's
extremely
important
for
synchronization.
B
Then,
once
you
do
the
synchronization,
you
can
do
data
filtering.
What
I
meant
by
that
was,
if
you
have
a
huge
picture,
but
all
you
want
from
the
picture
of
the
features,
then
you
can
filter
down
the
data
size
by
one
thousand
fold
then
send
to
the
back
end
on
the
back
end.
What
we
have
is
a
gpu
and
a
multi-core
cpu.
The
gpu
is
reserved
for
deep
learning.
As
usual.
Cpu
is
for
planning
and
control
tasks.
Then
we
connect
that
to
the
canvas
to
control
the
vehicle
and
so
on.
B
B
B
Then
we
summarize
two
kinds
of
autonomous
machines.
It's
in
two
graphs
on
the
left.
You
see
a
level
four
times
vehicle,
different
modules
have
different
blocks
or
functions
in
that
stage.
On
the
right
side,
you
see
the
simplest
form
robot,
which
is
a
robot
vacuum
cleaner,
it's
quite
simple,
but
they
demonstrate
the
same
pattern.
B
You
have
a
huge
amount
of
data,
rushing
in
on
the
left
hand,
side
and
then
for
each
stage
you
build
it
down
with
the
data
size
and
at
the
end,
you
have
a
small
amount
of
data
going
up
to
control,
so
both
graphs
exhibit
a
data
flow
pattern.
We
have
summarized
the
pattern
in
the
paper
at
the
bottom.
I
welcome
to
check
it
out,
but
it's
quite
interesting
for
different
kind
of
mobile
robots.
When
you
deal
with
mobility,
it
follows
the
same
path.
B
So
we
summarize
the
pattern
here.
First
pattern.
Is
that
there's
a
continuous
input,
so
it
provides
abundant
parallelism
in
traditional
computer
architecture.
What
we
deal
with
is
sometimes
there's
a
huge
spike
in
parallelism,
but
you
don't
have
enough
resource
resources,
computing
resources
to
deal
with
that
kind
of
parallelism,
so
it
kind
of
stalls
the
whole
pipeline
and
at
other
times
you
have
not
enough
parallelism.
B
So
we
are
wasting
a
lot
of
resources,
that's
very
hard
to
deal
with,
but
in
robotics
you
don't
have
that
problem,
because
data
just
continuously
rushing
second
of
all,
it's
flexible
dependencies.
What
I
meant
by
that
is
that
usually
in
the
producer
consumer
type
of
relationship,
the
consumer
has
to
wait
for
in
every
data
item
from
the
producer,
but
in
robotics,
since
we
are
dealing
with
the
real
time
system,
we
only
take
the
latest
item.
So
if
there's
a
queue
of
the
buffer,
we
only
take
the
latest
and
we'll
flush
the
buffer.
B
B
You
know
the
connection
of
different
different
blocks,
they're
quite
deterministic,
and
you
know
how
much
profit
you
need
to
allocate
between
different
blocks
and
at
the
end
there
are
decks.
There
are
no
loops
inside
the
deck.
So
when
you
schedule
a
dag,
it's
much
easier
to
schedule
something
with
loops,
so
those
are
the
summaries
that
we
make.
B
So
we
propose
to
write
a
language
like
this.
We
have
a
bunch
of
required
statements
to
set
up
each
hardware
block
to
set
up
a
sensor
to
say
how
much
frame
rate
you
want,
and
you
could
set
up
say
the
perception
block
to
say
how
much
processing
power.
What's
the
resolution
you
want
once
those
are
set
up,
then
you
have
a
few
in
this
case
three
lines
of
code
to
set
up
the
computation
graph.
Then
that's
it
when
data
starts
flowing
in
and
you
have
data
continuous
coming
out.
B
That's
the
beauty
of
robotic
computing,
but
to
achieve
that,
you
need
fpga,
at
least
in
this
stage,
to
achieve
that.
You
need
fpga.
The
reason.
Why
is
that
first
step
you
have
a
software
specification
say
if
I
want
this
piece
of
hardware
to
do
640
by
480
resolution,
then
the
hardware
needs
to
configure
itself
to
achieve
that
and
then
once
it's
configured
it
has
to
tell
the
software
system
that
I'm
done
and
I'm
ready.
So
the
second
step
is
hardware
configuration
you
can
see
here.
B
The
third
step
is
that,
okay,
now
all
the
pieces
of
hardware
are
self-configured,
then
you
can
do
buffer
allocation
to
form
the
graph.
So
once
the
graph
is
formed,
computing
starts
it
in
run
time.
You
can
do
self
optimization
such
that,
for
example,
if
you
are
a
feature,
detection
block
and
you
are
in
a
very
clean
environment,
you
might
want
to
turn
down
your
frame
rate
such
that
you
don't
have
to
consume,
that
much
or
resolution
to
conserve
power
to
reduce
the
amount
of
heart
rate
you
use.
B
So
we
summarize
that
configurable
part
in
dcs
micro
conference.
It's
called
architects,
so
that's
our
basic
high
level
design,
but
what's
key
here,
I
think
it's
partial
reconfiguration.
It's
such
a
beautiful
feature
when
I
was
at
microsoft,
research
about
almost
10
years
back
10
years
back,
that's
the
project.
I
did.
I
figured
out
that
that's
quite
a
beautiful
feature
such
that
you
can
divide
your
hardware
view
estate
into
different
parts.
Then
you
can
dynamically
configure
these
parts
depending
on
different
features.
You
need.
B
So
you
can
have
these
different
partial
bit
streams,
for
example,
for
localization
you
can
have
big
streams
for
different
localization
algorithms
to
swap
in
and
out
at
run
time.
It
takes
less
than
a
few
milliseconds
to
do
the
swapping,
so
that
would
help
you
to
adapt
to
a
very
dynamic
environment
as
well
as
to
reduce
power
consumption
at
runtime.
B
I
don't
think
that's
acceptable,
especially
if
science
is
trying
to
enter
the
field
of
robotics.
So
a
quick
summary
I
I
do
think
fpgas
are
the
best
platform
for
robot
computing
for
several
reasons.
First
of
all,
robotic
workloads
are
highly
dynamic,
so
you
need
to
change
at
run
time
either
at
configuration
time
or
at
run
time.
You
have
to
change
it
to
adapt
to
the
environment,
to
adapt
to
the
future
requirements.
B
Second
of
all,
unlike
asics
fpga
offers
scalability
of
field
programming
without
going
through
manufacturing
experience.
Today
we
are
going
through
a
chip
shortage.
If
you
can
reuse
a
lot
of
fpga,
that
would
help
a
lot
and
you
can
do
over
the
air
update
of
hardware
and
so
on,
and
then
partial
configuration
again.
It's
so
underrated,
so
under
it
today
and
then
to
do
partial
configuration,
it's
very
challenging.
B
You
have
to
map
out
the
real
estate,
you
have
to
cut
it
into
parts
and
then
you
have
to
generate
bit
file
and
they
have
them
swap
at
runtime.
If
we
have
software
support
to
make
it
easier,
for
example,
make
all
the
rods
blocked
as
one
partial
bit
file,
then
it
would
be
a
beauty
to
see
those
functions
running
on
fpga,
another
takeaway.
That,
from
our
experience
is
software.
Support
for
fpgas
is
too
weak,
if
say,
for
example,
a
company
like
silence.
B
If
it
wants
to
be
successful
in
the
age
of
robotics,
then
you
probably
need
a
something
like
a
cuda.
The
full
fpgas
take
off
for
hardware
perspective.
I
think
fpga
is
very
suitable
for
robotic
computing,
but
from
a
software
perspective
it's
really
hard.
We
don't
have
enough
fpga
talent
to
do
that.
Kind
of
programming.
B
Third
thing
is
lost.
Support
today
for
hydrogen's
computing
is
still
very
weak.
I'm
glad
to
see
this
group
of
colleagues
working
on
a
hardware
solution
for
loss.
I
think
that's
required.
If
for
a
operating
system,
you
don't
have
that,
then
you
cannot
survive
the
age,
but
because
ross's
support
for
heterogeneous
computing
is
very
weak.
That
would
give
a
lot
of
new
opportunities
for
newcomers
say.
B
If
another
company
brings
up
a
new
operating
system
for
robots
with
very
good
hardware
support,
it
will
immediately
gain
a
lot
of
popularity,
so
I
think
for
new
generation
of
robotic
investments
to
support
a
lot
of
algorithms,
which
ross
is
doing
very
well
and
a
lot
of
computing
platforms,
which
is
doing
not
quite
well
in
this
regard.
I
think
today,
at
this
time,
the
opportunity
for
a
new,
robotic
os
are
still
unlimited.
B
So
at
the
end
I
want
to
make
an
advertisement
of
my
collaborator
he's
been
doing
a
lot
of
robotic
stuff,
especially
hardware
acceleration.
I
do
believe
he's
a
world
expert
in
this
field.
I'm
glad
that
we
all
joined
the
meeting
today,
then
we
can
talk
more
and
hopefully
to
collaborate
more
in
this
field.
It's
very
exciting
and
that's
the
end
and
thank
you
so
much
and
now
it's
time
to
let's
communicate.
If
you
have
any
questions.
A
All
right,
thank
you
so
much
so
sean.
I
enjoyed
it
very,
very
much
your
presentation
and
I'm
sure
there
will
be
a
few
questions.
Thank
you.
You
have
also
for
for
joining,
and
it's
also
great
to
see
you
there.
Okay,
so
I
have
a
few
questions
you'd
have
in
here,
but
I
would
like
to
give
the
chance
to
everyone
else
before
I
go
questions.
A
B
Let
me
bring
up
the
paper,
so
basically
we
do
it.
You
have
some
software
protocol
in
linux
that
you
can
use
that's
for
software
synchronizing,
but
you
need
hardware
and
then
hardware
needs
to
be
triggered,
for
example
lidar
they
they
need
to
be
triggered
in
order
to
do
sensor
synchronization
for
cameras,
sometimes
it's
passive,
so
it
depends
on
the
sensor.
You
have
to
design
different
mode
of
different
mode
of
operation
to
do
the
synchronization,
but
let
me
see
if
I
can
bring
up
the
page.
B
Yep,
that's
the
we
put
our
hardware
and
software
solution
over
there.
So
if
you're
interested,
I
I
think,
that's
the
one
to
read-
I
I
so
for
us.
We
implemented
the
the
hardware
logic
in
in
fpga.
So
basically
you
need
a
trigger
for
some
of
sensor.
You
need
a
trigger
to
tell
it
to
capture
frames
and
the
frame
would
come
back
so
it's
a
trigger.
Then
you
need
to
send
it
to
different
sensors
for
other
passive
sensors.
They,
the
data,
just
rush
in
their
firmware,
will
have
a
timestamp.
B
A
So
I
was
just
wondering
if
you
were
using
such
a
thing
across
the
different
sensors,
where
each
one
of
them
had
essentially
synchronized
timestamps
or
if
you
were
doing
this
on
the
on
the
go
within
the
fpga
once
you
receive
all
of
the
data,
but
it
does
sound
like
the
answers
are
in
this
paper.
So
I'll
I'll
do
my
reading,
and
maybe
I
will
follow
up
with
some
questions
for
you.
B
Yeah,
a
short
answer
is
that
we
use
ntp.
I
do
believe
the
problem
is
ntp
such
that
it
gets
the
signal
from
gnss,
it's
a
satellite
atomic
clock,
so
that's
the
the
source,
that's
where
you
get
the
source,
then
we
distribute
the
source
to
different
sensors
who
have
pga.
So
that's
the
short
answer
to
your
question.
B
F
So
the
question
I
have
a
question
I
mean
the
methodology
you
use
is
an
explicit
synchronization.
It's
not!
You
are
not
taking
advantage
of
any
streaming
programming
of
the
fpga,
so
with
the
fpga
we
have
the
ability
to
to
create
producer
and
consumers
of
information
that
can
self-synchronize
themselves
in
in
streaming
with
a
streaming
mechanism.
Are
you
taking
advantage
of
this
particular
methodology,
or
are
you
explicitly
sending
a
synchronization
signal.
B
F
B
Would
love
to
check
it
out,
but
the
problem
seen
a
few
today,
at
least
still
today,
is
that
the
sensor
highly
heterogeneous,
so
they
do
have
different
interface
and
particle.
So
once
the
data
cross
that
interface,
then
we
can
use
streaming
but
before
they
cross,
we
need
to
timestamp
them.
So
I
think
that's
the
difficulty
comes
into
in
this
scenario.
F
Just
correct
me:
normally,
this
type
of
approaches
is
generating
an
adapter.
That
is
a
model
that
takes
the
information
generated
by
the
specific
sensor
and
then
translating
this
information
into
streaming.
So
it's
taking
care
of
connecting
and
then
producing
the
data
stream
synchronized.
But
I
mean
it's
something
that
you,
you
might
investigate
a
little
bit
a
little
bit
deeper
sure
sure
we'd
love
to
check
it
out.
A
A
So
so
one
of
the
ideas
we've
been
discussing
internally,
and
also
we
can
extend
it
to
the
group-
is
to
what
extent
in
real
applications
to
what
extent
robotics
is
like
you
would
benefit
from
having.
Let's
say,
data
breaches,
data
translators
from
one
data
link
layer
to
another
data
link
layer,
because
often
what
you're
doing
is
just
passing
the
data
from
one
source
right
from
the
camera
bus
to
a
different
bus,
so
that
it
goes
to
the
corresponding
processing
system
and
often
you're
doing
this.
A
All
the
way
up
through
the
osi
stack
going
to
the
application
layer
and
then
all
the
way
down
into
the
corresponding
other
data
link
layer.
So
so
you
do
waste
lots
of
compute
capabilities
in
going
all
the
way
up
and
down
in
the
stack.
So
I
was
just
wondering
to
what
extent
is
this
a
problem
in
your
use
case
and
and
how
much
would
you
actually
save
latency
wise
on
having
such
things
happening
literally
in
hardware
in
the
fpga.
B
Well,
I
think
that
would
definitely
be
doing
the
community
a
favor,
huge
favor,
for
a
lot
of
companies.
I
think
a
lot
of
companies
today.
They
are
spending
a
lot
of
effort
doing
synchronization
either
in
software
hardware
for
two
two
door
blocks.
First,
not
many
talents
who
understand
synchronization.
B
If
there's
a
unifying
interface
like
you
mentioned,
and
then
connect
data
link
layer,
they
will
save
a
lot
of
trouble
if
we
can
standardize
that
and
in
the
future
all
the
sensor
developers
will
conform
with
this
interface.
It's
a
huge
impact
in
the
field
and
it
will
be
doing
the
community
a
huge
favor.
The
paper
I
just
show
you,
I
mean
the
only
reason
we're
doing
it.
It's
it's
paying
at
the
ass
right.
B
You
have
to
do
so.
Many
different
heterogeneous
sensors
then
manually
pretty
much
right
to
program
your
piece
of
hardware
to
do
it
manually,
and
things
should
not
be
like
that.
If
you
want
a
scalable
robot,
especially
as
you
add
more
and
more
sensors
into
the
system,
if
we
had
an
interface
like
that
that
that's
the
first
thing,
I
would
do.
G
Question
one
question
sure
it's
more
about
in
your
architecture:
choice:
you
talk
about
also
using
a
gpu,
so
my
question
is
when
you're
working
so
exclusively
with
fpgas.
Why
are
you
still
using
gpu
for
deep
learning
and
why
not
use
the
algorithms?
Also
on
the
fpga
and
a
second
follow-up
question
is:
what
is
the
compute
capacity
of
the
fpga
that
you
are
using,
so
how
many
lookup
tables
can
it
support?
Is
it?
Is
it
a
small
one?
Is
it
a
really
big
one?
So
yeah?
Can
you
tell
me
something
about
that
sure.
B
The
second
question
is:
is
it
zinc,
so
it's
not
that
big,
it's
embedded
fpga
the
lookup
table.
I
don't
remember
detail
numbers,
I
think
it's
all
in
the
vapor,
but
it's
a
small
one
for
gpu.
Well,
it's
answer
is
simple.
They
have
cuda.
So
it's
very
simple.
You
just
compile
a
program
running
on
cooldown.
That's
it
right.
If
silence
had
a
similar
programming
environment,
I
would
definitely
use
silence.
But
again,
at
the
end
of
the
talk
we
mentioned,
that
the
software
support
today
is
really
far
behind.
A
I
can
I
can
try
again
so
in
one
of
one
of
your
papers
at
least
one
of
the
ones
that
captured
my
interest,
which
I
will
share
here
with
the
group.
Let
me
do:
oh
there's
lots
of
people-
okay,
everyone
in
here
so
in
that
paper,
particularly
I
I
found
interesting
how,
in
the
conclusion
section
you
you
mentioned
pretty
much
what
you
summarized
at
the
end,
the
fact
that
you're
a
strong
believer
that
the
capabilities
of
pgas
fit
nicely
with
robotics.
A
But
then
you,
you
follow
up
by
saying
something
like
that.
Any
asic
based
accelerator
will
be
months
or
even
years
behind
the
state-of-the-art
algorithms,
and
I
think
that
the
rationale
is
pretty
well
understood
on
why
you
are
you're,
saying
that,
because
of
how
closely
time-wise
and
resource-wise
it
is
to
produce
basics.
The
question
is
by
by
when
do
you
think
if
you
ever
think
it
will
happen
in
robotics
by
when
do
you
think
like
asics
would
start
making
more
and
more
sense
in
the
field?
Based
on
your
experience.
B
I
think
it
would
take
a
while
right
if
you
look
at
autonomous
machines,
each
algorithm
or
each
module,
it's
its
own
field.
If
you
go
back
a
few
years
where
perception
is
its
own
field,
pretty
much
all
the
cnn
paper.
Today's
perception
planning
control
is
by
itself
a
localization.
I
mean
if
you
check
out
the
robotic
papers,
the
slam
paper
by
itself
its
own
field,
and
we
are
talking
about
generating
a
block
by
block
by
salvatore
and
putting
all
of
them
together
into
peace.
I
think
at
least
five
to
ten
years
of
effort.
B
If
you
ever
go
that
route,
who
knows
right,
if
fpga
is
good
enough,
then
you
might
not
have
to
go
that
route
and
then,
if
software
optimization
technique
is
good
enough,
you
might
not
have
to
go
that
route.
But
today,
from
the
the
point
we
see
it,
it's
a
small
design
that
represented
a
data
flow
architecture
to
connect
different
blocks
and
the
premises
for
that
is
that
you
do
have
to
have
this
individual
blocks
ready.
It
took
us
two
years
to
do
localization
right.
A
A
B
A
Well,
we
are
keeping
minutes
for
every
one
of
the
calls
and
we
try
to
track
sausage
and
the
where
each
person
that
joins
the
course
is
coming
from.
I
I
can
tell
you
from
top
of
my
mind.
Most
of
the
participants
are
from
industry.
Yes,
however,
if
you
want
specific
details,
there's
full
transparency
who
is
joining
the
calls,
and
you
have
the
recordings,
as
well
as
the
names
that
we
collected
so.
B
So
you
can
double
check
it.
Oh
no!
I
I
just
want
to
raise
the
point
that
if
we
can,
for
example,
I'm
from
the
industry
right,
but
I
do
a
lot
of
research,
I
do
feel
that
if
we
can
mingle
more
with
people
from
academia
like
professor
bihau
and
so
on,
we
can
generate
much
bigger
impact
because
a
lot
of
good
projects,
they
start
from
school,
start
from
research
project
and
then
with
good
papers
flowing
out
and
people
come
to
you
and
contribute
to
more
solutions.
B
I
do
think
this
is
a
very
important
effort
that
you're
doing
here
and
congratulations
for
for
this
effort
and
then
particularly
the
the
item
that
you
brought
up
for
a
unified
interface
for
all
census.
I
think
we
need
to
work
on
that.
I
think
that's
such
a
great
idea
and
then
we
can
publish
the
idea
to
let
people
from
academia
know
about
it.
For
me
that
should
know
about
it.
I
think
that's
at
least
a
one
small
step
forward
towards
your
end
mission.
I
think.
A
Thank
you.
Thank
you
for
the
feedback
and
and
very
well
taken
yeah
sure
I
mean
what
I
would
say
is
so
we're
meeting
monthly.
If
people
keep
up
coming
so
often
we
may
increase
the
frequency,
but
for
now
we're
meeting
once
per
month
we're
being
reasonably
active
in
ros
discourse,
which
is
kind
of
like
the
de
facto
place
where
to
discuss
such
ideas
and
definitely
that
encourages
both
academics
as
well
as
industry
players
to
interact,
engage
so
more
than
happy.
A
G
A
Wrapping
up
in
here
I
just
wanted
to
remind
everyone.
Please
please,
please,
if
you
haven't
filled
up
the
form
do
so
again.
This
is
for
the
sake
of
everyone
in
this
community
and
everyone
participating
in
the
meetings.
So
we
promise
we'll
digest
the
data
after
a
few
weeks.
I
don't
know
how
many
weeks
we'll
give
it,
but
I'll
see
how
how
many
responses
start
coming
up
and
once
things
stop
increasing
well,
we'll
just
cut
it
and
at
that
point
process
the
data.
A
Thank
you
everyone
for
showing
that.
Thank
you,
especially
those
folks
in
the
u.s.
I
know
it's
a
bit
early
and
I'm
sorry
about
that,
but
we
we
had
to
move
the
date,
sorry,
the
time
for
our
guests
from
asia
this
time,
but
I
think
the
participation
was
nice.
So
if
this
keeps
upcoming,
we
might
be
rotating
with
a
friendly
time
zone
for
a
us
one
week
and
then
the
next
one,
maybe
in
asia,
but
we'll
see.