►
From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #7
Description
Seventh meeting the ROS 2 Hardware Acceleration Working Group. We discuss the group's progress during the last month showing how ROS 2 perception pipelines run in GPUs and FPGAs, comparing results. We review the methodology for ROS 2 Hardware Acceleration and REP-2008 and then discuss a new subproject of the group: the Robotic Processing Unit (RPU). The session will also includes a guest talk provided by Hyunjong Choi who tells us about how although ROS 2 claims to enhance the real-time capability, ensuring predictable end-to-end chain latency still remains a challenging problem. To address it, their group proposes a new priority-driven chain-aware scheduler for the ROS 2 wherein callbacks are prioritized based on the given timing requirements of the corresponding chains so that the end-to-end latency of critical chains can be improved with a predictable bound.
For more including source code, check https://github.com/ros-acceleration.
00:00 Community contributions and group's progress
05:28 Hardware acceleration integration in ROS 2 Humble progressing
05:56 REP-2008 update
10:30 Hardware accelerated ROS 2 perception pipelines
18:50 ROSCon 2022 Hardware Acceleration Workshop
22:25 Robotic Processing Unit (RPU)
27:32 PiCAS: New Design of Priority-Driven Chain-Aware Scheduling for ROS 2
52:20 Q&A
For a discussion about this meeting, refer to https://discourse.ros.org/t/hardware-acceleration-wg-meeting-7/24603/1.
A
So,
let's
start
with
old
business,
so
in
a
in
a
nutshell,
great
traction
of
community
contributions.
Over
the
past
month,
approximately
I'm
just
gonna
mention
three
of
the
tickets
that
have
been
most
active,
but
there's
actually
a
bunch
more
so
kudos
for
those
of
you
in
the
community
helping
supporting
and
adding
value.
Thank
you
very
much,
and
please
keep
doing
so.
A
A
The
stereo
image
proc
part
of
the
image
proc
package,
which
is
probably
one
of
the
most
relevant
packages
in
the
perception
stack
of
ross,
and
he
is
bringing
it
to,
I
believe,
hls
using
vitis,
so
amd's
technology
and
he's
doing
that
itself.
There
is
a
an
already
available
implementation
of
of
the
monocular
version
of
the
stereo.
A
Sorry,
the
image
proc
part
of
it,
but
he
is
porting
things
to
image
prop
stereo,
which
is
an
awesome
contribution,
and
things
like
this
are
definitely
what's
going
to
get
us
faster
into
more
and
more
use
cases
and
into
speeding
up
our
computational
graph.
So,
thank
you,
derek
and
keep
an
eye
on
this
project
if
you're
interested
the
second
one
is
this
project
which
is
being
led
by
sierra,
so
he's
he's
actually
working
on
pouring
and
bringing
up
to
speed
in
an
fpga.
A
According
to
what
he's
saying,
the
orc
slam
shoe
raws
to
a
note
in
in
pieces
of
code.
Actually
so
he's
doing,
I
think
pretty
interesting
work.
In
the
background
I
gave
him
some
pointers
and
now
he's
provided
a
full
description
of
how
to
create
your
own
platform
by
this
platform,
which
is
not
the
easiest
piece
to
do
and
definitely
deserves.
Maybe
additional
documentation
and
walkthroughs
he's
done
a
fantastic
job,
providing
his
recipe
and
and
thumbs
up
to
him
and
and
definitely
worth
checking.
A
If
this
is
something
you
guys
are
are
working
with.
The
third
one
is
the
ultra
96
progress,
which
I'm
hoping
to
hear
some
of
you
particularly
joe.
I
think
you
may
have
some
interesting
comments,
so
I
thought
that
pedro
was
out
of
the
game
for
a
while,
at
least
that's
what
I
got
from
him.
But
apparently
he's
he's
now
back
in
the
game
and
he's
he's
got
some
cycles.
A
Apparently,
as
far
as
I
know,
pedro
did
build
a
preliminary
firmware
layer
and
I
have
not
had
the
chance
to
test
it
fully.
To
be
very
honest:
I've
been
occupied
with
other
matters,
but
it
sounded
last
time
we
spoke
that
there's
a
push
needed
to
generate
the
firmware
artifacts
and
the
conversation
is
long.
I've
provided
lots
of
input
about
how
to
do
it
happy
to
continue
doing
it
if
needed.
A
B
C
A
Also
to
try
to
maybe
group
together,
join
the
efforts,
so
so
I
created
acceleration
firmware
ultra
96
here
it
is
so
I
created
this
repo
for
pedro
a
while
ago
and
I
think
he's
he
hasn't
made
a
release
just
yet.
So
that's
one
thing
he
should
consider
he
actually
sorry
there's
a
releasing
here,
which
is
a
draft.
I
I
I
didn't
see
that
okay,
so
maybe
this
is
something
to
check,
but
regardless
I
gave
pedro
access
to
these.
A
A
Cool
cool,
okay,
so
so
that's
awesome
and
what
I'll
do
is
I'll
mention
you
as
well
jorge
and
I
do
have
your
github
handle
so
so
I'll.
Keep
that
handy
and
also
maybe
add
you,
okay,
joe,
so
just
maybe
paste
that
as
a
suggestion
in
the
minutes
and
I'll
make
sure
that
that
happens,
and
we
move
this
forward.
I'm
excited
to
get
this
ongoing
and
it
would
be
fantastic
to
get
the
first
community
contributed
board.
A
That
would
be
pretty
cool
awesome
cool,
so
that
was
with
regard
some
of
the
community
contributions
and
traction
generated
out
of
that.
I
also
wanted
to
mention
the
fact
that
there's
been
good
progress
and
we
are
on
track
for
the
integration
with
humble.
As
you
may
know,
the
next
big
ross
release
roster
release
story
is
happening
in
may
ross,
humble
huxby
and
that's
gonna
be
actually
the
first
release.
That's
gonna
be
maintained
for
five
years,
so
it's
gonna
be
pretty
relevant.
A
The
two
contributions
I
wanted
to
highlight
the
first
one
is
the
fact
that
I've
made
some
additions
at
the
at
the
rep
2008,
which,
for
those
of
you
who
are
new
to
the
group
or
haven't
been
in
past
meetings,
it's
a
joint
effort
to
try
to
somehow
standardize
and
provide
a
reference
architecture
and
conventions.
How
to
do
hardware.
Acceleration
involves
two
in
a
vendor
agnostic
manner.
A
We
don't
want
to
be,
you
know,
starting
with
a
or
b
and
the
intent
is
to
try
to
be
as
fair
as
possible
across
vendors
and
also
to
encourage
others
to
jump
into
the
architecture
and
and
bring
their
own
boards
and
their
own
technologies.
I
added
these
contributions.
You
see
them
summarizing
here.
The
first
one
is
is
pretty
silly,
but
the
the
ones
after
including
adding
a
methodology
which
you
can
see
described
in
this
figure
are
somewhat
interesting,
and
I
encourage
you
to
check
them
out.
A
A
It
has
limited
capabilities
at
this
stage,
frankly
speaking,
because
I've
been
putting
my
my
site
time
on
it,
but
the
overall
blueprint
and
architecture
I
should
be
well
defined
enough
to
build
upon
in
future
efforts
and
and
to
build
up
additional
capabilities.
So
so
pretty
excited
with
this.
You
can
right
now
run
various
of
the
examples
that
are
living
in
acceleration,
examples,
which
is
one
of
the
packages
that
we
are
maintaining
and
yeah
together
with
with
those
last
contributions,
I'm
hoping
to
to
get
this
landing.
A
Finally
into
a
formal,
rep,
ross,
enhancement
proposal,
I
encourage
you
guys
once
again
to
have
a
look.
I
I
think
I
heard
that
joe
did
take
a
look
and
joe
feel
free
to
dive
and
put
your
comments
or
ideas
super
open
to
to
traditional
beats,
and
I
would
say
the
same
for
everyone.
This
is
a,
I
think.
It's
a
good
read.
I
don't
want
to
say
great,
but
it's
it's
definitely.
I
definitely
put
quite
a
bit
of
effort.
A
A
I
encourage
you
to
do
so
and
and
send
your
input,
and
we
keep
up,
of
course,
with
the
intention
of
merging
first
accelerators
into
official
roster
stacks
so
that
just
by
pseudo
ipt
minus
get
whatever
roster
stack.
You
actually
do
get
the
accelerators
or
capabilities
to
actually
build
accelerators
from
those
resources.
A
So
far,
the
effort
in
the
past
month
has
concentrated
into
the
release
processes
of
each
one
of
those
core
packages
in
the
architecture
of
hardware
acceleration
into
rolling,
so
that
again
they
get
shipped
into
humble
and
that's
happening
in
the
open.
And,
of
course
you
can.
You
can
follow
the
progress
and
provide
your
input
if
needed.
On
that.
A
quick,
actual
comment.
A
Sorry
I
haven't
pushed
this,
but
there's
gonna
be
an
update
on
the
community
repo
by
essentially
adding
more
clarity
at
the
sub
project
lists
so
that
we
we
can
segment
also
that
down
the
road
by
core
architectural
components,
ros
packages
and
examples
and
tools
to
actually
do
hardware
acceleration
appropriately.
That's
that's
landing.
I
just
didn't
have
the
time
to
do
it
today,
but
I'll
do
it,
maybe
tonight
or
or
tomorrow,
okay,
so
that's
it
for
what
concerns
all
business.
Any
comments
points
about
old
business.
A
So
everything
is
open
source.
You
can
check
it
out.
You
can
reproduce
it
if
you
struggle
reproducing
it
open
a
ticket
open.
An
issue
complain
shout
and
I'll
do
my
best
to
support
you
in
principle
again.
This
has
been
tested
not
just
by
myself,
but
also
by
others
in
various
teams,
actually
within
amd
and
other
groups,
increasingly
so
yeah.
In
principle.
A
This
puts
together,
as
I
explained
in
the
past,
a
very
simple
two-note
computational
graph,
which
is
doing
rectification
and
resizing,
and
what
I'm,
showing
today
to
you
all,
is
actually
a
step
forward.
Comparing
it
across.
I
think
interesting
and,
I
would
say,
leading
solutions
for
hardware
acceleration.
A
In
particular,
we
are
comparing
amd's
krs,
the
kriya
robotics
stack
and
nvidia's
isaac,
walsh,
gems,
which
are
also
open
source
and
and
ready
to
be
used,
and
we
are
comparing
that
against
the
roster
baseline,
which
stands
for
the
default.
Roster
perception
stack
and
we
can
see
some
interesting
numbers
in
here.
I
will
not
get
into
lots
of
details
for
the
sake
of
time,
but
you
can
see
how
you
can
see
in
a
nutshell
that
there
is
some
platforms
that
do
better
in
this
particular
computational
graph.
A
Of
course
I
want
to
highlight
this
is
computational
graph
specific.
There
might
be
other
computational
graphs
where
we
might
actually
see
a
reverse
situation
where
maybe
gpus
actually
outperform
fpgas,
but
for
this
particular
case
and
as
far
as
my
benchmarking
and
strictly
speaking
following
the
architecture
we've
been
defining
and
the
benchmarking
approach
defined
and
methodology
we
actually
get
up
to
or
we
get
approximately
an
8x
8x
improvement
by
using,
in
this
case,
an
fpga
versus
using
gpus.
A
And
if
you
compose
that,
all
together
and
generate
performance
per
watt
plots,
you
get
something
like
this,
which
which
is
significant,
especially
in
robotics
and
especially
in
mobile
robotics,
because
you
definitely
want
to
get
performance
with
low
power
consumption
to
ensure
that
the
autonomy
goes
goes
as
smooth
as
possible.
So
I
just
wanted
to
show
this
and
open
up
the
mic
for
others
to
to
comment
and
and
share
ideas
and
thoughts.
Particularly
gordon.
I
don't
know
if
you
have
any
comments
about
what
could
be
done?
C
Yeah,
I
think
we'd
have
to
dig
into
what
was
run
and
what
hardware
was
run
for
it.
I'm
not
entirely
surprised
that
the
you
know
the
performance
might
not
be
great
because
there's
a
lot
of
data
movement
that
has
to
happen
from
cp
to
gpu
and
back
and
that's
super
inefficient,
and
that's
one
of
the
things
that
we
we
have
to
fix
with
this
working
group.
A
So
this
is
leveraging
fpga
specific
streaming
interfaces
for
accelerating
communication,
so
that
node
to
node,
actually
interoperates
throughout
the
fpga
and
the
information
doesn't
go
back
and
forth
to
the
to
the
cpu,
which
is,
I
think,
what
gordon
was
pointing
out.
I'm
I'm
somewhat
excited
to
to
test
out.
I
think,
what's
coming
soon,
regarding
the
optimizations
of
leveraging
a
type
adaptation
in
boston,
I
don't
know
how
optimistic
I
am,
though,
to
see
like
this
improving
significantly,
but
I'm
definitely
up
for
running
this
again.
A
If
you
guys
from
the
video
have
some
hints
on
how
this
can
can
get
better,
I,
frankly
speaking,
have
used
the
I
believe,
the
official
instructions
you
have
provided
using
both
the
root
file
systems
provided
as
well
as
the
hardware.
I
didn't,
modify
the
hardware
boards
but
yeah
this.
This
is
something
we
should
revisit
and
definitely
something
that
the
working
group
is
tackling
and
should
tackle
very
well
set
on
that
so
yeah.
So
here
you
have
a
few
more
notes
about
what
I
just
explained.
A
The
super
topics,
and
also
you
have
a
discussion
around
how
super
topics
are
not
just
great
because
they
optimize
the
loss
to
message
passing
infrastructure
and
overhead,
but
also
they
are
great,
because
there
are
two
approaches
here,
specifically
specifically
for
fpgas,
but
they
may.
This
may
also
apply
to
other,
like
architectures,
including
gpus.
One
is
to
completely
re-architect
your
computational
graph,
smashing
all
of
the
source
code
together
and
then
defining
the
kernels
as
most
appropriate.
A
That's
kind
of
like
the
approach
that
I've
been
told.
Many
have
been
following
an
industry
for
quite
a
while
many
using
gpus
and
many
using
pgas
for
robotic
applications,
and
that's
also
what
I've
seen
in
this
last
year
involved
in
robotic
use
cases
with
a
big
semiconductors
company,
but
frankly
speaking,
that
has
a
huge
overhead
for
the
systems
architect.
So
what
super
topics
introduce
actually
is
complete
transparency
and
alignment
with
rush
to
default
apis.
You
need
to
modify
actually
anything.
A
All
of
the
complexity
is
handled
directly
by
these
axes
for
streaming
interfaces,
and
you
don't
need
to
re-architect
anything.
So
that's
pretty
cool.
It
comes
with
an
overhead.
It
does
slightly
worse
than
re-architecting
everything
you
can
see
that
re-architecting
everything
gets.
You
mean
wise
at
least
a
bit
better,
and
it
gets
you
with
a
with
a
more
generous
speed
up
overall,
however,
I
think
it
is
definitely
worth
the
overhead
because
it
essentially
speeds
up
significantly
also
the
development
process
by
not
having
to
modify
anything
in
your
computational
graph.
A
I
must
say,
though,
that
the
current
implementation
of
super
topics
is
very
limited,
so
a
word
of
advice
in
there.
Currently,
it's
just
a
first
prototype
which
can
evolve
and
will
evolve
as
interest
is
shown
and
as
funding
is
thrown
into
it
and
right
now,
it's
just
prepared
to
do
single
producer,
single
consumer
interaction.
A
So
if
you
actually
have
various
topics
or
various
notes,
subscribing
to
the
same
topic
and
expect
the
super
topic
to
comply
with
that,
you
actually
would
need
to
in
hardware,
make
some
modifications,
such
as
duplicating
the
streams
and
so
on
and
so
forth.
This
is
something
that
doesn't
come
right
away.
A
A
I
don't
think
all
of
you
will
be
able
to
present,
but
at
least
some
of
you
presenting
if
you
want,
if
you
will
and
if
you
wish,
I
have
a
long
list
of
contributors,
many
of
which
I
think
you've
met,
which
have
attended
to
past
working
group
meetings
as
guests
similar
to
hong
kong,
this
time
and
I'm
definitely
up
for
pulling
from
that
list,
and
some
of
them
have
actually
shown
a
product
of
interest
on
participating.
But
before
that
I
just
wanted
to
make
sure
to
count
with
everyone.
A
That's
attending
these
meetings
in
case
they
have
something
to
share
and
and
and
they
plan
to
be
in
kyoto,
because
this
is
going
to
be
an
in-person
event
and
that's
also
what
I'm,
what
I'm
rooting
for
and
what
I'm
pushing
for
I'll
be
there
for
sure,
unless
basically
it's
impossible,
but
I
I
definitely
plan
to
be
there.
A
word
of
advice
is
I'm
particularly
looking
for
these
three
topics?
Okay,
these
are
the
three
topics.
A
I'm
interested
to
hear
about
proposals,
and
so
there's
a
ticket
in
here
which
you
can
track
and
which
is
actually
going
to
evolve
in
the
coming
weeks.
Eventually,
all
of
it
will
land
into
a
google
docs
and
that's
what
will
be
used
to
ship
the
formal
proposal
for
the
workshop.
But
if
any
of
you
have
some
ideas
or
would
like
to
make
a
presentation
in
the
workshop,
please
comment
in
here
or
reach
out
to
me.
B
Can
I
make
one
point
for
people
thinking
about
attending?
I
believe
that
roscon
precedes
iros
each
year,
so
this
is
a
great
opportunity
to
double
dip
with
one
of
the
best
academic
conferences
and
robotics
as
well.
So
I
plan
on
attending
and
I
think
that
if,
if
you
have
interest
in
the
working
group,
it
would
be
great
to
go
and
attend
iras
as
well.
Well,.
A
Well
said
well
said,
yeah
and
plus
one
for
that.
That's
that's
totally
correct.
As
far
as
I
know,
and
it's
been
more
than
I
think,
eight
years
nine
years
for
me
attending
roscon,
so
quite
a
while
yeah,
it
typically
tags
along
iras
and
if
you
haven't
been
in
eyebrows,
it's
a
fantastic
venue,
more
academic,
but
there's
lots
of
industry
representation
in
there
as
well.
So
I
definitely
encourage
you
guys
to
to
be
there.
A
Roscon
this
year
is
going
to
be
awesome,
so
you
should
definitely
come
if
you're
interested
in
these
topics
cool
all
right.
So
I
want
I
still
much
more
time
for
this
topic,
but
I
I
will
be
pinging
some
people
to
ask
for
proactively
for
their
participation
if
they
want
again,
it's
totally
up
to
you.
My
objective
is
to
put
together
a
list
between
four
and
eight
talks
that,
hopefully,
is
gonna
put
a
timeline
of
about
four
hours.
A
Approximately
I'm
gonna
try
to
get
representation
from
as
many
actual
vendors
as
possible,
so
I
would
love
to
get
like
them
right
in
there.
But
besides
that,
as
I
said,
I'm
definitely
very
interested
in
getting
people
to
speak
about
use
cases
demanding
hardware
acceleration.
I
think
that's
going
to
be
very
interesting
and
also
about
new
computer
architectures
in
robotics.
It
actually
I'm
even
willing
to
consider
things
not
using
roster
in
there,
but
we'll
see
what
comes
all
right
and
last
topic
before
we
hand
it
to
the
guest
talk
is
the
robotic
processing
unit.
A
Some
of
you
might
be
thinking
that
rpu
is
a
horrible
acronym,
because
it's
real-time
opera,
sorry,
real-time
processing
unit
or
it's
a
radio
processing
unit.
I've
gotten
already
those
complaints,
and
if
you
do
have
them,
please
do
suggest
something
better.
That
would
be
my
my
mention
right
now,
I'm
not
too
pushy
on
the
acronym
itself.
A
Actually,
I
I
don't
really
care
at
this
point
and
if
we
can
come
up
as
a
group
with
something
better
than
cool,
I
think
what
what
I
do
care
about
is
to
getting
to
get
this
project
ongoing.
The
vision
itself
is
actually
to
try
to
get
together,
architectures
that
lead
to
improved
performance
in
the
usual
robotic
tasks,
particularly
sensing
perception,
mapping,
localization
motion
control,
low
level
control
and
actuation.
That's
kind
of
like
the
those
are
the
actual
targets,
and
we
will
try
to
tackle
those
by
various
use
cases.
A
Some
use
cases
may
tackle
one
or
multiple
of
this,
but
that's
essentially,
the
objective
one
aspect
to
consider
is
that
the
objective
is
not
at
least
it's
not
initially
and
we'll
see
what
happens
down
the
road,
but
the
initial
objective
is
not
to
design
a
new
physical
device
or
new
hardware.
That's
not
the
objective.
A
The
objective
is
to
access
to
use
sorry
existing
of
the
shelf
hardware,
acceleration
development
platforms
to
put
them
together,
maybe
to
compose
essentially
boards
from
different
vendors
as
the
overall
robotic
processing
unit
and
then
to
ship
or
to
use
sorry
each
one
as
appropriate
to
accelerate
the
part.
That's
more
convenient
from
a
roster
computational
perspective
that
is
going
to
be
the
goal.
At
this
stage,
the
announcement
of
the
project
has
been
made,
I'm
receiving
some
interest
already.
So
thank
you.
A
All
of
those
of
you
who
are
showing
interest
there's
a
forum
in
here,
and
it's
also
publicly
available
elsewhere.
Where
you
can
show
interest,
you
can
actually
throw
some
ideas
and
I
very
much
appreciate
if
you
can
do
it,
because
it's
it's
gonna
be
considered.
I
promise
that
my
next
step
and
our
next
step
would
be
to
try
to
define
use
cases
that
will
drive
the
development
of
the
robotic
processing
unit.
A
I'd
like
this
project
to
be
driven
by
essentially
use
cases
that
allow
us
iteratively
to
benchmark
to
know
exactly
what
we
are
improving,
how
we
are
improving
it
and
to
land
into
the
right
architecture.
Maybe
you
know
this
right
architecture
ends
up
being
using,
you
know,
fpgas
for
io
and
then
gpus
for
the
rest
or
maybe
not.
Maybe,
actually
we
use
all
only
gpus
or
only
fpgas,
we'll
see,
but
definitely
driving
things
through
use
cases,
and
particularly
through
acceleration
examples.
A
In
a
few
meetings,
I
will
start
partitioning
the
work
and
aiming
towards
fulfilling
those
use
cases
through
demonstrators
and
we'll
see
how
the
partition
happens
and
who
can
put
timing
in
what,
and
that's
also
why
it's
important
for
you
to
show
interest
and
to
show
will
I'll
mention
this
a
few
times
in
the
future
as
well.
The
project
is
going
going
to
be
constantly
open
for
sponsorships
and
collaborations.
I
think
this
is
super
important
to
ensure
the
the
fact
that
the
project
continues.
A
Otherwise
it
will
just
die
if
we
just
pull
like
free
cycles
from
from
me
or
from
someone
else.
So
that's
also
something
to
consider
and
if
there's
any
organization
out
there
with
interest,
please
reach
out
and
we'll
find
a
way
that
you
get
represented
or
that
you
can
contribute
while,
while
also
getting
something
in
exchange,
that's
pretty
much
it.
I
personally
am
very
excited
about
this
project,
and
I
know
many
people
out
there
is
as
well
so
expect
good
updates
regarding
this
all
right.
So
what
we'll
do
now?
A
E
Okay,
first
thank
you
for
inviting
us
to
present
our
work,
so
my
name
is
hanjong
choi
and
I'm
a
postdoc
at
the
university
of
california,
california
riverside
and
I'm
a
member
of
real
time
embedded
and
network
system
laboratory.
So
if
I
introduce
our
labs
shortly,
our
lab
actually
focuses
on
real-time
and
embedded
and
cyber
physical
system.
E
So
let's
get
started.
I
think
the
audience
today
here
are
already
very
familiar
with
ross
right.
So
ross
is
a
flexible
framework
for
writing.
Robot
software
and
one
of
the
most
popular
open
sources
in
academia
and
industry,
so
this
video
was
captured
in
february
this
year.
As
you
can
see,
the
number
of
users
of
ross
is
still
increasing
for
several
different
metrics,
since
it
had
been
introduced
in
2007..
E
This
is
because,
as
we
already
know,
ross
has
a
strong
software,
modularity
and
compatibility,
so
it
provides
lots
of
software
tools
and
libraries
and
best
practices
across
a
wide
variety
of
robotic
platform.
However,
over
the
decades
the
first
version
of
ross
has
exposed
the
limitation
in
real
time
supports.
E
Then
the
question
is
why
real-time
capability
is
important
for
ross
framework.
So
here
the
last
video
describes
the
architecture
of
authorial
ai,
auto
air
ai
is
the
all-in-one
open
source
software
for
autonomous
driving,
and
that
is
built
on
was
one
so
on
top
of
a
ros
as
a
middleware
authorial
consists
of
individual
modules
in
an
application
layer
such
as
the
sensing
and
perception
and
planning
and
control,
and
those
modules
are
connected
with
data
dependency
and
forms
of
processing
chain.
E
So
here
is
a
good
motivational
example
from
autonomous
speaker,
let's
suppose,
a
scenario
that
a
beaker
tried
to
stop,
because
it
recognizes
an
obstacle
in
front
of
you.
Generally,
the
total
stopping
distance
consists
of
two
phases.
First,
at
the
time
when
a
driver
or
machine
recognizes
an
obstacle
and
apply
a
break.
E
I
call,
though,
which
is
the
perception
reaction
distance.
The
second
phase
is
a
deceleration
from
the
initial
speed
of
the
vehicle
after
the
brake
starts
to
operate,
and
this
braking
deceleration
distance
is
typically
determined
by
the
initial
speed
and
the
dynamics
on
and
physics
of
the
vehicle.
So
actually
we
cannot
control
it.
However,
in
autonomous
ecosystem,
the
perception
reaction
distance
can
be
varied
by
the
response.
Time
of
the
perception
pipeline
of
the
system
and
such
a
pipeline
is
composed
of
complex
information
flow
in
the
form
of
a
chain.
E
So
this
motivates
the
development
of
second
generation
of
laws
in
the
community,
so
ross
2
has
released
its
first
long-term
support
version
in
2017
and
already
on.
Most
concepts
are
inherited
from
the
original
voice.
Roster
has
completely
reflected
the
original
loss
framework
and
moving
from
los
1
to
verse.
2
has
many
benefits
related
to
real-time
capability,
quality
of
service
and
security,
and
as
arena
wants
to
support
the
dds.
However,
I
was
to
steal
some
limitations,
especially
in
the
real-time
perspective.
E
E
Second,
in
addition
to
that,
currently,
rost2
has
no
systematic
resource
allocation
policy
and
does
not
define
any
property
on
chain.
That
means
that
there
is
no
general
guideline
about
how
to
allocate
those
two
executors
and
how
to
assign
excuse
priority
and
maps
those
executor
to
available
cpu
code
so
on
improving
entertainment,
latency
of
change
for
loss.
2
is
not
a
trivial
problem,
because
we
need
to
consider
the
entire
entertainment
behavior
of
all
scheduling,
related
entities
at
the
same
time,
so
this
actually
motivates
us
to
design
a
new
real-time
scheduler
for
bus
2..
E
So
before
introducing
our
new
scheduler,
I
will
first
explain
more
about
watson
architecture
as
a
background,
especially
what
are
scheduling
related
abstractions
involves
two
and
what
are
the
challenges
from
them
now?
Basically,
our
roster
consists
of
multiple
abstraction
layers,
and
what
we
are
interested
in
is
fundamental
scheduling,
related
entities
across
all
those
layers.
E
The
first
there
is
a
callback.
The
callback
is
a
minimal
schedulable
entity
involved
and
we
modeled
a
real-time
callbacks,
either
a
timer
or
regular
callbacks,
based
on
whether
it
is
triggered
by
a
timer
or
external
events
and
callers
are
scheduled
in
a
non-preemptive
manner.
Within
an
executor,
a
node
is
are
collection
of
callbacks
and
node
is
a
minimum
allocation
unit.
Two
executors,
then
these
nodes
are
allocated
to
an
executor
which
is
an
os
level
scheduler
entity
running
on
cp,
of
course,
like
a
draft,
so
these
keywords
are
preemptable
in
our
work.
E
So
we
assume
that
the
start
callback
is
a
timer
convex
and
others
are
regular
content,
and
we
also
assume
the
chain
has
a
semantic
priority
given
by
the
system
designer
based
on
the
criticality.
In
the
system,
so
the
scope
of
our
work
here
is
how
to
schedule
and
allocate
schedular
entities
to
improve
latency
of
change.
E
So
with
these
scheduling
entities
a
lot
of
challenges
involved
too,
we
first
conducted
a
simple
experiment:
we
used
two
safety
critical
chain
using
tank
callbacks
here
and
assume
that
chain.
One
is
more
critical
than
chain
two,
so
here,
tau,
1
and
tau
4
are
timer
corrects
and
they
arrive
at
every
one.
Second
and
all
the
other
callbacks
are
regular
quarterbacks
and
they
are
triggered
by
its
prior
callbacks
in
the
same
chain.
First,
we
perform
the
case
where
all
corvettes
are
in
a
single
secure.
E
So,
as
discussed
in
the
radius
of
our
wall,
2
has
a
unique
scheduling.
Behavior
within
executor,
as
a
timer
corrects,
are
always
scheduled
first.
So
we
also
have
observed
this
behavior
in
our
experiment.
For
example,
here
tile
4
and
tau
1
are
execute
first,
whenever
it
is
ready.
This
actually
makes
the
other
regular
profits
priority
ineffective.
E
E
So
then,
this
time
we
allocate
the
corvettes
of
each
chain
to
a
separate
executor.
Actually,
there
is
no
general
guideline
to
assign
priority
or
executors
on
invoice2.
However,
in
an
intuitive
way,
we
can
allocate
all
callbacks
of
chain
one
to
the
to
the
highest
priority,
secure,
because
chain,
one
is
more
critical
than
chain
two.
Therefore,
as
we
can
see
in
this
scheduling,
video
chain,
1
always
preempts
chain
2..
E
However,
this
method
does
not
resolve
the
problem
because
in
this
case,
chain
2
without
unexpected
high
latency
here.
So
this
is
mainly
because
for
single
chain,
a
prior
chain
instance
interferes
other
instances
that
were
released
later.
This
is
like
also
called
the
self-interference,
so
a
priority
assignment
for
executors
is
another
challenge
in
current
lost
framework.
E
So
we
proposed
a
new
real-time
scheduling
and
analysis
framework
for
roster,
or
we
called
it
a
pickaxe
which
is
a
priority-driven
chain
of
rail
scheduling.
So
the
key
idea
is
that
this
approach
prioritizes
a
critical
computation
change
across
complex
abstraction
layers
of
laws
too,
to
minimize
the
endocrine
latency
of
critical
change.
So
a
picasso
is
redesigned
from
laws
to
deport
scheduling,
to
resolve
these
issues.
I've
mentioned
in
the
previous
slides.
E
Basically,
it
has
two
goals.
First,
a
critical
chain
execute
first
to
avoid
fairness-based
scheduling.
Second,
if
we
consider
a
single
chain,
a
chain
instances
should
be
scheduled
in
their
arrival
order
to
prevent
saving
appearance.
So
to
achieve
these
two
goals.
Picas
framework
has
a
correct
scheduling,
strategy
within
and
across
executives,
and
to
realize
these
scheduling
strategy.
E
E
First,
we
proposed
six
product
scheduling
strategy
within
and
across
exclusives.
Also,
the
idea
is
that
for
single
chain,
the
priority
of
high
indexed
callback
is
higher
than
low
indexed
code,
so
we
can
guarantee
that
prior
chain
instance
completes
its
execution
before
a
newly
released
chain.
Instance
start
execution.
E
So,
for
example,
if
we
say
there
is
this
a
newly
released
call
callback
taiwan
here,
then
it
cannot
start
the
exclusion
until
previous
release.
Powerful
here
completes
its
execution,
because
the
priority
of
tau
1
is
lower
than
tau
4
and
if
there
exists
multiple
chains
like
here,
are
we
assigned
higher
priority
to
call
backs
that
are
from
more
critical
chain?
So
we
also
propose
the
strategy
for
change
of
running
across
executors
in
the
bottom
part.
E
So
to
realize
these
scheduling
strategy,
we
found
solutions
in
two
aspects.
First,
a
prior
to
assignment
of
callback
and
second
are
loaded
location.
A
prior
to
assignment
of
callbacks
can
be
done
intuitively
to
realize
the
scheduling
strategy.
We
first
sort
a
change
in
an
ascending
order
of
the
semantic
priority
then
assign
priority
to
callbacks
from
low
price
number
to
high.
E
Then
we
also
proposed
a
chain
of
real
node
allocation
algorithm.
Basically,
this
algorithm
allocates
given
nodes
to
executors
and
then
maps
those
executors
to
available
cpu.
Of
course,
this
algorithm
does
not
just
simply
allocate
the
scheduling
entity.
The
main
goal
of
this
algorithm
is
to
minimize
interference
between
change
by
allocating
all
nodes
to
it,
to
one
chain
to
the
same
cpu
core
whenever
it
is
possible,
but
by
doing
so
it
can
actually
reduce
the
endocrine,
latency
or
critical
change.
E
So
so
far
I
explained
the
overall
design
of
our
new
scheduling
framework.
Now
we
run
the
same
vocal
set,
which
has
the
two
chains.
I've
explained
at
the
challenges
slide
previously.
The
first
case
is
using
a
single
executor,
and
the
second
case
is
two
executors
for
each
chain.
For
both
cases
we
have
observed
the
first
chain,
which
is
more
critical
chain,
always
execute
first
so
or
we
can
say
that
pick
us
does
support
prior
to
delivery,
scheduling
and
not
a
value.
E
Space
schedule
and
also
a
prior
chain
instance,
always
completes
its
execution
before
newly
released
chain
instance
starts.
So
we
also
can
say
that
the
pickaxe
supports
the
chain,
oreo
scheduling
so
based
on
these
results,
picas
outperforms
the
walls
to
default
scheduling
significantly,
especially
in
terms
of
the
entertainment
latency,
and
also
overcomes
the
aforementioned
challenges.
E
Picas
also
has
equipped
entertainment,
latency
analysis
framework.
Latency
analysis
consists
of
two
steps:
first,
computing,
the
worst
case
response
time
of
each
segment
of
a
chain.
Here,
a
segment
means
that
a
subset
of
a
chain
on
one
cpu
core,
then
in
the
second
step
we
can
add
up
all
segments
worst
case
response
time
for
the
same
chain
to
get
the
entire
latency.
E
So
here,
as
we
can
see,
right
bottom
figure,
I
actually
tested
the
same
workload
set
for
was
to
default
and
was
to
pick
us.
The
green
bar
here
is
analysis.
Time
of
picas
and
the
yellow
bar
represents
the
latest
latency
analysis
technique
for
roster
default
scheduler,
so
because
analysis
time
is
much
faster
than
the
latest
analysis.
E
E
We
use
the
case
study
that
is
inspired
by
indoor
navigation
stack
of
front
test
speaker
here,
as
illustrated
in
in
the
bottom
here.
The
workload
for
this
case
study
consists
of
six
real-time
safety
critical
chain
and
then
we
add
another
six
best
apple
chain.
So
totally
the
12
chains
are
running
on
the
four
cpu
equal
system.
E
This
figure
shows
the
maximum
observed
latency
and
the
analyzed
latency
here
the
y-axis
means
the
androgen
latency,
so
the
lower
is
the
better
here.
The
purple
color
here
represent
the
measured
latency
result
of
pickaxe
and
orange.
Here,
a
bar
is
measured
latency
for
was
too
deeper,
so
as
we
can
see,
picas
outperforms
the
others
on
most
real
time
change,
but
as
the
semantic
priority
of
chain
is
getting
low,
what
I
mean
is
going
to
the
best
apple
chain.
E
So,
as
we
can
see,
the
analysis
on
provides
tighter
upper
bound
for
real
time
change,
which
means
that
pickaxe
is
more
practically
usable,
so
we
also
integrate
our
pcas
framework
into
so-called
reference
system,
so
reference
system
was
developed
by
apex
ai,
for
was
to
execute
benchmarking,
and
it
was
first
introduced
and
roscon
2021
real-time
secure
workshop
and
we
we
actually
was
were
invited
to
introduce
our
pickaxe
work
to
that
workshop.
So
I
integrated
picass
to
their
reference
system
and
evaluated
on
a
raspberry,
pi
4
platform.
E
E
So
here
the
bottom
left
figure
describes
the
average
latency
latency
result
of
that
hot
topic
path
and,
as
we
can
see,
lost
to
pickaxe,
with
a
single
executor
achieved
up
to
86
percent
reduction
in
the
latency
compared
to
was
to
default
a
single
threaded
executor
and
the
right
figure
shows
the
jitter
of
behavior
planar
node
or
which
is
a
blue
color
node
here
and
the
desired
timing.
Behavior
of
this
node
is
as
cyclic
as
possible
and
we
said
that
its
frequency
is
100
milliseconds
here.
E
E
E
So
here's
the
conclusion
in
this
work:
we
actually
proposed
a
new
practice
event
chain
over
scheduler
with
its
entertained
latency
analysis
framework.
So
our
experimental
results
represent
that
our
proposed
scheduler
outperforms,
the
existing
lost
scheduling
with
respect
to
the
entertained
latency
under
practical
scenarios.
E
A
Thank
you.
That
was
a
fantastic
presentation,
very,
very
impressive
and
even
more
impressive.
As
the
paper.
I
encourage
everyone
to
go
and
read
it.
I
enjoyed
it
very,
very
much.
I
think
these
folks
go
into
a
very
complex
topic
into
very
details
and
and
in
my
humble
opinion,
this
is
one
of
the
best
pieces.
I've
read
so
far
and
I've
been
working
on
ross,
2
real
time
for
for
quite
a
while
now
so
so
kudos
and
congrats
to
you
and
to
your
team,
because
I
think
this
is
this
is
very
impressive.
A
Maybe
to
keep
to
kick
off
the
q.
A
I've
got
a
quick
question
regarding
your
case
study
shoot
the
reference
system
slide.
You
you
showed
you.
You
mentioned
the
fact
that
apexi
disclosed
reference
system
indeed,
and
you
showed
a
comparison
between
the
default
scheduler.
Sorry,
the
default
executor
enrolls
you
and
your
picas
one.
Can
you
comment
on?
How
does
your
implementation
compare
to
other
executors
because
there
are
others
available
out
there
in
the
community?
I
think
apex
also
has
their
own
executor.
E
Actually,
for
the,
if
we
run
the
reference
system,
actually
they
provide
the
the
the
key
performance
indicator
inject
like
the
hot
topic
path,
latency
and
the
behavior
planner
is
the
jitter
and
the
number
of
dropped
messages
and
so
on,
and
so
we
actually
create
our
kickass
framework
to
their
reference
system
and
compare
it.
Is
the
default
bus
to
execute
so
on
that
workshop?
Actually
there's
another
teams
from
other
on
the
industry
in
the
university
and
they
implement
their
own
security
as
you
can
see,
so
they
are.
E
They
also
imply
compared
their
own
skill
to
us
to
deport
these
cures,
so
we
have
a
bunch
of
the
experimental
results
from
that
workshop
and
so
yeah.
I
think
the
workshop,
the
the
there's
also
the
the
video
record
and
you
can
actually
revisit
the
workshop
with
the
video,
and
so
there
will
be
more
more
information
about
the
evaluation
result
and
so
on.
A
A
I
believe
we've
been
reasonably
successful
doing
so
because
it
allows
us
to
trace
not
only
sorry
trace
but
also
benchmark
out
of
information,
that's
generated
while
the
system
is
running
in
a
gray
box
manner,
so
we
can
generate
the
actual
data
and
then
afterwards
post-process
it
to
generate
the
benchmarks,
which
is
pretty
cool
and
also
lttng
introduces
a
super
super
low
overhead.
So
I
was
just
curious.
Can
you
comment
slightly
on
on
what
have
you
used
and
how
do
you
measure
things.
E
Is
that's
pretty
good
question
actually
for
our
own
experiments?
We
don't
use
the
httng
the
tracing
tools.
We
actually
implement
our
own.
The
tracing
like
that.
It's
very
simple
like
before
the
start,
the
execution
of
the
callback
we
actually
printed
the
the
time
and
the
end
of
the
latency
we
printed
another
timer
by
printing.
So
actually
we
just
simply
captured
the
entertainment
latency.
So
we
didn't
use
any
the
special
on
the
tracing
tools
and
for
this
the
reference
system.
They
also
used
the
similar
approach
like
we
did,
so
they
don't
use
the.