►
From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #9
Description
Ninth meeting of the ROS 2 Hardware Acceleration Working Group (HAWG).
We touch on a) ROS 2 Humble transition and b) analysis and visualization of the ROS 2 flow. As part of the meeting, we had Christophe Bourque Bédard telling us about extensions to `ros2_tracing` with the analysis and visualization of the flow of messages across distributed ROS 2 systems. Their method detects one-to-many and many-to-many causal links between input and output messages, including indirect causal links through simple user-level annotations, providing valuable timing and scheduling information to further study and improve the any ROS 2 system. This is very valuable for hardware acceleration purposes.
For more including source code, check https://github.com/ros-acceleration.
00:00 Welcome
00:20 Intros
06:15 Working group growth
11:03 Acceleration Robotics sponsoring the HAWG
13:00 REP 2000: ROS 2 Releases and Targeted Platforms, Yocto/OpenEmbedded
17:53 REP 2007: Type adaptation
18:53 REP 2008: Hardware Acceleration Architecture and Conventions
20:18 REP 2009: Type negotiation
25:10 Christophe Bourque Bédard, FogROS 2: Message Flow Analysis with Complex Causal Links for Distributed ROS 2 Systems
51:05 Q&A
For a discussion about this meeting, refer to https://discourse.ros.org/t/hardware-acceleration-wg-meeting-9/25499.
A
All
right
so
welcome
everyone
to
the
ninth
roster
hardware,
acceleration
working
group
meeting
happy
to
have
everyone
in
here
and
hopefully
we'll
have
a
great
meeting.
So
let's
kick
it
off
as
usual.
Let's
start
with
some
introductions
for
those
of
you
that
are
new
to
the
group.
I
would
like
to
say
a
few
words.
A
B
India,
so
I
have
a
robotics
consultancy
called
techno,
so
I've
been
working
with
ross
for
around
five
six
years
now,
so
we
are
actually
now
building
a
hardware
agnostic
solution
for
ams
more
like
an
os
on
top
of
ross-
and
this
is
quite
interesting
for
us,
because
optimization
of
any
open
source
software
is
the
big
challenge
which
many
face
and
of
course
you
can
do
it
on
a
software
level.
But
having
hardware
added
to
that
makes
will
add
more
value,
so
yeah
and
we
just
got
a
kv-260
board.
A
C
D
Also
I
can.
I
am
baker,
I'm
working
working
on
the
edward
robotics,
someone
wrote
wrote
up
to
there,
but
you
can
repeat
write
my
name
again,
but
it's
not
problem.
We
are
focused
on
intro
logic
sticks
except
especially
logistic
area,
and
I
am
focused
on
the
general
lead
navigation
tools.
We
are
working
on
that
and
I
I
want
to
participate
this
class
master
class
or
this
work.
Thank
you.
B
F
Hi
I'm
muhammad
from
pakistan.
I
am
currently
studying
bachelor's
in
software
engineering
from
referee
university
and
we
have
final
year
project
in
self-driving
vehicle.
We
just
simulate
a
vehicle
and
using
the
ross
one,
but
I'm
also
interested
in
this
class.
This
is
ross2
latest
updates,
so
I
want
to
join
this
workshop.
A
G
Hi,
I'm
sani
I'm
from
india,
so
I
have
been
working
prior
to
visti
vistion
corporation,
where
I
was
using
rose.
One
rose
two,
so
I
have
used
different
version
of
rose
two
as
well
and
right
now,
I'm
working
with
hair
maps.
Where
means
we
are
means,
I'm
part
of
some
autonomous
driving
group
as
well.
So
maybe
we
will
be
using
rows
rose2
again
so
y
means
I
am
here
to
explore
the
new
features
of
flows
2
and
for
networking
as
well.
H
F
A
A
A
There's
a
strong
participation
from
industry,
which
is
something
that
makes
me
pretty
happy.
There
is
big
names
in
here
that
you
can
see
from
big
semiconductor
companies,
big
technology
companies,
system
integrators
as
well
as
big
technology
makers.
So
super
super
happy-
and
this
is
by
the
way,
not
a
complete
list.
Of
course,
there's
there's
many
more
all
of
them,
which
are
relevant.
There
are
also
quite
a
few
of
you,
coming
from
academia
from
small
and
big
universities,
everyone's
welcome,
so
that
is
also
something
pretty
nice,
which
shows.
A
So
here
you've
got
the
sources
and
a
description
of
every
single
one
of
them.
And
again
I
wasn't
expecting
this,
but
there
seems
to
be
a
significant
amount
of
visitors
and
reactions,
so
so
looking
forward
to
see
where
this
brings
us
and
for
future
growth
now
looking
at
the
numbers,
certainly
one
asks
in
this
case
myself:
what
can
we
do
better
and
in
this
case,
there's
two
things
where
definitely
I'd
like
to
bring
your
attention
and,
if
possible,
maybe
some
contributions.
The
first
one
is
contributions.
A
This
is
something
that
I'm
particularly
keen
on
as
more
and
more
people
start
using
the
tools
we
are
generating
and
the
utilities
I
am.
I
definitely
would
be
very
interested
to
hear
opinions,
views,
reviews,
criticisms,
especially
from
those
ones
trying
to
bring
real
stuff
real
robots
into
the
market.
We're
having
guest
speakers
we'll
have
one
later
on
within
the
session
today,
which
I'm
very
excited
about,
and
typically
that's
going
to
continue,
but
within
those
sections
I
would
love
to
get.
A
So
if
you
would
like
to
take
a
more
active
participation
reach
out
to
me
and
we'll
look
into
possibilities
there,
so
that
is
for
what
concerns
the
working
group
growth.
Let's
look
at
the
second
administrative
or
old
business
item,
which
is
the
robotic
processing
unit,
which
is
a
project
we
kick
off
a
few
meetings
ago.
Unfortunately,
there
hasn't
been
a
lot
of
progress.
A
Over
the
last
month,
I've
been
very,
very
focused
on
the
humble
release
which
we
will
discuss
just
in
a
bit,
the
roster,
humble
huxby,
which
is
the
latest
rost2
and
the
essentially
the
lts,
the
long-term
support
version
of
ros2,
which
is
going
to
be
supported
for
five
years.
So
those
of
you
working
across
industry
or
for
academia
definitely
recommended
to
check
out
humble
for
new
projects.
A
So
essentially,
I've
been
super
focused
on
this
and
bringing
things
up-
and
I
haven't
had
the
time
to
take
a
look
at
contributions
as
well
as
to
contribute
myself
on
this,
so
pushing
this
out
to
to
later
meetings
again,
the
ticket
is
out
there,
so
anyone
interested
in
this
particular
sub
project
can
go
to
that
and
comment
or
contribute
jumping
into
progress,
review
and
new
business
I'll
touch
into
two
topics.
Actually
sorry
choosing
to
one
with
different
subtopics.
A
These
are
kind
of
like
community
conventions
or
community
standards,
from
a
ros
perspective
which
which
tried
to
standardize
how
things
should
be
done
in
the
ros
community,
rep,
2002,
sorry,
rep
2000
speaks
about
roster
releases
and
targeted
platforms.
If
we
take
a
look
at
the
one
that
is
supposed
to
be
for
humble
for
the
new
release
we
can
see
here
in
this
last
column
that
open
embedded
was
targeted.
A
Unfortunately,
there's
been
some
glitches
in
terms
of
support
for
meta
rose,
which
is
the
project
that
groups
together
the
yocto
recipes
for
enabling
ross
in
open,
embedded,
slash
kyoto
based
constructs
and
projects.
So
we
took
on
that
essentially
work
and
for
the
last
month
I've
been
pretty,
I
would
say,
intensively
working
on
ensuring
that
we
had
enough
capabilities
to
bring
a
preliminary
level
of
support
to
yokto
with
humboldt
the
contributions.
A
Sorry,
the
contributions
of
this,
I
think,
have
been
pushed
to
the
meta
ros
repo,
particularly-
and
you
can
check
this
out-
there's
still
an
open
pull
request
in
here
and
just
by
looking
at
it,
you
can
see
how
essentially
well
more
than
one
thousand
files
were
modified
and
more
than
60
000
lines
of
code
were
changed.
So
it's
a
it's
a
significant
amount
of
changes.
Within
this
repository,
as
I
said,
it
was
lots
of
work.
A
A
A
So
if
any
of
you
is
looking
into
building
production-grade
robots
using
yokto,
you
may
want
to
check
this
out
right
now.
There's
preliminary
support
tested
with
certain
boards
that
I
will
discuss
just
right
now.
Of
course,
this
is
by
no
means
final,
it's
still
under
review
and
everyone's
welcome
to
contribute.
A
Actually,
I
want
to
hide
like
that,
since
humble
is
possibly
the
recipes
of
humble
need
to
be
updated,
because
this
was
worked
out
for
the
humble
launch,
but
then
soon
after
that,
there
should
have
been
a
a
new
release
from
the
build
farm
coming
out
of
open
robotics.
That's
my
understanding
at
least,
and
that
possibly
just
modified
some
of
the
hashes
of
the
packages
which
possibly
will
lead
to
some
changes
in
the
recipes.
A
As
I
was
saying,
we've
brought
support
for
certain
boards
and
their
corresponding
bsps,
some,
which
are
officially
supported
by
the
manufacturer,
some
others
which
aren't
so
in
the
case
of
amd.
There
are
a
few
boards
which
were
tested
against
the
official
bsp
from
the
manufacturer.
In
some
other
cases,
it
was
tested
against
community
contributions,
community
bsps,
but
regardless
the
capabilities
were
tested
and
led
to
root
file
systems
which
included
roster,
humble
with
fully
fledged
capabilities
and
based
on
on
yokto,
so
yeah.
A
That
is
that
this
is
closed
right
now,
but
anyhow,
it
was
an
interesting
effort
that
led
to
some
some
interesting
results
for
each
one
of
these
boards.
You
have
a
write-up
that
we
made
at
acceleration
robotics.
That
explains
the
rationale
why
this
is
important.
Some
of
the
actual
features
that
you
get
by
building
your
own
linux
distribution,
and
then
I
walk
through
on
how
to
do
it
step
by
step
again.
It's
not
bulletproof
because
of
the
fact
that
some
recipes
may
change.
So
if
anyone
has
some
bandwidth,
contributions
are
super.
A
Welcome
now
wrap
2000
and,
in
particular,
the
enablement
of
yokto
base
flows
or
open
embedded
ones.
It's
certainly
one
of
the
big
highlights
hardware,
acceleration
wise
that
I
think
we
can
claim
for
what
concerns
humble.
The
second
one
is
rep
2007
claimed
as
type
adaptation,
which,
in
a
nutshell,
is
an
extension
of
the
ros
client
library
for
c
plus
plus
that
will
make
it
easier
to
convert
between
essentially
ros
types
and
custom
use,
certified
types
for
topics,
services,
actions
and
and
other
things.
A
In
a
nutshell,
let
me
bring
this
again,
which
moved
so
right.
2007
was
pushed
by
open
robotics.
You
can
check
it
out
online
in
here.
There
are
some
examples
and
also
open
robotics,
released
some
examples
in
their
corresponding
repositories,
which
show
you
how
to
how
to
use
it
and
how
to
leverage
it
type.
Adaptation
in
nature
allows
to
build
computational
nodes
that
are
somewhat
agnostic
to
whatever
types
are
used
within
the
accelerator
and
make
lives
of
developers
easier,
so
encouraging
everyone
to
check
this
out.
A
Definitely
one
of
the
blueprints
of
what
we
are
building
in
here.
Hardware,
acceleration
wise.
The
third
item
I
wanted
to
highlight,
which
is
unfortunately
still
being
reviewed
as
a
pull
request,
but
we
have
discussed
this
in
the
past.
Meetings
is
red
2008,
which
is
the
roster
hardware,
acceleration
architecture
and
conventions
this
rep.
This
was
announcement
proposal,
aims
to
bring
together
all
of
the
other
aspects
and
reps
so
that
we
have
a
consistent
way
of
understanding
how
hardware
acceleration
works
in
the
bus
two
world.
A
So
you
can
expect
that
this
rep
will
grow,
will
improve
from
will
enhance
over
time,
possibly
including
and
pointing
towards
other
reps,
such
as
alright
2000,
already
2007
that
we
just
discussed
again.
We
discussed
this
in
the
past,
so
I
will
not
stop
too
much
in
here,
but
in
a
nutshell
it
from
a
developer's
perspective.
It
explains
how
essentially
building
accelerators
should
happen
in
a
somewhat
simplified
manner,
a
roster-centric
way,
while
at
the
same
time
for
silicon
vendors,
it
provides
a
recipe
on
how
to
attach
themselves
to
the
roster
ecosystem.
A
So
they
are
consistent,
interoperable
and
don't
break
up
this
assumption
of
trying
to
be
agnostic
from
an
accelerator
perspective,
so
that
developers
can
jump
from
one
accelerator
to
another
without
having
to
reinvent
the
wheel,
which
is
at
the
end
of
the
day.
I
think
what
we
want
to
deliver
for
our
vars
users
and
what
everyone
wants
to
use
to
avoid
lockdowns,
the
last
one
which
was
recently
displayed,
though,
frankly
speaking,
it's
been
contributed.
A
A
This
is
a
super
super
cool
feature
that
tackles,
I
think,
one
of
the
big
issues
for
what
concerns
the
current
roster,
which
is
cpu-centric
most
computational
graphs,
are
very
cpu-centric
and
breaking
that
assumption
and
bringing
it
to
a
more
gpu
or
fpga
friendly
environment
require
you
to
essentially
play
around
with
shared
memory
or
use.
Some
special
constructs
leveraging
xe4
interfaces.
A
So
I
think
type
negotiation
plays
an
interesting
tune
because
it
proposes
an
abstraction
layer
that
lives
at
the
top
and
which
simplifies
how
this
can
be
digested
from
a
roster
user's
perspective,
we'll
touch
on
rep,
2009
and
also
on
the
rest
of
them
in
more
detail
in
the
next
meeting,
because
I
was
actually,
I
was
actually
asked
repeatedly
to
speak
about
this
so
we'll
book
plenty
of
time
to
touch
on
this
in
the
next
session.
But
for
now
I
think
it's
important
to
highlight
these
are
some
of
the.
A
One
of
the
topics
we've
been
discussing
historically
and
repeatedly
is
benchmarking
benchmarking
in
the
sense
of
ensuring
that
we
do
know
what
is
our
cpu
baseline
from
a
computational
perspective,
as
well
as
comparing
fairly
against
accelerated
versions
of
dash
and
also
comparing
different
accelerators,
possibly
coming
from
different
silicon
vendors,
so
that
we
pick
what's
best
for
our
use
case,
both
from
from
essentially
latency
perspective,
as
well
as
from
a
power
perspective
and
performance
perspective.
So
on
that,
I'm
super
super
super
happy
to
have
with
us
today.
A
A
The
reason
why
I'm
so
excited
is
because,
when
you
work
with
hardware
acceleration,
you
do
realize
that
benchmarking,
and
and
and
essentially
getting
clarity
on
what
you're
doing
is
instrumental
for
the
development
of
accelerators
and
often
benchmarking
becomes
pretty
tricky
in
the
roster
ecosystem,
because
different
groups
and
different
roster
stacks
are
different,
are
using
different
mechanisms
for
instrumenting
the
code
and
benchmarking
staff.
So
the
exciting
thing
about,
essentially
what
christophe
will
tell
us
about,
is
that
they've
been
walking
in
towards
a
direction.
A
That's
been
said,
I
think,
for
the
last
I
would
say
quite
a
few
years
since
roscon
2016,
or
something
like
that,
where
I
think
ingo
first
spoke
about
lttng,
so
so
yeah
they've
been
using
the
linux
tracing
toolkit,
proposing
tools
on
top
to
facilitate
its
use
and
now
he's
tackling.
I
think
one
of
the
really
really
nice
issues
which
is
once
you
have
instrumentation
and
you
can
fetch
the
data.
A
Then
how
do
you
map
all
of
that
static
data
to
the
actual
computational
graph
right,
and
how
do
you
visualize
and
introspect
it
to
make
sense
out
of
what
you
want
to
optimize
and
anyone
that,
like
me,
has
been
creating
accelerators
for
the
last
year
would
definitely
resonate
with
these
words,
because
one
thing
is
getting
the
performance
up.
Another
thing
is
is
essentially
making
sense
of
that
in
correspondence
with
the
computational
graph.
A
So
christophe,
I
don't
know
if
I'm
saying
maybe
too
much,
and
you
will
tell
us
less
out
of
that.
I
hope
not,
but
anyhow
super
super
happy
to
introduce
christoph
and
his
speech
about
his
paper
message
flow
analysis
with
complex
casual
links
for
distributed
roster
systems.
So
crystal
the
stage
is
yours
and
and
again
kudos
to
have
you
here.
K
K
K
So
here's
a
summary
of
my
presentation.
I
will
start
with
the
short
introduction.
Then
talk
about
tracing
just
a
little
bit
just
to
make
sure
we're
on
the
same
page.
Then
I
will
shortly
quickly
introduce
roster
tracing
and
talk
about
tracing
routes
to
itself,
and
I
will
move
on
to
the
main
part
of
the
presentation,
which
is
the
message
flow
analysis.
K
K
So
of
course,
in
robotics
we
have
many
different
types
of
applications.
We
have
safety,
critical
applications,
they
can
be
distributed
and
connected
over
network
and
the
key
elements
here
that
is
that
they
for
systems
like
rust.
They
use
publish
subscribe
system
patterns
so
they're,
quite
modular
and
higher
level
scheduling
of
tasks
is
quite
challenging
and,
of
course,
the
systems.
We
often
want
them
to
hit
certain
performance
targets
and
they
can
have
also
performance
constraints.
K
And
so
to
do
this
we
need
debugging
tools,
so
we
often
debug
with
simply
with
gdb
or
we
use
sometimes
lag,
which
are
usually
human,
readable
strings.
So
quite
simple:
it
works,
but
it's
quite
rudimentary
so
for
distributing
for
distributed
systems.
Then
you
want
to
like
you
wonder
how
do
you
analyze
the
distributed
system
as
a
whole?
So
there's
also
some
real-time
constraints,
and
you
want
to
try
to
use
this
in
production
because
there's
some
otherwise,
you
can
hit
some
quite
serious
problems.
K
So
when,
especially
when
you
try
to
extract
timing
related
data,
if
you
influence
or
affect
the
application,
then
you
might
get
totally
wrong
results.
So
you
need
a
great
tool
from
this,
so
this
leads
us
to
tracing,
so
the
tracing
is
really
the
best
tool.
I
think
for
this,
so
it
allows
us
to
extract
runtime
execution
information,
which
is
usually
usually
low,
level
information,
so
not
humanitarian
strings.
It's
usually
raw
or
binary
data
like
integers,
arrays
and
sometimes
strings,
but
it's
usually
data.
That
is
not
meant
to
be
read
by
humans
right
away.
K
So
it's
quite
useful
as
when
issues
are
hard
to
reproduce.
So
you
can
usually
trace
for
some
time
or
all
the
time,
and
then,
when
you
hit
an
issue,
you
can
just
look
at
the
trace
and
start
debugging,
pretty
much.
Of
course.
As
I
said,
we
want
to
use
this
and
in
production
sometime
or
use
this
to
get
a
valuable
data.
So
you,
of
course
you
want
to
minimize
your
overhead.
K
K
And
so
raster
tracing
is
a
collection
of
tracing
instrumentation
and
tools,
that's
closely
integrated
into
rust2,
so
it
has
tools
to
instrument
the
core
browse
tool
with
calculating
g
again,
and
it
has
tools
also
to
configure
tracing
without
the
ltng
and
there's
a
paper
if
you
want
to
look
at
it
and
there.
So
I
will
also
have
a
link
at
the
end,
and
so
the
main
way
to
configure
tracing
using
the
rost2
ecosystem
is
using
a
trace
action
for
a
launch.
K
K
So
you
can
also
use
it
in
xml
and
yaml
files,
of
course,
to
use
those.
So
as
for
the
instrumentation,
we
wanted
to
extract
information
about
all
the
layers
of
ros2
and
know
about
the
interaction
between
them,
but
the
layers
make
it
hard
to
get
the
full
picture.
So
we
need
to
gather
the
small
bits
of
information
here
and
there.
So
this
is
usually
the
case
for
tracing
so
but
especially
the
case
with
crosstalk
given
all
of
its
layers,
and
then
we
analyze
this
data
externally
at
the
end
are
usually
offline.
K
When
messages
are
taken
from
the
middleware
to
be
executed
to
be
given
to
callbacks
pretty
much,
we
can
also
extract
information
about
the
executor
to
get
information
about
it
states
or
over
time,
and
then
we
can
also
there's
other
information.
But
for
for
this
presentation,
these
are
the
main
data
points,
and
so
we
use
this
quite
raw
data
and
we
need
to
analyze
it
analyze
it
so
to
process
it.
K
The
specific
the
one
one
important
thing
we
need
to
keep
in
mind
with
distributed
systems
is
that
we
of
course
need
to
combine
traces
from
all
the
systems.
So
usually
after
execution,
you
can
synchronize
the
systems
beforehand,
using
ntp
or
pdp,
or
you
can
also
synchronize
the
traces
afterwards
using
trace
compass,
and
then
you
can
interest
components
with
model
the
roster
objects
and
the
instances
using
your
tweezer.
K
And
since
we
are,
we
wanted
this
to
work
with
distributed
systems.
There's
some
things
good
to
keep
in
mind,
so
the
trace
points
they
use
printers
as
unique
ids,
usually
to
simplify
payloads.
So
those
pointers
are,
of
course,
memory
addresses
so
for
it
to
be
valid
across
as
multiple
processes.
Then
we
come
to
combine
it
with
the
pids.
K
So
this
leads
us
into
the
main
part
of
the
presentation,
which
is
the
message
flow.
So
the
message
flow
is
a
graph
of
the
path
of
the
message
across
the
distributed
roster
system.
So
if
you
think
of
the
rus
rus
one
or
two
composition
graph
as
a
directly
directed
acyclic
graph,
then
it
is
from
usually
the
root
of
the
graph
to
a
leaf
of
the
graph.
So
from
message
to
node,
then
another
message,
and
so
on.
K
K
It
was
working
but
was
not
really
reliable
for
multiple
messages,
and
it
also
only
considered
direct
one-to-one
links
between
messages,
meaning
that
for
one
input
message
into
a
node,
it
only
considered
one
output
message,
so
this
was
limiting
and
it
that
really
applied
to
most
real
news
cases.
So
this
is
why
this
this
new
paper
is
really
an
improvement
on
this.
K
So
yeah,
so
this
message
for
analysis
now
is
again
the
same
thing
but
applied
to
the
distributed
roster
system
and
to
build
this
we
need
to
combine
multiple
segments
and
links.
So
the
one
one
segment
we
need
to
get
is
a
subscription
timer
in
callbacks,
so,
for
example,
this
one
and
then
the
following
segments
after
this
are
usually
message
publication
instances.
K
So
in
this
case
this
these
messages
were
published
during
the
timer
callback.
Then
the
next
segment
would
be
in
this
case,
for
example,
here
at
transports
links
so
over
the
network
and
then
so
in
this
case,
for
example,
this
this
message
went
over
the
network
then
was
received
and
then
the
callback
was
executed.
Then,
during
the
callback,
a
message
was
published
and
so
on.
So
the
graph
continues
and
we
also
have
a
causal
message-
links,
okay,
which
I
will
explain
shortly
so
they're,
just
the
link
between
input,
messages
again
and
output
messages.
K
So
the
first
link
to
consider
is
a
transport
link,
so
it
is
the
link
between
a
publication
instance
and
the
corresponding
corresponding
subscription
callback.
So
in
this
example,
we
have
two
systems,
so
the
first
system
below
and
then
the
another
system
above
in
this
representation.
This
is
this
is
from
the
resulting
trace
compass
output,
pretty
much
so
one
system
here
and
then
the
other
system
there.
K
So
the
first
system
has
a
periodic
timer,
so
5
milliseconds
and
during
its
callback
it
just
publishes
a
matches
on
a
message
on
topic
a
so
like
here
and
then
this
message
is
sent
over
the
network
to
the
subscription
on
the
other
computer.
So
then,
then
it's
received
and
that
the
callback
gets
executed
so
that
transport
link
is
really
this.
This
error
right
here,
so
it
should
be
noted
that
it
includes
more
than
just
the
network
time.
K
So
in
this
case
it's
network
time
plus
the
the
time
delay
between
the
reception
by
the
actual
middleware
and
the
the
actual
callback
execution.
So
there
there
should
be.
If
we
put
improve
this
method,
we
would
add
some
additional
segment
right
here
before
the
economic
execution
and
by
definition,
this
this
link
is
really
a
one-to-many
link.
So
a
message
can
be
is
published,
published
once,
but
it
can
be
received
by
any
number
of
subscriptions.
So
this
is
important
to
keep
in
mind,
so
this
is
already
an
improvement
on
the
previous
method.
K
So
then,
we'll
talk
about
causal
message
links.
So
it
is
the
link
again
between
input,
messages
and
output
messages,
and
it
can
be,
of
course,
many
too
many.
So
when
a
message
is
received
by
a
subscription,
something
happens
and
it
leads
to
some
output
messages
that
are
published
by
a
publisher
and
we
define
here
the
causality
between
those
messages
permanently
on
message
data.
So
the
data
of
an
input
message
is
used
to
generate
the
data
of
an
output
message.
K
So
this
is
strictly
this
without
any
strict
constraint
on
time,
so
it
can
be
directly
related
and
time-wise
or
it
can
be
really
indirect,
which
explains
pretty
much
the
rest
of
the
this
presentation.
So,
for
example,
here
direct
links
between
messages,
there
is
one
here
so
in
red.
So
this
the
message
for
this
subscription
callback
directly
led
to
this
output
message
that
was
published.
So
this
is
a
direct
causal
link,
but
we
also
have
again
in
red
these
indirect
calendar
links.
K
So
this
in
this
case,
this
message
was
received
and
eventually
it
was
used
to
publish
a
message
later
during
this
time.
We
call
back
here
and
say
same
thing
here.
This
message
was
was
received
here
and
eventually
was
used
for
it
to
publish
this
message
at
the
end,
so
it
can
be
directly
indirect
and
so
for
direct
causal
links.
K
As
I
said
it's
for
in
this
case,
it
should
be
really
in
inside
the
subscription
callbacks.
So
message
publication
happens
in
this
case,
for
example
here
inside
the
subscription
callback.
So
it
can
be
pretty
easily
inferred,
so
we
don't
need
any
additional
information
or
instrumentation,
so
the
instrumentation.
It
tells
us
that
the
callback
started
at
some
point
and
then
ended
at
some
point
so
that
if
a
publication
happens
on
the
same
thread
within
those
time
points,
then
we
can
refer
to
the
messages
are
linked.
K
So
that's
pretty
pretty
easy
to
do,
and
now
the
the
more
complex
cases
are
for
the
indirect
cause
of
links,
so
the
unknown
here
is
that
user
level
call
is
difficult
to
include
in
the
execution
model,
so
it's
difficult
to
model.
So
this
is
usually
done
above
the
eurostar
api,
so
user
level,
for
example
people
you
often
use
in
real
applications
message
caches.
It
can
be
as
easy
as
just
message
that
you
copy
into
a
cache
or
you
could
just
keep
a
message
printer
or
something
like
that.
K
So
it's
hard
to
keep
track
of
those
messages.
So,
as
I
said,
it
leads
to
indirect
causal
links
which
are
in
synchronous.
So
again,
there
is
no
direct
time
link
and
they
are
again
many
too
many
links
and
this
kind
of
link
requires
additional
user
information,
user
level.
Information
which
we
collect
using
add
simple,
trace
points
so
for
inside
nodes,
you
user
or
simply
have
to
give
the
type
of
a
link
that
they
are
using,
give
all
the
input
subscriptions
and
the
output
publishers.
K
K
So
the
first
one
is
the.
What
what
I
call
the
periodic
asynchronous
link,
so
messages
are
received
and
then
cached.
So,
for
example,
here
this
is
the
the
node
with
the
special
link
and
it
receives
messages
from
two
nodes
on
topic
a
here
and
topic
b.
So
the
first
messages
is
published
and
received
by
the
special
node.
So
it's
just
the
subscription
callback
is
quite
short,
but
the
message
is
simply
put
into
a
cache.
K
Then
same
thing
happens
for
a
topic
b,
so
the
message
is
received
and
it
is
put
into
a
cache
and
then
periodically
a
timer
gets
called
and
it
simply
takes
the
messages
in
the
caches
and
then
then
computes
the
result
and
publishes
the
result
inside
a
message.
So
the
the
input
messages,
then,
should
be
linked
to
the
this
output
message
to
an
indirect
causal
link.
K
K
So
then
the
second
link
is
quite
similar.
It's
the
partial
synchronous
link,
so
again
messages
are
received
and
cached,
but
in
this
case
messages
are
published,
I
mean
they
may
be
published
inside
the
timer,
the
subscription
callback
directly.
So,
for
example,
here
a
message
is
received
from
topic
b
and
then
a
message
is
received
from
topic
a
and
in
this
case,
since
a
cache
for
a
message
from
topic
b
was
available.
K
The
cache
and
a
message
from
topic
a
was
available
since
this
is
the
subscription
callback
for
the
message
on
topic
a
then
the
output
message
can
be
computed
and
published.
So
then,
the
this
message
is
linked
asynchronously
through
the
output
message,
but
the
message
from
topic
is
linked
simply
directly
to
the
output
message,
so
this.
This
is
why
I
called
it
a
partial
synchronous
link.
K
So
again
this
these
are
two
simple
links
that
we
define
and
model
using
by
looking
at
existing
systems,
but
we
could
either
define
other
links
if
there
are
like,
if
you
have
other
special
links
in
your
system,
you
can
add
them
to
this
model,
and
so
the
result
of
all
of
this
information
put
together
is
the
message
flow
graph.
So,
for
example,
here
I
took
the
outerwear
reference
system,
which
is
a
a
system
that
is
based
on
the
ottawa
computation
graph,
and
I
split
it
up
over
multiple
processes.
K
K
K
K
And
now
a
short
section
on
the
executor,
so
you'll
see
why
it's
related
quite
soon,
and
the
executor,
of
course,
is
a
high-level
task
scheduler.
So
it
fetches
new
messages
from
the
middleware
and
then
calls
your
user
provided
callbacks
with
those
messages
and
also
executes
timer,
callbacks
and
so
on,
and
the
challenge
with
the
executor
is
that
it
does
scheduling
on
top
of
the
os
scheduler,
which
can
sometimes
be
inefficient
and
non-deterministic.
So
there's
a
lot
of
work
on
this.
That's
been
done
in
the
past
few
years.
K
There's
many
solutions
depending
on
your
application
requirements.
You
can
use
other
executable
designs
that
are
better
suited
to
your
application.
You
can
optimize
scheduling,
policies
and
priorities
and
of
course
you
want
to
study
and
compare
executive
designs
and
optimize
overall
application
performance,
which
applies,
of
course,
to
hardware
celebration.
H
K
Orange
is
just
waiting
for
something,
so
you
can
see
that
some
executor
instances
are
busier
than
others.
So,
for
example,
in
this,
the
first
system
above
is
quite
not
really
busy.
It's
mostly
waiting,
but
the
system
below
is
some.
Some
processes
are
busier
than
the
other
than
the
other
half.
So
this
can
cause
message,
processing,
delays
and
bottlenecks.
K
So
in
this
case
we
can
we
can.
For
example,
we
had
a
pid,
it's
not
the
same
pids
but
oh
well,
oh
yeah,
so
it's
not
another
experiment
runs
and
it's
different
pids.
But
in
this
case
we
took
one
process,
so
in
this
case
690
and
we
used
instead
of
using
a
single
threaded
exhibitor,
we
we
use
a
multi-threaded
executor
with
two
threads,
so
it
of
course
it
spreads
the
the
load
over
two
threads.
K
So
in
this
case
it's
less
busy
and
it
can
execute,
receive
message
callbacks
for
received
messages
more
quickly,
so
we
can
take
a
look
at
the
results
on
the
computation,
the
message
flow.
So,
for
example,
here
we
have
three
nodes
on
the
same
process
so
690
again,
and
these
these
two
nodes
are
in
here:
rayground,
filter
and
voxel
grid,
dom
templar.
Here
they
receive
the
same
messages,
the
same
message
and
they
have
to
execute
callbacks
and
with,
of
course,
with
a
single
executor.
K
They
would
execute
callbacks
one
update
the
other,
but
in
this
case
using
the
multi-threaded
executor,
they
can
execute
the
callbacks
concurrently.
So
then
the
the
end-to-end
latency
is
shortened.
So
this
is
quite
a
simple
example
and
of
course
you
would
want
to
look
at
more
complex
executor,
and
but
this
is
just
a
simple
illustration
of
that
of
the
method.
K
So
if
we
talk
about
runtime
overhead,
if
we
extrapolate
from
previous
overhead
results
so
from
lttg
and
from
roster
tracing,
it
should
be.
We
expected
this
to
be
very
small,
so
we
executed
500
nodes
first
without
tracing
and
then
with
racing
which
has
a
total
engine.
We
can
see
about
260
milliseconds,
and
then
we
just
took
the
difference
in
the
engine
latency
and
it's
about
like
less
like
less
than
0.06.
K
So
it's
quite
small.
So
I
think
it's
quite
quite
good
for
other
applications
and,
of
course,
it's
likely
challenging
to
measure
on
more
complex
systems.
I
spend
a
lot
of
time
measuring
this
kind
of
data,
so
if
you
measure
entrained
latency
for
distributed
systems,
you
hit
all
of
the
the
jitter
and
all
of
the
variation
scheduling
and
io
and
kernel
and
everything.
So
it's
pretty
challenging
to
measure
since
the
already
is
quite
small.
K
We
can
use
this
information
to
compute
and
enter
in
latencies
and
intermediate
latencies,
and
we
can
use
this
method
to
study
and
improve
performance
on
for
rescue
applications,
and
I
have
some
notes
for
future
work
if
you,
if
you're
interested,
but
I
would
invite
you
to
read
the
paper
if
you're
interested,
since
we
are
a
bit
short
of
time.
So
again,
this
is
the
paper
I
presented
and
there's
a
qr
code
on
the
left
above
and
then
for
the
tracing
paper
is
also
available,
so
yeah
questions.
A
A
The
first
is
so
I
would
love
to
pick
your
brain
christoph
about
essentially
the
ongoing
activity.
You
you
guys
and
particularly
you
have
been
leading
in
using
lt
tng
and
these
open
source
tools,
including
trace
compass
and
the
proposal
of
roster
tracing
which
build
on
top
of
lt
tng,
I'd
like
to
know
how.
A
How
do
you
see
the
future
growth
of
those
tools
for
what
concerns
hardware
acceleration,
because
we
are
seeing
some
ongoing
friction
with
some
proprietary
tools
like,
for
example,
amd
has
its
own
proprietary
tools
to
have
introspection
into
the
kernel
interactions
with
the
with
fps
with
the
processing
systems,
and
we
have
their
own
tools
for
also
doing
gpu
introspection.
So
how
do
you
see
the
landscaping
here?
K
Well,
I
don't
know
much
about
their
like
vendor,
specific
tools
like
from
amd
or
xilinx
or
nvidia,
and
but
yeah.
That's
a
good
question.
I
know
that
so
other
people
in
my
research
lab
they
work
with
amd
and
so
on,
gpus
and
fpgas,
and
they
from
what
I
understand.
They
just
use
the
vendor-provided
tools.
There
is
a
pretty
easy
like
conversion
between
the
information
they
get.
K
They
can
extract
from
those
tools
and
like
information
that
into
tg
generates,
but
so
you
can
pretty
much
easily
use
those
tools
and
then
generate
the
ctf
information,
so
trace
data
and
use
the
same
analysis
tools
just
such
as
the
one
I
presented
but
yeah.
I
don't
really
have
an
easy
answer
for
you.
A
No
problem-
and
I
would
be
I
would
be
interested
if
we
can
follow
up
if
you
can
give
me
a
pointer
to
to
actually
those
things
you
guys
are
using,
because
so
I
am
aware
there
are
some
like
utilities
to
convert
formats,
but
the
main
friction
I've
been
finding
is
not
so
much
about
getting
things
consistently
into
ctf
format.
It's
more
about
getting
the
clocks
actually
right
and
making
them
make
sense,
because
you
may
get
actually
things
somehow
data
consistent.
A
K
So
if
you
use
the,
if
for
the
two
cases
I
presented
so
for
the
the
two
indirect
causal
links
I
presented,
if
the
same
message
gets
reused
for
two
output
messages,
it
will
be,
it
will
be
supported
quite
easily
soon.
But
if
you
have
something
else
then
we
will
need
to.
We
would
need
to
take
a
look
but.
A
K
A
K
On
this
yeah
sorry,
it
might
be
worth
looking
at.
So,
of
course,
running
that
trace
it
out
to
disk
is
also
an
overhead,
so
sometimes
it's
less
somewhere
overhead
to
send
some
the
data
directly
over
the
network
for
another
system
to
basically
ingest,
but
in
most
cases
it's
simpler
to
just
write,
write
it
on
each
system.
A
A
A
All
right
cool
so
awesome,
so
I
think
we
reached
the
top
of
the
hour
unless
there
are
any
other
questions
we
will
thank
christoph
thank
christoph.
I
was
fantastic
to
have
you
kudos
again
for
the
paper
actually
the
papers,
I
should
say,
because
I
think
you
guys
published
a
couple.
I
saw
a
second
one
touching
on
on
the
perception
analysis.
A
A
I've
been
following
christoph's
work
for
quite
a
while
now
I
personally
believe
that
hardware
acceleration
definitely
demands,
let's
say
a
tracing
and
benchmarking
centric
approach
a
to
further
optimize,
the
accelerators
and
b
also
to
have
a
always
graph
centric
view,
so
that
you
don't
essentially
damaged
parts
of
your
craft
that
you
do
not
necessarily
want
to
do.
The
complexity,
though,
blows
up
at
the
moment.
A
You
start
seeing
the
tracing
information
and
christoph
today
just
presented
a
super
nice
set
of
contributions
that
essentially
do
much
of
the
work
for
you
digesting
it
proposing
some
data
structures
and
then
presenting
it
nicely
for
visualization,
so
so
yeah
again
kudos
christoph.
Thank
you
again
for
your
time
and
to
everyone.
We
will
have
one
more
session
before
the
summer
break
next
month
by
the
end
of
june,
and
the
announcement
will
come
up
soon
in
this
course.
A
Next
session
actually
is
going
to
touch
in
more
detail.
Some
of
these
unique
new
capabilities
that
brought
to
humble
brings
in
from
a
hardware
acceleration
perspective.
I
will
have
a
session
discussing
particularly
a
framework
for
facilitating
hardware
acceleration,
so
definitely
encouraging
everyone
to
check
that
out.
It's
going
to
be
again
the
last
session
before
we
break
for
the
summer,
and
after
that
we
will
reengage
in
september
with
more
exciting
stuff.
Thank
you.
Everyone
for
joining.
Thank
you
christoph
once
again
and
to
everyone
have
a
fantastic
rest
of
your
day.