ROS 2 Hardware Acceleration Working Group, 4 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RobotPerf subproject - meeting#4

Description

Weekly meeting of the RobotPerf (https://robotperf.org/) subproject of the ROS 2 Hardware Acceleration Working Group (https://github.com/ros-acceleration).

RobotPerf is an open reference benchmarking suite that is used to evaluate robotics computing performance fairly with ROS 2 as its common baseline, so that robotic architects can make informed decisions about the hardware and software components of their robotic systems. The group meets weekly at this room and discusses robotics computing architectures and benchmarking across various compute substrates.

Minutes of the meetings are available at https://docs.google.com/document/d/185Cy1xjpAOgJygEOnlf5OCgOQTywmF0qgSpS3GiW16Q.

A

All right welcome everyone to the fourth uh robot Perth uh weekly meeting, um let's as always start with the minutes, so I'm sharing those in the chat, and let me also share my screen and get started with that um here. We are.

A

All right so um yeah um we had a few uh actions uh from last week, so uh just knocking down. We have advanced a bit on our side uh with the continuous integration pipeline, as reported last time, we're focusing on um AMD and Nvidia uh fpg and GPU socs for starters, and we will be skating to more platforms. uh Last week we pushed uh and managed to get uh the infrastructure, uh let's say to a decent stage, but still it's not fully ready uh to Output stuff directly to GitHub to the benchmark.yaml files.

A

It's going to take a bit more of hacking to get that done and I think we'll continue today after today's uh working group meeting. Actually so um so, yeah um expect more updates on on that regard, certainly been progressing and and getting ready for um running as many benchmarks as as possible. So uh so yeah we'll report next week on this um again, no don't want to share specific expectations because we're encountering sorry hurdles on the way, but in principle um the pipeline is working.

A

It's just a matter of structuring it for supporting more and more ports at scale. So yeah we'll continue uh updating everyone.

A

Sorry! So, uh um besides that I think we have exciting updates from Jason Jason. Do you want to take it over sure.

B

Of course, let me go ahead and share my screen.

B

All right there we go.

B

So um yeah, so we have gotten in a uh third Benchmark in the perception category of robot perf, uh so I thought it would be helpful to go through the Benchmark demonstrate it. uh So then, for people who want to uh you know contribute their own benchmarks. They can have a better idea uh like how to set a benchmark up, and if anyone has like questions, we can talk about them and then I'd love to get your feedback about this Benchmark.

B

What we should change- uh and we should add to it before we- you know- push it into the main uh branch of robot version. So uh having said that, uh I'm gonna introduce this Benchmark, it's the um it's a stereo image uh Benchmark.

B

So basically we are taking a look at the stereo image process uh package in the image pipeline of Ross, and this specific package takes in two images: a left and a right image and it outputs uh some information like a disparity method and basically a disparity map is uh showing like how deep images or how deep objects in front of you are using uh the to the left and right image that you pass it and then that allows you to like. uh You know know how far away you are from things.

B

uh So that's a little bit of a background about what, like a stereo image, uh what the stereo image package does so I'm gonna go ahead and and uh like demo, it I have all of the steps in the readme uh how to install it and then all the commands, you're gonna, need to uh run through everything. You're gonna see. Let me demonstrate uh so: let's go ahead and yeah get started so the first step, I'm gonna, do is I'm going to um play back a Ross Bag that I've recorded okay.

B

um So we can just take this Command right uh and replay uh Ross Bag in a loop uh and then I'm also going to like for visualization purposes of this uh demonstration, I'm also going to start like an rpt graph uh to see like what's going on so right here we have like the Ross Bag player. Node.

B

uh Next thing we can do is visualize. The feed uh the Ross back player is uh publishing. So let's go ahead and do that.

B

All right so uh here you see a left and a right image of a little robot, navigating through a gazebo World um I've set up like the configuration files for uh this um arviz um I. Don't know what you call it for yeah. This configuration file for Arvest so then, like all of these uh settings, will come up. uh If you run the command. I showed you uh to like better understand if your like Ross back player, is working things of that sort.

B

uh So yeah, as you see here, we have like the robot navigating uh in the world uh and and uh to if you want to reproduce the simulation. I have all the steps here. uh Basically, it's launching um like a gazebo simulation and then recording all the topics uh that are being published and the world looks something like that cool, so um yeah.

B

So as of now, let's refresh our graph and let's see, what's happening now, so we have the Ross Bag player node, and we can see that it's publishing a left image and the info for that left image. And then we also have a right image and the info for that right. Image uh and that's you know being produced from the Roswell all right.

B

Let's see what the next step is um so next we can go ahead and start our benchmarks, um and we have a launch file inside of um this director here that you can just do like Ross to uh and then the package name um and just watch it. So let's take that command, uh throw it in here, okay, cool all right, so our Benchmark has technically uh been initiated. Now, so let's refresh our graph and see what we're working with.

B

um So this is what our Benchmark added to our graph. So I'm going to go through it quickly and explain, what's happening here, so we have a uh node that subscribes to the topics being published from our Ross Bag player. So it takes in the images uh both in the Raw image and the info about the camera, um and then what this does is every time a new image is being published into these topics.

B

It has a callback function um which basically start a trade point and then publishes that same image back to like another topic. So the purpose of this node here is to start a trace point. Every time like a new image is coming in okay, so uh and and that will better help us uh like contain everything that we're measuring inside of the benchmark, which brings me to like this part of the graph. uh So here we are like once we are starting to like Trace uh each input.

B

um We can then take the information, that's being published onto these topics uh and then our stereo image uh node is going to use that information right and, as I mentioned earlier, the stereo image uh package you know takes in this information and then outputs a disparity map uh and that's and that's what we're benchmarking so it'll take this information and then uh you know, do all the calculations necessary, output, disparently and then.

B

Finally, uh when that happens, we also have like an output node, which starts another Trace Point, uh so that we can like measure uh like how much time it took an image to like first be inputted into this node. All the calculations happen and then uh what time like the calculations ended um basically we're just like measuring latency, um okay cool and, let's also visualize, what's happening just to make sure like we fully understand uh what our Branch Mark or like what this node.

B

This package is outputting, and to do that, we can run this command. uh Basically we're just going to be viewing uh the despair development. So let's go ahead and pop that into terminal.

B

Here you can see uh it's the same images of the robot navigating of the world, but this time, instead of seeing like a regular raw image, you're seeing a disparity map which shows like how close you are to objects so like when you get closer to objects, they can heat map, will change cool, um so yeah, that's what's basically being outputted from our graph here uh and we can like refresh this, and this is basically the I'm subscribing to the uh node, to like view this disability button.

B

So that's the full, the full graph of our Benchmark. um We can go ahead and end end our Benchmark and go look at the results. um So to do that, we can take a look at this command um and to my understanding, I think like Victor. You can correct me about log here, um but, like most of the information is stored uh within this directory, um so you can navigate to that um and then we can take a look at all the output that's being produced from those Trace points uh by using this command.

B

All the information from the trace ones uh I have yet to like, take this data and analyze it, which is like a Next Step, um but that's basically the full pipeline, uh the uh stereo image.

B

So if you guys have any feedback, um you can either provide it now.

A

B

Yeah, if anyone has any feedback um or questions happy to take them, uh but there is a current like a pull request uh to add this Benchmark into our uh repo. So if you want to like take go there and um you know add some comments feel free, um but yeah.

B

uh So any questions about this Benchmark Let's uh I'll sit there.

A

Well, first of all, I think. Congratulations, sorry, Northern, right! I think this is a fantastic contribution. uh Jason in the team in Harvard, so very nicely done. Folks, I think that, uh with this work, crackling will finally be um the pace at which we can contribute more and more benchmarks. So now we just need to get 100 and um we'll have a hell of a uh Benchmark Suite I guess um no! This. This is looking great uh well done, uh good work overall, um maybe a few remarks.

A

um I I need to look at the pull request. A bit deeper I just had look at it when I woke up this morning, which is I, think exactly at the time you were going to bed because we were exchanging uh messages, uh I think very late for you very early for me, it was a bit fun um and I. Think I kept you awake a bit longer! So sorry about that. uh But um you know natural.

A

Thank you. Nevertheless.

A

um So three remarks, I guess: first, um the rhythmics are out generated. Sorry, if you guys remember um we uh we have this vision of having the Benchmark dog yaml file, um grabbing everything and then from there um generating the readme uh periodically. The rational behind that is because the Benchmark data, where we contributed to these yaml files I mean every time. Sorry, every time it gets changed, we will update the readme automatically and reproduce it so uh I. Just look at your branch and I didn't see the Benchmark or yaml file yeah.

B

Exactly yeah I realize I still need to do that. I was doing the readme's manually um a question. A good question is: does it have to like in my readme I added like a simulation reproduction uh and uh like section, does it have to follow the exact sections here? Can we add some more to the CM file, or should we just like stick to this um and not like add anything extra? We.

A

Can add, we can add more sections to it. uh We just need to revisit the source code that takes care of and producing automatically uh the corresponding um thingy. I can point you to um to where that is happening. It's in the repo as well. There's nothing I! Guess we just need to make sure that um that's contemplated um it's particularly um as living in I. Think if you go to.

A

I need to I probably need to double check this.

B

A

Yeah no problem, no problem yeah, but nevertheless, um just just keep in mind that please so adding The, Benchmark or yaml file is probably important. Yes, yes also take take note of the fact that if you put like fancy gifs or images that they they may need to get added also into the yaml file in a specific way, um something worth worth noting as well and I know that uh in your source code, that you rewrote uh input and output components from The Benchmark.

A

So at the end of the day, we, uh the idea is that we should have a common sort of like package which has input and output General uh components. If it's, if it's general purpose images uh going in and out, uh we should reuse this, because.

B

Yeah yeah, so um the reason I rewrote the image component is so that I can like subscribe to two different uh topics of images, but like now that I'm thinking about it, maybe I could have just created two of those image components exactly uh okay, but then I also created like a disparity input, uh output component, because the uh message type that's being published by the stereo image. The like process package is of type like disparity or something. It's not like an image yeah.

A

That with that I agree, I was uh I was. My remark was more towards: let's try to reuse this technology components because at some point we will need to um revisit this input and output bits and just push it to a common boss package.

A

I have have most of the benchmarks, reutilize things or contribute uh input and output components to that instead of, like you know, uh enlarging um the the amount of code that that just gets written again and again so uh for the input components you can reuse the one that we have in I think it's A1 uh and you can just use remapping for changing the topic. Names. That's that's as as far as As It Gets.

B

That brings me to like one quick question, sorry to interrupt, um but uh wait where's. My bookmark uh uh I can just like go together here, um so I added a trace point for each of the images like two Trace points. Is that okay, or should we just have like one Facebook.

A

um That's a good question and I guess that I guess that brings me to whether we need a new uh component or not for input data um I need to think about that a bit more um in principle: I! Guess! If it's just an image, Trace point: it's an image: Trace, Point um I!

A

Don't think we um I don't think we need to know which one is what we just need to account for the fact that there's two Trace points before the actual Benchmark starts like two identical image: Trace points, but that that is something that you will figure out once you start analyzing the data and that's the.

B

A

uh Which was uh some sort of like uh input regarding um you do uh produce your uh script for analyzing things, uh take a look at what was produced for by A1, which is essentially building a launch file which just leverages Babel, trace, and much of that you, you would be able to uh just uh re-utilize yeah yeah over there and analyze exactly.

C

A

In much of that could be reutilized, uh much of which might not be useful, uh but you you, you decide that on your own in the introduction, so yeah, no overall um I'm excited about this um I, don't mind uh getting this inside and now that has been cleaned up, um but it's up to you Jason. Would you like to maybe work a bit more on the uh on the analyzing and then we merge it all. Or um would you prefer that we get this in right now.

B

I don't mind either way honestly uh I'll leave that up to you. Whichever way you think whatever workflow you think is best um I I can go ahead and like clean it up, and then we can just do one.

A

Let's, let's do it like that, then, uh let's um just have a look at it uh see if we you can re-utilize the components for images at least and then ping me and then we'll get it in.

B

Okay sounds great awesome, uh yeah.

A

I think I think one minor bit is I. Think I also saw the template that you added as part of this pull request is that is that right, or did you remove.

B

That, yes, yes, yes, oh yeah I was going to talk about that. um Thanks for reminding me, uh let me just pull it up real quick.

B

So uh yeah so last time we were talking about, we don't want to have overlapping like people working on the same benchmarks. We kind of want to know what What's um what the community is doing right. So we created a template uh to create an issue. So if you go to like uh once, this is like March with a real Fork, you can go to the issues uh Tab and then create a new tab.

B

I haven't created a new issue um and then you can use the new robot curve Benchmark in progress announcement, uh issue templates uh and basically, what that will do is have you answer a few questions which is like specify the Ross package you're working on provide a link to that Ross package. Like what category does it fall under um like what metric are you planning to measure the estimated date of completion, Hardware you're, going to use to run the reference implementation and then some information about the person who's submitting Ben Benchmark?

B

But this is I just uh this is a draft, so uh you know happy to get some feedback about what an issue template should look like.

C

A

So my feedback on this is uh this is fantastic I, think it's what we want um and I'm very happy to get this in right away. My only uh comment would be uh if you can remove this from the existing pull request and push it into a different pull request, we'll get that right away inside and if anyone or if anyone has any contributions uh to this like template, do feel free to submit any other uh follow-up for pro request on it.

A

um I can see some like minor bits that I probably will um sharpen a bit, um but uh but overall I think this is fantastic, uh and this is exactly what we want uh so so yeah if you can submit that in a different pull request, um just just to make sure we we keep things tidy. That would be fantastic.

B

Awesome yeah that sounds great uh yeah I.

D

Think we have so many of them.

D

Why why you need to know about the organization and then the person engineer? We don't we don't. To be honest, we don't care too much. We care about the content and what is the issue like more about the the context and then uh yeah is Victor said. Beginning is perfect, I think.

B

Yeah I mean that's a great question.

B

um The reasoning behind this is that we would like uh robot perf to be um you know, utilized by our community, and we think that you know one of the major steps to get a benchmark uh like popular is to get some like the big players in the room, um not to say that, like we're, restricting people from you know submitting benchmarks, uh but I think it is definitely helpful uh for robot perf to be like taken very seriously uh to have like some of the big uh robot.

B

uh You know uh community members contribute to this benchmark.

B

Hey Victor! Let me know if that like uh aligns with your vision, yeah.

D

I can give you another and, like an example, later offline, maybe because I use, we use this one from the OE 40 like template like this, when people want to create issues and then give us like the context which, like machine, has used, as you mentioned, like the hardware, how yeah you can produce like the steps you, if you can to reproduce these things, Etc yeah I can give you some insight. Definitely cool yeah, I'd love to chat regarding your benchmark, which platform you use.

B

uh I was using uh like a RTX uh well, actually, I wasn't using like GPU. It's just using a regular CPU um I can add, like the specifications of it to The Benchmark once I had the results. I was planning to do that once like I kind of analyze uh and like get a number for that latency.

D

About the your PR but uh yeah, we.

B

Will do yeah that'll be awesome if you can throw it into. uh You know that pull request, and we cannot, you know, continue the conversation there thanks thanks.

C

C

One option here, like forgetting the information we could just keep like just mentioning bracket, that organization name is optional. Something like that. So we just cater to both.

A

Yeah I mean so so on that regard uh again, everyone's everyone's welcome to include whatever they want. We won't punish uh someone who who doesn't want to like describe his organization or her organization, but I I do think that there's Merit in what in what Jason was describing in the sense that, um just just to make sure we coordinate amongst ourselves, but also down the road provided this gross in popularity. I. Think it's great that you know.

A

Maybe there are various divisions within your big organization that are working on benchmarks actually, and you would want to know uh who is working on that. um So that's One, Good Reason, also uh for competitive reasons, it's fantastic to show that there's like big or small players, uh putting their resources into a specific uh direction. That gives not only a business um insights but also um interest insights from a community perspective. So I think there's good reasons why you know it's.

C

A

To everyone opening a ticket again, it's it's just a way to show interest and describe that you are working in this direction. Hopefully people will maintain their tickets and they won't give updates and so on and so forth, but uh yeah that we cannot force anyone.

A

This is a community and- um and it's kind of like a best effort at this stage at least um so yeah no uh I I would support uh let's, let's nevertheless move the discussion to the pull request same with regard the Benchmark and uh just kudos for uh Jason and Tim uh nicely done. Keep it up.

B

I also want to thank you pratik, uh you know. Definitely. I was definitely stuck at that like uh publishing the censor data. So thank you so much for that. Help definitely want to give you a shout out for that. That's.

A

Awesome that we have uh interact corporations in here within the group so uh very much appreciate it all right, so um I think we've reached the top of uh the half an hour folks, so let's meet next week, uh hopefully we'll bring some uh more updates uh and move forward with more benchmarks looking forward. Thank you. Everyone have a good one. Thank you. Bye.