ROS 2 Hardware Acceleration Working Group, 1 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ROS 2 Hardware Acceleration Working Group - meeting #15

Description

15th meeting of the ROS 2 Hardware Acceleration Working Group (HAWG, https://github.com/ros-acceleration).

We discussed the Robotics MCU (robo-v-mcu) project, RobotPerf benchmarking updates and REP-2014 contributions.

For more including source code, check https://github.com/ros-acceleration. Minutes of the meeting available at https://docs.google.com/document/d/185Cy1xjpAOgJygEOnlf5OCgOQTywmF0qgSpS3GiW16Q

00:00 Welcome
03:05 Updates on Robotics MCU project (robo-v-mcu)
10:45 Updates on RobotPerf, robotics computing benchmarking suite
32:45 REP-2014 review

For more including commercial support on hardware acceleration topics in robotics, refer to https://accelerationrobotics.com.

A

All right welcome everyone to the 15th, Ross 2 Hardware, acceleration working group meeting. uh Today we have a bunch of topics we'll be covering, uh so let's get into it, um as always: I'll be sharing first, uh the minutes in the chat.

A

So let me push it uh right in here for all of you and then let me share my screen.

A

Okay, so um again, um 15th uh thing of the working group um to get started. I wanted to share a couple of updates with everyone. The first one is that the working group is growing actually significantly. We are increasing, also the periodicity of both um previously. If you remember, we were meeting once a month and we will continue doing so. Typically, at the end of each month, we will have uh limitings, but now with the uh Advent and increase as well as increase of popularity of some of these projects of the working group.

A

We are moving also to weekly meetings, especially for the rubber curve. It's a project so very happy to share it with everyone. Last week we had the first uh robot perf weekly meeting uh with a pretty decent attendance for being the first one, um the recording and material. As always, is available. You can check it out it's in this same resource, um so we're going to be using the same resources of the working group just to make sure we leverage and make everything, um usable and accessible.

A

So again, the resources are in here check it out. uh So that's one first bit of excitement. The second one is that we are holding also today as part of this meeting uh concurrently, the first meeting of the architectures and processors working group of the Spanish Association in semiconductor industry, we're doing lots of stuff that relates to essentially building up and bringing both as soft cores and hard course.

A

So um we thought it was uh going to be a good opportunity to bring uh Isme folks, which we are part of uh within the Spain Spanish domain uh into this International Community we've been leading now for uh quite a few months. I would say officially more than a year now, so they can take part of it contribute collaborate and maybe also bring their two cents.

A

uh That's about it. With this um I'd like to welcome everyone uh to the meeting we're new to the working group, you can check out the attendance uh in the linking event. We will not be recording every single participant because there's already uh more than 30 persons that join today, so uh to save my time and and the time of my people helping me uh admin, this working group, uh you can just refer to the linking event and and that's gonna, be faster. Let's start with the uh progress review on various of the projects.

A

The first one uh I'd like to uh start uh with is the robotics MCU or the robot 5 MCU project. uh That's LED, together with Plan B. uh We have today with us Mustafa, which can give us an update. uh Most of our stage is yours.

B

Can I share my screen yeah.

A

Please go ahead sure.

B

No, not this one.

B

Can you see no uh sorry.

A

Yeah, we can see a presentation right now that looks that looks very well yeah.

B

Awesome: okay, thank you. It's the first time I'm doing this presentation on this Ubuntu computer, so it was kind of problematic, uh so we have added the Ethernet Mac on the general structure with the udma part just right here, that's mostly the only uh modification we have done to the general system, except we had to create uh 125 megahertz rjmi clock, so we also modified this clock generator uh other than that we didn't. We did not yeah modify any of the other projects so that it can go smoothly.

B

um So the the inside of this of our module looks like this. We have this resistor interface and two cross clock domain controllers for uh the forensics, IPS interface and and then the buffers this buffers and the controllers are also required to switch from this uh regular udma interface between regular udma interface and axi stream interface.

B

Controlling the module is pretty much the same with the other peripherals in the system by just like uart SPI I, Square CD, you have those peripherals already. We kept the same register structure. Actually we just tied the yeah. We just type the connections and stuff and added some more status information.

B

You can yeah enable disabled model module start the transmit and yeah set the transmitter address, transmit size and set the receiving address in order to work smoothly.

B

You can either pull the receive, receiving status or receiving interrupt is much a better way uh after receiving the after getting the receive interrupt, you have to switch the receive address to not lose the next data.

B

So as the conclusion actually, on the RTL side, we are mostly completed, uh we are having it tested for the software driver. We just began to work on it. We are glad to. uh We will be glad to receive the feedback on the RTL.

B

I'm, very sorry, uh it's like our hands are warm right now. So, if we, if we are required to add more functionality or do something differently, uh we'd be glad to know it immediately, so that we can like change it fast and we need the clarification of the API requirements.

B

And since we are starting to build a driver like it might be the right time to tighten the core cooperation with acceleration robotics so that they can uh okay, they can provide the necessary information about the API requirements.

B

Then that that's all thanks.

A

That's awesome, thank you so much yeah. So uh maybe a couple of of uh questions in here to follow up from this conclusion. Can you can you uh share with us the uh pointer to the RDL so that we can have a look uh and and if appropriate, just provide feedback? That is uh that is kind of like a a first one and then I had another question regarding the uh testing that you're currently performing? Are you currently using any sort of like uh verilog uh test bench environments such as Coco, TV or related.

B

uh We are using the test bench already given in the uh project in.

A

The very long yeah.

B

Yeah, we are just uh uh controlling this controlling the modules via software yeah.

A

Yeah, so if you're, if you're testing uh with the same tooling, that's provided in the very log ethernet uh repository, then you're using Coco, TV, uh definitely and and possibly also leveraging, uh eCards verilog, which is the simulator that's used in there.

B

No I'm, sorry, we are not using the Coco TV. We are uh I'm I'm using the same environments with the core 5 MCU.

C

A

B

You're, so it's already integrated uh it's it cannot it's yeah, it's! It can already be uh controlled via software, so we uh I actually do like this uh skip the booting stage and like uh just before the booting stage, I uh write my code and test it so that I don't have to wait. The booting during the simulation.

A

Okay got it got it um so um that that makes sense, I guess um so uh yeah. If you can provide in the minutes, I'll just put a placeholder in here a URL for the RPL.

A

um That would be fantastic in case it has changed or anything and then uh yeah we from our side, at least from from acceleration, robotic side. We will definitely follow up and and try to provide feedback, as well as as things so that we can meet in the middle, but yeah uh fantastic progress, Mustafa very, very excited uh in- and this is this- is super cool, uh very much looking forward.

A

I I think I did uh sing with uh you guys, especially with Max, on the uh Hardware that you're using for the setup, so we should be in the same position um and having the same tools and the same things. I should be able to represent pretty quickly so uh yeah looking forward for that. Thank you. So if you can drop that link in the minutes, I'd appreciate that otherwise I'll I'll chase you to try to get it there.

B

uh I, don't know if I can just uh I I'm going to. We are going to reach you for this. Okay.

A

Okay sure, no problem all right, so um um anything else to comment on the um on the robotics MCU project. Folks, any questions uh that we may have right now. Otherwise we can jump into robapurf.

A

None all right thanks a lot man yeah that was fantastic, all right, so uh maybe back into uh robot perf. um That's the second topic of the agenda today and again: I'm sharing the minutes. I know some of you guys just joined, uh so there you go with the minutes um again.

A

uh Roller perf uh we're looking at uh decent progress, and now that we are increasing the uh periodicity of these meetings and Robert Murphy's is walking or meeting weekly, so you can expect definitely uh more and more contributions, hopefully from many more um I wanted to touch on a few topics today.

A

um The first one is a Hands-On uh session on essentially how to build and launch a first uh benchmark. um I was planning to do it today. However, in the last week's meeting of the working group, we actually uh recorded it.

A

So um what I've done is just place the link in the minutes uh that points to the recording of last week, so if anyone's interested in a deep, diving source code wise on how you can build uh in a reproducible manner, um as well as create your own new benchmarks, please refer to that particular pointer.

A

That way, we optimize time second topic that I wanted to touch on is uh the Benchmark specifications, so uh we've requested feedback last month on the actual specification on how no benchmarks should be created and again this is hosted within the benchmarks. Repository um of the benchmarks, uh repo within the organization, I believe Matt I, don't know if Matthew joined us today. Jason do you know if he's with us.

C

uh He's not with us, but um I can speak on what we did um go for it, oh yeah, so there should be a pull request coming soon, um but basically we organize the definitions of what each part of our Benchmark uh means, um because a lot of the definitions and the words we're throwing out can get pretty complicated. So we took a stab at uh like what is a benchmark. What is the Benchmark Suite?

C

It's a workbook, what's a reference implementation, um so we clearly defined all of those things um and then soon when we uh you know when we get that pull request in if we can get the community feedback on that. um That would be really awesome, because all these definitions are not Sandstone. We just started to be uh like place, the a starting point uh that we can discuss and change the definitions around. uh If we want to.

A

C

A

Awesome awesome so looking forward to get that pull request and um I know that Matthew I reached out asking for um privileges I think it gave him uh what he needed. However, for sending up a requests, you don't need any privileges. So um yes, so just uh yeah just feel free to shoot and uh that'd be awesome, all right, so uh back into um I, guess, uh sharing and and going into some more topics that we will discuss today.

A

um So I wanted to touch on two more things: at least benchmarking wise. The first one is I wanted to report on some work that I did recently on creating a benchmark, CLI, tooling extension of the Ross to CLI tools uh that I'll just uh show you just in a bit right now and then I wanted to so we have with us today also Michael uh Carol from intrinsic now hi, Michael uh and uh so Michael has been uh spearheading. I.

A

Think that's fantastic initiative on Boss 2, profiling, I'm, actually pretty excited about it as well. So maybe we can touch on that Michael bit uh and try to align on on some of the aspects that I think we could benefit from each other.

A

um Just for the sake of being super brief, um I did prepare an environment to show you guys this contribution this week on the roast UCLA, tooling.

A

um I won't touch again too much on the structure of the benchmarks. Today, that's recorded and and thrown last week, so you can go and check that out what I'll show you is, and what we are doing right now is getting into that uh depth containers so we're using uh the same repo itself. Has the containers prepared so that we can reproduce things systematically, regardless of which operating system you use?

A

If you use BS code, you can just open things up um in the way that I just did uh technically speaking, what's happening, is that you're actually entering and developing from the container, so the container, the docker container, the virtualization itself is sharing volumes and it's sharing links with the host system so that you can reproduce the same experience that you can have in your host system. That way, we get a Linux um kind of like a base line set up with a series of um things installed that allow us to allow service benchmarking.

A

So uh what I wanted to show? You is how um so, let's just build the workspace that uh it gets uh out right away, so I'm.

D

Just gonna take.

A

A second so right now we're building it and then we're going to see how uh we're gonna we're gonna be able to visualize this new contribution this week, which includes a series of tools that we will be incrementally uh maturing and adding to manage and um essentially play around with the benchmarks.

A

So yeah, that's it now we can Source um this as an overlay workspace and finally, we can see how to the typical tools that we get uh in the default was to CLI installation. We now have a new one uh Benchmark one uh which reads as various performance: benchmarking related uh subcommands right, and so, if we watch to Benchmark now we have two suburbs or two verbs, one that allows us to list the benchmarks and one that allows us to The Benchmark.

A

So um the first one was to uh Benchmark list it actually um recursively goes through each one of the first. Two packages goes searches for benchmarks. Each one of these benchmarks is defined by the corresponding, um as you know, benchmark dot, yaml. So, technically speaking, each Benchmark is our last two package.

A

The only difference is that there is this extra file benchmark.yaml file, which complies with the specification that Matthew and his team are reviewing right now, uh and this is a super simple yaml file, which just specifies a series of aspects that are useful for reproduction launching and then reporting aspects in mind.

A

So hopefully, this is something also we can touch on with Michael now in the discussion, so the tooling that we've created the least verb or the update, verb, uh help you navigate around benchmarks in that sense, so once we know which benchmarks are available right now build in this overlay workspace, we can pick, for example, the second one watch to Launch, uh and then we can just and that's gonna um once the computational graph uh and then as we throw in input into the sorry data into the corresponding inputs.

A

It's going to start registering and tracing data that we can later on post process, with the corresponding analyze uh launch file. That says, AC or is it complex as it gets um so yeah more verbs will start showing up in this CLI tooling the update, where right now updates automatically the readmiss of each Benchmark as well as the top readme one, and this is going to get connected to a CI job within the GitHub repository that Updates this every time we change anything and over time, we'll be adding more and more functionality for now.

A

This is what I prototype uh and delivered, um and there are a few uh next aspects that will come down the road, uh including prototyping, this CI job that I just described so with that. Maybe we can spend some time uh discussing um ross2 profiling in in alignment with this uh Michael I. Think that maybe you can say a few words about the the F4 you're leading in in where you guys are heading foreign.

D

Respond to that first there's a message in the chat awesome.

C

um I was just gonna: ask uh if there's a way you can add like a readme uh inside that Docker uh folder to kind of help us get started, oh yeah like develop with soccer and how, like Dev container, that Jason uh like integrates with it, because I just don't have that much experience with it.

A

Yeah yeah, sorry, um yes, I can I can um Jason um and it nevertheless that's kind of like uh not something that that I I thought. Certainly not it is. um It is something that's pretty standard uh in vs code, it's uh so the best resource that you'll find out. There is, if you type um Dev, containers, vs code and you read through the documentation, that's the best resource, but nevertheless I'll. Take note of this and I'll see what I can do uh probably hard to make it better and the official documentation, but yeah yeah.

B

uh Reach out to me otherwise, okay, thanks.

D

Oh um yeah so Victor we kind of interacted a little bit on uh discourse earlier this week. uh So this this Ross II profiling tool, kind of grew out of the desire to be able to uh inspect. You know a an entire running Ross graph and be able to derive a set of metrics or analytics from that which I think probably has a lot of overlap with the same kind of metrics.

D

That you'd want to get out of a benchmark and then be able to do analysis and um and eventually write kind of unit tests and assertions against the performance of a system, and so some of the kind of core tenets of of what we were trying to do. uh One was kind of be minimally invasive, uh so not a substantial amount of recompilation or modifications to your existing ross2 code base. uh So that's kind of how I arrived at the Ross II tracing, which obviously in the rep 2012 or 20 14.

D

How many reps am I reviewing right now.

D

um You know you kind of arrived at the same conclusion right, so ros2 tracing is more or less Universal at this point, uh hopefully by the time that iron comes around we're going to have it enabled by default in the Debian packages um and it's using lttng, which is a kind of an agreed upon standard way of doing tracing.

D

So what this essentially does is builds a framework on top of that for doing the types of analysis that you would want from the Ross to tracing data, and so the big things are being able to look at latency and Jitter on topics. So if you have Publishers and subscribers in your graph, you want to know like who's publishing who's subscribing. How frequently are they sending at the time that they claim they should be and then also being able to trace uh events all the way through a graph which the ross2 tracing the trace tools?

D

Analysis right now, so Kristoff and and some of his uh colleagues have have done a whole bunch of work on being able to trace like message flow through the graph and use uh eclipses Trace Compass, to be able to like Analyze, That and basically I'm, just kind of surfacing the same data, the same information in a way that you can make um CI like tests against uh so I I, think yeah, I think there's a lot of kind of overlap between maybe the analysis that I'm doing.

D

Maybe you could just leverage that as part of the benchmarking tool, um but yeah really I'm, trying to stay in use using things that are available in ros2 core. Nothing super crazy or are out there um and and kind of surface the type of metrics that we'd be interested in um cool.

A

Cool thank you for explaining all that yeah. uh We are, as you said, we are building up on the same baselines.

A

um We are both leveraging roster tracing, which is uh the right thing, in my humble opinion as well, so um others disagree uh just to be clear for everyone. Roster tracing is not specifically tied to any uh particular Tracer. You can use lttng or any others.

A

So those of you, Windows users, you can actually use other tracers cross street racing is, is prepared for that actually, and it was validated in that context um and and I love the fact that you're looking into graph uh tracing um the way it's been done right now for us in the group has been literally by um not leveraging directly the information model that Kristoff built initially as part of his roster tracing Authority.

A

So he builds this information model and then you can query that information model um to try to extract the flow of the data. My understanding and please correct me on this uh Michael, but um from what I've read I I feel that what you've, what you're planning to do is push that into the graph um right is that correct.

D

uh Put push it into the graph in what sense in.

A

The sense that so so Kristoff has some sort of like representation of the uh timings that's stored locally, uh but kind of like in a in um in a particular memory. uh Section. uh I was under the impression that what you were going to do was uh kind of like push that data back into the computational graph into the Ross, to graphs that you can query it, uh but maybe I just got this from reading your comments. uh That was.

D

Yeah, so so not necessarily pushing it back into the roster graph, but at least having a an API that closely matches what you would expect um from from the roster graph, so I guess uh I can I can actually share. I've got just like a little notebook here, um so this is basically um I was not not prepared to present, but I've got something uh so so this is basically using the recorded events that that Kristoff had set up. So there are all these various Trace points.

D

uh You know when the Publishers and subscriptions are initialized when Publications and subscriptions actually happen um when timer callbacks happen, and so from that I build this graph data structure um from those events, so it basically gives you a different way of introspecting uh the events that have happened so one like one way of looking at it is an event sequence through the graph, and so here you can see this is using the reference system from like the autoware uh reference system, so that it's kind of got a bunch of Publishers and subscriptions, and so what's happening here is if we want to figure out why a certain callback is firing at a at a time and that's kind of the end point, and what this lets us do is Trace all the way back through the graph and see what sequence of events allowed us to arrive there and it was from a from a timer um at the you know, four nodes back, uh so this kind of information would be really helpful and if you especially, if you have like long processing chains uh like you, would see in like an image pipeline um or if you have like complex data dependencies, it would let you sort of trace and see what the cycle time between maybe uh a perception input and a control output would be um because you you get this chain of events for every time that that callback, fires uh or every time a particular test point or at tracing point is reached.

D

You could do this kind of analysis and say uh this: this control output is attached to this perception input um and it should happen every 100 milliseconds, and what we found is that it's happening. You know not every 100, milliseconds or whatever.

A

And so just just for me to confirm this, uh the you are uh you're. Moving from the Chris from Kristoff's information model, you're, creating your own graph like structure. Is that correct, correct? That's that's pretty awesome, yeah and that's going to be very useful down the road so just to give you back, maybe some feedback.

A

um So far. Most of the analysis we've been doing on crafts have been uh reasonably simple and has been done manually directly from the trace files, so by leveraging Babel Trace python API right, which is very cumbersome, assuming you know uh very, very cumbersome, but at the same time it gives you a huge amount of granularity and data.

A

So if you want to do fancy stuff, especially when related to Hardware acceleration, which is what we're doing here, uh you often the case need that level of of details and and especially uh because we are often the case not just tracing things at the CPU side of things. We're also mixing things up with tracers that run in accelerators enough. In the case, each vendor has a specific tracers and profilers, and so we need to mix it all together at the CTF level and the common tracing format level.

A

So that's why uh most of the cases, even though it's kind of like um very low level I'm often the case just handling things at the uh CTF level, with bubble Trace directly digesting uh lttng Trace files. uh That said, uh as we evolve into tracing, maybe more complex, graphs, I think we would leverage uh much of what you're doing right now. So it sounds to me like we will connect uh quite nicely um very exciting man. Thank you. So much for sharing this.

D

Yeah happy to uh you, know, de-duplicate and not not reinvent the wheel, wherever possible right, um and so uh this was primarily for the idea is that this could connect to a system under test um without modifying the system under test, because that's a pretty big component of kind of these safety standards is that the the system as it goes in the field needs to be the system that you test. uh So in that way, you know, there's this isn't modifying the system under test, and then it allows you to.

D

You know, make sure that the things that you say in your safety case are actually being met as part of the process. So I think it's very, very similar. You know to to benchmarking, um just just a slightly different end goal right, um so I think there's a lot of overlap. I guess you could call it a benchmark right, um it's it's all about where exactly you want to draw lines. Yeah.

A

I mean so definition wise. There are some bits between profiling and benchmarking, uh but they are very, very close. I mean we are using profiling and tracing uh to produce our benchmarks. So, uh technically speaking, uh Benchmark has more that comparison um side of things, but yeah we're very close and, and we will diverge everything. No, this is fantastic.

A

um So uh what I'll do is I'll be trying to uh follow up your progress, Michael and maybe I'll ping you from time to time so that we can sync um yeah I would love to um not right now immediately because we're running out of time, but um at some point, I'd love to um pick your brain on how to do data collection and and then analysis, so that we can, maybe um you know, get the beginning and the end uh aligned and that way we can.

A

We can build up on each other right now again we're doing things very raw uh in the sense that we're just grabbing the CTF Trace data um in in going with that uh which is bubble, Trace, python, apis uh and and again we're not Reinventing the wheel. We're just leveraging uh launch files which just use uh bubble. Trace, python, API but still I, do see the value of of uh maybe leveraging that uh graph like um understanding, you're getting yeah.

D

So the other, just the other part of this is It's actually kind of similar to your command line tooling as well, um and that it's kind of a three-phase so there's a there's, a launch and basically you can give it an established launch file without modifying there's a configure that tells which tracing things you want to enable which instrumentation points you want to turn on um and those two things. It will write to a particular place on disk, uh and then there is a convert that will kind of um pre-pre-process.

D

Like you said, the CTF is kind of brutal to iterate through um it's it's much better, to put it in an intermediate format and there's some other discussion on ross2 tracing about potentially picking a better intermediate format and then there's a final stage which basically lets you take a converted data log and run it against a python unit test.

D

And so this graph data structure is available in your unit test, and you can just write assertions like assert that this latency is under a certain amount, or this Jitter is under a certain amount, I'm still working on some documentation um for for general public consumption.

D

um But there there is some Tooling in terms of you, know, running a system under test with the collection, doing the processing and doing the analysis as well.

A

Awesome once again, and and this is fantastic- uh we're going to meet in the middle, um having heard you and again we for the for the reasons that I explained, because we are dealing off in the case with very low interface, low level interfaces and accelerators. We need to deal with things at the CTF level, but then I think down the road as we start exploring more and more complex use cases. uh Definitely uh we want to build upon your stuff so uh we'll be paying um close attention. Cool thanks.

C

A lot thanks for.

A

Having me anytime so yeah, um uh we are pretty much. uh We reached the totally half an hour, folks I'm. Sorry, uh we dragged it a bit longer today, but it was definitely worth it. I just wanted to um very quickly just give a heads up about two uh last bits um that I think are worth uh you knowing the first one is, and it was mentioned by Michael- also rep 2014.. So uh we in the past asked for feedback regarding web 2014. I will just do it once again. Please guys um do comment.

A

Do provide feedback, try to contribute we're very open to have new Authors come join us in in writing, I'm very contaminated every time. I read it again: uh I I get more contaminated and I've spent so much time looking at it that right now, I think I need some um extra feedback.

A

um I wanted to say kudos for uh rifa, who is doing a fantastic work. uh Reviewing this is working on this branch um and so uh that the Outlook of that is is very, very positive. Just wanted to uh send word uh to those of you that uh want to contribute to maybe consider uh converting things into something that saves us all uh myself yourself.

A

Also, as a reviewer and contributor, sometimes so uh one thing you can do and and I encounter often the case- people- that's not so familiar with with this tool, so GitHub give gives you a great uh tool to actually make suggestions. So if you are going through that text and I'll just demonstrate it just right now, if you're going through the text- and you encounter one bit that you think is worth modifying- let's just say this line- you can literally press in here and then make your suggestion in here.

A

For example, let's just put an extra T in here right, and so this is going to come out as a suggestion and if you, uh if you launch this, it's gonna land into everyone, and so we can accept or reject or comment on the suggestion. I know this sounds silly, but uh given the complexity of this piece of text and the amount of people involved, this is actually the best way to move things forward. So I'll say I'll, say just for anyone interested in getting familiar with this um aspects of benchmarking and Hardware acceleration.

A

Please go through the document and try to be constructive by sending comments in in this manner. I spoke with uh with Raisa about his approach. Hopefully rice, you can send in uh some reviews um this way and and that way we can get them. We can get them in really fast.

A

Okay. So, um besides that and encouraging again everyone to just please uh send some additional feedback, uh just last words uh from my site regarding uh the group growing. So as I reported, um the group continues to have quite a bit of uh let's say attendance.

A

We are 30 persons today and again, we're increasing, also the periodicity, uh so um that made us reach out to the uh TSC to ask whether it made sense to make this an official working group, uh because it's right now, possibly the most popular working group in terms of attendance and and contributions. uh So the TSC reacted at least uh intrinsic folks, Catherine Scott, a colleague of Michael, reacted very positively. They suggested to apply for the TSC um and that's what we did, what they suggested.

A

uh We made a presentation for the TSC and we're still pending on on knowing the results, um I guess cat uh and everyone will inform us, uh regardless of that result, we'll continue with our work meetings and so on and so forth, but anyhow hoping good news on that. On that side, um yeah not much more uh to share from my side uh questions comments. Anything from you folks,.

A

I know we're 10 minutes over time, so possibly people want to uh get out and prepare dinner. In my case, um nothing else all right folks, so uh we'll close it in here thanks everyone uh thanks Mustafa thanks Michael for joining and Reporting thanks also Jason and everyone from the team in Harvard for supporting, with with bubble perf uh we'll continue uh next week with uh weekly meeting of Robert perf and next month uh with a week with a review meeting of the progress of the working group uh thanks, everyone have a great one.