CHAOSS Metric Models Working Group, 20 Jun 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Metrics Models Working Group June 20, 2023

Description

Meeting summary can be found here: https://chaoss.discourse.group/t/metrics-models-working-group-summary-june-20/189

A

There oh wait, maybe because I don't think, we've officially announced it Dawn.

A

So for those of you that don't know, uh Don Foster is joining the chaos project uh in August, as the Director of data science, which will undoubtedly have uh probably pretty big implications for this group in particular, I would suspect, as we think, about metrics and metrics models in practice, I'm still recovering so don. If you want to say anything, I think everybody on this list probably knows, but nonetheless.

B

um Yeah I I will say that I'm I'm super excited I I. Think like like Matt, said I'm gonna start sometime in in August, so so what I'll do is my kind of first first task as the Director of data science to start putting together kind of a plan for what I'm, what I'm going to do and let everyone provide feedback on that um so that you all can tell me whether or not there are some additional things. I should do.

B

Help me prioritize, um so I will I will look forward to that, but that that's probably the first thing. That's probably how I'll spend a chunk of Chuck at the time uh when I start in August I will be going to Fosse in Portland um and I'll, probably in July, so that's July, 16th 17th ish somewhere in there.

B

um So I'll probably do that under the chaos Banner, because I'll I'll have left VMware so so Matt. That's. Why I'm trying to convert my presentation over to the template so.

A

That I can get it if.

B

I can get my presentation at Posse using the chaos template.

A

Actually I didn't see if there was a response in the chat this morning in yeah.

B

I I looked and I I, don't think so there last I checked they were. They were pinging people who they thought might something: okay,.

C

You need this slide template. Is that what you need? Don? No.

B

I have the slide template um the the issue is the way the slide template was built. I can't get um high resolution images of the backgrounds that they used and I need to convert it to Keynote, because that's what I use? Okay.

C

B

So I need I need the images that they used to build. It is what I'm, what I'm actually looking for got.

C

B

Or someone who knows more about Google Slides, who can somehow extract those the high-res images that were used to build it, um because I couldn't figure out how to do that? But don't.

A

B

I was kind of looking at Elizabeth, but yeah.

C

I was gonna, say: I can't mess with it. I don't know that I've ever tried to do that, um but I can't around with it for sure.

B

Or we can wait for someone from Africa to to get back to us, yeah.

A

Maybe just give it a day or so.

D

A

um All right cool, so we had a couple metrics models from last time: one was sustaining contributors, one was business Readiness and then one was this influence metric model um yahui. You were gonna. Take a look at this first, sustaining contributors. Did you wanna? Did you have any comments on that? Did you make any progress? It's? Okay, if you didn't too.

E

Actually, I prepare a short demo about the whole contributors. I actually had just attached. These slides, yeah I can share my screen. Go quickly through what I call it there.

A

E

Go yeah I'm sharing my screen. Can you.

F

See it yep yep.

E

Yeah yeah I I, actually I I, just prepared this very, very short, slides quickly this afternoon and and but actually the whole things was, would would discussed for for our romance together with the professor young, who also attended today's meeting, and it's about contributors profile and the contributors matrix model, uh we're thinking. We are thinking about uh a lot, how to create the full view of a contributors profile and how to use that in the different matrix model related to the contributors. So, first we create this contributor profile.

E

We divided the whole profile into three dimensions. First, is about a milestone or contribution frequency and from this Dimension they category provides the contributors into different role like status, who's, never touched a project or Community before Cairo. You know, contributor alleged a little bit more or less, not so frequently regular. Who is a long, sustaining a contribution, make contribution to to the project and core contribution who is taking the main workload in this project and dormant, which means who is not active anymore as one state.

E

The other dimension is about the ecosystem and we think all the community, with the composed of a leadership and participant a leadership, could be organizations could be one or more a group of organizations or could be an individual person or a group of individual person and the similar to the participant concept categorized into organizations or individuals.

E

The the last Dimensions is about types contribution types we distinguished.

E

The different contributions as a Starfall coach, which is the most simplest, uh simple contributions for community and issue track platform like we're, using GitHub or bugzilla, to track the issues and and and the code contributions and its pull requests, and also about Forum, the different Forum like a q, a forum and also like a q, a forum like or this course we use in chaos and also we use in some chat platform like uh like us, like at the media channels, Twitter's Facebook and online offline events.

E

We also do some contributions under these events, so we creates a whole contributor profile into the into three dimensions and around these three dimensions. We create the matrix model, A, contributor, matrix model, and especially I emphasize here, with the events impacts, so uh the whole contributor Matrix models, I I, divided into the a couple of the different sub models like track models, activity, model mutation, model, promote model for fit out model and re-engage model, and under this each of single contributor matrix model combined with Pro contributor profile.

E

We continue to to divide one sub model into another second level: sub model like Milestone retention, model, Ico rotation model and types retention model, or what does that mean? So a track model means the people who never touch the project before who got chance to come into the project and and make the first contribution. We focus at this part and the activity model.

E

We will focused on the people who who is native, who is active or not active in this contributions in one specific project or communities, contributions intention you know just in the context words, it means how many people left, how many people live and how many people reserved into the community and and become the core contributors.

E

uh Something like that and promote and fill out and re-engage means the different thing, and so the contributor could be get promoted from a Cairo uh Cairo contributor to uh to a regular or to a core contribute contributors or Fade Out. They could you know, from core contributor fit out into a regular or or or career contributors really engage, because some contributors would you know, step into the silence for for over years or months, but because of some event Trigger, he will re-engage into this specific community.

E

So I I listed an event impact here. I divided the events into the three types. First is the regular events, for example, the some important holidays, such as Christmas, Chinese, New, Year and some planned events.

E

You know some online offline activities and uh priority periodic version release and some emergent events. uh You know some big news uh in open source world or in the industries, something like uh chat GDP just to come up yeah.

E

So the whole thing is to describe the all the metrics model around these contributors and together with profiles and and event impact. So next exercise, I I would like to uh you know, give us an example. So here is the contributor milestone in the different model. Here you can see.

E

They step into the different state like first people. First, these people who never touched or contribute to this community before and we treat it like the state race and after uh because he makes some first contribution like focus on or make pull requests or even ask a question on on some Forum. Maybe he will be attract, become a Cairo and Cairo would be promoted to the regular and even so, to the call and all the states could, you know, become a dormant because of the turn and also call regular, Cairo and Donald.

E

These four states could be changed from one to another because of the contribution change and their their states change, and they use this concept and doing some verifications on some big projects or communities, so we're using three communities, as our first try out examples. So here first, is the rust project. This is the programming language and focus on the focus on the memory security things which is very popular, not only some big company. How? How interested on that many many individual person make a great contribution into this community.

E

The interesting thing is that um I can I can give you a general interaction for each a bar means it's a different year and the current one means the country has regular contributors, and this purple bar means the the new regular country new, regular contributors attracted this year. So- and this is the current core contributors and the the blue one means and the new call contributors attracted in this year.

E

This is the new core contributors so which you can say that the core contributors from this color, yellow color become more and more no matter the count and also- and they keep the very stable increase into change and the regular amount also Inc keep increasing, and this increase change has been kept up over more than 10 years from the earliest history, from uh 2010 into the current year in the in the last year.

E

2022- and this is the first community- and we can see another community- it's a this- is best code, another very famous uh open source project, including myself, also using it quite a lot of times, and you can say that in the in the 2070 and the contributors I mean the core contributors together with regular computer builders has become, the Harris has achieved the highest number and after that it's decreased. But so far you can say it's it's come up again, which means some uh some big news or some good news happened into this area.

E

Like vs code integrates some uh useful plugins, something like a copilot, so many people start using under three contribution back to this community and another. Another example is uh kubernetes.

E

As you can see that and actually a we started, the history data from the 2014 and the and you can see in the 2019 and the code, core contributors and the underlying contributors achieved high risk numbers in this year and after that, it's decreased. But the interesting thing is that the amount of contributors I mean the call contributors.

E

They are very stable, not decreased. They are the rest, almost keep the same. So if weaker, if you treat this curve, uh something like a gardener curve, I mean in the 2019. Maybe they have changed. The highest um amount attracts a lot of different people into this project, but our flight.

E

It will decrease its normal things, but uh we believe that, because of the core contributors keep the same size, they will come up again into a stable curve, something like vs code here and but still the the First Community I mean the rust gave me a shock after I saw this thing.

E

Thank you graph. In the past. Over 10 years, this project incredibly increased, no matter the regular contributors and core contributors. So uh this is just for a three three examples: I use the Milestone or frequency profiles and, together with with some attraction, some traction and and the reservation, our orientation model to describe the different change or change. I think, and we can see some interesting information from here and in the future. I would like to extend and Implement more uh Matrix models around around contributors. Maybe there's more.

E

Some interesting information would would give us yeah that that's basically it's a whole presentation. I give you.

A

So, thank you. um You make beautiful things. I can say that they're very easy to look at and very easy to understand. I love it yeah.

B

This is really. This is really fascinating. I really um I really love the way. You've done this, because uh what I? What I, really like about it is, is that you can see what happened to the regular contributors that left. You know you can see that you've attracted some regulars and then they've stayed regulars.

B

um You know, and some of your cores have have left and gone other places. It's it's really. It's really interesting to see um to see the evolution and and this type of visualization I I yeah I, find it really interesting.

A

So I'm I'm kind of curious, like folks in ospo's like would this be helpful at all in an ospo and, if so like how or is it more of a community perspective? You know how. How might this be useful.

E

I'm thinking, uh if this question asks me um how to use that uh the for for the first scenario or case is that for the different uh they have the different promote of result pass, and so, if we could abstract the general pattern from the different communities, they could clearly know what happened in one Community, especially for to to promote more contributors into this community, and also, if we're thinking about, is what I mentioned earlier about event impact.

E

What kind of thing would have a big impact to this community if we add some time time tag into these communities, change I mean the whole Trend change. We could find some interesting thing and also, if we add ecosystem contributor profile, that dimension, we clearly know who have a big interest on some specific community, no matter as organizations or individuals.

E

So we clearly know that.

B

Yeah and I I think that this is really useful both for both our community managers, because I think you know from a community manager perspective looking at the projects that you are responsible for, managing being able to see whether your core contributors are sticking around and whether you have enough regular contributors to replace them and whether regular contributors are actually becoming core contributors is something that I think is important for for Community managers to look at I think from an osbo perspective.

B

I think that this would be particularly um also interesting to look at because I think it's important to you know, especially for the projects that you're maintaining. So you know, kind of the company originated open source projects. um What's what's happening, you know with that. Are you are you attracting contributors, or is it kind of the same? The same group of people continuing to do all of the all of the work and I think also from an Hospital perspective.

B

It might be sort of interesting to look at kind of an aggregated view across um you know certain types of projects or sets of projects and I think this is a this. Would be a good way of understanding whether or not you have enough contributors to sustain the work that you're trying to do over time? I think it would be an interesting way to look at it from from ospo's I'm curious, I'm, really curious. What Gary thinks.

G

Yeah I mean I, think uh this model of how we think about contributors is a very in-depth and uh comprehensive way to think about how people get involved in projects how they may manage their relationship with the project over time and I'm interested like what metrics we can attribute specifically to the transition states there, because I saw that really um that awesome like graph that flowchart of they start here they go here. They.

E

G

Go here and they.

F

G

Up here and I feel like there's a lot to dig into on how each one of those States happens if we have metrics for each. uh That would contribute to the likelihood of that happening, um because this, like idea, feels um kind of like a super model. The same way that uh don you had mentioned viability feels big enough that maybe we should break it into like three or four different things.

G

um It feels like large enough that I would want to know when am I at risk to see a maintain or churn or when am I at risk to see a maintainer come back or when am I at risk to see folks turn into uh from casual to regular maintainers. You know, and is there anything that projects can do to uh you know lean on different metrics or different strategies to try to create that change depending on their state and where they are in their maturity.

G

uh As far as like how it fits into viability in general, I think that contributors uh is a very. um This would be great to roll into viability, because the proxy that I have for what this does is not as uh clear. It's not as concise as this model with I. Think you're really just amazed me with this graphic because it looks so cool um but yeah. It's I hope I said enough to to not lead with that foreign.

A

Sorry I was taking some notes.

F

um I have a technical question. This is like so amazing that just looking at it and the flow it's so clear, like you know, the message is Crystal Clear from from a technical perspective perspective. How did like? What is the criteria that you used to classify like okay? If they have made 10 commits, then they are turning into core and then or regularly? What is the criteria that help you to move the particular contributor from a court or regular or uh different state like?

F

Are there any uh particular data points that you choose to decide those things I I.

E

Think I think this question is a very good question. That's that's. Why, like us, four four or five weeks ago, I I got questions on slack to Dawn, so how to like how to distinguish the call contributors and code or on so I mentioned a pirate concept and and the bus Factor. Something like that, and but after that, I I do understand how to distinguish the core contributor or regular, contributed.

E

Regular contributors really have some context upon different communities, so I would like to so in my case I still using the you know the whole contributions divided into the fifty percent. Fifty percent sorry 80 contributions in a whole year which was taking or making and by the smallest group of people. We treated things as a contributor for the regular contributor, Matrix Matrix thing, who should give a sustaining contribution like at least one month in the in the 12 months of each year.

E

They have to make at least one contribution in each of single months, which means the regular regular so but I I do believe that this distinguish is not a common solution to distinguish or to give them the con definition of regular or call so I I'm, very open uh to to make any new definitions for the regular and and the core and I would treat it as a flexible way to define the different definition based on the different contacts. That's my answer. May not. Okay,.

F

Yeah no thank you. Thank you. This is still helpful, like I was more thinking of how we can contextualize it to different communities. Some has a different. You know pattern of contribution, but still the way you have adopted it like keeping 50 and the other at least one which makes a sense in different context. So I I still feel like this is still a valid way of classifying them.

F

A

Yahoo, can you go back to the slides for just a second.

F

A

Can you go up yeah to that one right there, so the different types on the left.

F

A

Are those are those what go into identifying people as regular core.

E

I only make exceptional case for the star folk watch, which would not content into the contribution either as a core or or regular contributions, okay, but for the rest of the other contributions, this should be, in my case, I recognize things either as a one type of contribution. Okay, in the background call so.

A

In that the graphs that you were showing those two four six.

E

Honestly speaking, I only because of the data sources limitations so far, uh currently we only considered issue, part proquest part and the slack part select, Channel, part and and Twitter.

A

Yeah, okay, so those those four are: what constitute the graphs? You were showing okay and then do do. You vary the weights of those between regular contributor and or contributor. Actually,.

E

No I think all the contributions I so far as I know, should share the same weight. Okay, until we got some more, uh you know more suitable way to to give them the different weight.

F

A

That makes sense. Thank you.

A

Yeah go ahead, go ahead.

D

Thank you for the introduction and I think the model is really impressive. My suggestion is, maybe we can develop a family of metrics based on the uh based on the model. For example, the number of new members joining the project, the percentage of poor members staying in the project all based on the the gravity graph model is just shown as yeah yeah I I just have a suggestion. Okay,.

E

Yeah yeah I I have to mention again actually all the thinkings behind the contributor profiles and all those matrix model with the with the what discussed deeply with the with the professor Liang. Actually, we made the whole contribution together under all those Concepts.

A

Okay, so this is I think we could probably talk about this the entire time, but we do have a few other things on the agenda. um I guess one one question for you, Yahoo and Liang is where, where do you want to go with this? So I mean it seems like it's extremely positively well received from this group, and so how? What do you think the next steps for this are yeah.

E

Actually it's the next step. I was thinking. This is my first time firmly to to to to introduce this metric model uh openly.

E

So you know you, you could guess what I might want to do in the in my next type, I want to contribute this all those Concepts into the chaos and the working together with you guys to to make it more useful, no matter in the community or in for the hospital uh people yeah.

A

So I kind of like to Gary's point: do you think that the contributor metrics model is a super metrics model and what we need is metrics models for say, attract a metrics model for activity. You know like contributor, attract.

F

A

Activity, contributor retention and so on, like an individual model for each one of those yeah collectively, so that would essentially be six models that I'm looking at right here, as we currently Define them in chaos. Is that how you see it? Yup.

E

Because you can't say that, for the h0 single model, we divided it into again into the Sub sub uh models like Milestone retention, High code retention and types of retention model. uh So in total count so far we can say it's um how many like 18.

E

matrix model we have around these contributors matrix model, the the a lot yeah.

A

Do you think that they would be the models themselves would be quite similar, just with slightly different descriptions as to what we're trying to accomplish with the composition of those types, those contribution types.

E

Yeah they could be, but they are ready to support for the different use case, for example, um Echo retention model.

E

um Would you already care about how many people has been reserved in one Community because of some reasons like like? We have released some very good fantastic features in the in one project, and we will see okay, they speak things or events has has helped me to return some people and a lot of people in that period has been retained and because of I gave some some very quiet features make me lose a lot of core contributors.

E

That's very important for one communities, yeah.

A

So maybe the first step then would to be there. Maybe just ask for the next meeting that maybe what I'm looking at, like with Milestone Echo and types to maybe bring some really preliminary thinking forward description, yeah document would that be okay.

E

Short short, actually doctor, actually Professor, Liang and I have written some something in Chinese just to make it convenient make make both of us communicate with each other. But it's really really nice for us to translate into English. Okay.

A

Yeah that'd be super helpful, um at least for me, to just kind of get my sense on how we can build this out. Yep yep sure, okay, great thanks, um all right. That was pretty cool, um any any last comments for you, Hui or or Liang.

A

E

F

Yeah yeah all right.

A

um Well, we're I promise. We won't get through our entire agenda, but that's okay!

A

It's well! It's great when we like, when we have a great thing to talk about, and so that's really cool.

F

I have missed it because I uh missed a meeting. I I thought our meeting is next week, so I I have not got a chance to work on this, though it's not big thing, I missed it in the the scheduling in my mind, so.

H

F

Definitely next in the next meeting, it'll be done for sure.

A

Actually I mean vinod two, if you can I mean, maybe if you can just do it, asynchronously yes, I mean because it like I said it seems like this. One was pretty much ready to go right perspective, so maybe you could just post in the metrics model, Channel. Okay, when you think it's good to go and we could just carry on the conversation there right thanks finad um and then there was this influence, metrics model and it's currently a metric.

A

So is Elizabeth I'm kind of curious. What you think one of the suggestions was to kind of remove influence as a metric and start looking at it as a model. Do you logistically, like honestly, just like from a WordPress perspective like what would what would be required here super.

C

A

C

I mean we would have to change the template and you know obviously build that document out in a different from a different perspective. But as far as WordPress goes it's it's literally just changing the path to the GitHub doc. Changing the title, maybe changing the keywords. It's totally fine, not.

D

C

C

I would say, most of the work will will come in just the uh rewording, the doc that we have into and fitting it into more of a metrics model. uh Template okay,.

A

D

A

And I can work on that. I. Just put you and me down, yeah.

D

A

Okay um and then maybe before we do it, we could kind of like what Bernard would do just take a look at this like share it asynchronously with folks.

C

Yeah sounds good: okay,.

A

That sounds great.

A

um All right, I I was going through.

A

This is for yahui as well, so I was I kind of went through through this I cleaned it up a little bit just to get everything towards the top, um but one of the requests was to take a look at the different metrics models that we have with respect to the different metrics that are in those metrics models and take a look at which of those metrics are probably higher priority and which are lower priority. Is that rightlying or I'm? Sorry?

A

Is that right, that's kind of what you're asking for yep, and so it took a little bit of time to go through I had a few questions that came from this um and it kind of raises a a another question. So one was I think from the starter project. Health metric model perspective, I, put all of these as high. We have these all pretty well defined.

A

There are, you know like this is technically um time to First Response, that's technically one metric that we have released in chaos and you have it broken out as two that's I have that's fine, you know, and that happened just a few times.

A

um There were a few metrics in here. So, for example, like code contributor count, I just put. Is this the same as contributors based on our list that we have over here and.

B

A

Code, merge ratio is this the same as change request acceptance, ratio.

E

E

Similar definitions- actually they all are originate from the chaos metrics. Okay, yeah, just just I changed the name little bit because we have change request, but here in GitHub or gitty we call it pull request. I haven't made the things understable.

A

Okay, yeah so I, guess that should I should I care too much I, don't know what your thoughts are on like these are the metrics that are inside of that are being used by Compass, like if I click on any one of these These are. The metrics that are in chaos should I care too much. What do people think should I care too much about the fact that these don't align perfectly.

A

If you just say that's okay, these were inspirational, and these are the metrics that Compass is using. You.

E

Know what I mean yeah I understand, because in our chaos metrics we really have um there are metrics have a different level. Some some Metric is very atomic.

E

We can clearly know the definition and how to how to deploy an implement it and some Metric is kind of general and of course we have some filter to filter out some specific case for the environment.

E

So in that case we I have to add some more uh correct words to to to to to describe this special case. That's why I use the different name, yeah, yeah, yeah, yeah, so I don't know how to handle, but uh but, as I mentioned, actually they're all coming from our chaos. Metrics.

A

I'm curious: maybe what other people think here.

A

D

A

F

F

Like it's a time if these things are like uh as a context, they feel the different name is more suitable, then we can really think our metrics that we have because it's a we defined on the contextual based on more on the thinking process, but the implementation might need a revised name. So maybe we can look at the Matrix, for the revision in the name can be one option too. Okay,.

A

Other other thoughts.

A

My my reaction is I'm, a like At. First I was like oh, we should keep these things aligned perfectly, but then the better part of me is like that is, is different people to play different metrics in different ways like that would be kind of an errand or a task in the chaos project. That would probably drive us crazy to be honest with you as different pieces of software kind of, are inspired and deployed different metrics in different ways.

A

That's my reaction and then for us to keep try to keep these things perfectly aligned um like we. We have a hard enough time in the chaos project like keeping our own community in order, let alone in order with other communities like that alignment, I think would drive me. Maybe.

E

I can either for each of single metrics deployed in the compass and add one exactly the same reference chaos metric link yeah to.

A

Tell people yeah this was inspired by this metric or these that would maybe that would be probably perfect and that I do think as part of this. um So that's a great suggestion there. There are a couple metrics in here so, for example, commit PR link Ratio or there was another one like opr issue, linked ratio that we don't have in chaos and I think this is interesting because I we've talked about this before and I think some of these are quite quite interesting. I.

E

Remember Elizabeth has already helped us to create many new metrics, especially for those models, but I I can't remember to do. We already have those metrics in chaos. I I cannot thank.

A

You correctly I, don't think we have linked ratio- okay, okay, but so like this is also inspirational for, for me, like I, look at some of these I'm like well. This is silly. Why don't we like this was such a straightforward metric? It's probably something we should think about anyway, so um so with that in mind, I'll kind of continue to go through here and think about um anyway, with with this conversation in mind, I'll continue to kind of work through this list and just consider what I own my own personal opinion is.

A

These are all based on me. Thank.

E

A

E

Important yeah.

A

Kind of what I'm, what I'm, thinking and I only put these as medium at the moment, just because I needed some clarity on this conversation that we're having right now, yeah. Okay, we.

C

Can also maybe have the common working group look at the list too, and certain ones or you know like go through them as a group. If you don't want to do it on your own map, we could maybe take five minutes or ten minutes as a group and look through it and just decide.

A

Which we should surface okay? No, that's that's great!.

H

I was gonna ask just because this is what I was working on this week was also like.

H

You were talking about how the software project might relate for for the chaos project, whether we might consider um for some of the models that the especially like I was looking at the starter model, whether linking that to actual implementation could be a priority for places that we're looking at as starting places for people so that when they open something like the starter model, it's like really clear how to get to the pieces of software that have implemented it.

H

So it would be only you know what is it four or five metrics that we're really trying to keep um some kind of some kind of documented link to implementation in place.

A

Would you want to Jen? Do you think that would be in the model itself so like if I was to come here like there? That's.

H

That's what I well I! Guess! No I guess when I so I as like, as a user of this and I, was doing.

F

H

I was clicking through to each model and looking for how to deal with it on that page, so I guess it would be more on the model page, but yeah.

F

H

On the metrics page, but that those metrics were part of this model, where we're saying it's potentially a starting point for people, we might take extra care about the links to the software implementation for.

A

H

A

Have in the past we had in the metrics themselves, we would have this like tools providing the metric I, don't know if you ever saw that, but it doesn't it's really hard to maintain. We found over time.

D

H

That's what I was getting and that's why I was wondering about doing it like just for the starter, metrics or.

A

H

So that it was just a smaller list, but even that might just still be too much yeah.

A

No, it's it's a fair point, though I'd like to bring people immediately to like, like yourself like just show me the model in practice, so that I can ask questions against the communities that I care about. um It's probably something that I like that idea. Something definitely to think about.

A

um Okay, we have negative one minute left in the meeting, so we're done.

A

um Thank you, everybody for joining us. uh It was great there's more to talk about and I think we have some things that we can start off with next week. I will say this next twos. The Tuesday two weeks from now is that the Fourth of July.

A

Yes, yes, it is okay, that's the Fourth of July in the U.S, so I'm pretty sure we're not going to be having meetings on the 4th of July.

A

um If, if you all want to meet that's great I'm going to be out of town anyway, like I'm, taking my daughter to Atlanta and so I will not be available, but the meeting will be open and available if you'd like, and maybe we can talk about it asynchronously as well. But my guess is it's going to be pretty lightly attended in, particularly in the United States. So, okay, great thanks, everybody, it's good to see, see y'all I, really.

F

A

It bye-bye okay, take care bye.

D

Thank you, bye-bye.