CHAOSS Metric Models Working Group, 17 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CHAOSS Metrics Models Working Group 5/10-5/11/22

Description

Links to minutes from this meeting are on https://chaoss.community/participate.

A

First and I'll do the live transcript as anyways as well all right! Well, welcome to the may 10th may 11th metrics model meeting for the chaos community, it's great to have everybody here. um We had a few action items from last time and it looks like there's going to be some demos today um around safety and funding, so that would be cool to see those we've just kind of. As a note too, uh for everybody on this call.

A

We've been really talking about the metrics model, uh working group and the work, that's being done and kind of the structure that's occurring, and we were talking about that in the community call today so really well received. I think the work that we're doing is right on target with a lot of things that people are interested in. So it's good validation that I think a lot of people like the work that we're doing so um regava. Do you wanna? Do you wanna? Could you put the link to your pr in the minutes.

A

So regava was gonna, start structuring the folder around welcomingness the way that we talked about last time. The proposed structure, I think june- you I forget, I forgot who had done it.

B

And she just uh add the pr for for the foundation.

C

B

June reality later right, drinking gotcha.

A

Well and following the model that xingxi had put forward, so thank you for that.

A

I'm going to share about the screen here.

A

C

A

The obviously the structure that that we had talked about last time that sheng chi had put together and- and we had talked about and agreed upon- um there is a pr here so is this: is this the pr regawa, no you're, muted regava, or I can't hear you.

A

C

My pull request.

A

I see yeah you're. I.

C

Committed 18 yesterday.

A

D

Is somebody merging that german emerged that.

A

Yeah, could you tell us maybe what it's about the pr just a little bit, it looks like you're merging, okay, yeah.

C

And then I have committed a ipython sale about a jupiter notebook. I will share with you later about a founding metric model.

E

C

The following and to jason it's the uh some data, helped me to analyze the.

A

C

Magic model.

A

Okay gotcha, so this is for a new.

B

Metric model yeah, actually this matrix model is uh it's created by us by ima and yes, yeah. It's a it's part of sustainability.

A

Okay, okay, so these sean. I think this is easy enough to merge and you can go ahead and do that. Alright, emerging.

D

D

A

Are you back nope still can't hear you.

A

That's okay, regatta, but we'll come back.

A

Nope still can't hear you at home, I see you, I see you just fine yeah.

A

All right, maybe the the next thing this is. These are leftover action items from last time, so one was how to use a notebook and json file to see community results, and if I recall this conversation, it was about how do we really kind of think about other people having access to these notebooks and json files to to see the results that we're talking about so sean? I know you've, given this some thought as well how to like.

A

How do we bridge.

A

um The metrics that are developed as documents into you know something along these lines right, so we have this to get it to people that can actually work with or see the results.

D

You know what I.

A

D

I think what the exam the example there in that notebook that you were just showing is a really good example of how to do that, because effectively it can function without a database connection or having to collect data, and you can see what it does and that's one of the reasons I really like this approach. The last time we talked about it.

D

I think I think the other question you might be alluding to is we have all of these metrics, many of which, upon reflection and having a metrics model working group, might actually evolve into metrics models from metrics. Were you also talking about that?

D

Or did I jump the shark yeah? No, I think so yeah yeah. So I mean, I think. In those cases it's a it's a process and the process starts. I think, with updating the metrics as metrics, just so that they're current and have current examples and over time I think we have to move.

D

We would move some of the more complex or composite metrics into metrics models, but I don't think we want to do that all at once or throw all of those potential metric models that are now metrics quote unquote over the wall, because this group and the other working groups would be overwhelmed by that, and I think it would put some of the core value that chaos brings, which is this taxonomy of metrics.

D

It would confuse people, and so I just think we would I'd suggest we proceed slowly and deliberately and not quickly, because I think moving quickly will cause more confusion. Then value okay,.

A

I I agree, I think the utility that comes from the metrics model working group is in deployed solutions and if we just create a lot of noise around a lot of potential metrics models, that's not going to help the cause really helping people understand how to use metrics in practice yeah.

A

I agree um so basically this this conversation, uh yehui and others on the call really what this comes from is. In the past we had developed metrics, which we would call atomic metrics, and so those were metrics. There were kind of these low level granularity things like I always go to like age of an issue. You know it's like this thing that you can just measure, and it's this it's very low level item before the metrics model working group.

A

We would also develop what we call um composite metrics and they were exactly like what they sound like. They were metrics that were kind of a collection of other metrics, and so what we're learning is we kind of go through and review our old metrics?

A

What have been released as metrics you've seen this right when you've done the translations, that these they're more complex, metrics and there's more atomic metrics. We ultimately want to move these these composite metrics into the metrics model, working group, because that's really what they are. They are metrics models, but we need to be careful not to just say to the metrics model working group: here's 40 comp composite metrics, you know have fun with that and you fix it. So we're trying to be pretty thoughtful on how we do that.

A

um The other. The other kind of question I have I think around this is- is how do we like? What's the end of the line that we provide in the metrics model, working group in terms of deployment for other people?

A

So if we have an individual from a company that comes to the metrics model working group- and they say I would like to understand in this case um you know community activity.

A

So how far do we go? Do we just say here's a jupiter notebook like here's, here's the folder that you need to do this deployment, or do we say, here's a a public facing instance where you can run this model? Do we say we're happy to work with you, as kind of a consultancy group, to deploy this for the repositories you're interested in.

B

A

B

I think here we give some uh give the uh verification with the real world data from from some communities and also, as you mentioned, if there are some new community managers like people from new uh any other communities, let me come up to say: okay, how can I use this and matrix model?

B

uh I think we can give some um consultancy service, something like that. Let's first, let's that's one way, and another way is that, because we provide some data schema in our notebook. So if they already have such ability to to collect such data, they can easily to collect by themselves. No matter use other group life or any other tools.

B

Okay, we already have got, but if they have any writing, I think shouldn't you already provide some other service for that and- and in my part, we can also provide similar the green live service or something like that to help people to collect some data to lighten to to show the stability of one matrix model.

C

A

I, like that um verna you had your hand raised. Did you want to say something.

F

Yes, I had one thought like uh looking at listening to the previous discussions too, so the discussion was like. We have some model to show that what is the applicability which we can test it on our own repository, like on chaos, repository to show what is it like and maybe a second step? If somebody is really interested and want to continue, then we can either provide an instance or some support in implementing them or guiding them and hook them with the software team will be a two-step option for them.

A

D

A

Think that's so you're kind of agreeing with what yahui was saying. I think yes, okay, sean did you have thoughts on this too? I don't know if you've been.

D

I think I'm I I I am in favor of that approach.

D

But just give the thumbs up. Yes,.

A

Okay, um could I ask, I think this is kind of important.

A

Would any I mean I could probably take this out. I think it's important that we that we message this believe it or not as to what we are providing in the metrics model, working group and- and I think we're coming.

A

I think we agree kind of what we can provide, but we need to be really clear to people that this is more than just a markdown file like we are providing sample data and fully deployable notebooks, and if that's not enough for you, we can provide additional service to help you kind of localize this for your needs, and I think we need to articulate that so that people fully understand what it is that we provide, because I.

C

Think there's some.

A

Inconsistency as to how people understand what we are doing in the metrics model working group and if we all.

C

A

That's great, but then we can just be consistent, that a metrics model includes the documentation, it includes sample data, it includes a notebook and we can do there's kind of a range of things that we can provide.

B

I I and I think we got a lot of new new people joined our community. I would like to invite them to help us do so. Yes, and I I totally agree, I we provide matrix model and we provide a verified result with some real world data and and the result and the reading, and we also provide consultancy service, something like that. Yeah and that's the whole things we provide for people. You know from our matrix model group.

A

Yes, because I think from a metrics group perspective, the farthest we go is we provide a definition of a metric, that's it and then we stop, but we don't really provide consultancy around a metric or we don't provide really deployable metrics. That's not what the metrics model or I'm sorry, that's not what the metrics working groups really do, which is fine, yeah totally. That's completely fair. We define 70 metrics. So I like I like this.

A

Okay, okay, good um I'll I'll kind of write this up for next time.

A

Just so we're consistent on what we're providing okay.

E

Do we have to provide json file, is the requirements and do we support other other format, file.

B

Sure, if it's uh it, it can be processed by notebook or jupiter. It's I think it's okay.

B

Do you have some any other new ideas for the data format, except for electricity.

E

Because I just read some studies: uh safety, safety, safety, meter, module in this module and the most dead from survey.

B

E

Yes, so they have not just json like this.

D

I I can also share, as just a brief sidelight, that I think there's a group of folks in open source, including red hat software and the folks in kubernetes, who are working on a project called san diego, where they're going to deliver chaos metrics using something called dash, which is a python implementation of plotly and there may they may join us at some point to share what they've done.

D

I am not advocating for that, but I think that the work that they're doing is closely aligned with the work that we're doing and when they are ready to share or talk, and they have the brain power to engage. I think it might be interesting.

A

You have me intrigued, I don't know about this.

A

D

Yeah this is the these. Are the folks at red hat that I've been primarily with I've, been working through red hat, but there's a lot more folks involved, um I'm showing them how to they've been doing the dash part and I've been showing them how to get data. Okay,.

F

A

F

A

Yeah I I agree, I want to know more um all right, so all right cool. um I know we have a couple of demos that we want to take a look at today, but I just had one more one more item from last time: um an ability to get community data. So it's part of this conversation that we're having it's about providing real world data against the metric model, so that people can understand what the model looks like in practice. I think is really what this is about.

A

um I do know that at the chaos community we are happy to to be a place of of questions around metrics models, so I mean, I think the chaos community would be a a good community to potentially take a look at do do other people have thoughts on how we could go about getting community data.

A

That would be this real world implementation.

D

I mean it's: if it's an open source project, it's pretty easy.

D

Do you mean by community data? Do you mean something other than the trace data? Are you talking about surveys, for example, because when I say it's relatively easy, I do mean the trace data from the platforms. Me too, I think obviously, it's harder if we're talking about survey, data or or demographic.

B

For the sorry data, if this, if some communities they would like to uh to contribute to to to the chaos or to our metrics model, they are also welcome to contribute such data, but the most in the most cases, uh the data we mentioned. It's like election. You mentioned it's true status from the public area,.

D

Yeah- and I think we would- we would be fairly challenged from just a pragmatic, github hosting perspective if we tried to provide large scale data within our repository, because at that point I don't get super technical, but you you're going to have to use something called get lfs, which then also makes the repository more complex to clone and use.

D

So you can store large data in the get large file system format, but you have to have that installed locally and know how to operate it. And if you don't, then it can cause problems. So I do think we want to have small samples in this repository and perhaps ultimately, maybe some more hosted kinds of solutions which.

B

Exactly uh for in the in in this structure, I mean in under the implement implementation. Folder we have data. Such data is the sample data to to use to verify the matrix model.

B

But if, if you are going to to figure out the more more enriched data, you need to get some place to store such a big amount of data like we use we in the last two meetings we we mentioned that we would like to provide some services from open ruler and also the storage service, from open course to provide such service to store the data.

B

If we need such service and we can doing that step by step, but as far as we can as far as we know that we can just start from the notebook sample data and later on, we can store more data from in our data data storage service.

E

Yeah we make our load notebook, um we make our notebook example more useful and then we will start to.

E

Implement our provider server.

B

Yeah, actually, during the interface uh with with these communities, I mean the open course she has talked to the community manager. They would like to contribute some some.

F

Of their authority.

B

To to chaos so later on, uh I mean, after that we can. We can see how to use this data storage service to store our.

E

A

D

A

Conversation, so thank you for that and I think sean.

D

Has an interest yeah I mean we've. um We've explored um microsoft's as your service. We, we don't have credits that last beyond the end of this month, though so it it's, um it's not been as uh it's it's. It was a very short-term commitment, so it was uh very hard to implement and and show utility in the time period that we had those credits, so I'm still searching for uh other opportunities. Well, it fits more.

A

Hostages, yeah.

D

Yeah for hosted services, okay,.

A

Maybe you could talk with june and through slack and just see what that is, that might yeah conversation for them. Yeah, I'd.

D

Be happy to do that and okay, I'm happy to help get the auger part set up and and perhaps configure some some ways that we can dump the json files following the example that was set in their metrics model.

A

Okay right on, thank you one, quick question on this, so I did bring this up so if we rely on for the notebooks, if we rely on freely available trace data, I'm thinking about what you had put together yahoo with respect to your own insights that you were drawing on some of the data that you were looking at, you don't mean you had written those those paragraphs around insights that could be drawn.

A

Is it do you think it's going to be necessary for us to talk to the communities to draw out what those insights are or do you think that's something that we can do as a group here, you know what I mean.

B

Yep yep, I think it would be great. We can show that you mean we can demo it on in the community meeting or.

A

Yeah, so I'm wondering so like my concern is: is that, like I've been in talks before where I've seen somebody who's, not part of a community, give a talk about that community based on data that they received from online trace data all right? So there's a person say it like me: I'm giving a talk about a community that I'm not a member of but I've. I've collected online trace data and I'm giving a presentation and I'm telling the audience.

A

What's what about that community, but in the audience, are members of that community and they say things like. I see what you're trying to say but you're wrong, because.

C

A

Talk to us to truly understand the context of the data that you're looking at. So I'm wondering how much we need to talk with the communities as we draw insights from the trace data that we're looking at. That's all yeah. That would be great.

A

So we may want to think about that as we're collecting data sample data and putting together metrics models that have insights that we may want to try to speak to some community members and.

C

D

A

Appreciate it to be honest with you, so yeah yeah, something to think about.

D

Yeah, I think, for a lot of us- it's I think, important to get evalu. So you know it's an important evaluation step to have the community if the insights were generating heaven sort of faithful directions.

D

Well, there's yeah that they hold face validity at least and yeah, ideally that they provide an insight that they otherwise were not aware of sure, and that.

A

They agree with.

D

Yeah a fair conclusion to draw right.

A

Right: okay, okay, great! uh This is very helpful. I'm sorry for kind of getting stuck on some of these details, everybody, but I just it's. I get asked these things throughout the week. I just want to make sure that we're clear on our messaging. That's all um all right! So we have a couple uh demos june.

A

Did you want to go first? Do you want me to to just pull this up, or do you want to share your screen what's easier for you.

E

A

Okay, I'll we can do that. I will stop my share and I will make you co-host. Okay, you should be good now.

E

Okay, I wish you my screen. Can you say this.

B

B

A

I see your minutes, I see the minutes not here.

E

um I just write and show the demo about the safety match module and we could see our safety matrix module and we have a lot of metric in safety. It means um in this metro model uh we have to.

E

We have to know uh what is the signal. um Second, one equal project safety and in this metric module we have, you can see. We have one two, three, four, five six. We have six 6 metric uh in this metric module and in in this match module we the most of the debts, also from the service.

E

It means that if we want to get the inclusive, such as if we want to get inclusive leadership, metric that and we have to, we have to get survey debt- or maybe we have to get a checklist. So um in my demo I choose a server survey.

E

E

For example, I just I just use this inclusive leadership survey as an example- and this is our this- is my our steps, the step one. We will um give some survey to to which people to this people to to these people, for example, maybe project maintainer or some other contribute contributor in our community.

E

So we give give several people survey and our state survey. Our server content is like this and we will ask in serving. We will ask um maybe maintainer and what the review or renew is. Maybe we will ask in this question and our leadership rules limited by time and we will give them some score in this question.

E

We will give them one two one: zero zero to five and he can get one number about this question. Maybe he think the leadership role limited by time is and not good, so maybe he can give the number two or some three. So this is our step, so you can guess we list.

E

This is our that we distort that. Maybe in this in this metric um we separate this structure, one two, three, four, five, six, seven yeah seven and in each in each metric we have maybe this three smaller metric. So we can give give this a survey number.

E

And we use this server number, we can use our nomination normalization, so we could get the the in each metric. We can get a score.

E

And we can compare from, we can compare from two different uh survey.

E

This is the this survey will be a little different from other metric because uh on this because this metric model, this metric is, will not change faster. So we only have to get survey dates from each month or maybe half maybe half year like this. So so um we can say our so we can see our the results.

E

We can say our realization and we can say each each metric and we and four different different months- and you can see in this distribution of leaders- is a.

E

And then we can use the in in this visualization. We can see different amounts. This result.

E

In this we can see only one month's result for each metric, so um we have another.

E

We have another angle and we can also say the we implementation. The safety metric is because implied uh projector safety. We have for um five fiber metric, so we can say our inclusive leadership.

E

What what changes and we can easy to get what's the um board, what the board, or maybe inclusive, governance, what changes like this and so.

E

So we can use it to see what what the oncological safety metric is and decreasing.

A

Yes, so survey can't be that frequent, like yeah um so jun. This is great because it really helps me overcome something that I was struggling with, which is how do we approach metrics models that are survey, driven, that's something that I've been struggling with for a long time, and I think this this really helps in that regard.

A

So thank you for that. um I think it was easier for me to see metrics models that were driven by trace data.

A

You know what I mean like I could visualize those better.

A

I think surveys are really challenging to me and this. What you're telling me is that that this can be captured in in a notebook still and and be distributed. The same way that we distribute other metrics models.

A

The execution of the model is obviously a little bit different. You know what I mean that it's the collection of data is is going to be different, but this really helps me overcome a challenge I was having. So thank you for that.

D

I think the periodic execution of the survey monthly would probably lead to survey fatigue in the long run, but you know a couple times a year. I think you, you know the ability to store and see trends is, I think, incredibly valuable.

B

Yeah also, you have to prepare for three for a longer time into just choosing the data on the public area. You have to prepare the question and to prepare the actual answer mechanism.

B

You already start this survey every half years or, like you mentioned every year, not that.

E

Yeah we could and use our results to change. Maybe change like yeah, we said. Maybe we could change. Our question use the more useful, useful way to describe how to how to describe our question.

B

Can be can be translated with uh with value I mean yeah.

A

B

A

Super is super fascinating. Thank you. So much for this.

F

I have uh june I have a comment. I really like the approach of taking this and shown this data. I just wanted to know. Is this a dummy data that you are showing just for the understanding purpose, or is it a real data that you have gotten from your community.

E

Yeah, this is for our maintenance and the contributors as a server survey.

F

So this is an actual data.

E

F

Oh great, that is wonderful.

E

But but we get a small small serving debt, so maybe we could get in the last. We could get more.

A

I do I I do wonder, and this could be something to think about, but I do wonder about in the data repository. You know how we have the sample data that is provided there are some metrics models that communities may not want to share their data, so if every if every year they showed that their project is getting less safe than it may not be something they want to show. But it's just something to think about.

B

Yeah, I agree if so, for such data, we have to actually uh see to waiting to contribute such sample data to our chaos community. Otherwise, we cannot.

G

Just say: look at how badly they're doing so.

A

Okay, but I think that's, I think, that's all fair great. This is great jun. Like I said you really helped me overcome a challenge that I was having by showing me this. So this is really wonderful.

A

Agreed this is great, so is the notebook, then what you're, showing us is this? How is this deployable so you? How does this work mechanically? So, do you deploy a survey using your favorite survey tool and then import? The data make the data available to the notebook like. How does this work.

E

For survey, we have our survey 2 server, 2 together. The original dates.

A

So what tool was that? Is there a form that the data needs to be in.

E

This um and other tools, some sentence survey.

B

Like survey monkey where you're, okay.

E

B

To do that and we can go to excel format, okay and then we we do some translate to the to the data like during share here on this folder structure: okay and they imported imported in into the notebook yeah.

A

So do we need to have a process by which data is transformed from surveymonkey or from google forms or from.

B

Sure I think we can add some explanation on that to.

B

E

So yeah there are a lot of tools.

A

That would be good because I know that, like with auger, I mean the work that sean does pulling the data from github and then transforming the data into something. That is is a tremendous amount of work, and I would imagine we need probably a similar process here that, irrespective of the data collection tool that we need, we need to get it into a standard format.

B

Maybe during the you you can, you can uh lose refactor your the whole uh code structure to make it more uh easily understand and and create a pull request to like like us to review it. It would be very helpful for us, I think so,.

E

Okay, maybe next time.

A

Yeah yeah, but I was I was thinking.

E

A

Dude june, are you going to be in the asia pacific call on wednesday of next week too?.

E

A

I mean personally I'd like to start using some of that time to continue to talk about metrics models because we're all here.

C

Because we can keep.

A

It going on a week by week basis, sometimes I feel every two weeks is. I mean it'd be great to talk about this then as well. I mean we could pick up the pace, not not that you're being slow because you're being very fast, but I'm just saying keep the tempo going um week by week, so yeah very cool.

E

I just want to answer matt's question about why why community will show their massive, maybe because they know how to resolve it and maybe the next time they will show them how to resolve it and raise their um self safe score and increase their safe scope. Maybe.

A

Yeah, I agree, I mean if they can show improvements over time, yeah and then we could really start drawing what those insights are as to what was what we're seeing and maybe what was done. I mean then then we're getting into that that the graphic again from information to knowledge, to insight to wisdom and if we can show improvements. I agree, that's great cool, any other questions for june.

A

Great thank you june. um We do have one more. We have a minute and this always happens with this group. um uh We have a about funding, so um do you wanna? Do you have time to show us what you've been working on.

B

20, maybe you can quickly go through your solution. I I think it's more similar. Similarly like what we have done for community activity. Okay, 20. Could you share your screen and quickly share with us your solution.

C

I will share my solution quickly and I try to analyze some funding magical model and I have committed a pro request and in this symmetrical model and they pay more attention to organization.

C

So I try to analyze the following metrical metrics, because the data the raw data has studied in my data source. So I can get the result about this metric. In a short time, field structure is still similar to community activity and there are data inside data set and the algorithm and the algorithm is empty. Now uh I will try to implement it in the next period uh and I have a show.

C

I have employment on some metric about her organization, this university and the first. uh It is about the the contributions of each and each organization, and this is this figure. uh We couldn't see the percentage of contributions by different organizations and the following matrix is about the type of contributions and the first is about a issue.

C

You should contribute organization and we could see the growth of the count of organization and the second is about the poor request, contribution.

C

Another third is the number of organizations of git committee contributions by this figure. We could see the growth of organization count by different data source and um then and these matrix models.

C

Are all be implemented in a short time? Because when we have the definition of this symmetrical model and we could see, we could get the visualization in a short time.

C

Okay, that's what I want to share.

A

So, thank you for sharing this. This is wonderful, so this is with the open, euler community right.

C

um Okay, it's about open roulette.

A

Yeah, okay, and so is this: is this the funding? Is this a metric model that has been built out by this group? Quite a bit, I'd have to pull it up.

C

And the magic model is built in this documentary. Okay,.

A

I see it okay, so we have contro organizational influence types of contributions, okay, so all right! This is interesting because this'll here I'm going to share my screen for a second, I'm sorry go, so this is the metric model.

A

This is the funding metric model.

B

Yes, it's uh it's a from it's defined by emma and some people from us. Okay, we we selected some of the metrics from this matrix model. To give this verified result.

A

So what are your thoughts on like something like if I pick this one like labor investment, so like shanxi? I can see what you're doing and by selecting some of them like. How do we accommodate a metric that maybe we can't get for the.

B

um um Yeah for for the limitation of the data data source, we only can get some public data from the public area so like labor investment investment, it's kind of hard for us to to to to to fetch those data, especially from the different communities.

B

So so the most easy way for us is for the trace data yep and that's fair. So do you think that there's I'll go ahead here? We go.

A

B

For the internal usage, I mean, of course we have much more data for the uh for the community yeah we are not going to for, for example, in one community, we are not going to uh explode those data publicly yeah they just use matrix model or in our community internally share with people internally, because it is some privacy data sure that's the best one. Okay, so.

A

Do you think, do you think it's worth for the metrics model, saying from a from a chaos deployment perspective, here's a jupiter notebook like what chiang chi had put together and here's the collection of some of the metrics. Please understand that there are other metrics that yeah. Of course we don't collect them, but so that's one option. The other option is to to simply remove this from the model and just say we don't we don't care about it. You know what I mean so one option.

A

One option is to remove the getting a lot of feedback. All of a sudden one option is to remove the metric from the model. The other option is to add a disclaimer that we don't collect that metric.

B

Actually, for for, as I mentioned, some data is a it's a not publicly, so I, but it's still useful okay, so I I would not to. I would avoid just simply removing okay.

A

I agree, I agree we added disclaimer by saying as part of the chaos release. We don't provide this, but you may want to provide that as you look at this model, specifically okay, I agree. I just wanted to make sure we were all in agreement: okay, cool um shanxi! That's that's great. um Are you uh in june?

A

Can you, I think, are you ready to think about some prs or we have the startup funding here.

B

Yeah, I shouldn't just help 20 more to the code: okay, okay, okay, good right, perfect.

A

All right great um well, as always we're we're out of time we're seven minutes over. um Thank.

C

You for the demos.

A

Shang ching and jun that was really wonderful and, like I said I really helped me overcome a challenge that I was having it'll be nice to think about that.

B

A

B

Maybe we can say the dragon's um company's presentation: yeah, oh yeah, yeah you're funny your iphone is uh it's through camera.

A

That'll work all right till next time, everybody thank you so much and have a great day. I hope the rain stops for you. Okay, take care! Thank you. Everybody bye.