Node.js Benchmarking Working Group, 18 May 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Node.js Community Benchmarking meeting 18 May 2017

Description

https://github.com/nodejs/benchmarking/issues/107

A

Yeah, hello, so welcome to the node.js community benchmarking meeting May 18 2000 17. So far we have in attendance myself, Michael Dawson, uh Canole, Pathak and Kyle Fernan. Oh hey! There is with me and I'm, okay, sorry and Taylor wall as well. Okay,.

B

I just walked in okay.

A

Great thanks: that's for all for you all joining and yeah. If you can update the the minutes so that we keep track of always here, that'd, be great I! Think that's right! So on the agenda today we have stand up just generally going around table as to what we've been working on since the last time we got together. I also threw on discussing adopting the TSC participation guidelines and I'll go in a bit more detail when we get to that, we can go over actions from last meeting.

A

We can take some time to review the open issues to see. If there's anything we should talk to on that front, and then you know we can put together our actions for the next time we get together.

A

Is there anything that people want to add to the agenda before we.

C

Get started on that also I have few questions around Oh memory, um the memory consumption um numbers that we collect for akmed like the footprints or matrix. Oh, but we can. We can talk that in the other business yeah.

A

Okay, so let me well, let me just add that so we don't forget into the after the open issues review, maybe actually, maybe before the open issues review, we can go into questions on acne air right yeah, here number.

C

Report and then even I wanted to check about the status of landing the EIS benchmark or that's been there for a while right.

D

C

You know also wanted to do thoughts about taken power, or do we want to add that right.

E

A

Okay, great we'll talk about those.

D

Anything else hi Michael.

D

My name is cap Andrew. We have talked in the meeting, yeah yeah I, don't wanna, say hi to anybody here, that's great. If.

A

You can also add in if you can go like go to the link in the issue for the minutes and add yourself in in terms of like who's present. Okay, just you know your name and your github handle so that people know how to contact you afterwards if they want to uh-huh.

D

Okay, Delta is a faintly usual. The meeting issues right.

A

Yes, you're in the meeting issue, there's a link to a Google Doc, oh okay, and in that Google Doc. That's where we're taking minutes and you can see like there's four of us so far you can add yourself as a fit person. Okay,.

D

Cool I'll do that great.

A

Okay, um so onto the agenda then stand up so my sell off start I'll volunteer to start haven't had as nearly as much time as I would like to work on benchmarking related things. A few things I have done is set up the charts for version note version. Eight I still have a little bit of work to do there in terms of there were some comments on the color, so I have to think about whether I can find some better colors and then land that p.r.

A

um The other thing is, there have been a few conversations. It seems that Intel is donating, a benchmarking machine like a like a hardware, machine native hardware, machine through kneel form, and so at some point in the next couple months. I expect that to show up we'll work on hooking it up to the CI- and you know, I think the first. The discussion so far is the first thing. We'd probably want to target that at is letting collaborators run, the benchmarks that are built into the node core they did.

A

They tend to run for a long time like they could run for a couple days, but that would be a way where you know, because it's a dead machine, it wouldn't be starving out or existing benchmarks. Those could be run. You know as needed.

A

F

A

As well go in the order of the people in the list, so canal you'd be next yeah.

C

Actually, given all right, oh so for me like a mostly I, was like looking into like Achmed numbers and how we can, how those footprints look like the memory footprints and how we can like go through those the memory print, the memory growth trend. Do we do that or not today, and that's what like, we can discuss it after.

C

We discuss I, think around the stand-up, so you're not like not specifically in the benchmarking like I, contributed, but I was just taking a snake in two different benchmarks that we run today and how we can like improve them. Sounds.

G

C

H

Kyle I'm just getting up to speed on this particular working group. Okay,.

A

Okay, Taylor I.

B

In a pretty simple abode, actually I, don't I, don't know much about this working group, I just kind of was sitting in to see what you guys are talking about. Okay,.

A

See just come to find out.

A

D

Okay, perfect Jim Peng, oh hi Erin, my name is Jeff Andrew. Today is my first time to tone fifth meeting I'm, a master's student to University of New, Brunswick, Canada and I'm. Getting involved means the Michael's no Jeff forget for now I'm working on the implementation and the collection. A set of no TS forget ability, oriented benchmarks, and this benchmark suite is to measure and evaluate the no G a scalability yeah I. Don't wanna try to introduce myself and glad to see you guys here today. Thank you, okay, good.

A

Thanks, okay, so that's the rocket! We got everybody there. um Next thing like I, you ever be: alright, oh did Annie Lendl done from engine. Ok yeah! Can you guys so yeah I'll, try and add youth in them, so yeah, you guys, can add starting me to the purpose of people present wrong. Who else was that with you well dump and a.

G

F

And I mean yeah I mean okay, yeah.

A

If you could go to the minutes, the Google Doc and add, like your github, fill out your names and github by these that'd be great. So, okay, so I will add you into the roundtable. So what have you been up to since last summer time so.

G

An ill can it should the water status since we kind of worked together, but.

F

Yet we would like to earlier, we have we come here and we presented the node DC, yes, workload which can be used for scaling as well as different stressing.

F

So we are ready with almost one point overs and now, and we wanted to see how we can make it part of the node benchmarking harness and we are ready to meet those script and whatever action needed.

H

F

Because we were trying to find out if what script needed so we can add to the environment. Let us know that that part is not on the website.

A

Right so I guess yeah. We we've got that on the agenda to talk about the details, so we do that when we get to that part of the done, but that's okay, yeah.

F

Absolutely okay and I: don't have the question to jipang. He was talking with a scalability etcetera if you could talk later in the open, because we are interested in that part and we can see if there's a time and or I can contact him separately and talk to okay.

D

Yeah, actually, my initial idea is also on a coward you're. The benchmark and not you've called no TV ëif right yeah, you kind of covered in my haha bad mathlete yeah.

F

I generally suggest yeah. Thank you baby. One week you should have one find all my friends just worth it, but we can talk later sorry, okay, yeah.

A

I definitely suggest anjaii thing to make sure that you know that benchmark is included in either researcher or investigation that he does. Okay, yeah.

D

A

um Great okay, anybody else on the call I missed no okay, um so next is discussed. Adopting TCC participation guidelines um I just bring this up in that the TSC has adopted a set of guidelines, for you know when people are considered active or not active in terms of being a member of the working group find the case of the TSE. Is you know, being a member of the TSE itself? I thought?

A

Maybe we could discuss whether you know we think that makes sense for the benchmarking workgroup as well and basically you know I pasted the text in there. That's that's applicable, you know it's. If the member you know if, within three months you attend less than 25% of the regularly scheduled meetings, you don't participate any the discussions or don't participate in the votes to take place. Then you kind of automatically be dropped for them from the membership of the workgroup.

A

Of course, you can continue to participate in meetings in the observer and in our case, for the benchmarking workgroup. You know we haven't been that formal about bringing you know what it takes to join membership. So you know, if you came back and we're active, you would probably just become a member again in some other work groups.

A

It's if there's a more formal, I, wouldn't say formal, but it's like you know, you need to come and be active for a while before you would be added to the work group in the first place, I mean I think the main reason the TSE did.

A

This was that you know you sometimes end up with, and even if you look at our you know the members for our workgroup, a good number of people we haven't seen for a while, and it would just be one way where you know we could make sure that the you know the membership list, at least, is the people who are active, I.

F

Think Mike attended IRB planning to have some voting other things on like different benchmark or the area. We want to do. I think that even you, Polly Nita, okay, Fe members, right yeah, whether that's the only time where yeah too much that Ling into like. Are you going to give one company one word and all individual one word? It is start becoming funny becoming complex. There's.

A

Existing rules already that you know that we, the initial governance for the working group, was, was adopted from the the templates that's used to cross and there are rules in terms of you can't have more than two thirds earth. I forget what is it something like thirty percent from a single company I have to go by I mean we can look at the document to see what it is. It's either a quarter or thirty percent of the member.

A

The voting members can't be from a single company, and otherwise it ends up being you know, a single vote per per person in the work group, but.

F

I think you'll see when we start making those decisions. I don't think we can be more strict about it, but I think the foot. Now we want to encourage more participation, yeah.

A

I, certainly I have no no issue with. We want to get as many people as we can so anyway else I'll throw that out there and if it sounds like you're, not necessarily in favor of us pulling that in in necessarily right.

H

C

So also I think the decadence of the three months, because I think that we do like meeting in like once in two months on one once in three months. So if I just miss one meeting, I will be out. Okay, see.

A

Ya, no I I know that we have some challenges on that front, so we need to to tune it to make sure that that's not like it actually is. If you miss yeah, we need to do it to make sure that that kind of things don't don't happen. It there's no intention that it's you know you miss one meeting in you're out, it's more like you just haven't shown up for a year or something like that right, I'm, but yeah I, don't think it's a big deal.

A

I just thought: I'd put it on there, because the TSE has made that that choice and it may end up applying to other working groups and whatever, but uh in any case, any other thoughts on that like we don't have to decide today. I just thought: I'd put her on the radar.

B

What about people who are not active, do we have to ask them before we can vote this in? Oh.

A

That's a good question. um We would have to maybe have our first vote so you're right, probably by the but by the the existing. You know, rules we wouldn't have a vote on it to say yeah, let's vote this in and and we'd have to I'd have to go back and look and see what the rules say in terms of you know what quorum is and all that kind of stuff.

A

Often the rules are that you don't you don't need a vote unless there's an objection, although I'd have to go back and ask Williams our exported procedure. So we could ask him whether this is something where we need a vote regardless, as opposed to that. The other approach is like you go out there and say: does anybody object and if nobody objects that can go forward, that's out there right it gives it gives everybody an opportunity to say: hey, wait a sec.

A

I don't agree, but if at least you don't get stalled just because people don't answer at all, I.

B

Think I'm okay, baby, not like like is one maybe getting it right.

A

I mean the other alternatives. We could try and have more reason these not more frequent meetings, which would be good too right, but yeah I'll think about how we could adjust it so that, regardless of our our meeting, frequency right, says tens, yes for tens, listen to 25% of the regulars 50 anyway, okay I'll think about. If we can adjust that and then come up with something good.

A

Any other comments.

A

It's not okay. Let's move on to actions from last meeting, so this was from quite a while and so Michael and Gareth really just had an action to work on additional benchmarks.

A

You know, as we've discussed in terms of the matrix Ali was going to open an issue on ignition I. Think I'm, going to delete that in terms of you know, I we haven't seen him for a while ignition is actually almost already in so I. Don't think that if that issue really applies anymore, so this is like no longer longer really.

E

C

It was the talk about like testing the ignition engine like against this benchmark and see how it performs so it.

A

Would be nice to do that? I have run acne air on canary, which has ignition turned on and reported some initial results back in the thread within the node.js repo, where that's being discussed had any other any more time to to do anything else, but I think if people had time that would be a very good thing to do would be to run benchmarks and provide feedback on. You know how things look for the ignition builds.

A

So, for example, you know the I state benchmark if you were to run the canary build and the existing master, and even you know six it. That would be an interesting comparison.

G

A

So I, you know I think that any of that work I would say if people have time time and cycles. It's like open an issue say: hey I'm, going to go off and do this and then capture the results, and that would definitely be good. You know admission is going to be a big change, so the more benchmarking we can do up front. The better.

C

A

Yeah so fact mayor, I, think we've you know, I've got an easy way: we've got a CI gel bike and run that, but any of the other ones, people adding in there would be good. Actually I did so sorry, I did it for all of our nightly benchmarks, because I created a job that ran the same things record those results but, like I said, additional things would be good too. Oh.

C

I'm just wondering what would it take to have a CI job for Eggman for no check record.

H

Does it run on Linux it does okay.

A

um I think then it would be a matter of cloning, the existing one and just have it build chakra core. Instead yeah.

C

A

C

Either, do that live in a little bit, but just won't do my courses on and you know be, have enough resources to do. Those runs yeah.

A

I think we definitely have run. You know we don't keep the benchmarking machine busy round box. Oh I'd say yes, what I've also started to do slowly in the background, is you know in the community we've defined a way where we can give benchmark members the ability to work on their own benchmark jobs and I? You know so I've actually enabled some of the jobs so that you know anybody in the benchmarking team could actually go and modify them.

A

What I'd like to do probably to suggest, is that maybe you know I, could you and I could set up a time where we spend a half an hour. I could give you the quick head start for, like you know, like the one thing that the the benchmarking members can't do is create new jobs, but I like I, can actually do that. So I can clone a new job. I.

F

Can spend a little like woody until that's all. We also want to achieve the same goal right at the no DC and Kuna are wanted to the chakra and Kunal. We could do takara with acne or as well a chakra with no TC. We could unable same thing with a and.

C

That sounds like plan like we can twist it together and and then just share the instructions to each other. What it needs to run this PC era, oh yeah, I'm, gonna, say.

F

This some pizza and unique it is special that away we had religion, more activity, there's nothing secret. Here we go. Do you have a good update sense for anyone else? Also, alright, right some.

A

Discussion on how to create new benchmark charge right so um but probably well, you know setup.

F

Ok might look at Michaels submitted or anything at the node summit to talk about more about the benchmarking should be. You could say why don't we submit a joint presentation or something what the node benchmarking doing a different benchmark and ask more people to have used cases? Another thing at the node summit, yeah I, think we.

A

Could do that is and I guess the the call is still.

F

Open in North submit content open, they made it rolling call and they were even asking like oh who they should go and ask for the submission. Profess was I'm, sorry strings onto thing.

A

We're going to finish off the critic, drugs Michael doing some work to enable the broader.

F

Would be interested we can even have if you want I think they were two choices: panel or presentation we could for both models, so I can take a look and and the you and the Microsoft and us we can three of us can put combined presentation there. Sure okay might be greedy get Google. Also, there would be awesome from the v8 perspective.

A

Right: okay, so note some of CFP I'm going to discuss again there was this discussion around the group submitting either talk or panel, and it's right that was who is that speaking me.

F

A

Mean I know: okay, I know is going I knew you.

A

Volunteered set up a meeting with with interested parties.

C

So Anita you get it dad unleash gender, oh he's a PM from Microsoft and he I think he has some presentation notes of it. So I check with him.

A

Okay, yeah so I. You know, I think that the what I suggest is create an issue, um and you know just to say who who's interested in collaborating on this and then we should set up like a time to start to talk specifically about that, come up with what we want to present and at presenting them submit it. That sounds good, but Haley you're volunteered to set up that meeting yes, I can Rick, sounds good, um so creating new jobs, and so I just took notes in that.

A

You know, I've been doing some work to enable broader team, the broader team to be able to manage Jenkins jobs. We do need to be a little bit careful with what we do in terms of you know, resources and all that, but I'll set up a meeting to kick that off again. I'll create an issue. People who are interested can join and we'll go through and, and you know, get get people started on that.

A

The next thing we had on the agenda was questions on acne err, numbers that are reported.

C

Yes, so there is a something called footprint or that we report on the website on the benchmarking website, crying a little bit on like what exactly is done right.

A

So, let's open that up, I'm going to go to benchmarking, no dia org.

A

Right so we have footprints and.

C

Before load and footprint after load right, alright, so is it like a peak working set, Oh memory working set or is it like collected over, like let's say, Acme, runs for five minutes to be collect a snapshot of memory problem working set. What the.

A

Footprint after load would be after we've run the benchmark and it runs for I think it's five minutes. Okay,.

C

A

C

Memory, the working set after five minutes and move booking said before five minutes like before the show. Yes, now you.

A

Can go into the repo and look at the scripts that collect that to see exactly um if you go into github second I'm, just benchmarking.

A

So you have benchmarking under experimental and all the names not great anymore, but under experimental benchmarks, acne air, okay, there's a called rock run: acne air dot, Sh.

I

C

I think I remember like using this crib, but it doesn't spit out the foot cream I think it's. There was another footprint folder outside which might be the one.

A

Yeah, if it doesn't capture the footprint itself, it's another script within that exact same directory, okay, I'll, then those in the makers F.

H

Need, oh yeah? Okay, so if you look at F P dot, Sh.

A

Okay, so that doesn't exactly I think what's being used to capture that footprint, okay and they might what was going with me.

I

So I said it can be paint. You.

G

Look little github link again here. There.

A

Are you just pasted in the chat? Okay? Oh.

C

But the reason I was asking was because I was experimenting this for comparing the memory usage for like v8 versus chakracore, and what I saw was like. Let's say that there is a there is a trend in the memory growth, so it might be like a huge memory that is like allocated allocated and then, after a while it gets free. So I was wondering like.

C

Will it be good to capture the trend of the memory growth in those five minutes instead of just before, and after, like after the benchmark is run sure like garbage collect every round, it will collect some memory.

C

We might see some thing if there any memory leaks, we might see it there, but can it be good like if we can capture the trend of the memory that is being like there for like, while running the benchmark right support macular, basically like when you are running your app, how much memory it is getting consumed at like when the app is running right so.

A

My my only concern is like how do we you know? How do we show in an easily consumable way the data? So if you, if you actually could capture the peak, that might make sense so, like you know, if we captured the single number, which was the peak memory that you hit I, can easily create a chart that shows that over across builds, if we try and get the whole thing, I don't know how we boil that down into something we can show a nightly I.

C

Think the problem with peak is like let's say it like like after a tense again, the peak for let's say I am talking from v8 versus check record perspective of both went up to like, like, let's say, 1.5 cake, but then v8 draw and chatter could continue to be like 1.2, gig well,.

H

C

Like cheddar burger is consuming more memory or v8 not, but the peak working side are same for both, so we won't be able to see like. There is a problem.

G

So go ahead, this wanted to say it's something that, for the note DCI as we build the test harness also like a is in a Python script, which generally drives the workload so that the way the way it once is it collects is memory footprint every one. Second, so in a way no DS see there is a callback you you make callback on separate thread in Python we just written the existing. What is the memory state at every one?

G

Second, and then, eventually, it kind of displays you in a graph, so total RSS heap and although usages okay.

C

So but like like I, also put a thought on how we can like compress that in, like twelve put just a single number, which will tell you whether to nightstand was better than yesterday's run or not yeah. So we can do some kind of like a geometric mean or something of those of those like collection of numbers. And then we can just get a number which will say like lower the better right.

A

Even like I, don't I, don't know what the geometric main relates to this, but you could think of one where it's like, if you, if you it's sort of like you, want the area so like if you sampled, every five every you know one second right, you can then add all of those together and like.

C

How many times and maybe was 100 night or how many times it was 200 make. So you collect all this like bucket eyes, yeah all those things and then like somehow do math and come up with the number. The.

G

Nose, vcq Israel gives you min max and kind of mean what was the minimum memory. What was the max, whatever max happens, to be a or that period if we chose collected over one second, every one second, and then you just give you three numbers. Basically yeah.

A

So that would that would work well, and we could add that extra number as a you know something else. We showed yeah I.

C

Think just but collecting just so max, won't or might not solve the purpose because, as I said, like I was checking in my like other, like just a sample app where you are like yesterday, and today's binary both went up to two gigs. But then yes is binary, dropped to like one gig and it stayed at one gig. But today's binary was just like flat on 1.5 gig, so today's Valerie actually regress because it is consuming finder, make more memory over the period like over the benchmark run, but that doesn't show up in the result.

C

Right still says like both yesterday doesn't consume too big of maybe the highest working set. That's.

G

Why we are gonna, have three numbers? As Michael said, there are three numbers: one is the minimum the maximum and the mean the average over N, and hopefully, though, if you collect this number so for long enough time, like five minutes, a good enough time for the Acme air, oh, we were hopefully very stable enough to give the kind of a right kind of information. So.

C

The other guy show me we can experiment and see and then tweak it to get the right number right. So I.

G

Just wanted exploring and asking you kind of put it into low dc's test, harness that we mentioned that okay, so.

C

The visible so basically like when you run your benchmark. There is a cost for collecting those numbers from your benchmark code. Well, ya know.

G

What of one of the compact you can think about? Yes, right.

C

So there is a cause, basically, so can't it can't you do it like outside or like in the Python. Let's say you get a process ID of your running node process and then use your like UNIX tools to get those memory numbers from Python script itself, instead of node calling like node having a callback, because in that way, then your benchmark code won't have a penalty of collective memory, numbers yeah.

G

That can be done all right now, right now the I think it. That's. That's a good suggestion. Okay, when all on.

F

That one that becomes a bit OS and architecture is specific. Allen. Okay, you.

C

Can write a load module to do that, but there are like node models which say like take this PID and give me like memory status of this PID. So it does like all four windows. It will run some portion, I, don't know come on and for UNIX. If you run something else, but yeah sure means it's not like a super critical thing, but something something too good to have.

F

Yeah, we want to make it useful, because many people in the enterprise would like to have all this information for different reasons. Okay, in.

G

Terms of minutes here, I'm taking five if it is given in a configuration file, there is adjacent file which you can change this time instead of one second ten seconds, and since if these runs in a parallel process of Python, the client is keep on sending requests. While the you gave like 200 to 500 concurrency. That means your server. We check sure does the job is really busy and the penalty hold. Time is much less to collect this information, but.

H

A

I

G

Created yeah, okay, so.

C

G

Run sorry how long the benchmark runs yeah so right now what we are working towards is. The final version will have two modes where you can say run with million requests and 500 concurrency over to what our concurrency or you can say, run for ten minutes. So both modes use over to have yeah.

F

Usually, you should be able to get pretty reliable data in ten to fifteen minute run. Okay, so.

G

The the what we are working on this is right now it runs on both mode, as I said the time based on what we are thinking is the system should that you run for 10 minutes and you should come back and say: hey I ran in for 10 minutes. This is me, this mini request, I process and- and this is the best concurrency I can get out of the system. That's ideal, Sina! That's we working towards right! Okay, it's like a what seized us today right right, something like that. Okay,.

C

So this by phone run, pools are like Windows and Linux our choice for their acts. We.

G

Have tested on Windows, it runs on Windows, the benchmark runs on Windows for sure the script, our Python scripts, and there are some mechanism it. The client is a little bit I.

F

Need to be Linux yeah.

G

If it is, it definitely works on Linux, but there's some Python modules we use them. You need to find a phobic equivalent or for the graphing purpose over it may or may not work as it is out of box on Windows at the client. The cursor would definitely works. Okay,.

A

Go ahead now we are in terms of the community environment. We.

E

Only have Linux right now, anyway, yeah.

H

E

Yeah, that was my next question to you. What we died on into.

A

Every employer plan we have one dedicated machine from soft player. So it's like you know it's dedicated, so there's no interference with anything else, but it's the one we have and it's it's got Linux installed on it. Okay,.

F

One question that from Intel we do a open source language performance lab which has like dedicated machine for different work, different runtimes, like Python, SVM, node, etcetera. We could run the and the machine is not used when the node test is done. If you like, we could move the current node benchmarking infrastructure to those machines right.

A

Now, typically, the you know, the stuff that's hooked up to the community. Cis is something where the community build work. Group has access like root access to the machines configures them. We then connect them up to the height know to the Jenkins infrastructure. I, don't know if that's possible in that.

F

Case again can just the root access part we like to have control. So if you want some company, they could have one on the on the other side and we could run the same copy and the data could come usually from Intel. But if you need any modification, we can mirror yeah.

A

I'd have to bring that up in the build workgroup like it, certainly like the the issue might be connecting something to our Jenkins, interested in instance, which isn't under control of the build team. Members I think is potentially an issue right and the.

G

A

I

G

You talk about the too much issue is sent to near forum, I think they the what we heard that they already received it and they're in the place.

A

They are configuring.

G

A

Those ones as I understand you know we will just connect them right up to the CI, and this group can use them. However spits um so you know, those ones are certainly will be directly used. Okay,.

A

um So it was I'm. Sorry again are not great at picking up, since you guys have logged in under the same in the same room, I'm not great at picking up names. Is it Canal who was asking those questions? Yes, I saw it I've taken. Tell me if I'm wrong here but I'm saying that as an action for next meeting, you've taken the to do to create a PR to add either something like a mean or something else to the Acme airline here, yeah you're giggling.

A

So you know you can look at those scripts create the PR data mean then I can quite easily add it into the with the charts that are reported. What.

C

All sure so the footprint yeah I lied to like I'm, just like trying to get things around like I like to make sure that I run those footprints like every one. Second, and then gather those numbers and then calculate a mean and dumb dat number somewhere. Yeah.

A

Something like that make make sure you try and analyze whether that taking that those numbers is going to have any effect on the yeah mark itself, but if you you know sort of, if you can come back to us and say well, here's your proposal, you know every second is fine, because it has no effect on the results or maybe it's every five. So I can fight or no, you know come back with what you suggest and you know, yeah.

C

And that's all I got to have it like a outside benchmark or like donate like externally, like I, have a PID and then some other tool which can just query those memory numbers I'm.

A

Pretty sure that's what FP does yeah yeah, it's just a it's just a bad script and I can see that it's you know, cutting proc mem info to get the the footprint.

C

That a patient can be I, mean yep yeah, so.

A

I think that's pretty much what it already is: yeah, okay! Well, yeah! You know, take a look at it, make us a judge, and you know it sounds. It certainly sounds like a good idea. I think everybody else sounds like a good idea to immediately.

C

Give us they go sense of memory cream like the memory growth trend over the board. Okay, yeah yeah.

A

No definitely and having that extra number yeah because, like you said if one that would be another nice single number, we can easily look at and see well did the the characteristic changed significantly or not? Okay, okay! So on any other on that that agenda item any other discussion: Oh, nope, okay, so EOS, benchmark, yeah.

C

That's it I think with them already like a water. The next steps to life as.

A

I remember the next step was, as you know, I was kind of asking, if you guys could provide something that says that covers these things in our matrix yeah.

G

So how about I open the issue and kind of say this is what we want to integrate or donate to our group and I'll say what are the things it covers? What are the use carrier dressing and the github link right now? It's already on github, it's open source, select the add a link to that project and then can discuss again. Yeah.

A

That that sounds good to me and and like even you know at some point, if is we were looking at, you know, there's this under the experimental directory. What we'd likely want is a benchmark. You know in there there's benchmarks you know, would there be another directory which would be the DDC eis and have all scripts that are needed to actually run the the benchmark? Okay,.

G

I mean for today, in fact, once we have a we 1.0, what done I can in fact go ahead, and at least he should pull request all the code, and maybe the script needs to change so that it can be hooked up easily with Jenkins existing.

I

Structure yeah exactly so be you know. Basically, if you put a.

A

Pr 4 ad right, you know new directory in there and it's as simple as like for Acme air. There is a single script. Where is it.

A

Run Acme air right so like if there's an equivalent run, DCE is yep and then that spits out you know in the output it spits out things they can be easily parsed into like the specific numbers we want to report.

A

That's all that's needed right, because the Jenkin running in Jenkins there's nothing special there beyond those scripts so like for Acme are all it does. It says it runs that script. It parses the output to pull out the things and then the actual publishing into the databases. You know a couple lines of bash: scripting: okay, okay, so yeah.

A

I but key for me- is that, like we want to build up that, you know matrix coverage to say well we're running this one, because it gives us good coverage in this area and so that at the end we can say, look: we've got recovering all these different areas, so yeah.

G

So that the issue I will go to open I'll, add all the details saying what are the only? It also creates a summary text file that our existing office, what it looks like what are the things it covers?

G

A

That sounds great and then even like when I set up the meeting to go over like the the jobs you know, it should be something where we can clone the acne err job and then it you know, I can do whatever transfer so that you even know how to be able to then call those scripts once they land and stuff like that. Okay, so that was boom.

E

A

Submit ship you are with info / scripts: okay, I will.

A

And under actions.

A

Dcis: okay, that's great okay, tech and power. Yeah.

C

Do you think every dislike every discuss earlier in the benchmarking group, whether we want to add taking 400 points or because there are lot of free bugs that are compared on this tech and power and no Jess seems like very low on that compared to other frameworks right.

H

C

Like if we start having taken power numbers, maybe it will drive people to a cone or yeah.

A

I haven't I to be honest, I kept Tekken Power has been discussed, I can't remember if it was in this workgroup or internal die BM. um It just never has gotten to the point where it's you know been high enough priority to make something happen. I would, in my mind, I would drive it from you know. Does it fill in some more of the gaps in our matrix and if yes, then we should run it right. Yeah.

C

Because a set up is very easy, especially compared to a command right.

A

So did you have a good good feeling for what kind of things they would be measuring and telling us I mean.

C

Like it's again, it's same lighting, Ric system response, but this one is a I can remember, but this one actually measures in different or type of files. I guess like one, is JSON versus text response and there are like two or three other, but it's it's again that overall, it's like a reverse response: I guess.

F

I can I get for you acted and we, what did the we have looked at a couple of them, I think and what it mostly because you it's written in so many frameworks, it's pretty much the HTTP response, while using plain text, file or JSON object or different kind of object, and that is why it have been implemented.

F

So many because the main idea you just sending and receiving the different object over the HTTP, so it is going to exercise only that component, just the HTTP response, all right, even response, and what you may find that, because that part is full. That part is almost 100% execution path.

F

When you start tuning for it, you may find the tuning, which are not necessarily good for the typical workload or the products and environment it becomes very it become very sensitive to your your network, our tuning, your the number of thread for the network, part tuning, because most of the time you are executing only that part right.

A

Okay, you think there's any like one. One thing that came to mind is a potential value. Is you know if no js' is compared on that basis, then? If, for example, we did something that half the performance to what it used to be before, that might not be a good thing.

F

I think, as far as we find, the same thing is benefiting like no DG is or acne or something which are little more traditional and, as we go add more. If the similar thing benefiting others to then only we add because it may give us something or the network stack, but because of the HTTP response type, you may find more blocking requests much faster than the asynchronous approach right. Yeah.

A

So I guess: what's the I, are you suggesting, like it's less useful or just to be careful? If we do it yeah.

F

Be careful about the conclusion you could draw based on the optimize and you find right, but if it is simple we could look at canal the initial data, if you, if you are planning to run with jab cross over then it would be interesting to see the initial data yeah.

A

Like is it simple enough that we can just run it and does it add value if something, if there's a big change in it, is that useful for us to know yeah.

C

I'm running is like: oh, you just run this over and then like use the load tester to the Ubuntu or I can't remember the name, but that's pretty much it, and it doesn't even run for like five minutes like 10 seconds around. That.

F

Is firing it? We have to look for the variable, 80 and yeah there. If you run anything you run like 10. Second I would be little worried of the repeatability of the number system. Yeah yeah, that's really. Okay,.

A

Well, good now, I don't know if you want to look at in sort of experiment and then come back and say hey. This looks like it's stable and useful. It sounds reasonable. Yeah.

C

But that like that brings me another thing like so this is basically just definitely checks for the HTTP stack. Like all these benchmarks. Do we have any benchmarks that verifies like other things like buffer and I? Don't know like domain, and things like that. Well,.

A

That's that's where, if you look at the the goal of that matrix was to try and call out the specific things we should be calling covering, um so we don't and then those are mostly higher level things or key attributes. So we don't have like micro benchmarks, which are the benchmark yeah.

C

A

Thing we did have on our list that you know we haven't figured out yet is if there was some way we could boil down the core benchmarks which cover all of those mm-hmm there's some reasonably reportable numbers. That would be useful as well. A problem is right there you know. There's two problems with the existing core benchmarks is one they can take so long to run yep until you end up with so many numbers like. How do you report them such that people don't actually figure out? We need to do something right: hey girl,.

F

My god I have a question about the buffer question. You asked that, and with ample to speak more so known, core itself has the micros about the buffer, etc yeah and in acne Rand no DG is do use these buffers anyway. So what exactly? Are you looking for stressing more or I think so you have to know? Oh.

C

No I'm, like I, don't know like to what exchange um the Eggman and like the empty is a benchmark, uses a buffer but I think like in the core. Again then Michael can correct. Me like. There are like different, like I, didn't buffer sizes, that we verify it again. Yeah.

A

There's I'm pretty sure, there's a whole set of micro benchmarks. That would, you know, cover the challenge. We have I think as a group is like. How do we usefully leverage those into something that you know you can run regularly report numbers on and catch regressions because it generates. You know a fairly large set of numbers and I. Don't think people are going to be looking at that all the time and and the other thing is like it takes this may be, maybe with the new machine we can actually run it.

A

You know once a week or something like that, but it can take like, depending on the configuration days weeks, to run through a whole, so there's some thought that we need to put into in terms of well. How are we going to run it so that we can, you know, get the data. What subset of the data can we report and make it consumable and.

G

Internal workloads, we are running it at intel's performance lab website. If you go there, you do see some of those are added and we are in the process to add more to their a buffer, related and quite a bit. So that's so we do have it I mean- and it's already we announced it at the last interactive. How.

A

Did you choose those which, how did you choose, which ones to run? Oh, that.

G

Was a big exercise? Actually we ran into a few weeks and we found that of them have a very huge variability in the run, and then we pick some which are very stable, like buffer index of data, if they are quite quite a bit stable right. So we for output a but as we so busy as I said- and you are right, if you run all of them with all configuration, is going to take weeks to run complete it. So the process is just find something which is stable, so we can run lightly basis.

G

Then gets a meaningful date out of n. Hopefully try to see a trend or on what the trend is going up or down right.

A

Because yeah, that sounds good. If you know that that could be, you know that could be one of our group's efforts is to say: well how do we do that? You know, as they explain what we picked this subset. We run these regularly. You know I think that it's just a matter of having the time to make that happen and.

G

Here you guys as I understand it. Look at our last meetings. We try to see what are the real use cases, the nodes, no Jacek getting used web casing react or whatever different forms of guessing. Also our database condition all those things we are trying to cover here. That's what my understanding is yeah.

B

G

A

Okay, so our time is we're down to two minutes left um I. Think in terms of open issue review is we won't have time so maybe we'll defer that to the next time we had lots of good good discussion. Is there anything that that anybody thinks we absolutely have to cover this week? So.

C

So--But, you will be taking a like at least like looking into having a some kind of CI job. We can configure for no trick record like we use like yeah.

A

So here the actions I've taken okay, I'm, going to set up a meeting for CI job configuration, so I'll send out an issue in a doodle to say: hey, let's get together, I'll walk the interested people through the jobs I can create like sample ones um that you know, individuals can actually then take off and work on. I'll give you the caveats of.

A

You know we're kind of being trusted to do this, so we can't go crazy and we got to be very got to be a bit careful, but you know it's worked for the other work group. So I think we can do that as well, and so hopefully that will be the starting point. We're like okay, if we say yeah, let's run these other benchmarks, you guys can help to implement those as well, which would be great.

A

ah The next action I took was an is going to set up a meeting to flesh out a submission to no summit. So again, you know pull together the right people. We can brainstorm and agree what it is. Get that submitted canal is going to create a PR to add a mean or something else to the Acne air runs looms going to submit a PR with info scripts for DCE is the explanation. How covers the the different areas of the matrix like even as part of the PR?

A

If it includes updates to that matrix, would be great to say: yeah did I miss anything there.

A

No okay um and I will take the action to definitely you know, we've kind of let it go a little too long, thanks to the Pope to get the meeting going. I'll make sure I set one up like this week for a month from now, yeah try and send out the doodle. You know today, probably tomorrow and we'll make sure we get back in another month and we'll.

D

Have those arms.

G

A

The middle as well so, okay.

F

That would make our North summit thing through the meeting, so you can have more active than even a month, so we could attract more yeah.

A

Yeah well, I would say like set up that meeting for next week, sometime right, yeah, so I'll turn.

F

You personally deal then.

A

Yeah, that sounds good and if there's any you know if we need to have one sooner, we could do that too, like you guys, think a month is right or what do you think I think.

F

Other things are picking up and we want to add more detail. I would say bi-weekly we could go, look like kunana genders to have the chakra related thing we want to and then or DC related thing. So we might have any two weeks. So how about two weeks you can cancel. If people don't have the agenda item or something sure I guess that does come. You know the other approach.

A

Versus the the doodles, which sometimes is hard to find some time, we could try and choose.

H

The green line.

A

Okay, let's let's try and do that now: what's the best is dead, I guess, maybe a doodle is a good way to do that still or is there a better way that we want to fix the time every to lead? What working? Okay.

F

So I'll start.

A

Out a doodle, it says: okay, what time can we do it? Every week, I'll pick a few, or at least I'm available I'll try and pick a good number. The the challenges is, you know finding one where maybe the people from the UK and everything can attend as well, but we'll yeah we'll do the next doodle knife for a ring.

F

And if it picked like 12:00 to 1:00 lunch time, I think most of us guys probably keep their lunch. Open, love everyone to lead, and that is still like 9 o clock for UK guys or something probably still okay. But we could go early if you want to go. If you don't have problem early yeah.

A

I'll try and find one which is like you know: 9:00 a.m. Eastern, 12:00, Eastern, 5:00, Eastern and I. Guess even something outside. You know something that covers the full range now I. Don't know what like what time would be the best for everybody. Do we have? We have I done you're lucky that no one from China Russia here yeah. We have that problem on the TSC, we're like there's one. That's like 4 a.m. in the morning right, which you know it's like no I can never make that one.

F

You know I think 9:00 a.m. Eastern becomes hard for us on the west coast is yes 6:00, so anything after 7:00 is reasonably ok. So, like the.

G

10 can Eastern I. Do the doodle also good, because you give these multiple options. I mean. Does that's a really great way, yeah.

A

So 10 Eastern 12, Eastern, five Eastern and yeah. Ok I'll, try and find a bunch across those things, and but it's good to have the feedback that says like Oh 10:00 Eastern is you is the earliest that's good for you guys, yeah, okay, yeah, all the way able to be groggy. Yeah, no I understand it's like the time zones are a killer right, like I.

A

Think 12 covers like 12:00 Eastern covers, is around 9 o'clock, free, Wesley, oath, right right and that's about five for the UK people, so that one, maybe, and as long as we don't have yeah.

F

A

F

Will they many times rather than 5 p.m. many? You could be like 9, because then they're done with the family thing and they join from home. That's.

A

Right, yeah, you're, right, ok, so sometimes a little bit later is even good too so. Okay, yeah I'll send out a doodle to try and get something if we can do a repeating way, because that would make it easier, I think to have a good cadence, okay, okay, that sounds good and I. Think between now and then there'll be a bunch of other meetings that were, you know, there's at least three at least two other ones that you know us. We should be setting up so we'll be talking before then, anyway.

A

Okay, anything else again before we close up for today, okay, nope, nope, okay! Well, thank you very much and we'll talk to you guys all soon yep. Thank you. I.