Node.js Benchmarking Working Group, 20 Nov 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Node.js Benchmarking WG Meeting - 20 Nov 2018

Description

https://github.com/nodejs/TSC/issues/622

A

Yeah, so welcome to the OGS community benchmarking team meeting for the 20th of November 2018 we're going to go through our standard agenda, which I'll just pick pull up in a second here. Okay,.

A

Okay, so investigate the time taken, Markson core I, guess we don't have Gareth this week, so I think we'll just need to skip that. One.

B

Let me just do that. Take that note, skipping.

B

A

Okay, the next one is required, perf much lower, so I know you've been looking a bit at that yeah, so I.

C

Updated that if you look at the issue, 50 no, not 252 so updated yesterday, what I did I just ran separately required perf and require no required new and required cashed care and I see huge difference. The catch one really much faster than the new, and we can look at that issue, but that that should be right, like cash, should be way faster than new right, so account. So that means the way our workload. The way the run benchmarks runs.

C

I think there is some disconnect between the way it reports, the numbers and the way internet I did a small change looking into the required purchase file. Okay, so it has a run benchmark as an object, and you had two of those functions required new and required. Cached exchange the order. I ran the cached version first and then I ran the required new version. Second, and then I saw different numbers I, don't.

A

C

What they were requiring.

A

The same modules like I, wonder if the require cash just messing up the require new button.

C

With the require new, we are definitely resetting that in the module constructor undefined that file name right time, we're on we set it to undefined, and that means every time you go and do a required. It will know but find that key, and you should just do a new required right. Okay, so there's something going on which I I clearly don't understand the run benchmark but yeah so that what I found. So what that's exactly for that reason, I just took it out of the run benchmark ran separately.

C

Okay, and if you look at the issue, if you can open that issue, OH yeah.

B

Here, I could just share that for a second I had.

A

B

Hold on I want to not share that for a second I'll close a few other tabs. First.

B

A

Right so I've got there there at the bottom right. Alright,.

C

So if you look at those numbers yep and which makes sense that guys, one should be much much faster yep and then the non-cash one, but somehow the wrong benchmarks has something the way it runs. I guess I, don't exactly the how what is supposed to do when things like the, but this looks okay to me.

A

So it looks like we're right, so that shows the right running separately.

A

Just get that to choose me the.

B

Right thing: so what the heck just happened? There.

A

The running separately seems to give the expected behavior right, yes, which would explain why we haven't heard people saying: hey, wait. A second seem so slow. So that is correct seems like it is related to how.

A

So I guess that's the next step to try and dig in a little bit more to see. What's going on when it's running the benchmark, yeah and.

C

I just said you know, so if I, if you look at the code, we can, we can probably look at that. If you want to, if I change, the order so run benchmarks, add the add required cache first to run and then you then I get the expected numbers, but if I reverse the order, then I get this supposed to regression with the required cache.

C

So that mean there is something in that run benchmark what are the set of scripts? Are they don't really hmm like.

A

I don't know the like the framework, that's used to run them. I didn't do any like I haven't worked with those at all other than calling them from the scripts right. So I, don't know what they're doing, but it's it's I, don't know if, like you said, I, don't know why the order of those two would make a difference exactly. That was my question.

C

A

Used it myself, so unless the numbers from one or getting bundled into the numbers from the other one.

C

Could be that is possibility, so so.

A

Maybe that's the like what happens if you just comment out one of them then.

C

I get the okay I, don't remember now, but I don't remember now.

A

Cuz, maybe that's something to try like, maybe it's just that there, although the time, how is the time calculated the time is calculated with.

C

The what you said, the heard some kind of ads API ops per second, so I was just going to show you this. Is there something some way I can share here.

A

I'm, just navigating into yeah, you can just say, share if you want I'll stop sharing and you can share.

C

A

The thing is, I, don't quite understand, show, know those the graphs are run separately.

C

And you see yep.

A

C

So, if I change this order, instead of run or bench Mars cache before the new I get the different numbers, they should not be different, but they do.

C

And this then I when I ran them separately. As you saw from my example, we get expected behavior I.

A

Wonder if it's like, maybe something in the framework confusing the names like, maybe the dots and the names are causing a problem.

C

A

I can take a look again like maybe try just changing that to require new and require cached without the dots in them or that's. The only thing I can think of. Is that, like somehow it's confusing them and you know, maybe we maybe we gotta, install the new version of the test runner or something like the framework that has a bug or whatever, and that's why we saw a difference. Okay,.

C

A

Yeah, that meant my only my only looking at that there is the like. Somehow is the naming with a dot.

C

Okay, take a look: okay,.

A

Yeah, if you could figure that take a look at that, that's probably the next best step.

B

C

But at least for now it looks like there probably is not an issue with this to the cash and non-cash P yeah.

A

That's a good, that's good news, I mean if it was a real issue. You think we would have been getting some some.

C

A

This point reports somewhere else we're seeing one of the other, the other benchmarks showing something as well. But it's good to see that, yes, you can show that it really isn't an issue yeah.

C

And he's been there since at least eight, so yeah, it's in a long time, yeah.

A

Okay, yeah, if you can just maybe try playing with the names cuz, that's gotta, be something in the benchmark. In the worst case, we could refactor these to just run them each individually.

A

Okay, but you know, maybe playing with the names to start with, would be easier. Okay! Okay, ah do you want to unshare? Yes, okay,.

B

A

Me back to okay: there we go I, just wanted to get back to I, see I think is maximized on me. Okay, I'm, just taking a few notes here.

A

Yeah with the reversing orders thing, the only thing I can think of. Is it like somehow it's adding them together or something.

A

Okay, so ghosts gonna, assume still that's next yeah.

C

That's why I did not get chance to work on ghosts. Some other work items I thought that the Jamie was going to take a look at as a part of adding a new slips right, yeah. That is an example to set up his scripts, how to run, and he he would be the one to he's learned it I guess. But since he's not here, I can't really same.

B

A

Okay, simplify muscle script, logs infrastructure, so 155 is that Jamie's that was Jamie's tool, yeah, okay, so we'll skip.

B

B

Okay, candidate benchmarks.

B

B

Just opening that one up right.

A

Okay, I think that's really Jamie added.

A

Okay, so that's what we had on the list where there are other things you want to talk about this week or no. That's it just wanted to give that one update yeah. No, that's definitely good. It's good to know on that front.

A

Now I'm gonna have to look forward to. When would our next meeting be calendar because I'm actually on vacation a good part of December I.

C

Imagine everyone is I'm.

A

Yes, so the 20th, actually that's the one week I am no. Is that right? No sorry, that's November I'm looking at so when is our next meeting.

A

It's right, I will be away on the 11th.

C

A

Be around on the 18th: if we want to move it till then I'm, fine, I'm, fine plating, okay! So why don't I move that to the 18th that way, I can make it you can make it, and hopefully it's no better or worse for Jamie yeah.

B

Okay, I'm going notice that one.

B

Move next meeting from.

A

Ok, um so let's see we can look at YouTube, see if anybody's watching 9 watching. So if anybody is watching, you can ask questions, I guess, there's one question which is: what does it mean by benchmarking, nodejs and so I'm gonna type, a quick lost, say, quick, quick, quick, quick answer, so I mean the the the mandated the benchmarking workgroup is to track performance of the node.js runtime.

A

We do that through a standard approach of trying to document the key use, cases and attributes of node coming up with benchmarks that will help us attract those getting a bit of an exit.

A

Ok attract those attributes. Then we try and come up with benchmarks that will help us track results on those identify. If there's regressions like we were just discussing you know, one of the benchmarks we do run nightly reveal the regression looking like it's not a real one, but something in our in the way we're running. We basically run those and address issues that come out as an example.

A

You can go to benchmarking node.js org, which shows the benchmarks that run every night, as well as the documentation and the repo talks about all the infrastructure that we have set up to to do that so I'm hoping that answers the question. If there's any other people watching on YouTube will give you another minute to ask any questions so I'll just wait, 30 seconds or so.

A

And you know basically well I, do that if you, if you're listening and you have an interest in benchmarking performance, you know you're welcome to join the meeting and it's get involved in. You know the different activities that we have on that front. So, okay sounds like we answered the question. That's good and, like I said, if you're interested, you know we're always looking for more people to get involved as well. I think that that will give it a we'll call that for call it a day.

A

So that will end the meeting and we'll talk to everybody and see everybody in github and at our next meeting on the 18th talk to you later. Thank.

C

You Mike Thanks, bye.

A