Node.js Diagnostics Working Group, 27 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Diagnostics WG - Performance Deep Dive - May 27, 2020

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

A

Should be wait for someone else, mister.

B

We should probably get started all paying Jewish just to see this.

C

Meeting we are going to be discussing the last one of the performance use case as the last time we discuss it.

C

Cost for memory bound applications, and this week we are going to be discussing. The application has a use case.

C

So when we were brainstorming use cases this one, we described the symptoms for it as applications legacy is high and I have already confirmed that the bottlenecks not.

D

My dependencies.

C

Like type databases or downstream services, so I suspect that my applications is spending significant time writing code or processing information.

C

You are satisfied. The pure application forms in.

D

C

But who like, who understand, which part of your case and mean prove this to run faster or more efficiently? It can be useful when we want to improve the user experience or save computational cost.

C

C

C

There is a disconnect between the ballistic use cases and the headers like.

C

The header here is: medication, has a poor performance, but I don't see a lot of difference between these and the other two are at head-butting, the leaves of use case in the first page. The third use case is my patient shows spikes.

C

So maybe we get some.

B

You just call up Oh. Is the poor pay still linked to the? Maybe that's you already did. Okay nevermind got it I'm, just looking to see.

A

Are you saying that this one and the first one is the same or or what sorry.

C

So I think it was from my eighth pass iteration of this document, and it should be remove it. The use is that we are going to be discussing today is.

C

Which are different than, for example, constant.

B

Yeah that makes sense.

D

Can you please bring me the link to the document.

B

So we're all just replaced in the chat for the wrong one. Sorry taste got it.

C

C

What are the symptoms for spice or.

A

An application is this: Ponzi temporary.

A

High latency temporary.

C

C

Which can be either.

C

Fight for a few minutes or a few seconds so.

C

If it's a moderation that the spike happens, you might see it as a phrase latency sometimes.

C

In any nine requests.

B

Now the the thing that makes me think like I, don't think any of the previous sections we covered late and see at all. Did we. It was mostly investigating either memory or CPU.

B

C

B

Some of the so go ahead.

B

Right but in terms of debugging, the problem like I think that's where, like that description, that we'd written the was trying to get at the the latency part versus the CPU part.

B

Just since you're mentioning latency ia, we should come back to that after we get through this, but like I, don't think. We've talked about any tools for looking at the event loop plate and C, or anything like that.

A

What about unresponsiveness is there just another say of saying latency or like incompletely time out requests I.

B

Think that's possibly the extreme case of the latency so long that you then start timing out or.

C

Any other symptoms.

B

Well, the the title says: spikes of CPU usage, so I kind of miss the symptoms like maybe that title needs to be more. We have indoor intermittent performance issues right which could.

C

B

And then like a symptom would be spikes of CPU usage. Perhaps.

C

C

B

Two or gets lower over time.

D

So here by and large, when we say late latency, we are essentially meaning delay in the transaction or the responsiveness, not necessarily the latency, in internal component components like UV loop or any other runtimes activity. It's an externally visible delay in some of the users initiated process.

D

D

When we say latency, it is a visible delay from a user's perspective on some of the activities which they initiated. For example, web transaction takes more time than expected, as opposed to latency, that is internally measurable, like UV loop or anything, which is an internal component. We are not measured. We are not talking about that. We are talking about an external web based transactional. Latency am I, correct, I, don't.

B

Think it has to be web-based necessarily, but I would agree that the like the use case is you're having you're seeing in intermittent cases where you know, the response that you expect in terms of latency is greater than you expect whether it's a web service a web page.

B

Some other sort of call yeah.

D

It makes sense in either case the latency is something which is perceived by the end user through some of his tools.

B

Yeah I know that's where we are mixing in the symptoms like CPU. Spikes isn't necessarily going to be a symptom that the end user sees, but it is a symptom you can see. If you look at your system and in that context, like a longer event, loop latency would be something you might, you know, might be a symptom on your system. So I guess it's you know, I, don't know if you're leading towards the question being.

B

Should those symptoms strictly be the external ones or things like CPU spikes or whatever is I, don't know if that was the question yeah.

D

Yeah, my point is: when you say you have a performance issue, you could actually experience the performance issue in two two broader categories. One is that you see the throughput is reduced. That is the operations per second of whatever parameters you use is the substandard or suboptimal. Second, one is the overall efficiency looks good, but there are. There are actions which are taking longer time than expected.

D

So then this need to be qualified better. You employ some tools like a few profiles or some graphs, which has time in the x-axis. They eventually use start seeing the spikes and other things. So the latency would map from external perspective as well as internal. If you are using tools, tools will show internal component level granularity, whereas if you are not using the low level tools but the application specific tools, you will see whether the transaction or the user level operation specific latent.

B

Right in the symptoms, it might be like high request latency. If, if the the point is to qualify which latency I.

A

Will vote on the user symptoms yeah.

D

Yeah, that makes more that is more relatable to the user. I agree you.

B

Know similar, it could be in crest increase in requests. Time outs.

C

E

Think into the banking yeah.

C

Any other symptoms that we.

A

Not that I can see people.

C

Staring symptoms that you can observe on servers.

C

For example, on bundlers, maybe I.

A

Mean I see malignancy can be interpreted in a very general way like if it's some kind of asynchronous queue processing system, then not necessary would call latency. It would be time to handle the package, but I guess that's. Let's do the same. That's like I said the system is.

B

Yeah and things like you know, I think you said bundlers like so command-line programs, I'm, not sure that they're like use, often see intermittent performance issues. It's more like I ran it and it just didn't do what I wanted fast enough.

C

F

B

But I guess the there are a bunch of different tools, but it's in terms of our recommendations. I wonder if, like those can be very hard things to do, if you don't have instrumentation in beforehand right, so it's kind of like I wonder if our recommendations should be to prepare. You should always be outputting metrics through things like Prometheus, or you know something that you don't have to go and turn on when you, when you decide, there's a problem with that. Are there and you can use.

C

B

That'd be good, although I think we should maybe write it first for what it looks like for Diagnostics cuz, that that would be enough of a scope and I'm pretty sure.

B

Like Jewish in some of the sections we had that write like four core dumps, we included the you know, we recommend you do XY and z to make sure like your you masks are set correctly or not your you masks, but your you limits are set correctly and that, like you, have enough disk space, so I I think that would be consistent like for this section and then maybe we should pull all those out to like just a one consolidated document at some point that says: here's all the things we recommend you do and then they can have the links back to the sections that actually talked about the debugging and why you need those things right.

B

D

Mean say: consolidate all the preparatory phase activities, all the prerequisites respective the symptoms, put everything together and put it in the one document.

B

Yeah or but like I, think that stuff belongs nicely with the sections that explain. You know I think if I look at crash is it the read know there was one that you'd already written. Where was a normal termination yeah.

C

B

Then, at the bottom there is: where was that in there there was definitely a set up that M D right yeah.

D

B

So I think, like I, don't think we'd want to pull all of that into one common place, but having one common place that then refers to them all. It says: hey, we recommend you set up blah blah blah just to very high level and here's the instructions for how you set up in each of the section. Something like that would would probably be quite useful.

B

D

That makes sense, and- and we can actually provide sections to say that the this is for crash diagnostics. This is for performance diagnostics, and things like that right because.

B

This one right here is here: I'm, just gonna paste. It is the one that you wrote earlier and just posted that in the chat for that was related to what what are you gonna need to be able to have in place for abnormal termination, so we should write a similar set up in each of these sections like okay. What is it that we suggest you have in place already.

D

Yep, that's the only confusion. The only concern is that it, it might deviate from the original directory structure, which we decided, which is okay, I'm.

B

Just like in an additional file that pulls that all together, so if you look at a particular section, you still have the setup env, just like the structure we had but separately. There could be like a something that is a reference to all of those just to say.

D

B

If you're setting up a system so that you can do any one of these things when you need to this is this: is the master list of what we think you should set up, and it's just references to the actual files that house the details that.

D

Make sense, because for a production, the the user will be interested to make sure in the system is prepared against all the symptoms as opposed to. He cannot foresee that his application is going to be vulnerable only for crash, so yep.

C

B

Because the other thing you're gonna, need in that situation, is you want to be able to look at the you know the traffic or what kind of requests you're getting like in the abstract sense? You want to know how much load external load there is on the system, because is it just slow because you've got a lot of traffic or is something else going on right.

B

Like in fact, I I would almost put that as the first one.

B

Right, like the my first very first question, would be okay, it's slow at some intervals. What what's special about those intervals right? Is it that there's more traffic are there are a different mix of certain transactions.

C

Lizard makes sense, understanding.

D

C

Requests they're coming, which paths they're meeting in.

B

C

If there's any difference.

C

B

Those are all things I think you mentioned Prometheus, but that's like if you've got your app, if you either have you know if you either exporting those Prometheus or you're getting them through some other APM type solution, that's probably what you're gonna need I.

B

Think Prometheus is kind of the standard. If you're not you know, buying a service.

C

B

Guess separately, there's also things like Jaeger and open.

B

It's not open tracing anymore, but open.

B

Open API so go ahead, open.

A

Telemetry open.

B

Telemetry, that's it right! Yeah wish you were working which you were participating in right. So me.

A

uh Was yeah yeah I was participating with only at the beginning right.

B

Okay, I think that's the other sort of monitoring that would help in this situation to like cuz, it's helping you that helps you break down where and the overall transaction flow I.

A

Just got back in I was out in the last five minutes: I'm just reading. Okay,.

G

Yeah I, don't I, don't know I, don't have a context. Just continue. Sorry, we were just talking about like how.

B

You would like how you would investigate these. You know intermittent and I was I was saying like my first question is like, what's going on, that's unique about those intervals.

A

So the tracing is that fitting a good solution to confirm that it's not the done stream service and it's not the database and it's not your dependencies right. It's kind of it's not going to help you in the process, but it's going to help you to kind of narrow down to your own process. The problem right.

B

Okay, so maybe that's the pre, a pre step is to somewhere somewhere it's good to cover like okay. How do I make? How do I figure out, which component is causing the problem? If I have multiple requests.

B

But this one in particular could be hey, I've already narrowed it down to this component. I.

A

Don't think that's facing, we all have to do.

B

Right once you've already narrowed it down to this component that open tracing is no longer the solution. Right, yeah.

A

B

Ike yeah, so we we just want to cover it somewhere and I. Think I was you, you were maybe thinking that could be a separate use. Second separate part of the it might be another step, different step, or maybe it's just a different different use case to go through.

B

Because I would agree would be good to break that out into a different part right, because otherwise we get to.

A

So the tracing is limited in usually in two sense that one it's highly sampled, so I'm, not sure you will get the exact problem. Another thing is like the multiple levels, what you can capture and like many companies, literally only capture like their own process like one spam, and they don't go into what that process does so you're, just capturing basically dancing service calls.

A

So your service is just in black box and you will only see that the requests are typical test and star quarters and some companies capture more like they're capturing the design functions in this usually in general tracing is.

C

A

Some company or method that can help them but I think in 90% of the youth, identical excited.

B

Yeah I could kind of see that there's like a if you've got a if you got a system with a bunch of components. The first step is to use something like open, tracing, open, telemetry to say, okay, which is the component. That's the problem, and then once you've got that then separately you've got to dig into okay, it's a node process on. How do we investigate why it's in particular has has this intermittent behavior.

B

So the the the are there other. So there are others you're saying you thought there was something other than Prometheus I. Think that's where it got. We got.

C

B

There any other so whew you're getting intermittent behavior. If you've got those metrics, you can tell what's changed, what's different in terms of the.

B

You know the traffic or, what's going on around the time of your slowness or your performance issues, I guess still turning on things like sampling, the sampling heap profiler, not the profiler, but like the CPU profiler. If you can get, you know, turn it on for those periods would help. You potentially also see what's going on.

C

C

In the gifts that there are increasing, increase in CPU I mean yeah.

B

I guess it would also be good like looking at the event loop lag there. Any booth any of the tools help you get I know node has some API is to help you get that this clinic.

C

B

C

C

B

Definitely so what's the tool or the tools that you would see, people using.

C

And I think we covered this already in. We will cover two ways of using the CPU profiler to pass sections and I think they are buggy here is yeah. So if it's not it's worth looking.

C

Which I think we have on for most folks.

C

To determine bugs.

C

Yeah, it can be calculated.

B

Right now is that something, though, that you know something: there's there something there we should in recommend being pre-installed or.

C

B

That I guess we'll need to figure out like I'm, not sure if there even is a.

C

C

B

We should be exposing you think those so there are they low enough overhead that they can be like Prometheus is really just collect. Metrics keep sending a tail right.

B

Is it the kind of thing that we can just have like an event? Loop lag always available.

C

C

B

Usually a collection like it's it's in the metrics endpoint that then can't get hit. There are some ways to push push data as well, but you know the default. Is you got an endpoint and every time it gets hit, it provides some metrics.

C

But it's something that.

C

B

Yeah I think you're right when maybe you know something we should consider is like trying to get it in is the default metrics? For you know, prom client, which is one of the leading Prometheus noj's Prometheus libraries.

B

Or having some I think we could also like it also allows extension so having some like. You know, recommendations on how you extend the data that it reports by default might be the other approach.

C

Yeah, we should take a look at the official if they're, using no Jess permit use agent. So just just so, we know what.

C

B

The problem like prom client is the one that seems to to be most used, like we've done some like from our internal review. Looking at it, that's the one that we saw as the one we'd wreck we'd end up recommending just because it was widely used and has the base features so, but in terms of what it provides by default. I don't know that off the top of my head.

B

But that would be good to like you know. If we don't have it doesn't, have these things either confirm that it does have them or if it doesn't figure out how you would get them out.

B

Sings, like, though, like the event loop leg may or may not be there if it if it needs an experimental API, but.

C

C

B

Let's assume we can get those out for how we recommend you debug right, because either it'll be yes, it's there or here's how you would so that you get that in your environments.

C

C

Because I I'm, mostly familiar with you, usually I, look at the matrix.

B

Yeah I'm trying to think the event loop getting longer I.

B

Mean it's it's sometimes the the external dependencies which we're trying to factor out of this. In this case I guess it could also be CPU, though right like it's, that could be a reflection of some things are taking longer and therefore blocking the event loop.

C

B

It may be that, like you know, if you're getting, if you yeah so, are there Kate? The question is: are there cases where you have an event like it might be? If you have an event loop lag, although the the lag I think actually ends up being that wouldn't necessarily be affected by long external requests, I think it's more, you know I'm I'm event. Things are waiting for other they're, not getting processed because other other earlier things are taking the time right like is there.

C

F

C

It can be because some deaths are taking longer part because we have so many tasks. That's also possible right.

B

C

It's like more possible that we have.

C

B

I guess there's that there's the other one is, if you actually do a synchronous, call and block the event loop. That would be another potential.

B

Right, like you, actually do something which you wait on sync, you wait for the response synchronously. Instead of doing it, asynchronously.

B

I'm not aware of any way that you can say: yeah I'm, blocked on I.

B

Guess you could I.

B

Wonder if there's anything that tells like I, don't know if any of the performance things like you could look for things which are like running on the thread for a long elapsed time, even though they're not using a lot of CPU.

B

Right exactly so, you can kind of like. If you look at it, you see say through your penis Prometheus metrics. You see, okay, like if you basically see that that you have high CPU usage and a high event loop lag. It's probably best to just look at the take the CPU profiler out and see well, what's using all that CPU right, because hopefully that gets you there.

B

If, if you don't see the high CPU, but you still see the high event loop like, then you want to figure out what's in the event loop for a long period of time, even if it's not using CPU and what I don't know is I haven't looked enough at the performance timing API to see, if that, if any of those api's give you that or not.

C

D

C

C

B

I know I think I I think that's right.

C

What's interesting is.

C

C

Monitoring agent, rather than you simply need monitoring agent that will result in metrics, not in reports by and you're, going to see basically unreported metrics.

F

C

F

C

You, if you have a if your sampling rate is one minute and you have asynchronous.

C

C

But if you are doing.

B

Now sorry I have to leave five minutes early today, which is now.

D

B

I'll talk to everybody later all right.

C

Anything else on lamentable metrics that we can use between identified problems.

A

C

Yeah I think we should.

C

Separate from this meeting, we should look into adding ways to identify synchronous calls on processes, maybe add a trace inference. Boughs.

C

Anything else on this case.

A

D

So when you say identify synchronous chords, do you mean the method JavaScript matters which don't which are not a synchronous but consumes a considerable amount of CPU.

D

Okay, basically, the ones which blocks the thread.

C

D

C

So many fast spice mines.

G

C

A man must be caused by swapping.

C

When you swap the memory, you probably see a spike. If you swap intermittent frequency, your men see it as intermittent CPU usage or at least in terms of latency, so checking for swap metrics, also something that users can do.

C

Okay, we have one minute left. Is there anything else we can? We want to add to this case. I think you might want to talk about it a little bit more next time before moving to the next one.

C

Okay, so I think that's it. For today we meet again next week for the.

C

And then, in two weeks for the deep dive, continuing performances cases and probably starting another one, so we should decide what we want to discuss next, we can do it in the tracker.

D

Yeah yeah make sense.

C

D

You very much bye.