Node.js Node.js Interactive North America 2016, 15 Dec 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Node's Event Loop From the Inside Out by Sam Roberts, IBM

Description

Node's Event Loop From the Inside Out - Sam Roberts, IBM

Do you think Node is asynchronous because it uses a thread pool for network interactions? Do you there is a relationship between "event emitters" and the "event loop"? I will be working through these misaprehensions as well as others, showing how the Node.js event loop really works. There are many, many pictures and blogs on this topic, but many of them are subtly wrong, or outright misleading. Despite that its event loop that is key to Node's scalability and performance, it remains a misunderstood black box to many. I will show how it works - C code may be seen! - talk about why it works that way, and what some of the implications are for Node developers, as well as briefly describe some changes that the Node core team are considering with respect to how Node uses libuv.

About Sam Roberts
Node, communications protocols, deploying node, rock climbing, mountain biking, beer, and whatever they find really exciting.

A

Okay, my name is Sam Roberts I used to work for strongloop, but now I work for IBM, since we were acquired and working with node for about four four and a half years, and before that, I've done a lot of UNIX systems programming. My background is first embedded systems and then like UNIX and real-time operating systems. So I've worked at the system layer a lot and oh, oh.

A

So I'm, that's what I'm going to talk about so Doc's called the event loop from the inside out, and it was partly inspired by a series of interviews. I got pulled in to I'd, sometimes get cold and to end people saying you know: can you check out this guy? You know he or she looks like they know a lot about node. But can you ask them some questions see what they know about node core?

A

You know you're lower than Express and loop back and stuff, and one of my go-to questions is um what's a node event loop and my secondary question is: is Note single, threaded or multi-threaded and a surprisingly number of people had I, either unclear wrong or muddy answer to those questions. So the part of the purpose of this talk is to clarify what is an event loop.

A

What is the node event loop, so you can speak confidently about it and also it impacts how node works and you you will care someday when things go wrong, so I will.

A

So yeah, here's the questions. What's an event loop is node, single or multi-threaded. We hope to answer these questions so I'm going to jump right into it. There's going to be a bunch of pseudo C code here, please don't criticize my my pseudo C code. It's not always some tactically accurate, but it's hopefully enough to give you a sense of what's going on and if you really want real C code, I've got a guess somewhere you can play with so we're going to jump right into it. So networking on UNIX is is based around sockets.

A

It was socket, the socket is what's returned by the socket system. Call and it's a number it's an integer is integers, are usually they're, sometimes called socket, descriptors they're, more often called file descriptors. This is very confusing. It's not my fault.

A

They don't have anything to do with files necessarily they're just called that their files also have file descriptors. But anyhow, so, let's just be clear: the all of the system call documentation. Api's discuss things in terms of file. Descriptors I'll use that as well so file descriptor can refer to a socket or two other things, and it's kind of a it's a system way of doing object, oriented abstraction, a file. Descriptor is an integer offset into an array. That's kept in the kernel. Every process has an array of file descriptors.

A

That array, then, has a pointer to an object, which is the open control block for whichever resource is attached to the file descriptor, and that has a pointer to a virtual function table. So you basically it allows different things like sockets files.

A

Pipes to all support, read but read does different things depending on what the underlying resource is. So this is, you know: okay, fundamentals of UNIX programming. One file descriptors are not really files, so I'm going to jump into a kind of a classic way of doing TCP servers and it's I call it thread per thread per socket. So basically, what you do is you is you create a socket, so server equals socket?

A

You bind that socket to a port, so you know 80, for example, and then you call listen on it until you call listen, that socket could be used potentially for accepting connections or making connections it could be used for either side. Once you call listen, it will accept connections. So this is a your standard. Three part stands, I call socket, bind it to a port call, listen you're, now ready to accept connections so and the accepts you accept connections by calling except it's in a system call.

A

So the thing about system calls is they can block block system calls are effectively our library calls into the system, and if you, if you ask the kernel to do something for you, that it can't do right now or that it's going to take a little bit of time, it's going to D schedule your process, you're not going to take any more CPU you're, not going to get anymore, see. Pou can't do anything and sometime later, when whatever you asked for is as complete or ready, your process will get scheduled again.

A

This is great when, when you don't want to use CPU and you've got nothing to do, but it's not good if you actually wanted to do something else, so in particular, it's not good. If you want to handle multiple connections at the same time, so we're going to sew the thread per connection model, you call accept it blocks. Tcp connection arrives, it unblocks. You now have a new socket. This socket is specifically for the TCP client. At that point you can, you could start reading and writing on that socket.

A

But if you're doing that, you couldn't accept any more connections. So you kick it off into a thread. You call pthread, create you pass it a function pointer, you pass it. Your socket, descriptor file, descriptor call! Well, you will and that that thread function can now start for. This is an echo server. It can read data from the socket write data back out the socket, and it just does that in a loop in its own thread, the main thread goes back to its while loop calls, except so this works fine.

A

A lot of Java servers work like this, but back in the bad old days, systems would often croak. If you had more than you know, four or five hundred a thousand thousand silent threads at the same time would take systems down. So operating systems are much better than that. Now you can easily handle a few thousand threads on Linux. There's no problem at all windows same thing, but the fact remains. A thread is heavy heavy weight, relatively heavy weight to the amount of data we actually need per connection.

A

All we really need to know to talk to a client is its socket descriptor, and we have to remember what we intended to do with that. Descriptor. That's it is, you know a dozen bytes of memory. That's all we take so to spin up an entirely new operating system resource a thread just to handle a single socket, doesn't doesn't scale well, doesn't scale to tens of thousands of connections. So there's a better way now.

A

I originally had a kind of a simpler piece of code. Here that used a traditional set of UNIX system calls called pull its pull and select they're. Two different system calls basically do the same thing, but I didn't have time so I'm going to jump right into equal, so equal equal is what Linux has BSD systems use. Kq Windows systems do things that are completely different, but have the same end effect, but I'm going to focus on equal and Linux, so free pull. You basically do the same thing as before.

A

You set up your server, so you call socket called by and called listen at an equal and then you create an e pull descriptor, and what we're going to do is we're going to use this descriptor to tell the colonel about the things that were interested in and the kernel is going to use it to tell us when the things when things happen, that we that we asked about. So we excuse me so to start off here we have a server. We only have one socket the server socket.

A

The only thing we can be interested in is: is there an incoming TCP connection? So so we ask that we create a file to script. They pull descriptor. We say: hey we're introduced to scripture to the e pull loop and we're interested in when there's incoming TCP sockets, so we set it up. Then then we go into the loop so we're in the loop we're going to we're going to once again block on the kernel, we're going to say we're going to call a pull.

A

Wait at which point we're going to get D scheduled, no CPU will take, will happen, we're not going to do anything and when the kernel sees something interesting happen interesting as in we were interested in it, it'll wake us up. So what kinds of things can it tell us about? Well, so far, we've only told it we're interested in the server socket. So the only thing it can tell us about is the server socket.

A

So that's that's one of the two things this really simple piece of code to tune do so when the server socket is ready. You pull return. What can we know it's available? We know there's a CP connection. What do we do? We accept the connection and once we've accepted the connection, we have a socket descriptor this one for the the TCP client, but there's not necessarily need data. Just because it's connected to us doesn't mean it sent us anything. So we can't do anything else with it yeah we can't read from it.

A

We can't write to it. So what we're going to do is we're going to add it to the loop, we're going to say, hey this client socket I'd like to know when there's data available so once again, I call I, pull control, add I pass the the connection socket to the leepo loop and I go back to the loop waiting. So this point two things can happen the next time the loop wakes up.

A

It will because of either there's a new TCP connection or there's data available on one of on one specific TCP connection so and that so that's the the other part of the loop. So if there's data available in a specific TCP connection, time to start echoing so I can read up, I can read or date some data out of the out of the connection and then I can write it back out to the client some you might be choking at this point.

A

It's actually much more complicated than that, because just because I can read, data from a socket doesn't mean I can write to it. You know. So what you actually have to do is read the data. Stick it in and stick it in memory somewhere. Add that socket back in saying: hey I'm interested in it, but when it's writable wait until it's writable pull the data back out, there's a lot of a lot of bookkeeping involved, and this is why people use libuv. It's the there's, a lot of race conditions, there's a lot of subtleties!

A

There's a lot of things. You can do wrong, so you really shouldn't be doing this stuff yourself use UV it does it really well, but this is what it's doing at its core. So this this is, that is the node event loop. The answer. The question is: what is the node event loop?

A

Is it's in a semi infinite while loop calling either a pull, wait or pull or k-q wait whatever you know the system call is, but it's a semi infinite loop calling on the server sorry on the kernel, I'm kidding blocking until there's something interesting. When something interesting happens, you know what happens is node. Is it transforms out into a JavaScript API, so something interesting? It might manifest as an event. It might end. You know as a callback and the FS thing, I think sorry, the most precise of language.

A

Here we have s module so that that's that's that now that you've seen the lo loop, you can ask another question is what when at semi-infinite, so when does it exit? Well, if think about it, work, we've got any poll Sat there's a set of events were interested in. We call wait. So that's too interesting, there's an interesting state. What if there are no events? Well, if there are no events, April wait will never give you an event so you're either going to it's always either going to always return immediately.

A

So you busy loop or it's going to block forever, which is silly and not very useful. So what node does? Is it exits? So that's, you know when does node exit, if you ever get asked that or you know have bugs and don't understand why node is or is not exiting. The answer is no two exits when there's nothing to wait on in the e-poll loop anymore, there's a bit of a caveat, there's also an unwrapped call on a lot of node objects.

A

So if you happen to create a row, something that it is being e-poll waited on time, a network socket something, but it's it's not core to what you're doing and you don't want to keep node alive just because that resource is there. You can unwrap it. So UV has a way of saying: okay, yeah I am waiting for those sockets, but I'm not going to count them. If that's the only sockets I have left or whatever resource it is I'm going to call it quits and exit the loop okay.

A

So no it has a you know, not a huge API, but it's a reasonable wide API. There's a lot of modules, a bunch of things so is everything in the node API eople able an answer is kind of yes, and no some things are directly eople. Some things you have to fudge around a little bit to make them available. Then there's basically three kinds of things so Pola will file descriptors. These are directly pulled.

A

Well, that's that's kind of what he pulled as a file descriptor in the loop boom pull for it time, I'm going to talk about time in a second time at least one time out is directly Pola ball and everything else. Well, everything else. You have to do some jiggery-pokery to make it work, and you know it's also very difficult if you're not really familiar.

A

If the underlying system calls right now, it's not documented in the node API what how things are implemented and it met, and it will matter to you eventually so I'll run through these things. So, first of all, socket so sockets include net diagram, HTTP TLS HTTP child processes, namely their pipes, standard out standard in standard error. Okay, about five minutes. All of those things are directly probable.

A

The timeouts you get one timeout, there's one timeout argument to the 2kq in Ipoh, so all timeouts anode are sorted from you know most the one that's going to happen next, to the one that's furthest away, and every time a poll is called it looks at you know when the next time out is- and it takes that time and sticks it in e poll. So that's that's dealt with by a node for you file systems. Everything everything in the file system layer is not it can't be. It can't be selected on I.

A

Don't have time to talk about it, but even though it is a file descriptor, you cannot effectively pull on them. So that always happens in a thread pool so UV had UV has a thread pool and.

A

So basically- and you can't- you can't wait on threads- these are in any pol loop. So there's a thing called there's this ancient UNIX programming trick called the self pipe, so basically create a pipe. It's got two ends when the threads done. It writes a bite into the pipe. The other end of the pipe is being waited on in the e pol loop and the e pol loop wakes up.

A

So this technique shows up over and over again when you've got something that's asynchronous or something off the main loop and you want to wake the loop up. You'll use a pipe to do it there's also event FTS, there's other mechanisms, Linux that are better so title is a fret, so DNS DNS is an interesting case, and if you read the docs that goes so, the docks are better now, but there's all these back bending loops.

A

It goes to try and describe it, but the basic deal is that there's one the system resolver for host names does not necessarily use DNS. A lot of you might know. There's a hosts file. You can write directly, write things in there's, multicast DNS. There's Sun Yellow Pages NIST, there's lots of ways of resolving host names. So if you want to resolve hostnames and have the system do it in any of the many possible ways, you have to call get a get host by outer as or get a tour infos a more recent one.

A

So that's blocking so it has to happen on a thread pool everything else in DNS is asynchronous, and so it's safe if you want to stay, stay out of the thread pool but, however, get hosts by outer is the default. So when you do an HTTP, GET or TCP or net connect, and you give it a host name, it's going to go through the thread pool.

A

So some quick notes about the thread pool so who uses it FS uses it. Dns uses it, but only for DNS lookup. Everything else is fine. Crypto uses it, but only for crypto, random bytes and crypto PB pbkdf2 password-based key derivation for algorithm HTTP, GET and request doesn't use it. Unless you pass a hostname, in which case it has to pass it through the thread pool.

A

So if you don't like that, if you know that your host name is DNS, look able do the lookup yourself using the other DNS functions that will happen entirely on the main loop and out of the thread. Pool and C++ add-ons can use a thread pool now. There's nothing wrong with the thread pool. It exists there, it's very useful. It does things, but there's only four threads by default.

A

So if you have a lot of a few of thousands of concurrent connections and each of them is doing a DNS lookup or doing some file system activity or doing anything that uses a thread pool, they hit contention for the thread pool and you can find your performance dropping through the floor. So you might need to increase the size of the thread pool or if you can get out of the thread pool so you're. Those of you two things.

A

So if you want to increase the thread, pool size, UV thread pool size, is your environment variable of choice, the only one and if you can get off the thread pool, maybe you can for some cases I'll keep those signals. I'm not going to talk about more, it doesn't happen on a thread pool they're, completely async to use self pipes.

A

Child processes don't use the thread, pool there, so they're safe. For you know, high concurrent use you don't have to worry about that. C++ add-ons C++ add-ons are a common source of problems here.

A

If they're they may not use the thread pool at all, because they don't need to that's fine, they may make blocking system calls and they should have used the thread pool that's the problem, so you should review your C++ add-ons and make sure they're not blocking your loop and then, if they do use a thread pool which is good, they didn't block you, but now you might have thread pool contention. So, if you're doing a lot of file system activities- and you have a C++ add-on that does some magical stuff for you- that's non-blocking.

A

Well, maybe that's all going through your for thread thread pool and you need to increase your thread pool size it. So it's good to keep a keep an eye on that and actually for that I would say. There's a really useful metric for applications is the loop time strong agent. Did that APMEX tricks? Did it I think lots of other monitoring solutions? Do it you? You want to watch your loop time because problems will arise, you'll see them.

A

Often in your loop times a loop should finish in like 8 to 12 14 milliseconds spending in your system. If you have stunk and hardware and stuff, but if you, if you see your your loop times in the hundreds of milliseconds you're, either doing something incredibly CPU can intensive or you're blocking and blocking that's a problem. You need to find it.

A

So that's I, sorry for a fast side to talk so hopefully that this makes you know all of you. Everybody knows now who didn't before or already knew what the event loop is when notes multi-threaded and the basic reason. We say that it scales well, which is low low resource cost for connection, and that's that's kind of ended. My talk like said I actually wrote real, see, code like compiles and runs it's buggy, it's not as good as UV, but it's good to play with.

A

If you want to there's a it's in a guest and also burp elder who's, a much funnier and charming person than me did a great talk. It note confidant you about the event loop. He talked about it from the what is called this inside out. He talked about it from the outside in what it looks like more at the node API surface and how it interacts with the famously miss named process. Next tick inside immediate, which do the opposite of each other. I won't go on about it.

A

Okay am I supposed to answer questions. Do we have time, for that? Is there any questions? This is at least one question.

A

Yeah in the slides, it's correct, so I think the my slides are in the whatever the application is so and if you can't find it find me, I can dig it dig it out. Yes,.

A

Well, it uses a thread. Yes.

B

A

So again, because it uses a thread, it does not block. Let us be clear: um it runs the stuff through the thread pool so the thread blocks, but who cares? The thread is only doing one thing so so the main event loop is not going to get blocked and it won't get blocked, because this so-called blocking call is being kicked out into the thread pool. But if there's no thread available, it's going to take longer than you wanted, then that will only happen if you have a hostname to HTTP, GET or Ornette dog connect.

A

B

Addresses in fine.

B

Guy calls out to different service no.

A

It would be imprudent to cash your own DNS. You should use a caching, DNS proxy and most Linux absolutely has one but or you can use you can do the lookup using the other. Dns calls yourself if you, if you do the lookup, that will happen with pure DNS and will happen on the main event loop, and in that end it would also be cached like if you use localhost as your what's a cook stop okay after we done. You could ask me afterwards, but there's there is a classic.

A

There is a classic Linux, proxying, DNS server and I. Would you should use that and not do caching yourself, otherwise you're going to screw up the timeouts and everything else? Okay,.