Node.js Node.js Live Events, 21 Apr 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Node.js Live (Paris) - Igor Soarez, Anti-patterns in Node.js

Description

Igor Soarez works for YLD, which provides consulting for large enterprises looking to adopt Node.js. After helping many enterprises transfer to Node.js, Igor and his team decided to research (based on their work) the common mistakes that happen when enterprises begin to adopt Node.js, especially if the developers are coming from a Java or .NET background.

Igor discusses the most common and problematic anti-patterns that arise when enterprises look to adopt Node.js and how to solve these issues.

You can follow Igor on Twitter @igorsoarez.

Thank you to Opbeat for sponsoring the videos for Node.js Live Paris, and IBM for sponsoring the Node.js Live Paris event

A

And nodejs any patterns and bad practices. Why is this guy? Talking about? This nodejs keeps getting more and more adopted by large enterprises. The benefits of note are getting established, so more companies are feeling more confident about adopting note.

A

Companies with big names have led the adoption of note and the rest of the enterprise is just following along. That's why it a wildly: we've been seeing more and more developers coming into node from other backgrounds and in most enterprise settings. These are developers that are looking to be productive as quickly as possible and, as we all know, when compared to Java and.net know, Jas has a different programming model and culture.

A

So it's no wonder that this clash of paradigms leads to the emergence of anti-patterns, I'm, Igor and I'm principal engineer to aldi. For more than two years now, we've been helping our clients adopt nodejs safely and as quickly as possible. That includes helping them contain and control the risks of this paradigm clash and a big part of our work is educating our clients. This includes advising them on best practices, and so we had plenty of exposure to mistakes. Enterprise developers make when rushing into node besides just telling them what to what they should. Do.

A

We also one of the things, a list of things that they should watch out for, so we took a deep look of our at our engagements history and we try to aggregate and distill the most common and the most problematic anti-patterns and enterprise node.js projects.

A

We ended up with a long list, but I've tried to select the most important ones for this quick talk and remember. This is based on our experience, so your mileage may vary to help with our narrative. Let's try to think of a fictional character named Jane she's, going to be our hero for the story.

A

Jane is an experienced java developer for a big company. Her experience is with javascript is limited to all jquery based web apps, and james mission is to build a nodejs prototype that will hopefully become a new first-class service. So let's try to place ourselves and change shoes and try to get a sense of what this journey looks like for. Her we've seen that the very first type of mistakes that chain is likely to make are related to the use of JavaScript on the server side.

A

It's not even know jazz specific things you have to remember her experience is limited to old jQuery applications and while these mistakes may seem silly to us, the reality is that we see them happening over and over again. First mistake: Jane learned that doing I, oh and node involves callbacks a chain wasn't used to having to change so many callbacks. The amount of chain died. Oh she needed to perform in browser applications.

A

Usually wasn't that big and I really don't want to be the guy telling you about callbacks today, it's the most basic anti-pattern of all, but we can't ignore it. We all know why this is a problem. The code base gets hard to read hard to change. We got it for extra references, etc, but Jane is learning this slowly.

A

This is a real world. It's a real world example. It's the type of thing we run into this was a module um file, module that fetched task from riders and the road we need to guide chain into look something like this Jane. It's a good idea to name your functions, Jane, there's this thing called hoisting, it's pretty cool for readability.

A

Jane functions should really go in the most outer scope as possible, and it's easy to do down that. You've done the previous steps and oh jane, you should try to beat. You should try to avoid being repetitive error, handling and now that you understand all this chain, you should really use async. It makes your life easy.

A

Alright, second mistake: coming from strong, strong typed languages, Jane used to rely on the compiler to verify that interfaces were being used with the proper types. So the way she starts, defining interfaces for constructors and functions in general is to provide an ever-growing list of arguments.

A

It actually takes some time until she learns that it makes it harder to change the API, that it makes it error-prone for the interface users and it makes it hard to set default values, and this is because, usually the API start small, but then options grow in complexity. Developers like Jane are used to strong typed languages like Java and C sharp we're implementing the options. Pattern implies defining class just for that and it just too much work.

A

That's until they get used to JavaScript. So Jane needs to understand that when a function takes more than two or three arguments, she should consider using an object. Instead. Third mistake: Jane is feeling a bit more comfortable using javascript. She learned that functions can be invoked with a different number of arguments. She also learned about the special arguments variable and she finds this useful because she was used to doing method overloading in Java, but if you abuse it, it becomes unnecessarily confusing.

A

It becomes hard to handle all the different cases. So if Jane wants to be successful, JavaScript she will sooner or later prefer just using an object, an option, object and said. Instead of trying to mimic um method overloading with Java Jane was forced to place each class in one file and browse the JavaScript. There are no rules and optimally. You want to deliver a big concatenated file and no there's no compilation step so you're not forced to do anything.

A

So this is why I believe this is another anti-pattern we're talking about huge file modules or separation of concerns, testing becoming harder than it should. The most frequent examples we find are having all the request, handlers on the same file, module or modules with the big series of unrelated functions.

A

So what Jane needs to realize is that it's much easier and much quicker to create file modules know than it is to create Java class files. So her gauge for when to put things in the different file need some adjustment. She'll see the small modules with clear interfaces are easy to test, and sometimes the complexity. Analysis too will help here coming from Java and translating classes into prototypes, Jane's used to classes and creating objects using the new operator.

A

Unfortunately, she thought JavaScript also has a similar, similar way of creating objects from classes, but the reality is that even the use of the new keyword in Java Script is mostly unnecessary. It's harder to introduce object caching, later on.

A

If you want to it's hard to switch to functional programming, you can't just return things anymore and, worst of all, if you forget to call a constructor without a new keyword, you'll get strange behavior with no warning and overall, as you can see in this example, modeling by relying on too many constructed objects, makes your code look like Java, which can annoy most node developers.

A

So we need to tell Jane about the revealing pattern.

A

Second, set of mistakes that chain is likely to make are more specific to the way node works.

A

So Jane spent a lot of time developing and debugging until she finally got a feature working and she ships it but turns out that some of those pesky call back error arguments were being ignored, which leads to some mysterious bugs, and this is unfortunately another anti-pattern.

A

It's too easy to forget about handling errors and the debugging experience usually ends up with a feeling of this is why note sucks so here using a linter that makes the right checks help Jane is still finding the best way to glue things together and she's coming up with small utility functions that she keeps on a utility. Is that file that just keeps growing so when she needs the same function on her second module, she isn't sure whether to copy the file module around or to promote it as a very weird node module.

A

It's also sometimes called misc or common. So Jane will need to learn that in node modules are cheap modules may be too big, but no module is ever too small. She should embrace the single responsibility principle and she should use individualized repositories for node modules, regardless of if they're, public or private.

A

Turns out this sink versus acing thing is still not very clear to Jane she copy pasted snippet of code from stack overflow and put it on a request handler. It all seems to work in her environment, but as soon as the service gets a bit of load, it becomes unresponsive and Janes thinking seems like no doesn't scale.

A

So what Jay needs learned is that there's a set of operations that should only be used in the initialization phase after the process starts handling concurrent requests. Those operations should not be called Jane learned about streams and she started using them and one instance: Jane had trees three streams and she parked them all together, but she missed some details of what would happen in the narrow scenario. Eventually, this makes for another mysterious bug. This is something that even some relatively experienced, no developers don't know when the stream throws an error or closes while piping.

A

The source stream is not properly disposed and therefore, resources may leak. This happens because the source stream isn't destroyed when the target streams closes.

A

So, let's take a closer look at the problem, so you have three streams: the source drain, the transfer stream and the right stream. If there is an error on the transform stream, the target stream gets destroyed, but the readable stream doesn't it's left dangling?

A

The same thing happens: if there is an error emitted in the right stream, the transform stream will not get automatically destroyed. This means that you need to carefully handle every stream pair that your piping together. That means this pair and this pair also.

A

So what's the right solution for Jane here Jane needs to learn that when using streamed up pipe, she either needs to listen for error and close events on every stream and do the proper cleanup or she can use the pump module instead of the natives instead of the native stream that pipe the first option when using stream that pipe she needs to learn. She needs to listen for error and close events on every stream and do the proper clean up something like this or she can use the pump module.

A

It will take care of properly handling the clothes and error events by disposing of all unclothed streams.

A

Changing how require works. The fact that a note the paths for file modules are relative is annoying jane. Coming from java, Jane was familiarized with a global namespace. She feels that using the default node way forced her to have ugly require pads, and she claims that it makes it harder to move code around so the default behavior just feels wrong to her, and eventually she found a way to fix it.

A

She found out that, if you can that you can fiddle with the note path environment variable or that there are these modules that allow her to change the way require works. All she really wants is for the require paths to be absolute. That is relative to the module root.

A

You need to understand that, for someone in Jane's use the idea that it will become easier to reason about the structure of the module. If you do, this can be appealing. Yes, it becomes easier to move file modules round in and out of folders, but the big problem is.

A

It also becomes easier to create dependencies across folders, and this is where a cohesion tends to get lost. So, what's the lesson for Jane here, if you look into most of the best modules out, there require paths with lots of dots, isn't something you find often so the right. The right approach is about how Jane organizes for code and models her problem, ideally node modules are small, so folders aren't even needed, but when they are, every folder should encapsulate the domain and abstract it away in a concise way.

A

Each folder should be what's described, that Eric was described by Eric Evan as a bounded context, so no file module inside a folder sure ever require another foul module inside the sibling folder.

A

Another painfully recurring pattern. The monolith use an API is that tent that end up cohabiting the same module services that perform unrelated functions, huge code bases notice great for prototyping, but that can sometimes be a trap. So Jane tends to stuff everything into the same code base, and we all know this results in poor test coverage. It gets harder to make changes, it's harder to collaborate, you get longer delivery cycles and you spend more time onboarding and hand-holding new collaborators.

A

So Jane story goes something like this. It starts off by being a small service that will talk to a back-end system. It starts by just holding a few resources, and now it needs some authentication and session session management. So, let's make that in it powers a single page application and coors is hassle. So, let's, let's make it also serve the static assets. Oh and now we have something that should really be another app, but it needs to use the same authentication provider and some of the API endpoints.

A

So it's more convenient to just put it there also, and then we need billing and support for a multi-country fact. So, let's add it there also on, and so it goes and when Jane realizes what she's heading into it's always too late she'll realize that she should have kept view separate from the AP I thought of modelling the problem with many small modules and not been afraid of breaking down services. Now it's just harder to deal with.

A

Jane is now doing unit tests back when she did Java. She would have a unit test for each Java class. Now she's writing a test. Suite for each javascript file and in order to achieve a relevant test coverage, james dean tends to expose some internal details of the module and then every time an implementation detail changes. Someone in the team wastes a lot of time. Updating the film tests, so Jane needs to keep the tests at the interface level. All that it's actually require is the main module.

A

She should focus on testing what the module should do and not on testing the implementation details. She should try to keep module small and externalize functionality as much as possible. She'll eventually learn to try as hard as possible to not conflate concerns in the same module.

A

His new members come in to collaborate on change. Project Jane, finds that there are issues when collaborators are setting up things, work well in her development environment, but not on everyone else's.

A

Here's an example of how it happens, developer rights, test suite using her favorite testing framework. She had its package Jason and adds a commanded using that uses that testing tool and everything runs great then the collaborator joins the project runs NTM test, gets a command, not found error and realizes.

A

She needs to install the tool it's annoying, but it doesn't seem like that bit of that big of a problem, but the reality is that there's just no control over the installed version on each developers machine they can have different behavior a flag might not be supported by all versions, and this gets even more complicated when you couple tools together, for instance, using a linter, go rich check, transpiler mini fire, etc.

A

So Jane eventually learns that when modules declare executables, they are made available under node module, slash bin and that this folder is added to the search path by NPM before it runs any NPM script.

A

Jane says that the team is testing their code, but bugs are still slipping into production and some teams, member some team members are even claiming that tests aren't that useful. The problem is that chain is just not measuring. Test coverage turns out that there's a lot of code that isn't actually being tested without code coverage, Jane can't be sure of what is being tested. We can. We also find that code coverage can be a very good indicator of quality.

A

The solution could be simpler. We just need to tell Jane about a tool like Instanbul.

A

The best thing for Jane to do is to make it part of the acceptance of the acceptance process using git hooks, for instance, the pre-commit module is great for that or she can also track it using something like overalls.

A

Sometimes we find the chain doesn't seem to be comfortable with the idea of having a long list of dependencies. She fails to leverage the rich ecosystem of public node modules, and she is aware of the price she's actually paying for this. Regarding the public registry, NPM has like a quarter million packages, and several hundred packages are created every day.

A

So why isn't Jane making good use of NPM? It could be that they don't know about the existence of a certain module. It could also be that they are reluctant to use third-party my packages in general or that they're unsure of how to deal with a lot of dependencies.

A

Sometimes it's just that they're not breaking the problem right. Imagine a requirement for a node module tool that goes like this. I need a testing framework that computes code coverage and since coverage stats to cover also do there's no module for that. But if they break the needs down, then it becomes easier. They need to learn to choose each tool. That was that does one thing it does it well choose tools that play well without and build just a bit of glue that exposes a simple interface congruent to your needs.

A

Sometimes Jane's biggest issue is just that: they're, not sure how to select the package. How to select the dependency is more of a problem in node than in any other ecosystem of public packages.

A

Okay, so James project is now ready for production. The team gets excited, but the cheering stops when they start getting production traffic and they start looking at the performance. Metrics processing large quantities of data was rarely a problem in jane's previous java days. She isn't used to be wary of these situations, but now her application is having some hiccups. A simple filter or aggregation over a large data set can delay the event loop, causing latency bubbles. There are different ways this can happen.

A

Jane needs to know that for large quantity of data she must either schedule small amounts of work at a time that might mean doing streaming or offload the workload to another process, preferably with the more appropriate runtime and last one for today, jane learn how to use async to manage iterations involving asynchronous I/o, but she didn't dig deep enough to be aware of the dangers of not putting a limit on concurrency, but Jane can later notice high latency at times and possibly only observed in production, which can seem somewhat correlated to request frequency, but not entirely for a high enough number of concurrency.

A

The event loop gets clogged with too much cpu work. Many operations started in parallel can lead an application to have hiccups. This is the kind of problem that makes circumvent the whole test, suite and CI checks and only show up in production. Here you can see an adaptive example of request handler on a messaging app. This handler is used for retrieving a conversation.

A

The service, this service call needs to get the entire user profile for each conversation participant what happens when the conversation regis reaches. Hundreds of participants use the profile that get operations will get started on the same event: loop tick and this clogs the event loop, making this application less responsive and also making the prone to memory exhaustion.

A

So in this case, Jane needs to sacrifice response time for the greater good and limit the concurrency. This way, she'll ensure that this will not overwhelm the event. Look, that's it. The main takeaway here is notice, fundamentally different from other technologies used in big teams and opting note means adopting newer practices. I just showed you some of them, but there are more- and I hope this talk helps. You form a better idea of some of the most basic problems you have to face when adopting note from an enterprise perspective.

A