Node.js Web Development

The new test framework built-in to Node.js 18.8.0

 (September 5, 2022) New in Node.js v18.8.0 is a test framework highly remniscient of Mocha. It is marked as experimental, but points to a future where our unit tests will not require an external dependency like Mocha.

Using Protocol Bffers with Node.js applications

 (August 22, 2022) Protocol Buffers is a Google-developed toolchain for binary encoding of data and objects that works between programming languages. It is the basis of gRPC, a cross-language remote procedure call system, but can be used separately.

Creating either CommonJS or ES6 modules for Node.js packages using Typescript

 (February 2, 2022) The core unit of software on Node.js is the package. Therefore using TypeScript for Node.js applications requires learning how to create packages written in TypeScript or sometimes a hybrid of JavaScript and TypeScript. Fortunately this is easy for any Node.js programmer, and amounts to setting up the TypeScript compiler.

Automatically handling runtime data validation in TypeScript

 (May 27, 2022) TypeScript does excellent compile time type checking. But that does not mean TypeScript code ensures runtime type or data validation or checking. TypeScript compiles code to JavaScript with no runtime checking, leaving the door open to runtime data problems.

Setting up the Typescript compiler to integrate with Node.js development

 (January 30, 2022) TypeScript is growing rapidly in popularity within the JavaScript ecosystem. TypeScript offers us important compile-time data type enforcement, for what is otherwise JavaScript code. Any large project written in JavaScript will, in theory, find this useful because, in theory, the larger the project the more likely the coders will have keeping track of data types. The TypeScript tools run on Node.js, and while it can generate code for Node.js, it can be configured for any JavaScript environment.

Generating API documentation for TypeScript packages with TypeDoc

 (March 12, 2022) With TypeDoc, one adds structured comments (JSDoc style comments) to TypeScript source code. With data in those comments, TypeDoc generates nice API documentation you can easily publish to a website. With the addition of another tool, gh-pages, it is easy to publish the documentation to GitHub Pages.

Complete guide to using TypeORM and TypeScript for data persistence in Node.js module

 (July 16, 2019) TypeORM is an advanced object-relations-management module that runs in Node.js. As the name implies, TypeORM is meant to be used with TypeScript. In this article we'll learn about using TypeORM to set up Entity objects to store data in a database, how to use a CustomRepository instance to manipulate a database table, and to use Relations between Entity instances to simulate database joins.

Data validation decorators in TypeScript

 (February 3, 2022) Decorators are an interesting feature of TypeScript, which may become a standard part of JavaScript. They resemble annotations in other languages, in that they're attached to the declaration of something and add behavior or information to that thing. To get some familiarity, let us use a library of data validation decorators along with TypeORM.

Using HTMLParser2, DOMUtils, to process HTML and XML in Node.js

 (December 7, 2021) HTMLParser2 is part of a cluster of Node.js packages (domhandler, domutils, css-select, dom-serializer) that enable powerful manipulation of both HTML and XML DOM object trees. These packages can be used not just for web scraping, but for server-side DOM manipulation, and they form most of the underpinning of Cheerio, the Node.js package for jQuery-like DOM manipulation on Node.js.

Implementing WebHooks and RESTHooks using TypeSript for Node.js and ExpressJS

 (June 30, 2022) The WebHooks architecture lets one webservice send event notifications to another. RESTHooks build on them by advertising available events, allowing software to manage event subscriptions without human intervention.

Deep introduction to using and implementing TypeScript decorators

 (February 10, 2022) Decorators allow us to add additional information to classes or methods in TypeScript. They're similar to annotations such as in Java. With decorators we can add a wide variety of functionality, with a very succinct notation. It is planned that decorators will become a standard part of JavaScript, making it important to learn how they work.

Deep introduction to method decorators in TypeScript

 (February 17, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Method decorators are applied to methods defined in classes in TypeScript. With them we can record information about methods, or modify method execution.

Deep introduction to parameter decorators in TypeScript

 (February 22, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Parameter decorators are applied to method parameter definitions in TypeScript. With them, we can record information about parameters, including customized information, using that data in other features.

Deep introduction to property decorators in TypeScript

 (February 12, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Property decorators are applied to property definitions in TypeScript, and can observe them.

Using the Reflection and Reflection Metadata APIs with TypeScript Decorators

 (February 20, 2022) Making full use of TypeScript decorators requires understanding the Reflection and Reflection Metadata API's. The latter lets you store data in object metadata so that decorators can work together to create complex behaviors.

Runtime data validation in TypeScript using decorators and reflection metadata

 (February 22, 2022) TypeScript decorators allowed us to intercept calls to both accessor functions and methods. That let us spy on data passed to either type of function, or even to supply default values for any that are missing. A practical use we might have for this is automatic validation of data arriving at an accessor or method. By instrumenting these functions, a validation package can act as a gatekeeper ensuring that data in objects is always correct.

Using automated runtime data validation decorators in TypeScript

 (March 10, 2022) A new package for TypeScript enables us to automatically check validity of runtime data, without having to explicitly call a validation library. Instead, the runtime-data-validation package lets us attach decorators describing data validity, and arranges for validation to be automatically performed on assigning a value to a property, or when calling a method.

How to install NAVE, an alternative Node.js version manager to nvm

 (July 25, 2019) Those of us wanting to keep up with the latest Node.js version, or to test our code on alternative Node.js versions, probably have installed nvm. It is an easy-to-use tool for quickly installing the latest Node.js version, or to easily switch between Node.js versions to test our code. But nvm doesn't fit all use cases, and that's where nave comes in. Its purpose is extremely similar to nvm, but it satisfies a slightly different set of use cases that one may find interesting.

How to use npm/yarn/Node.js package.json scripts as your build tool

 (September 1, 2020) To Node.js the package.json simply tracks a few things about a package like its main module. To npm and yarn this file serves more purposes, from package management data, to dependencies lists, to package installation and project management scripts. The scripts tag may have a humble originating purpose, but by overusing it package.json becomes powerful platform for running command-line software processes like building websites or building software products.

Simplify your npm publish workflow using Github Actions

 (September 12, 2019) Github's recently added new feature, Github Actions, promises us a powerful Workflow system to handle a huge range of tasks. Those of us who publish Node.js packages can use Actions to automatically run tests and then publish the package to npm. Let us see how to use Github Actions to simplify our lives.

Using Git submodules to streamline Node.js package development

(December 13, 2021) When developing a Node.js package, we typically create a test application with which to exercise the package. That means for every source code change in the package, we must reinstall the package into the node_modules directory of the test application, which can be very frustrating. Git's submodule feature can streamline this by letting you directly edit both the test application and package with no reinstalls required.

Yarn versus npm, what's the big deal?

 (Tue Sep 19 2017 17:00:00 GMT-0700 (Pacific Daylight Time)) A couple years ago Yarn came onto the Node.js scene promising to be a better npm than npm. Apparently a lot of people have taken to Yarn, given the frequency with which I see instructions to install packages using Yarn. Since npm is installed along with Node.js, and since Yarn just uses the npm package repository, I'm left wondering what all the fuss is about Yarn. If it's just doing the same thing as npm then what is its reason to exist? What advantage does Yarn have over npm? Let's take a look.

Deep introduction to accessor decorators in TypeScript

 (February 15, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Accessor decorators are applied to accessor method definitions in TypeScript, and can observe, or modify data accessed through get/set functions. With care, they can be used for advanced features like runtime data validation.

Deep introduction to class decorators in TypeScript

 (February 10, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Class decorators are applied to class definitions in TypeScript, and can observe, modify, or replace the class definition.

Implementing hybrid decorator functions in TypeScript

 (February 20, 2022) The documentation on TypeScript decorators describes function signatures for each decorator type. It implies that each decorator must be implemented for a specific target object type. But, with a carefully defined method signature we can construct decorators to handle decorating any object type.

Is Node.js one of the most widely used scripting languages on the Internet? Really?

 (2011-03-31 20:21) I can believe JavaScript is one of the most widely used scripting languages, but Node.js? Huh? But, over there on embed.ly they have published two libraries implementing the embed.ly API for PHP and Node.js, and claimed "These two libraries give you server-side access via the two most widely used scripting languages today."

Could Node.x unseat Node.js? Event driven asynchronous server side platform duel in the making?

 (2011-07-11 16:25) Node.js is getting a lot of excited attention, but is it the only way to build an event driven server? For example the other day I summarized some work by Twitter to re-architect their systems including an event driven asynchronous server, but that server was written in Java and in general the article proved that a single-language solution like Node.js might not be the best because there are plenty of useful languages besides JavaScript (see Java, Twitter, and asynchronous event driven architecture). Today on Twitter I saw an announcement of Node.x, which is a general purpose framework for asynchronous event driven programming but rather than built on top of Node.js/V8 it's built on top of the JVM. And by being on top of the JVM it immediately opens the door to a multi-language solution, where Node.js is a single language solution.

How to use Passport in ExpressJS to handle Node.js app login/logout functionality

 (July 15, 2019) Almost every web application requires login/logout functionality. In the Node.js/Express world, the Passport module handles that functionality.

Real-Time Machine Learning with Node.js by Philipp Burckhardt, Carnegie Mellon University

(August 26, 2018)

High Performance Apps with JavaScript and Rust, It's Easier Than You Think

(August 26, 2018)

Node.js Performance and Highly Scalable Micro-Services - Chris Bailey, IBM

(August 26, 2018)

Automating Node.js/JavaScript code quality checks with eslint

 (Wed Dec 20 2017 16:00:00 GMT-0800 (Pacific Standard Time))

With eslint, coders can save a lot of time (in theory) by checking code for simple (but hard to find) errors. Take a function where you typed one parameter name, and used a different name, and then your unit testing did not touch that function. You'll find that error after a customer reports a bug. With eslint such an error can be caught before shipping the code.

Resizing images using Node.js image processing tools, without external dependencies or image servers

 (September 18, 2019) While getting ready to post a set of images on another blog, I wondered how to most effectively resize the images. They were high resolution product images that came with a press release, and to be nice to my readers it's important to shrink the image size to promote faster downloads. What if I want to supply several image sizes based on the device used by the visitor? It's not enough to resize the image on my own, it would be better to generate several image sizes. Which led to exploring Node.js tools for resizing images, and finding them lacking.

Microsoft proposes bringing TypeScript-like type syntax to JavaScript

 (March 9, 2022) It's not every day you see TypeScript as a trending topic on Twitter. But, a proposal by Microsoft to bring type syntax to JavaScript is certainly a big deal worthy of being a Twitter trending topic. Types coming to JavaScript?

A popular Node.js package, request, has been deprecated

 (February 13, 2020) What do we do when a venerated elder is supplanted by young whipper-snappers? Node.js is barely 10+ years old, and the availability of async/await and other JavaScript features means older packages must either adapt or be replaced. The community around the request package chose the latter, to let Request become deprecated, and to officially recommend folks to use other packages.

Express enters Beta 1, proving the project is still alive

 (February 17, 2022) Express is the most popular web application framework for Node.js, even while its development has seemingly stagnated for years. Work on the Express repository is very slow, and the Express 5.x release has been promised for years; for example 5.alpha.1 was released in November 2014. But if you look under the covers, there is significant work happening in Express sub-projects.

Regarding the recent security vulnerability in event-stream and other npm packages

 (June 18, 2019) Recently security vulnerabilities were discovered in the event-stream package, and at least one other. Malicious code was added to specific packages in a way that could be done much more broadly. While the specific vulnerability was tightly focused on one specific target and did not affect most of us, the problem could have been extremely wide-spread. As a result we, the Node.js community, need to rethink how packages are managed.

Bootstrap v5 will drop requirement for jQuery, drop IE10 and maybe drop IE11 support

 (March 11, 2020) The Bootstrap is working on the Bootstrap v5 release, and the team recently committed changes to drop support for at least IE10 (and possibly IE11). Judging by pull requests, their plan to drop the requirement to use jQuery has been a long time coming. On the one hand it is heart-warming to see Internet Explorer finally be dropped from support, and on the other hand we recognize the necessity of letting go of tools that have outlived their usefulness. Finally we'll note that the predicted release date for Bootstrap v5 is "late spring 2020".

Github buys npm: Might cause more angst about npm as de-facto package manager for Node.js?

 (March 16, 2020) From the beginning of Node.js, npm has been a faithful companion providing useful package management service to the Node.js community. Node.js would not have risen so high without a good package manager, and npm served that role. By rights we should celebrate that Github is buying npm since a big question mark about npm has its level of funding. But - it raises a big question mark about the continued independence of the npm registry once it falls into the clutches of a big corporation (Microsoft owns Github).

How do function(err,data) callbacks work in Node?

(2013-12-17 17:38) Those new to Node programming might find the callback function pattern a little difficult to understand.  For example how does the data get into the arguments to the function call, and how does the callback function get called?

Node Cookbook is great for deeper understanding of Node.js programming

(2012-10-31 12:53)

Node Web Development 2nd edition has been released!!

(2013-07-30 12:05)

Books and videos so you can easily learn Node.js programming

(2018-06-14 11:19)

Have you heard about Node.js, but aren't sure what it is? Maybe you know about this exciting new software development platform, Node.js, but don't know where to start learning it? Maybe someone told you Node.js is JavaScript for server-side environments, which left you extremely confused? Or maybe you've never heard of server-side-JavaScript, and think JavaScript only runs inside web pages? The following resources will help you learn how to develop in JavaScript not inside a web browser, but on the server, using the Node.js platform.

Node.js is an exciting new platform for developing web applications, application servers, any sort of network server or client, and general purpose programming. It is designed for extreme scalability in networked applications through an ingenious combination of server-side JavaScript, asynchronous I/O, and asynchronous programming. Its claimed that the event-driven architecture gives a low memory footprint, high throughput, a better latency profile under load, and a simpler programming model. All this makes it an attractive platform for web application development.

Learn the MERN stack with this video tutorial series

(June 27, 2018) MERN (MongoDB, ExpressJS, ReactJS, Node.js) is a popular alternative to the MEAN stack, and that has a nicer-sounding name. Therefore many of us want to learn the MERN stack, and this video tutorial series is an excellent way to start.

Node.js toolkit for mobile iOS and Android devices announced by Janea Systems

 (Tue Oct 24 2017 17:00:00 GMT-0700 (Pacific Daylight Time))

Node.js is no longer limited to server-side application development. The Electron platform, popularized through the Atom editor, is an excellent way to develop desktop applications. Now it's possible to target mobile devices running either iOS or Android using a Node.js implementation. Janea Systems is offering a "library" for both iOS and Android systems allowing an app to host a full Node.js execution environment, and offering UI implementation either with Cordova or React Native.

npm version 5 has major usability bug with installing packages locally

 (Mon Nov 27 2017 16:00:00 GMT-0800 (Pacific Standard Time))

With npm version 5 we gained a lot of welcome new features and performance improvements. I've been happily using npm@5 for several months, but recently discovered a major problem that dramatically affects my workflow. When I'm updating a package, I want to test that package locally WITHOUT pushing changes to the Git repository. To do so, I found it best to install that package into another project to test/run the code. This worked great with npm versions prior to npm@5, but now I have two major problems. First, npm modifies the package.json to insert a "file:" dependency, overwriting the existing dependency, and second it makes a symlink to the package rather than doing a proper installation.

Node.js 10.x released - What's NEW?

 (April 24, 2018)

After a year of development Node.js 10 has been released. It represents a huge step forward because of some high profile features and fixes. In October Node.js 10 will become the active Long Term Support branch, with 11.x becoming the new experimental branch. Let's take a look at what's been included.

Node.js Web Development, 4th edition, coming soon

 (February 25, 2018)

With the rapid advances in the Node.js platform, Packt Publishing (the publishers of Node.js Web Development) and I both felt a new edition was required. The 3rd edition published in mid-2016 updated the text to support Promises and some advanced techniques like deployment using Docker. Since then, async functions have emerged on the scene, and with Node.js 10.x we'll have ES6 modules available. I just submitted the first draft of all 12 chapters to the editors, meaning that the book is about a month from being finished.

The rumors of Express.js's death are greatly exagerrated

 (June 26, 2019) Wandering across a question on Quora (Is Express.js dying? What are the alternatives?) had me stop and take a look at the facts. I'd noticed a few months ago that contributions to the main Express repository had dried up, and therefore I was worried the Express project was dying from neglect. So let's talk about this, because it is important to clear up the air.

Installing some native-code npm packages on Ubuntu 20.04 fails on not finding python command

 (April 20, 2020) Node.js 14.x was just released, as was Ubuntu 20.04. Testing my application on that combination failed because native code npm packages fail because python is not found. WTF? I followed the instructions and installed build-essential which supposedly brings in every compiler tool we would need. But ... fortunately there is a simple solution.

Avoiding rogue Node.js packages by using good version dependencies in package.json

 (January 22, 2022) Several times now, widespread failures in Node.js applications were due to Node.js packages containing malware. If bad code makes it into our applications, what sort of trouble can ensue? That bad code has the same privileges your application has. What if the bad code isn't immediately obvious, but is stealthy malware that steals data?

New Book: Asynchronous JavaScript with Promises, Generators and async/await

(2017-01-01 11:19) The JavaScript language is changing, and among the changes are three new features which will revolutionize the way we write asynchronous code in JavaScript. The problem with the old callback-oriented asynchronous coding practice is the Callback Hell resulting from stacking callbacks within callbacks within callbacks. Between Promises, Generators and the proposed async/await feature, JavaScript programmers can largely free themselves from Callback Hell, and write clean asynchronous code that's easier to write, easier to understand, easier to maintain, and more robust.

How to run JavaScript in the command-line environment

 (June 15, 2018)

JavaScript is an extremely popular programming language deserving of a bigger role than browser-based programming. Fortunately we can use JavaScript outside web browsers, because of several non-browser JavaScript implementations. The most popular, Node.js, is not the only choice for running JavaScript outside the web browser.

Managing Node.js servers on Mac OS X with forever - works best for development

(2014-01-03 17:07) If, like me, you're doing Node.js development on a Mac, you might have a yearning for a tool like MAMP but which works for Node.  A couple weeks ago I wrote a blog post covering the first step, (nodejs.davidherron.com) setting up a Node and npm instance on your computer.   If you don't know what MAMP is, go read that blog post, and then come back here.  What I want to go over today is a way to manage/monitor one or more Node processes on your computer.

Crunching/minifying HTML, CSS and JavaScript in Node.js with the minify module

(2013-01-01 12:33)

I've just added HTML/CSS/JavaScript minification to AkashaCMS (link is external), to minimize the size of static website files rendered by AkashaCMS. This will reduce page load times by decreasing the number of bytes required to be downloaded. We humans like blank space because spreading out information makes it easier to understand. With programming languages that becomes all the white space and indenting conventions. But the white space makes no difference to the computers that interpret the web pages, in fact it makes the web browsing experience slower because it takes longer to download or execute the web pages, or the CSS or JavaScript referenced by the web pages.

How to correctly calculate path to a Node.js module to access files in that module directory

(May 9, 2019) Many Node.js modules contain files meant to be accessed as data. For example the Bootstrap, jQuery and Popper.js modules do not contain Node.js code, but exist to distribute the libraries. We are supposed to configure Express (or other framework) to serve the files in those directories to a web browser. The question is -- what is the best way to calculate the filesystem path of a file installed as a Node.js module inside the node_modules hierarchy?

Can I monkey patch a Node.js module installed from the npm repository so the patch is maintained after npm install?

(April 7, 2018)

Suppose you've found a Node.js module that almost does what you want, but it needs another feature or two, or some other tweak? How to modify the package, and perhaps temporarily modify it at runtime? Technically, monkey patching is not simply maintaining a local modification to someone elses code, but to dynamically modify that code at runtime. Some say this is evil, but it is a practice that is required at times. Let's see how to do this.

Simple data export and manipulation using Node.js and the node-mysql module

(2012-07-10 16:04) What follows is a fairly simple data processing script to extract data from a Drupal installation in a MySQL database. The purpose was to export nodes from a Drupal site, reformatting them for some software I'd written to import content into a blogger blog. In other words, this is the sort of simple data extraction and manipulation tool we write all the time.

Using SSL to connect to MySQL database in Node.js

(March 7, 2018) Encrypting database connections can be extremely important for security. The documentation for the Node.js MySQL driver briefly mentions SSL support, and does not give adequate documentation. What follows is an example showing how to connect using PEM certificates to a MySQL server that was configured with a self-signed root CA.

Fix NVM_NODEJS_ORG_MIRROR deprecation warning when running npm install

 (April 10, 2018) The nvm script is an exceedingly useful tool for maintaining multiple Node.js versions while keeping your sanity. Running "npm install", I noticed a strange warning message from node-gyp about "NVM_NODEJS_ORG_MIRROR" being deprecated. Looking for the cause, I found a cause centered on nvm. Let's see how to fis this problem.