Deploying Node.js application services

Using Git submodules to streamline Node.js package development

(December 13, 2021) When developing a Node.js package, we typically create a test application with which to exercise the package. That means for every source code change in the package, we must reinstall the package into the node_modules directory of the test application, which can be very frustrating. Git's submodule feature can streamline this by letting you directly edit both the test application and package with no reinstalls required.

Using HTMLParser2, DOMUtils, to process HTML and XML in Node.js

(December 7, 2021) HTMLParser2 is part of a cluster of Node.js packages (domhandler, domutils, css-select, dom-serializer) that enable powerful manipulation of both HTML and XML DOM object trees. These packages can be used not just for web scraping, but for server-side DOM manipulation, and they form most of the underpinning of Cheerio, the Node.js package for jQuery-like DOM manipulation on Node.js.

Getting image metadata (EXIF) using Node.js

(November 17, 2021) We can store extra data inside images, such as geolocation coordinates, text strings, and more. While there are fancy GUI applications for dealing with this data, we sometimes need to process it in our Node.js programs. For example, a static website generator platform, like AkashaCMS, might use metadata stored in images such as to automatically provide captions, or tooltip text. What we'll do in this tutorial is explore reading metadata in Node.js.

Troubles with Asynchronous code flows in JavaScript, and the async/await solution of ES-2017

(October 27, 2021)

Asynchronous coding is one of those powerful tools that can bite, if you're not careful. Passing around anonymous so-called "callback" functions is easy, and we do it all the time in JavaScript. Asynchronous callback functions are called, when needed, as needed, sometime in the future. The key result is that code execution is out-of-order with the order-of-appearance in the code.

Correctly match URL against domain name without killing yourself with regular expressions

(October 26, 2021)

The Internet relies on domain names as a more user-friendly humane address mechanism than IP addresses. That means we must often write code checking if a URL is "within" domain A or domain B, and to act accordingly. You might think a regular expression is the way to go, but it has failings because while a URL looks like a text string it's actually a data structure. A domain name comparison has to recognize that it's dealing with a data structure, and to compare correctly. Otherwise a URL with domain name "" might match the regular expression /$/i and do the wrong thing.

Simplify catching uncaughtException, unhandledRejection, and multipleResolves in Node.js

(October 15, 2021) When native Promise and async/await support was added to Node.js, the process object gained some error events to notify the program about un-handled or un-caught error conditions. It's important to handle such errors, but of course we sometimes neglect to set up error handlers for every possible error condition. The primary purpose for these event handlers is so we know about program errors that we failed to catch. Node.js provides default error handlers which leave a bit to be desired. The log-process-errors assists with these issues.

Implementing worker queues for processing datasets in Node.js

(May 27, 2021) Processing large datasets can be done with a simple for loop. But, processing one data item at a time, eliminates possible efficiency gains from processing data in parallel. But, parallel processing is not scalable, since it can easily swamp CPU or memory resources. Worker queue packages help by constraining the number of parallel tasks executing at any one time.

Loading an ES6 module in a Node.js CommonJS module

(April 22, 2021)

ES6 modules are a nifty new way to write modules. They're rather similar to the CommonJS module format traditionally used by Node.js, but there are significant differences. When ES6 module support was first added to Node.js, it was an experimental feature with significant limitations. One limitation was whether you could use ES6 modules from CJS modules, and there was a 3rd party package to fill that gap. Today, ES6 modules are a first class feature in Node.js, and ES6 modules can be used from CJS without any add-on support.

Complete guide to using ES6 modules to create Node.js packages that are easily usable from CJS modules

(April 22, 2021) It is very tempting to write all JavaScript code on Node.js with ES6 modules. But that creates a problem of interoperability with the large number of existing CommonJS (CJS) modules. Namely, it is difficult to use ES6 modules from a CJS module. Node.js makes it easy to use both CJS and ES6 modules from an ES6 module, and CJS modules can easily use CJS modules, but a CJS module using an ES6 module is painful. At the same time ES6 modules offer enough attractive benefits to warrant rewriting our code to switch from CJS to ES6 modules. This means we need a strategy for easily using Node.js ES6 modules from CJS modules.

Serverless MongoDB, using ForerunnerDB to store local data in a Node.js application

(April 15, 2021) ForerunnerDB is a JavaScript library for storing data using a MongoDB-like API. It runs in both the browser and server-side in Node.js, and requires no setup of any server making it convenient for inline database functionality. It supports persisting data either to browser local storage, or the Node.js supports persisting data to a directory containing JSON objects.

Easily export or import data between CSV files and SQLite3 or MySQL databases with Node.js

(February 27, 2021)

It's convenient to import a CSV file into a spreadsheet to crunch numbers. CSV's (comma-separated-values) and it's brethren (TSV, for tab-separated, etc) are widely used to deliver information. You'll see CSV downloads available all over the place, whether it's your stock broker, a bank, or a government agency, and on and on. Spreadsheet programs like LibreOffice Calc offer direct import of these files, and then as a spreadsheet you've got a zillion tools available to crunch numbers, make graphs, etc. Sometimes, though, you need to load the CSV into a database to do other sorts of work. That's what we'll look at, the twin tasks of autoloading a CSV file into a MySQL database, and of exporting data from MySQL into a CSV file.

Memory-efficient CSV transformation into CSV or text reports in Node.js

(February 26, 2021)

Those of us who consume/edit/modify/publish CSV files must from time to time transform a CSV file. Maybe you need to delete columns, rearrange columns, add columns, rename volumes, or compute some values taking one CSV file and producing another CSV file. Or maybe you need to convert the CSV to a simple text report. You might have a raw CSV file with no column headers, and need to supply column names. Or your CSV might, instead, have header names.

Convert SVG to PNG in Node.js using Sharp, no headless browser in sight

(November 10, 2020) SVG is an excellent portable, XML-based, graphics format that lets us show high fidelity graphics in modern browsers without requiring large image files. It works by using graphics operations written in XML tags. But, sometimes we need to convert an SVG to an image format (PNG, JPG, etc) in Node.js for use in non-browser contexts. Unlike most solutions to this problem, the Sharp library lets you do this without using a headless browser, decreasing the overhead of SVG image conversion.

Fixing CannotPullContainerError when deploying to AWS ECS using ECR registry

(September 4, 2020) The AWS Elastic Container Service is supposed to be an excellent place to deploy Docker containers at scale. But unless you're careful with the configuration it is easy to get stuck with the dreaded CannotPullContainerError. AWS's documentation about this is unclear as to the precise cure for this problem. In this post we'll go over the correct way to configure a VPC and deploy a Docker container in an ECS cluster attached to the VPC.

How to use npm/yarn/Node.js package.json scripts as your build tool

(September 1, 2020) To Node.js the package.json simply tracks a few things about a package like its main module. To npm and yarn this file serves more purposes, from package management data, to dependencies lists, to package installation and project management scripts. The scripts tag may have a humble originating purpose, but by overusing it package.json becomes powerful platform for running command-line software processes like building websites or building software products.

Running the TODO application

(August 28, 2020) In this article series we've developed a Todo application using Node.js 14.x, Express, Bootstrap v5, Socket.IO, and using Sequelize for the database layer. It was developed to explore some of the new features in Node.js, and to explore the post-jQuery world that Bootstrap v5 offers.

Using Bootstrap and Socket.IO for the user interface in an Express Node.js 14 application

(August 26, 2020) The Bootstrap project has released the first Alpha for Bootstrap v5. It contains a number of major changes, such as dropping the requirement to use jQuery. Nothing is said about when the first official release will ship, but we can get a taste of what Bootstrap v5 will be like by using the Alpha to build a small application. This article goes over developing a Bootstrap v5 user interface for a TODO application.

Using Sequelize for the database layer in an Express Node.js 14 application

(August 25, 2020) Like many web applications, the TODO application we are developing with Express and Bootstrap v5 must persist its data somewhere. For that purpose we will use Sequelize to to interface with an SQL database (SQLite3). Sequelize is a high level ORM (Object Relations Manager) for Node.js that supports a long list of SQL databases.

Project setup for Node.js 14, Express, Bootstrap v5, Socket.IO application

(August 24, 2020) In this project we'll create a small TODO application using Node.js 14.x, the latest version of Express, Bootstrap v5, Bootstrap Icons, Sequelize for the database, and Socket.IO for multi-user real time interaction. In this posting we'll initialize the project directory, and put together the initial code.

Single page multi-user application with Express, Bootstrap v5, Socket.IO, Sequelize

(August 23, 2020) The Node.js world changed a lot this year with the Node.js 14 release, and its support for ES6 modules, async/await functions, and top-level async code. Another important change was the Bootstrap project which is preparing version 5 where the leading feature is to drop jQuery. These and other changes have come, and are coming, to application developers. This creates a good opportunity to revisit building web applications using Node.js, exploring not only the new features in Node.js but life with Bootstrap without relying on the jQuery crutch. To that end, let's write a little TODO application using Node.js 14, Express, Bootstrap 5, Sequelize, and Socket.IO.

Node.js Script writers: Top-level async/await now available

(August 13, 2020) Async/await functions are a god-send to JavaScript programmers, simplifying writing asynchronous code. It turns difficult-to-write-and-debug pyramids of doom into clean clear code where the results and errors land in the natural place. It's been wonderful except for one thing, we could not use the await keyword in top-level code in Node.js. But, with Node.js 14.8 that's fixed.

Installing some native-code npm packages on Ubuntu 20.04 fails on not finding python command

(April 20, 2020) Node.js 14.x was just released, as was Ubuntu 20.04. Testing my application on that combination failed because native code npm packages fail because python is not found. WTF? I followed the instructions and installed build-essential which supposedly brings in every compiler tool we would need. But ... fortunately there is a simple solution.

Terraform deployment on AWS ECS a multi-tier Node.js and Nginx system using Service Discovery

(April 1, 2020) Amazon's AWS Elastic Container Service (ECS) lets us deploy Docker containers to the AWS cloud. In earlier postings of this series we have hosted a NGINX/Node.js application stack using Docker on our laptop, and on AWS ECS, then we hosted it on AWS ECS using Terraform. In this tutorial we switch to hosting the application using two AWS ECS Service instances, and use AWS Service Discovery between the instances.

Terraform deployment of a simple multi-tier Node.js and Nginx deployment to AWS ECS

(March 29, 2020) Amazon's AWS Elastic Container Service (ECS) lets us deploy Docker containers to the AWS cloud. ECS is a very complex beast to tame, but Terraform offers a way to easily describe infrastructure builds not only AWS but on many other cloud services. Using Terraform is hugely simpler than any tool offered by AWS ECS.

Deploying a simple multi-tier Node.js and Nginx deployment to AWS ECS

(March 24, 2020) Amazon's AWS Elastic Container Service (ECS) lets us deploy Docker containers to the AWS cloud. ECS is a very complex beast to tame, but Amazon offers a method of using Docker Compose to describe an ECS Service. That hugely simplifies the task of launching tasks on AWS ECS.

A simple multi-tier Node.js and Nginx deployment using Docker

(March 22, 2020) In a world of Microservices, our Docker deployments are probably multiple small containers each implementing several services, with an NGINX container interfacing with the web browsers. There are many reasons for doing this, but lets instead focus on the how do we set up NGINX and NodeJS in Docker question. It is relatively easy, and to explore this we'll set up a two-container system with an NGINX container and a simple NodeJS app standing in as a back-end service.

Github buys npm: Might cause more angst about npm as de-facto package manager for Node.js?

(March 16, 2020) From the beginning of Node.js, npm has been a faithful companion providing useful package management service to the Node.js community. Node.js would not have risen so high without a good package manager, and npm served that role. By rights we should celebrate that Github is buying npm since a big question mark about npm has its level of funding. But - it raises a big question mark about the continued independence of the npm registry once it falls into the clutches of a big corporation (Microsoft owns Github).

Bootstrap v5 will drop requirement for jQuery, drop IE10 and maybe drop IE11 support

(March 11, 2020) The Bootstrap is working on the Bootstrap v5 release, and the team recently committed changes to drop support for at least IE10 (and possibly IE11). Judging by pull requests, their plan to drop the requirement to use jQuery has been a long time coming. On the one hand it is heart-warming to see Internet Explorer finally be dropped from support, and on the other hand we recognize the necessity of letting go of tools that have outlived their usefulness. Finally we'll note that the predicted release date for Bootstrap v5 is "late spring 2020".

A popular Node.js package, request, has been deprecated

(February 13, 2020) What do we do when a venerated elder is supplanted by young whipper-snappers? Node.js is barely 10+ years old, and the availability of async/await and other JavaScript features means older packages must either adapt or be replaced. The community around the request package chose the latter, to let Request become deprecated, and to officially recommend folks to use other packages.

Using Dynamic import in Node.js lets us import ES6 modules in CommonJS code, and more - UPDATED

(January 21, 2020) With ES6, JavaScript programmers finally gained a standard module format that worked on both browser-side and server-side (Node.js) JavaScript runtimes. Plus, ES6 modules offer many interesting features. But Node.js programmers need to know how to integrate ES6 modules with the CommonJS module format we're accustomed to using. Now that Node.js v14 is coming, and bringing with it full ES6 module support as a standard feature, it is time to update a previous blog post about Dynamic Import on Node.js. The Dynamic Import feature, a.k.a. the import() function, gives JavaScript programmers much more freedom over the standard import statement, plus it lets us import an ES6 module into CommonJS code.

Resizing images using Node.js image processing tools, without external dependencies or image servers

(September 18, 2019) While getting ready to post a set of images on another blog, I wondered how to most effectively resize the images. They were high resolution product images that came with a press release, and to be nice to my readers it's important to shrink the image size to promote faster downloads. What if I want to supply several image sizes based on the device used by the visitor? It's not enough to resize the image on my own, it would be better to generate several image sizes. Which led to exploring Node.js tools for resizing images, and finding them lacking.

Simplify your npm publish workflow using Github Actions

(September 12, 2019) Github's recently added new feature, Github Actions, promises us a powerful Workflow system to handle a huge range of tasks. Those of us who publish Node.js packages can use Actions to automatically run tests and then publish the package to npm. Let us see how to use Github Actions to simplify our lives.

A deep dive guide to JavaScript Symbols (ES-6 and beyond)

(August 29, 2019)

ES2015 added a ton of useful new features, and among them is the Symbol type. Symbol lets us avoid property name collisions in a way where we can almost implement private properties in JavaScript objects.

In Jenkins, automatically test an npm package, then publish if tests succeed

(August 14, 2019) We want to automate the boring stuff so we have more time for interesting stuff. Clearly running tests and administrivia like publishing a Node.js package to npm, well, that's boring stuff. It is the same tasks every time, and as an unautomated process you might forget a step, so let's look at how to automate this common workflow for Node.js programmers.

Serious security hole found in Electron framework, Electron app's can be easily backdoored

(August 7, 2019) An attacker that can gain access to a machine can easily patch an Electron application to run bad code that does bad things. The problem is Electron does not secure the contents of ASAR files, and they can be easily opened and modified.

How to install NAVE, an alternative Node.js version manager to nvm

(July 25, 2019) Those of us wanting to keep up with the latest Node.js version, or to test our code on alternative Node.js versions, probably have installed nvm. It is an easy-to-use tool for quickly installing the latest Node.js version, or to easily switch between Node.js versions to test our code. But nvm doesn't fit all use cases, and that's where nave comes in. Its purpose is extremely similar to nvm, but it satisfies a slightly different set of use cases that one may find interesting.

How to create a copy-on-write clone of a file in Node.js

(July 25, 2019) It's been refreshing to learn of an advance in Linux/Unix file system semantics over the last 35 years. In 1984 I first used symbolic links, on a Vax running 4.2BSD. That was a significant advance over the hard links that had existed for many years before. The other day I learned that both Mac OS X and Linux support a new kind of link, a reflink, that is a form of copy-on-write for certain file systems.

What is the most popular front end framework for web applications?

(July 25, 2019)

In my book Node Web Development (see sidebar), I have used Bootstrap to demonstrate the front end of the web application. The book is primarily about Node.js, and therefore it should focus on the back end rather than the front end. But it seems that Bootstrap's popularity is waning, and other frameworks are gaining popularity in the front end. A look at Google Trends confirms this.

Complete guide to using TypeORM and TypeScript for data persistence in Node.js module

(July 16, 2019) TypeORM is an advanced object-relations-management module that runs in Node.js. As the name implies, TypeORM is meant to be used with TypeScript. In this article we'll learn about using TypeORM to set up Entity objects to store data in a database, how to use a CustomRepository instance to manipulate a database table, and to use Relations between Entity instances to simulate database joins.

How to use Passport in ExpressJS to handle Node.js app login/logout functionality

(July 15, 2019) Almost every web application requires login/logout functionality. In the Node.js/Express world, the Passport module handles that functionality.

What is Node.js Exactly? - a beginners introduction to Nodejs

(July 12, 2019) Node.js is JavaScript running outside of the web browser. If you're expecting a Window or Document object because JavaScript has that, you'll be disappointed because those objects are side effects of running in a browser. This introduction to Node.js is a very basic intro, showing the bare bones of beginning to understand what Node.js is, and what its role is in the software industry.

Build A Restful Api With Node.js Express & MongoDB

(July 11, 2019) We are going to cover how to create a restful api using node.js express and mongodb together with mongoose. REST apis help us decouple our backend code from our front end so we can use it across multiple application (mobile apps, web apps, etc). We are going to learn how to build a simple blog post type api with all the useful methods(GET,POST,DELETE,PATCH).

The rumors of Express.js's death are greatly exagerrated

(June 26, 2019) Wandering across a question on Quora (Is Express.js dying? What are the alternatives?) had me stop and take a look at the facts. I'd noticed a few months ago that contributions to the main Express repository had dried up, and therefore I was worried the Express project was dying from neglect. So let's talk about this, because it is important to clear up the air.

Regarding the recent security vulnerability in event-stream and other npm packages

(June 18, 2019) Recently security vulnerabilities were discovered in the event-stream package, and at least one other. Malicious code was added to specific packages in a way that could be done much more broadly. While the specific vulnerability was tightly focused on one specific target and did not affect most of us, the problem could have been extremely wide-spread. As a result we, the Node.js community, need to rethink how packages are managed.

How to avoid vulnerability to malicious modules distributed through npm repository

(June 18, 2019) Recently a pair of modules in the npm repository became the vector for distributing malicious code. As much as npm Inc is working to improve security in the npm repository, there are steps we all must take as Node.js software coders to reduce the vulnerability of our packages. While its unlikely, in theory malicious code could be introduced through any package and how would we notice? It's not enough to trust npm Inc, we have our own work to do to strengthen the modules we publish.

How to create Node.js modules using Typescript

(June 16, 2019) To use TypeScript on the Node.js platform, a core thing to learn is how to create a Node.js module using TypeScript. Obviously using TypeScript on Node.js means transpiling the TypeScript code to JavaScript in a form that is useful on Node.js. Any use of TypeScript means transpiling it to JavaScript, tailoring the generated code to the targeted environment. To target TypeScript for client side use in the browser we need one set of compiler options, and to target Node.js we need a different set of compiler options.

How to set up Typescript compiler and editing environment with Node.js

(June 2, 2019) TypeScript is growing rapidly in popularity within the JavaScript ecosystem. Where interest in JavaScript interest is relatively flat, TypeScript popularity (gauged by Google Trends) is growing fast. TypeScript offers us important data type enforcement, implemented in the compiler, for what is otherwise JavaScript code. This is potentially attractive to any large project written in JavaScript. TypeScript is easy to use on the Node.js platform, so let's see how to do so.

How to correctly calculate path to a Node.js module to access files in that module directory

(May 9, 2019) Many Node.js modules contain files meant to be accessed as data. For example the Bootstrap, jQuery and Popper.js modules do not contain Node.js code, but exist to distribute the libraries. We are supposed to configure Express (or other framework) to serve the files in those directories to a web browser. The question is -- what is the best way to calculate the filesystem path of a file installed as a Node.js module inside the node_modules hierarchy?

Why publish Node.js packages to npm using Babel/Webpack?

(February 22, 2019) Babel and Webpack are widely used tools for front-end-engineering. Between the two you can code JavaScript using the latest ES2016/2017/2018/etc features, with multiple modules, and transpile it down to a simple JavaScript file using ES5 syntax. This is a big step forward in JavaScript tooling. But, since Node.js natively supports a huge fraction of ES2016/2017/2018/etc features, why do Node.js programmers even need to know Babel/Webpack? After reading a tutorial on using Babel/Webpack with Node.js modules, I have the beginning of an inkling for why.

How to fix Node.js require returning an empty module object

(January 25, 2019)

In Node.js the require function is synchronous, meaning you're guaranteed that when require returns the module object is fully populated. But what happens when that doesn't happen? It can happen that the module object is empty, and then what do you do?

Fix Webpack's ability to package an app using the mime module

(August 2, 2018) Application development with Vue.js, Electron, and other frameworks, requires using Webpack. Being a Webpack newbie I'm not sure what it does, but it is required for the tasks I'm doing right now. One of which failed with a strange error message while trying to build a Vue.js component which used the mime module. Namely, we have to instruct Webpack to recognize the JSON files included in the mime module.

Fix Webpack's ability to load CSS while packaging an app

(August 2, 2018) Web applications obviously will need CSS, and therefore Webpack must support packaging applications with CSS files. However with the sample Webpack build procedure I'm using, it fails while loading a CSS file. Let us take a look at the configuration changes required to fix this problem.

A file-system browser component for Electron/Vue.js applications

(July 30, 2018) Many kinds of applications need to browse files in the file system. For example most programmers text editors (I'm writing this in Microsoft Visual Studio Code) include a sidebar showing the file-system. Therefore it's important to have such a component to use in Electron/Vue.js applications. In this article we'll look at one way to implement such a component.

With a Node.js based REST service, is it okay to not use a front-end framework like React or Vue.js?

(July 26, 2018)

React and Vue.js are very popular front-end application frameworks. One might think either one is the best and only way to develop the front end of an application. Lots of folks will tell you, use React because it's the best framework, while others are flocking to Vue.js because whatever. A few years ago The MEAN Stack (Mongo-Express-Node-Angular) was all the rage, and now The MERN Stack (Mongo-Express-React-Node) is taking its place. Is it therefore ridiculous folly to skip using any front end framework? Or to skip using one of the popular choices?

Use Electron-Vue to build Electron/Vue.js application with Bootstrap v4 -- DISRECOMMENDED

(June 30, 2018) Using Vue.js in an Electron app means dealing with using Webpack to build/package the components and other assets. The most effective way to put this together is the Electron-Vue package, which builds itself as "Boilerplate for making electron applications built with Vue.js". What that means is Electron-Vue is a pre-built application template and build infrastructure to simplify running Electron/Vue applications in development mode, and to package Electron/Vue applications for distribution.

Quickly start a new Electron and Vue.js application project,

(June 28, 2018) It's relatively easy to setup a blank Electron application, but of course the modern best practice is to use an application framework when creating an HTML+JavaScript application. The Electron development model more-or-less-exactly matches the model in regular web browsers. That means we can use a framework like React or Vue.js to make a powerful Electron app. In this tutorial we'll set up a simple Electron app, then add to it a simple bit of Vue.js code, and see that it is pretty simple to bring the two together.

Learn the MERN stack with this video tutorial series

(June 27, 2018) MERN (MongoDB, ExpressJS, ReactJS, Node.js) is a popular alternative to the MEAN stack, and that has a nicer-sounding name. Therefore many of us want to learn the MERN stack, and this video tutorial series is an excellent way to start.

Quickly start a new Electron software project

(June 20, 2018) If Electron encapsulates the Chromium web browser to support running GUI applications, isn't it a complex platform to use? Isn't it complex to get started with Electron? Yes, there is a fair amount of complexity under the hood of Electron. Electron really is the guts of the Google Chrome web browser, encapsulated in a Node.js runtime, letting Node.js programmers easily develop desktop GUI applications using web technology. That was a mouthfull to describe, and getting a fully polished application up and running has some complexity. Thanks to starter projects maintained by the Electron team, getting a skeleton application running is quick.