Node.js Web Development

Automatically handling runtime data validation in TypeScript

(May 27, 2022) TypeScript does excellent compile time type checking. But that does not mean TypeScript code ensures runtime type or data validation or checking. TypeScript compiles code to JavaScript with no runtime checking, leaving the door open to runtime data problems.

Generating API documentation for TypeScript packages with TypeDoc

(March 12, 2022) With TypeDoc, one adds structured comments (JSDoc style comments) to TypeScript source code. With data in those comments, TypeDoc generates nice API documentation you can easily publish to a website. With the addition of another tool, gh-pages, it is easy to publish the documentation to GitHub Pages.

Using automated runtime data validation decorators in TypeScript

(March 10, 2022) A new package for TypeScript enables us to automatically check validity of runtime data, without having to explicitly call a validation library. Instead, the runtime-data-validation package lets us attach decorators describing data validity, and arranges for validation to be automatically performed on assigning a value to a property, or when calling a method.

Microsoft proposes bringing TypeScript-like type syntax to JavaScript

(March 9, 2022) It's not every day you see TypeScript as a trending topic on Twitter. But, a proposal by Microsoft to bring type syntax to JavaScript is certainly a big deal worthy of being a Twitter trending topic. Types coming to JavaScript?

Importing an ES6 modules over HTTP/HTTPS in a Node.js

(February 26, 2022) In Node.js 17.6.0, an experimental new feature allows us to import modules from an HTTP or HTTPS URL. That will close one of the differences between Node.js and Deno, which is that Deno allows packages to be imported using HTTPS. Further, ES6 modules in the browser allow importing modules over HTTPS. Until now, the Node.js team did not allow this, citing security concerns.

Runtime data validation in TypeScript using decorators and reflection metadata

(February 22, 2022) TypeScript decorators allowed us to intercept calls to both accessor functions and methods. That let us spy on data passed to either type of function, or even to supply default values for any that are missing. A practical use we might have for this is automatic validation of data arriving at an accessor or method. By instrumenting these functions, a validation package can act as a gatekeeper ensuring that data in objects is always correct.

Deep introduction to parameter decorators in TypeScript

(February 22, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Parameter decorators are applied to method parameter definitions in TypeScript. With them, we can record information about parameters, including customized information, using that data in other features.

Using the Reflection and Reflection Metadata APIs with TypeScript Decorators

(February 20, 2022) Making full use of TypeScript decorators requires understanding the Reflection and Reflection Metadata API's. The latter lets you store data in object metadata so that decorators can work together to create complex behaviors.

Implementing hybrid decorator functions in TypeScript

(February 20, 2022) The documentation on TypeScript decorators describes function signatures for each decorator type. It implies that each decorator must be implemented for a specific target object type. But, with a carefully defined method signature we can construct decorators to handle decorating any object type.

Deep introduction to method decorators in TypeScript

(February 17, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Method decorators are applied to methods defined in classes in TypeScript. With them we can record information about methods, or modify method execution.

Express enters Beta 1, proving the project is still alive

(February 17, 2022) Express is the most popular web application framework for Node.js, even while its development has seemingly stagnated for years. Work on the Express repository is very slow, and the Express 5.x release has been promised for years; for example 5.alpha.1 was released in November 2014. But if you look under the covers, there is significant work happening in Express sub-projects.

Deep introduction to accessor decorators in TypeScript

(February 15, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Accessor decorators are applied to accessor method definitions in TypeScript, and can observe, or modify data accessed through get/set functions. With care, they can be used for advanced features like runtime data validation.

Deep introduction to property decorators in TypeScript

(February 12, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Property decorators are applied to property definitions in TypeScript, and can observe them.

Deep introduction to using and implementing TypeScript decorators

(February 10, 2022) Decorators allow us to add additional information to classes or methods in TypeScript. They're similar to annotations such as in Java. With decorators we can add a wide variety of functionality, with a very succinct notation. It is planned that decorators will become a standard part of JavaScript, making it important to learn how they work.

Deep introduction to class decorators in TypeScript

(February 10, 2022) Decorators allow us to add additional information to classes or methods in TypeScript, and are similar to annotations such as in Java. Class decorators are applied to class definitions in TypeScript, and can observe, modify, or replace the class definition.

Data validation decorators in TypeScript

(February 3, 2022) Decorators are an interesting feature of TypeScript, which may become a standard part of JavaScript. They resemble annotations in other languages, in that they're attached to the declaration of something and add behavior or information to that thing. To get some familiarity, let us use a library of data validation decorators along with TypeORM.

Creating either CommonJS or ES6 modules for Node.js packages using Typescript

(February 2, 2022) The core unit of software on Node.js is the package. Therefore using TypeScript for Node.js applications requires learning how to create packages written in TypeScript or sometimes a hybrid of JavaScript and TypeScript. Fortunately this is easy for any Node.js programmer, and amounts to setting up the TypeScript compiler.

Setting up the Typescript compiler to integrate with Node.js development

(January 30, 2022) TypeScript is growing rapidly in popularity within the JavaScript ecosystem. TypeScript offers us important compile-time data type enforcement, for what is otherwise JavaScript code. Any large project written in JavaScript will, in theory, find this useful because, in theory, the larger the project the more likely the coders will have keeping track of data types. The TypeScript tools run on Node.js, and while it can generate code for Node.js, it can be configured for any JavaScript environment.

Avoiding rogue Node.js packages by using good version dependencies in package.json

(January 22, 2022) Several times now, widespread failures in Node.js applications were due to Node.js packages containing malware. If bad code makes it into our applications, what sort of trouble can ensue? That bad code has the same privileges your application has. What if the bad code isn't immediately obvious, but is stealthy malware that steals data?

Using Git submodules to streamline Node.js package development

(December 13, 2021) When developing a Node.js package, we typically create a test application with which to exercise the package. That means for every source code change in the package, we must reinstall the package into the node_modules directory of the test application, which can be very frustrating. Git's submodule feature can streamline this by letting you directly edit both the test application and package with no reinstalls required.

Using HTMLParser2, DOMUtils, to process HTML and XML in Node.js

(December 7, 2021) HTMLParser2 is part of a cluster of Node.js packages (domhandler, domutils, css-select, dom-serializer) that enable powerful manipulation of both HTML and XML DOM object trees. These packages can be used not just for web scraping, but for server-side DOM manipulation, and they form most of the underpinning of Cheerio, the Node.js package for jQuery-like DOM manipulation on Node.js.

Getting image metadata (EXIF) using Node.js

(November 17, 2021) We can store extra data inside images, such as geolocation coordinates, text strings, and more. While there are fancy GUI applications for dealing with this data, we sometimes need to process it in our Node.js programs. For example, a static website generator platform, like AkashaCMS, might use metadata stored in images such as to automatically provide captions, or tooltip text. What we'll do in this tutorial is explore reading metadata in Node.js.

Troubles with Asynchronous code flows in JavaScript, and the async/await solution of ES-2017

(October 27, 2021)

Asynchronous coding is one of those powerful tools that can bite, if you're not careful. Passing around anonymous so-called "callback" functions is easy, and we do it all the time in JavaScript. Asynchronous callback functions are called, when needed, as needed, sometime in the future. The key result is that code execution is out-of-order with the order-of-appearance in the code.

Correctly match URL against domain name without killing yourself with regular expressions

(October 26, 2021)

The Internet relies on domain names as a more user-friendly humane address mechanism than IP addresses. That means we must often write code checking if a URL is "within" domain A or domain B, and to act accordingly. You might think a regular expression is the way to go, but it has failings because while a URL looks like a text string it's actually a data structure. A domain name comparison has to recognize that it's dealing with a data structure, and to compare correctly. Otherwise a URL with domain name "loveamazon.com" might match the regular expression /amazon.com$/i and do the wrong thing.

Simplify catching uncaughtException, unhandledRejection, and multipleResolves in Node.js

(October 15, 2021) When native Promise and async/await support was added to Node.js, the process object gained some error events to notify the program about un-handled or un-caught error conditions. It's important to handle such errors, but of course we sometimes neglect to set up error handlers for every possible error condition. The primary purpose for these event handlers is so we know about program errors that we failed to catch. Node.js provides default error handlers which leave a bit to be desired. The log-process-errors assists with these issues.

Implementing worker queues for processing datasets in Node.js

(May 27, 2021) Processing large datasets can be done with a simple for loop. But, processing one data item at a time, eliminates possible efficiency gains from processing data in parallel. But, parallel processing is not scalable, since it can easily swamp CPU or memory resources. Worker queue packages help by constraining the number of parallel tasks executing at any one time.

Loading an ES6 module in a Node.js CommonJS module

(April 22, 2021)

ES6 modules are a nifty new way to write modules. They're rather similar to the CommonJS module format traditionally used by Node.js, but there are significant differences. When ES6 module support was first added to Node.js, it was an experimental feature with significant limitations. One limitation was whether you could use ES6 modules from CJS modules, and there was a 3rd party package to fill that gap. Today, ES6 modules are a first class feature in Node.js, and ES6 modules can be used from CJS without any add-on support.

Complete guide to using ES6 modules to create Node.js packages that are easily usable from CJS modules

(April 22, 2021) It is very tempting to write all JavaScript code on Node.js with ES6 modules. But that creates a problem of interoperability with the large number of existing CommonJS (CJS) modules. Namely, it is difficult to use ES6 modules from a CJS module. Node.js makes it easy to use both CJS and ES6 modules from an ES6 module, and CJS modules can easily use CJS modules, but a CJS module using an ES6 module is painful. At the same time ES6 modules offer enough attractive benefits to warrant rewriting our code to switch from CJS to ES6 modules. This means we need a strategy for easily using Node.js ES6 modules from CJS modules.

Serverless MongoDB, using ForerunnerDB to store local data in a Node.js application

(April 15, 2021) ForerunnerDB is a JavaScript library for storing data using a MongoDB-like API. It runs in both the browser and server-side in Node.js, and requires no setup of any server making it convenient for inline database functionality. It supports persisting data either to browser local storage, or the Node.js supports persisting data to a directory containing JSON objects.

Easily export or import data between CSV files and SQLite3 or MySQL databases with Node.js

(February 27, 2021)

It's convenient to import a CSV file into a spreadsheet to crunch numbers. CSV's (comma-separated-values) and it's brethren (TSV, for tab-separated, etc) are widely used to deliver information. You'll see CSV downloads available all over the place, whether it's your stock broker, a bank, or a government agency, and on and on. Spreadsheet programs like LibreOffice Calc offer direct import of these files, and then as a spreadsheet you've got a zillion tools available to crunch numbers, make graphs, etc. Sometimes, though, you need to load the CSV into a database to do other sorts of work. That's what we'll look at, the twin tasks of autoloading a CSV file into a MySQL database, and of exporting data from MySQL into a CSV file.

Memory-efficient CSV transformation into CSV or text reports in Node.js

(February 26, 2021)

Those of us who consume/edit/modify/publish CSV files must from time to time transform a CSV file. Maybe you need to delete columns, rearrange columns, add columns, rename volumes, or compute some values taking one CSV file and producing another CSV file. Or maybe you need to convert the CSV to a simple text report. You might have a raw CSV file with no column headers, and need to supply column names. Or your CSV might, instead, have header names.

Convert SVG to PNG in Node.js using Sharp, no headless browser in sight

(November 10, 2020) SVG is an excellent portable, XML-based, graphics format that lets us show high fidelity graphics in modern browsers without requiring large image files. It works by using graphics operations written in XML tags. But, sometimes we need to convert an SVG to an image format (PNG, JPG, etc) in Node.js for use in non-browser contexts. Unlike most solutions to this problem, the Sharp library lets you do this without using a headless browser, decreasing the overhead of SVG image conversion.

Fixing CannotPullContainerError when deploying to AWS ECS using ECR registry

(September 4, 2020) The AWS Elastic Container Service is supposed to be an excellent place to deploy Docker containers at scale. But unless you're careful with the configuration it is easy to get stuck with the dreaded CannotPullContainerError. AWS's documentation about this is unclear as to the precise cure for this problem. In this post we'll go over the correct way to configure a VPC and deploy a Docker container in an ECS cluster attached to the VPC.

How to use npm/yarn/Node.js package.json scripts as your build tool

(September 1, 2020) To Node.js the package.json simply tracks a few things about a package like its main module. To npm and yarn this file serves more purposes, from package management data, to dependencies lists, to package installation and project management scripts. The scripts tag may have a humble originating purpose, but by overusing it package.json becomes powerful platform for running command-line software processes like building websites or building software products.

Running the TODO application

(August 28, 2020) In this article series we've developed a Todo application using Node.js 14.x, Express, Bootstrap v5, Socket.IO, and using Sequelize for the database layer. It was developed to explore some of the new features in Node.js, and to explore the post-jQuery world that Bootstrap v5 offers.

Using Bootstrap and Socket.IO for the user interface in an Express Node.js 14 application

(August 26, 2020) The Bootstrap project has released the first Alpha for Bootstrap v5. It contains a number of major changes, such as dropping the requirement to use jQuery. Nothing is said about when the first official release will ship, but we can get a taste of what Bootstrap v5 will be like by using the Alpha to build a small application. This article goes over developing a Bootstrap v5 user interface for a TODO application.

Using Sequelize for the database layer in an Express Node.js 14 application

(August 25, 2020) Like many web applications, the TODO application we are developing with Express and Bootstrap v5 must persist its data somewhere. For that purpose we will use Sequelize to to interface with an SQL database (SQLite3). Sequelize is a high level ORM (Object Relations Manager) for Node.js that supports a long list of SQL databases.

Project setup for Node.js 14, Express, Bootstrap v5, Socket.IO application

(August 24, 2020) In this project we'll create a small TODO application using Node.js 14.x, the latest version of Express, Bootstrap v5, Bootstrap Icons, Sequelize for the database, and Socket.IO for multi-user real time interaction. In this posting we'll initialize the project directory, and put together the initial code.

Single page multi-user application with Express, Bootstrap v5, Socket.IO, Sequelize

(August 23, 2020) The Node.js world changed a lot this year with the Node.js 14 release, and its support for ES6 modules, async/await functions, and top-level async code. Another important change was the Bootstrap project which is preparing version 5 where the leading feature is to drop jQuery. These and other changes have come, and are coming, to application developers. This creates a good opportunity to revisit building web applications using Node.js, exploring not only the new features in Node.js but life with Bootstrap without relying on the jQuery crutch. To that end, let's write a little TODO application using Node.js 14, Express, Bootstrap 5, Sequelize, and Socket.IO.

Node.js Script writers: Top-level async/await now available

(August 13, 2020) Async/await functions are a god-send to JavaScript programmers, simplifying writing asynchronous code. It turns difficult-to-write-and-debug pyramids of doom into clean clear code where the results and errors land in the natural place. It's been wonderful except for one thing, we could not use the await keyword in top-level code in Node.js. But, with Node.js 14.8 that's fixed.

Installing some native-code npm packages on Ubuntu 20.04 fails on not finding python command

(April 20, 2020) Node.js 14.x was just released, as was Ubuntu 20.04. Testing my application on that combination failed because native code npm packages fail because python is not found. WTF? I followed the instructions and installed build-essential which supposedly brings in every compiler tool we would need. But ... fortunately there is a simple solution.

Terraform deployment on AWS ECS a multi-tier Node.js and Nginx system using Service Discovery

(April 1, 2020) Amazon's AWS Elastic Container Service (ECS) lets us deploy Docker containers to the AWS cloud. In earlier postings of this series we have hosted a NGINX/Node.js application stack using Docker on our laptop, and on AWS ECS, then we hosted it on AWS ECS using Terraform. In this tutorial we switch to hosting the application using two AWS ECS Service instances, and use AWS Service Discovery between the instances.

Terraform deployment of a simple multi-tier Node.js and Nginx deployment to AWS ECS

(March 29, 2020) Amazon's AWS Elastic Container Service (ECS) lets us deploy Docker containers to the AWS cloud. ECS is a very complex beast to tame, but Terraform offers a way to easily describe infrastructure builds not only AWS but on many other cloud services. Using Terraform is hugely simpler than any tool offered by AWS ECS.

Deploying a simple multi-tier Node.js and Nginx deployment to AWS ECS

(March 24, 2020) Amazon's AWS Elastic Container Service (ECS) lets us deploy Docker containers to the AWS cloud. ECS is a very complex beast to tame, but Amazon offers a method of using Docker Compose to describe an ECS Service. That hugely simplifies the task of launching tasks on AWS ECS.

A simple multi-tier Node.js and Nginx deployment using Docker

(March 22, 2020) In a world of Microservices, our Docker deployments are probably multiple small containers each implementing several services, with an NGINX container interfacing with the web browsers. There are many reasons for doing this, but lets instead focus on the how do we set up NGINX and NodeJS in Docker question. It is relatively easy, and to explore this we'll set up a two-container system with an NGINX container and a simple NodeJS app standing in as a back-end service.

Github buys npm: Might cause more angst about npm as de-facto package manager for Node.js?

(March 16, 2020) From the beginning of Node.js, npm has been a faithful companion providing useful package management service to the Node.js community. Node.js would not have risen so high without a good package manager, and npm served that role. By rights we should celebrate that Github is buying npm since a big question mark about npm has its level of funding. But - it raises a big question mark about the continued independence of the npm registry once it falls into the clutches of a big corporation (Microsoft owns Github).

Bootstrap v5 will drop requirement for jQuery, drop IE10 and maybe drop IE11 support

(March 11, 2020) The Bootstrap is working on the Bootstrap v5 release, and the team recently committed changes to drop support for at least IE10 (and possibly IE11). Judging by pull requests, their plan to drop the requirement to use jQuery has been a long time coming. On the one hand it is heart-warming to see Internet Explorer finally be dropped from support, and on the other hand we recognize the necessity of letting go of tools that have outlived their usefulness. Finally we'll note that the predicted release date for Bootstrap v5 is "late spring 2020".

A popular Node.js package, request, has been deprecated

(February 13, 2020) What do we do when a venerated elder is supplanted by young whipper-snappers? Node.js is barely 10+ years old, and the availability of async/await and other JavaScript features means older packages must either adapt or be replaced. The community around the request package chose the latter, to let Request become deprecated, and to officially recommend folks to use other packages.

Using Dynamic import in Node.js lets us import ES6 modules in CommonJS code, and more - UPDATED

(January 21, 2020) With ES6, JavaScript programmers finally gained a standard module format that worked on both browser-side and server-side (Node.js) JavaScript runtimes. Plus, ES6 modules offer many interesting features. But Node.js programmers need to know how to integrate ES6 modules with the CommonJS module format we're accustomed to using. Now that Node.js v14 is coming, and bringing with it full ES6 module support as a standard feature, it is time to update a previous blog post about Dynamic Import on Node.js. The Dynamic Import feature, a.k.a. the import() function, gives JavaScript programmers much more freedom over the standard import statement, plus it lets us import an ES6 module into CommonJS code.

Resizing images using Node.js image processing tools, without external dependencies or image servers

(September 18, 2019) While getting ready to post a set of images on another blog, I wondered how to most effectively resize the images. They were high resolution product images that came with a press release, and to be nice to my readers it's important to shrink the image size to promote faster downloads. What if I want to supply several image sizes based on the device used by the visitor? It's not enough to resize the image on my own, it would be better to generate several image sizes. Which led to exploring Node.js tools for resizing images, and finding them lacking.

Simplify your npm publish workflow using Github Actions

(September 12, 2019) Github's recently added new feature, Github Actions, promises us a powerful Workflow system to handle a huge range of tasks. Those of us who publish Node.js packages can use Actions to automatically run tests and then publish the package to npm. Let us see how to use Github Actions to simplify our lives.

A deep dive guide to JavaScript Symbols (ES-6 and beyond)

(August 29, 2019)

ES2015 added a ton of useful new features, and among them is the Symbol type. Symbol lets us avoid property name collisions in a way where we can almost implement private properties in JavaScript objects.

In Jenkins, automatically test an npm package, then publish if tests succeed

(August 14, 2019) We want to automate the boring stuff so we have more time for interesting stuff. Clearly running tests and administrivia like publishing a Node.js package to npm, well, that's boring stuff. It is the same tasks every time, and as an unautomated process you might forget a step, so let's look at how to automate this common workflow for Node.js programmers.

Serious security hole found in Electron framework, Electron app's can be easily backdoored

(August 7, 2019) An attacker that can gain access to a machine can easily patch an Electron application to run bad code that does bad things. The problem is Electron does not secure the contents of ASAR files, and they can be easily opened and modified.

How to install NAVE, an alternative Node.js version manager to nvm

(July 25, 2019) Those of us wanting to keep up with the latest Node.js version, or to test our code on alternative Node.js versions, probably have installed nvm. It is an easy-to-use tool for quickly installing the latest Node.js version, or to easily switch between Node.js versions to test our code. But nvm doesn't fit all use cases, and that's where nave comes in. Its purpose is extremely similar to nvm, but it satisfies a slightly different set of use cases that one may find interesting.

How to create a copy-on-write clone of a file in Node.js

(July 25, 2019) It's been refreshing to learn of an advance in Linux/Unix file system semantics over the last 35 years. In 1984 I first used symbolic links, on a Vax running 4.2BSD. That was a significant advance over the hard links that had existed for many years before. The other day I learned that both Mac OS X and Linux support a new kind of link, a reflink, that is a form of copy-on-write for certain file systems.

Complete guide to using TypeORM and TypeScript for data persistence in Node.js module

(July 16, 2019) TypeORM is an advanced object-relations-management module that runs in Node.js. As the name implies, TypeORM is meant to be used with TypeScript. In this article we'll learn about using TypeORM to set up Entity objects to store data in a database, how to use a CustomRepository instance to manipulate a database table, and to use Relations between Entity instances to simulate database joins.

How to use Passport in ExpressJS to handle Node.js app login/logout functionality

(July 15, 2019) Almost every web application requires login/logout functionality. In the Node.js/Express world, the Passport module handles that functionality.

What is Node.js Exactly? - a beginners introduction to Nodejs

(July 12, 2019) Node.js is JavaScript running outside of the web browser. If you're expecting a Window or Document object because JavaScript has that, you'll be disappointed because those objects are side effects of running in a browser. This introduction to Node.js is a very basic intro, showing the bare bones of beginning to understand what Node.js is, and what its role is in the software industry.

Build A Restful Api With Node.js Express & MongoDB

(July 11, 2019) We are going to cover how to create a restful api using node.js express and mongodb together with mongoose. REST apis help us decouple our backend code from our front end so we can use it across multiple application (mobile apps, web apps, etc). We are going to learn how to build a simple blog post type api with all the useful methods(GET,POST,DELETE,PATCH).

The rumors of Express.js's death are greatly exagerrated

(June 26, 2019) Wandering across a question on Quora (Is Express.js dying? What are the alternatives?) had me stop and take a look at the facts. I'd noticed a few months ago that contributions to the main Express repository had dried up, and therefore I was worried the Express project was dying from neglect. So let's talk about this, because it is important to clear up the air.