How to use Node.js features

Troubles with Asynchronous code flows in JavaScript, and the async/await solution of ES-2017

(Wed Oct 27 2021 00:00:00 GMT+0300 (Eastern European Summer Time))

Asynchronous coding is one of those powerful tools that can bite, if you're not careful. Passing around anonymous so-called "callback" functions is easy, and we do it all the time in JavaScript. Asynchronous callback functions are called, when needed, as needed, sometime in the future. The key result is that code execution is out-of-order with the order-of-appearance in the code.

Correctly match URL against domain name without killing yourself with regular expressions

(Tue Oct 26 2021 00:00:00 GMT+0300 (Eastern European Summer Time))

The Internet relies on domain names as a more user-friendly humane address mechanism than IP addresses. That means we must often write code checking if a URL is "within" domain A or domain B, and to act accordingly. You might think a regular expression is the way to go, but it has failings because while a URL looks like a text string it's actually a data structure. A domain name comparison has to recognize that it's dealing with a data structure, and to compare correctly. Otherwise a URL with domain name "" might match the regular expression /$/i and do the wrong thing.

Simplify catching uncaughtException, unhandledRejection, and multipleResolves in Node.js

(Fri Oct 15 2021 00:00:00 GMT+0300 (Eastern European Summer Time)) When native Promise and async/await support was added to Node.js, the process object gained some error events to notify the program about un-handled or un-caught error conditions. It's important to handle such errors, but of course we sometimes neglect to set up error handlers for every possible error condition. The primary purpose for these event handlers is so we know about program errors that we failed to catch. Node.js provides default error handlers which leave a bit to be desired. The log-process-errors assists with these issues.

Easily export or import data between CSV files and SQLite3 or MySQL databases with Node.js

(Sat Feb 27 2021 00:00:00 GMT+0200 (Eastern European Standard Time))

It's convenient to import a CSV file into a spreadsheet to crunch numbers. CSV's (comma-separated-values) and it's brethren (TSV, for tab-separated, etc) are widely used to deliver information. You'll see CSV downloads available all over the place, whether it's your stock broker, a bank, or a government agency, and on and on. Spreadsheet programs like LibreOffice Calc offer direct import of these files, and then as a spreadsheet you've got a zillion tools available to crunch numbers, make graphs, etc. Sometimes, though, you need to load the CSV into a database to do other sorts of work. That's what we'll look at, the twin tasks of autoloading a CSV file into a MySQL database, and of exporting data from MySQL into a CSV file.

Memory-efficient CSV transformation into CSV or text reports in Node.js

(Fri Feb 26 2021 00:00:00 GMT+0200 (Eastern European Standard Time))

Those of us who consume/edit/modify/publish CSV files must from time to time transform a CSV file. Maybe you need to delete columns, rearrange columns, add columns, rename volumes, or compute some values taking one CSV file and producing another CSV file. Or maybe you need to convert the CSV to a simple text report. You might have a raw CSV file with no column headers, and need to supply column names. Or your CSV might, instead, have header names.

How to create a copy-on-write clone of a file in Node.js

(Thu Jul 25 2019 00:00:00 GMT+0300 (Eastern European Summer Time)) It's been refreshing to learn of an advance in Linux/Unix file system semantics over the last 35 years. In 1984 I first used symbolic links, on a Vax running 4.2BSD. That was a significant advance over the hard links that had existed for many years before. The other day I learned that both Mac OS X and Linux support a new kind of link, a reflink, that is a form of copy-on-write for certain file systems.

Build A Restful Api With Node.js Express & MongoDB

(Thu Jul 11 2019 00:00:00 GMT+0300 (Eastern European Summer Time)) We are going to cover how to create a restful api using node.js express and mongodb together with mongoose. REST apis help us decouple our backend code from our front end so we can use it across multiple application (mobile apps, web apps, etc). We are going to learn how to build a simple blog post type api with all the useful methods(GET,POST,DELETE,PATCH).

How to avoid vulnerability to malicious modules distributed through npm repository

(Tue Jun 18 2019 00:00:00 GMT+0300 (Eastern European Summer Time)) Recently a pair of modules in the npm repository became the vector for distributing malicious code. As much as npm Inc is working to improve security in the npm repository, there are steps we all must take as Node.js software coders to reduce the vulnerability of our packages. While its unlikely, in theory malicious code could be introduced through any package and how would we notice? It's not enough to trust npm Inc, we have our own work to do to strengthen the modules we publish.

How to correctly calculate path to a Node.js module to access files in that module directory

(Thu May 09 2019 00:00:00 GMT+0300 (Eastern European Summer Time)) Many Node.js modules contain files meant to be accessed as data. For example the Bootstrap, jQuery and Popper.js modules do not contain Node.js code, but exist to distribute the libraries. We are supposed to configure Express (or other framework) to serve the files in those directories to a web browser. The question is -- what is the best way to calculate the filesystem path of a file installed as a Node.js module inside the node_modules hierarchy?

How to fix Node.js require returning an empty module object

(Fri Jan 25 2019 00:00:00 GMT+0200 (Eastern European Standard Time))

In Node.js the require function is synchronous, meaning you're guaranteed that when require returns the module object is fully populated. But what happens when that doesn't happen? It can happen that the module object is empty, and then what do you do?

Fix Webpack's ability to load CSS while packaging an app

(Thu Aug 02 2018 00:00:00 GMT+0300 (Eastern European Summer Time)) Web applications obviously will need CSS, and therefore Webpack must support packaging applications with CSS files. However with the sample Webpack build procedure I'm using, it fails while loading a CSS file. Let us take a look at the configuration changes required to fix this problem.

Fix Webpack's ability to package an app using the mime module

(Thu Aug 02 2018 00:00:00 GMT+0300 (Eastern European Summer Time)) Application development with Vue.js, Electron, and other frameworks, requires using Webpack. Being a Webpack newbie I'm not sure what it does, but it is required for the tasks I'm doing right now. One of which failed with a strange error message while trying to build a Vue.js component which used the mime module. Namely, we have to instruct Webpack to recognize the JSON files included in the mime module.

How to run JavaScript in the command-line environment

(Fri Jun 15 2018 00:00:00 GMT+0300 (Eastern European Summer Time))

JavaScript is an extremely popular programming language deserving of a bigger role than browser-based programming. Fortunately we can use JavaScript outside web browsers, because of several non-browser JavaScript implementations. The most popular, Node.js, is not the only choice for running JavaScript outside the web browser.

How to deploy Express applications to AWS Lambda

(Sat May 19 2018 00:00:00 GMT+0300 (Eastern European Summer Time)) AWS Lambda and the AWS API Gateway is a powerful combination for deploying Node.js application code on an automagically scaling deployment platform. Out of the box you do not have the Express API. This webinar shows how to setup API Gateway and Lambda to support Express applications.

Use Bash-like command-line environment variables on Windows in Node.js npm scripts

(Sun Apr 15 2018 00:00:00 GMT+0300 (Eastern European Summer Time)) It is best to automate all things, and in Node.js applications a convenient tool is the scripts section of package.json. Using the scripts section, npm helps you to easily record the necessary commands to build or maintain an application. However, npm does nothing to help with cross platform command-line compatibility, because it simply hands the command string to Bash (on Linux/macOS/etc) or to CMD.EXE on Windows. For the most part this is okay, but what if you need to set environment variables? The Bash way of setting command variables gives an error message on Windows.

Fix NVM_NODEJS_ORG_MIRROR deprecation warning when running npm install

(Tue Apr 10 2018 00:00:00 GMT+0300 (Eastern European Summer Time)) The nvm script is an exceedingly useful tool for maintaining multiple Node.js versions while keeping your sanity. Running "npm install", I noticed a strange warning message from node-gyp about "NVM_NODEJS_ORG_MIRROR" being deprecated. Looking for the cause, I found a cause centered on nvm. Let's see how to fis this problem.

Can I monkey patch a Node.js module installed from the npm repository so the patch is maintained after npm install?

(Sat Apr 07 2018 00:00:00 GMT+0300 (Eastern European Summer Time))

Suppose you've found a Node.js module that almost does what you want, but it needs another feature or two, or some other tweak? How to modify the package, and perhaps temporarily modify it at runtime? Technically, monkey patching is not simply maintaining a local modification to someone elses code, but to dynamically modify that code at runtime. Some say this is evil, but it is a practice that is required at times. Let's see how to do this.

Using SSL to connect to MySQL database in Node.js

(Wed Mar 07 2018 00:00:00 GMT+0200 (Eastern European Standard Time)) Encrypting database connections can be extremely important for security. The documentation for the Node.js MySQL driver briefly mentions SSL support, and does not give adequate documentation. What follows is an example showing how to connect using PEM certificates to a MySQL server that was configured with a self-signed root CA.

A recommended Node.js/Express application folder structure

(Sun Dec 10 2017 00:00:00 GMT+0200 (Eastern European Standard Time))

Neither Node.js nor ExpressJS have an opinion about the best folder structure. As a non-opinionated application framework, Express doesn't much care how you wish to organize your code. Since the folder organization is up to you, please consider the following as one among many structures that at least one programmer thinks is an efficient way to organize Express application code. The structure here more-or-less follows the Model-View-Controller paradigm, meaning we have separated the code for data models and data persistence from the business logic from the view code.

In JavaScript (Node.js), how do I read a text file from a different directory and store into a string?

(Sun Jun 18 2017 00:00:00 GMT+0300 (Eastern European Summer Time)) Built-in to Node.js are a few packages providing core services, including the "fs" package. As the name implies, it interfaces with the file-system, and among its functions is readFile that reads a file completely into memory. So, the short answer to this question is to simply call fs.readFile but we should take a longer look than that.

Useful reading to understand the Promises, Generators and the async/await feature for Node.js/JavaScript

(Sun Nov 20 2016 00:00:00 GMT+0200 (Eastern European Standard Time))

The long-awaited async/await feature for JavaScript promises to make our lives much easier. Instead of asynchronous JavaScript programming being tricky and error-prone, with async/await our code will look like regular synchronous code we write in other languages, but will accommodate asynchronous code execution while retaining the single-threaded nature of JavaScript. With this feature we declare async functions, and within their boundary magic occurs - we can put the "await" keyword in front of a function which produces a Promise, and automagically JavaScript will wait for the Promise to resolve and give us the result or else throw the error. What follows is a list of posts describing how to use async/await.

The advent of async/await for Node.js - Node.js v7 has now arrived

(Tue Oct 25 2016 00:00:00 GMT+0300 (Eastern European Summer Time))

With today's first release of Node.js version 7, we now have async/await as a base feature of the platform. This is a significant milestone that wasn't even mentioned in the commit message. The async/await feature is so important that I'm preparing a short book to discuss JavaScript asynchronous programming patterns, that highly features async/await.

Fixing "Maximum call stack size exceeded" in async Node.js code

(Fri Sep 09 2016 00:00:00 GMT+0300 (Eastern European Summer Time))

I've happily used the async module for Node.js for years to simplify asynchronous operations over arrays. This is a difficulty in the Node.js paradigm, since the "normal" way to process an array is with a simple for loop. But if the operation is asynchronous the for loop has no way of knowing when to iterate to the next item, or even when processing is finished. Instead the async.eachSeries function does the trick, because your code tells async.eachSeries when to go to the next item, when there's an error, and it knows when the last item is processed, etc. It's been great.

Node.js 4.0.0 is out - quick tip for use while testing compatibility

(Wed Sep 09 2015 00:00:00 GMT+0300 (Eastern European Summer Time))

Node.js v 4.0.0 was just released. This is a long-awaited release representing the healing of the schism in the Node.js community created when the io.js project forked Node.js. That had been done over a disagreement about the policies and maintainership of Node.js. Joyent had been in control of the project (because Ryan Dahl had been employed by Joyent) and not all in the community liked the decisions made by Joyent. They instead forked the project to create io.js, and implemented a bunch of useful ideas for example to adopt the latest-and-greatest version of the V8 JavaScript engine in order to get all the modern ES6 goodies.

The difference between Node.js require, and Wordpress plugins or Drupal modules

(Sun Jul 19 2015 00:00:00 GMT+0300 (Eastern European Summer Time))

Someone experienced with using Wordpress, or Drupal, to build websites are accustomed to "plugins" or "modules". Both are software modules which extend the functionality of Wordpress or Drupal websites, giving more features to the system than what's available out of the box. For example it's typical for a Drupal site to install Views to support building fancy data displays, and on ( my Wordpress blog I've added the PODS Framework to simplify defining custom post types along with custom fields.

Where should you put routes in an Express/Node.js web application for simple easy coding enjoyment?

(Tue Jun 09 2015 00:00:00 GMT+0300 (Eastern European Summer Time))

The Express.js app framework for Node.js is a pretty cool system that makes it easy to implement web apps and even REST API's. But the Express team doesn't give you any guidance on structuring the application code. They give you an API and it's up to you to decide how or even if you structure the model-view-controller paradigm or any other paradigm you wish to use.

Don't rip your hair out when Vows tells you "Errored callback not fired" - try this one weird trick instead

(Wed Jan 07 2015 00:00:00 GMT+0200 (Eastern European Standard Time))

When your vows.js based tests for a Node.js application says "Errored » callback not fired" -- well, it can be very confusing. In my case the code clearly handled all paths ensuring the Vows callback would be called. No matter how many tweaks I performed to try and catch some possible error in test or code, I couldn't figure out what caused this problem. But after some yahoogling, the answer was not only difficult to find, but surprisingly simple.

Easily offload your CPU-intensive Node.js code with simple Express-based REST server

(Wed Jul 02 2014 00:00:00 GMT+0300 (Eastern European Summer Time))

Node.js is horrible with CPU bound processing, supposedly. Why? Because CPU-intensive algorithms block the event loop from handling events, blocking the Node.js platform from doing its core competency. Actually, as I demonstrate in my book Node Web Development (see sidebar for link), it's possible to use "setImmediate" to dispatch work through the Node.js event loop, and perform intensive computation while not blocking the event loop. The example I chose for the book is a simplistic Fibonacci algorithm, and requesting "large" Fibonacci values (like "50") would take a loooong time. But, by recoding the Fibonacci algorithm using setImmediate, it can do calculation for any Fibonacci value without blocking the event loop.

Unit testing Express route handlers in isolation from everything, including Express

(Sat Jun 07 2014 00:00:00 GMT+0300 (Eastern European Summer Time))

When we write unit tests it's good practice to "mock" out extraneous bits to the code being tested. It's almost like the scientific method in that testing, in the unit testing paradigm, means exercising each small portion of your code in isolation if only to eliminate unwanted variables. While there are other testing paradigms, unit testing has its value. A big question for Node.js web application programmers is - how do you mock out HTTP requests for unit testing? In other words, how do you test the route handler method in isolation from the Node.js HTTPServer object, or Express, or whatever app framework used in your application?

Managing Node.js servers on Mac OS X with forever - works best for development

(Fri Jan 03 2014 00:00:00 GMT+0200 (Eastern European Standard Time)) If, like me, you're doing Node.js development on a Mac, you might have a yearning for a tool like MAMP but which works for Node.  A couple weeks ago I wrote a blog post covering the first step, ( setting up a Node and npm instance on your computer.   If you don't know what MAMP is, go read that blog post, and then come back here.  What I want to go over today is a way to manage/monitor one or more Node processes on your computer.

How to generate unique temporary file names in Node

(Tue Dec 17 2013 00:00:00 GMT+0200 (Eastern European Standard Time))

Often we want to write data to a file, don't really care what the file name is, but want to be sure our code is the only code accessing that data.  A trivial algorithm might be to form a file name like:

How to get URL params after '?' in Express.js or Node.js?

(Sun Jun 09 2013 00:00:00 GMT+0300 (Eastern European Summer Time))

Express makes it easy to route parameterized URL's to a handler function, and easily grab data out of the request URL.  This makes it easy to build a web application, that puts data into the URL, with clean URL's out of the box.  That's so much nicer than other application frameworks that require abominations like mod_rewrite to get clean URL's.  However nice the parameterized URL's are in Express, they do not handle other parts of the URL such as query parameters (the part of the URL after the '?').

Crunching/minifying HTML, CSS and JavaScript in Node.js with the minify module

(Tue Jan 01 2013 00:00:00 GMT+0200 (Eastern European Standard Time))

I've just added HTML/CSS/JavaScript minification to AkashaCMS (link is external), to minimize the size of static website files rendered by AkashaCMS. This will reduce page load times by decreasing the number of bytes required to be downloaded. We humans like blank space because spreading out information makes it easier to understand. With programming languages that becomes all the white space and indenting conventions. But the white space makes no difference to the computers that interpret the web pages, in fact it makes the web browsing experience slower because it takes longer to download or execute the web pages, or the CSS or JavaScript referenced by the web pages.

Overriding console.log in Node.js, and other thoughts about logging in Node apps

(Mon Aug 06 2012 00:00:00 GMT+0300 (Eastern European Summer Time))

How should Node.js programs best do logging?  When that question came up in the ( node users mailing list, Tim Caswell answered to just use console.log, and if you want to send the log to somewhere other than stdout then to override the function.  That makes the Java programmer recoil with "eeew, overriding a system function is wrong" but hey we're in a JavaScript environment.  This sort of thing is par for the course in JavaScript.  So, how do you go about doing this?

Simple data export and manipulation using Node.js and the node-mysql module

(Wed Jul 18 2012 00:00:00 GMT+0300 (Eastern European Summer Time)) What follows is a fairly simple data processing script to extract data from a Drupal installation in a MySQL database. The purpose was to export nodes from a Drupal site, reformatting them for some software I'd written to import content into a blogger blog. In other words, this is the sort of simple data extraction and manipulation tool we write all the time.

Convert JPG's into PNG's using Node.js and ImageMagick - Possible Photo workflow?

(Sun Oct 30 2011 00:00:00 GMT+0300 (Eastern European Summer Time))

I'm pondering a new photography workflow where, instead of working with the JPG's directly out of the camera, that I instead convert them to PNG first.  The reasoning is that JPG is a lossy image format meaning every time you edit a JPG it loses a bit of precision, whereas PNG is a lossless image format and you don't lose precision.  That idea might not be the best, I haven't checked how well Picasa works with PNG's, in any case it let me play with what kind of image manipulation we can do in Node.js.