How to use Node.js features

Asynchronous coding is one of those powerful tools that can bite, if you're not careful. Passing around anonymous so-called "callback" functions is easy, and we do it all the time in JavaScript. Asynchronous callback functions are called, when needed, as needed, sometime in the future. The key result is that code execution is out-of-order with the order-of-appearance in the code.

The Internet relies on domain names as a more user-friendly humane address mechanism than IP addresses. That means we must often write code checking if a URL is "within" domain A or domain B, and to act accordingly. You might think a regular expression is the way to go, but it has failings because while a URL looks like a text string it's actually a data structure. A domain name comparison has to recognize that it's dealing with a data structure, and to compare correctly. Otherwise a URL with domain name "loveamazon.com" might match the regular expression /amazon.com$/i and do the wrong thing.

It's convenient to import a CSV file into a spreadsheet to crunch numbers. CSV's (comma-separated-values) and it's brethren (TSV, for tab-separated, etc) are widely used to deliver information. You'll see CSV downloads available all over the place, whether it's your stock broker, a bank, or a government agency, and on and on. Spreadsheet programs like LibreOffice Calc offer direct import of these files, and then as a spreadsheet you've got a zillion tools available to crunch numbers, make graphs, etc. Sometimes, though, you need to load the CSV into a database to do other sorts of work. That's what we'll look at, the twin tasks of autoloading a CSV file into a MySQL database, and of exporting data from MySQL into a CSV file.

Those of us who consume/edit/modify/publish CSV files must from time to time transform a CSV file. Maybe you need to delete columns, rearrange columns, add columns, rename volumes, or compute some values taking one CSV file and producing another CSV file. Or maybe you need to convert the CSV to a simple text report. You might have a raw CSV file with no column headers, and need to supply column names. Or your CSV might, instead, have header names.

In Node.js the require function is synchronous, meaning you're guaranteed that when require returns the module object is fully populated. But what happens when that doesn't happen? It can happen that the module object is empty, and then what do you do?

JavaScript is an extremely popular programming language deserving of a bigger role than browser-based programming. Fortunately we can use JavaScript outside web browsers, because of several non-browser JavaScript implementations. The most popular, Node.js, is not the only choice for running JavaScript outside the web browser.

Suppose you've found a Node.js module that almost does what you want, but it needs another feature or two, or some other tweak? How to modify the package, and perhaps temporarily modify it at runtime? Technically, monkey patching is not simply maintaining a local modification to someone elses code, but to dynamically modify that code at runtime. Some say this is evil, but it is a practice that is required at times. Let's see how to do this.

Neither Node.js nor ExpressJS have an opinion about the best folder structure. As a non-opinionated application framework, Express doesn't much care how you wish to organize your code. Since the folder organization is up to you, please consider the following as one among many structures that at least one programmer thinks is an efficient way to organize Express application code. The structure here more-or-less follows the Model-View-Controller paradigm, meaning we have separated the code for data models and data persistence from the business logic from the view code.

The long-awaited async/await feature for JavaScript promises to make our lives much easier. Instead of asynchronous JavaScript programming being tricky and error-prone, with async/await our code will look like regular synchronous code we write in other languages, but will accommodate asynchronous code execution while retaining the single-threaded nature of JavaScript. With this feature we declare async functions, and within their boundary magic occurs - we can put the "await" keyword in front of a function which produces a Promise, and automagically JavaScript will wait for the Promise to resolve and give us the result or else throw the error. What follows is a list of posts describing how to use async/await.

With today's first release of Node.js version 7, we now have async/await as a base feature of the platform. This is a significant milestone that wasn't even mentioned in the commit message. The async/await feature is so important that I'm preparing a short book to discuss JavaScript asynchronous programming patterns, that highly features async/await.

I've happily used the async module for Node.js for years to simplify asynchronous operations over arrays. This is a difficulty in the Node.js paradigm, since the "normal" way to process an array is with a simple for loop. But if the operation is asynchronous the for loop has no way of knowing when to iterate to the next item, or even when processing is finished. Instead the async.eachSeries function does the trick, because your code tells async.eachSeries when to go to the next item, when there's an error, and it knows when the last item is processed, etc. It's been great.

Node.js v 4.0.0 was just released. This is a long-awaited release representing the healing of the schism in the Node.js community created when the io.js project forked Node.js. That had been done over a disagreement about the policies and maintainership of Node.js. Joyent had been in control of the project (because Ryan Dahl had been employed by Joyent) and not all in the community liked the decisions made by Joyent. They instead forked the project to create io.js, and implemented a bunch of useful ideas for example to adopt the latest-and-greatest version of the V8 JavaScript engine in order to get all the modern ES6 goodies.

Someone experienced with using Wordpress, or Drupal, to build websites are accustomed to "plugins" or "modules". Both are software modules which extend the functionality of Wordpress or Drupal websites, giving more features to the system than what's available out of the box. For example it's typical for a Drupal site to install Views to support building fancy data displays, and on my Wordpress blog I've added the PODS Framework to simplify defining custom post types along with custom fields.

The Express.js app framework for Node.js is a pretty cool system that makes it easy to implement web apps and even REST API's. But the Express team doesn't give you any guidance on structuring the application code. They give you an API and it's up to you to decide how or even if you structure the model-view-controller paradigm or any other paradigm you wish to use.

When your vows.js based tests for a Node.js application says "Errored » callback not fired" -- well, it can be very confusing. In my case the code clearly handled all paths ensuring the Vows callback would be called. No matter how many tweaks I performed to try and catch some possible error in test or code, I couldn't figure out what caused this problem. But after some yahoogling, the answer was not only difficult to find, but surprisingly simple.

Node.js is horrible with CPU bound processing, supposedly. Why? Because CPU-intensive algorithms block the event loop from handling events, blocking the Node.js platform from doing its core competency. Actually, as I demonstrate in my book Node Web Development (see sidebar for link), it's possible to use "setImmediate" to dispatch work through the Node.js event loop, and perform intensive computation while not blocking the event loop. The example I chose for the book is a simplistic Fibonacci algorithm, and requesting "large" Fibonacci values (like "50") would take a loooong time. But, by recoding the Fibonacci algorithm using setImmediate, it can do calculation for any Fibonacci value without blocking the event loop.

When we write unit tests it's good practice to "mock" out extraneous bits to the code being tested. It's almost like the scientific method in that testing, in the unit testing paradigm, means exercising each small portion of your code in isolation if only to eliminate unwanted variables. While there are other testing paradigms, unit testing has its value. A big question for Node.js web application programmers is - how do you mock out HTTP requests for unit testing? In other words, how do you test the route handler method in isolation from the Node.js HTTPServer object, or Express, or whatever app framework used in your application?

Often we want to write data to a file, don't really care what the file name is, but want to be sure our code is the only code accessing that data.  A trivial algorithm might be to form a file name like:

Express makes it easy to route parameterized URL's to a handler function, and easily grab data out of the request URL.  This makes it easy to build a web application, that puts data into the URL, with clean URL's out of the box.  That's so much nicer than other application frameworks that require abominations like mod_rewrite to get clean URL's.  However nice the parameterized URL's are in Express, they do not handle other parts of the URL such as query parameters (the part of the URL after the '?').

I've just added HTML/CSS/JavaScript minification to AkashaCMS (link is external), to minimize the size of static website files rendered by AkashaCMS. This will reduce page load times by decreasing the number of bytes required to be downloaded. We humans like blank space because spreading out information makes it easier to understand. With programming languages that becomes all the white space and indenting conventions. But the white space makes no difference to the computers that interpret the web pages, in fact it makes the web browsing experience slower because it takes longer to download or execute the web pages, or the CSS or JavaScript referenced by the web pages.

How should Node.js programs best do logging?  When that question came up in the node users mailing list, Tim Caswell answered to just use console.log, and if you want to send the log to somewhere other than stdout then to override the function.  That makes the Java programmer recoil with "eeew, overriding a system function is wrong" but hey we're in a JavaScript environment.  This sort of thing is par for the course in JavaScript.  So, how do you go about doing this?

I'm pondering a new photography workflow where, instead of working with the JPG's directly out of the camera, that I instead convert them to PNG first.  The reasoning is that JPG is a lossy image format meaning every time you edit a JPG it loses a bit of precision, whereas PNG is a lossless image format and you don't lose precision.  That idea might not be the best, I haven't checked how well Picasa works with PNG's, in any case it let me play with what kind of image manipulation we can do in Node.js.