; Date: Thu Aug 13 2020

Tags: Node.js »»»» Asynchronous Programming

Async/await functions are a god-send to JavaScript programmers, simplifying writing asynchronous code. It turns difficult-to-write-and-debug pyramids of doom into clean clear code where the results and errors land in the natural place. It's been wonderful except for one thing, we could not use the await keyword in top-level code in Node.js. But, with Node.js 14.8 that's fixed.

Writing scripts probably involves reading or writing files, retrieving external data, to work on that data and produce a result, which might be written back to an external file or database. In Node.js, that requires asynchronous code. And in a script, where the code is at the top level of a JavaScript file, that prevented us from using the await keyword.

Consider:

const fs = require('fs').promises;

const txt = await fs.readFile('test.txt', 'utf-8');
console.log(txt);

A script like we just described might do some processing on the input file, then write its output somewhere else. But let's focus on the asynchronous act of reading the data file.

This is a traditional CommonJS style of Node.js program. The programmer said to themselves, let's use await because that is such a useful keyword. The programmer was even careful to use require('fs').promises to get the promisified fs package. But, running the script they see this:

$ nvm use 14.7
Now using node v14.7.0 (npm v6.14.7)
$ node m1.js
/Users/David/nodejs/top-level-async-await/m1.js:8
const txt = await fs.readFile('test.txt', 'utf-8');
            ^^^^^

SyntaxError: await is only valid in async function
    at wrapSafe (internal/modules/cjs/loader.js:1172:16)
    at Module._compile (internal/modules/cjs/loader.js:1220:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1277:10)
    at Module.load (internal/modules/cjs/loader.js:1105:32)
    at Function.Module._load (internal/modules/cjs/loader.js:967:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
    at internal/main/run_main_module.js:17:47

Throughout this post I'll use nvm to manage the Node.js version being used. The importance of choosing the 14.7 release will become clear later.

That's what we've lived with for the last couple years in Node.js. We could use the await keyword inside a function marked with the async keyword, but we could not use it in top-level code. In other words, that script would have to be written as:

const fs = require('fs').promises;

(async () => {
    const txt = await fs.readFile('test.txt', 'utf-8');
    console.log(txt);
})();

This wraps the asynchronous await invocation inside a light-weight async function that is immediately invoked. This is a simple technique that lets us write async/await code almost in the top level of the Node.js script. IMPORTANT NOTE: this simple script is missing some important error handling.

What we really wanted was to write code like the first snippet in this post. This was the closest we could get, until now.

Top-level async/await in Node.js 14.8

Node.js 14.8 was released today, and contains a very important advancement. In an ES6 module we can now use the await keyword in top-level code, without any flags.

That means in a file with the .mjs extension we can write this:

import { promises as fs } from 'fs';

const txt = await fs.readFile('test.txt', 'utf-8');
console.log(txt);

This import statement is how we access the promisified fs module in an ES6 module. Otherwise this is the same as the example at the top.

And we can run it like this:

$ nvm use 14.8
Now using node v14.8.0 (npm v6.14.7)
$ node m1.mjs
Hello, world!

This works as we want it to work, await executing perfectly in the top level.

This is how the same example worked with Node.js 14.7

$ cat test.txt 
Hello, world!
$ nvm use 14.7
Now using node v14.7.0 (npm v6.14.7)
$ node m1.mjs
file:///Volumes/Extra/nodejs/top-level-async-await/m1.mjs:6
const txt = await fs.readFile('test.txt', 'utf-8');
            ^^^^^

SyntaxError: Unexpected reserved word
    at Loader.moduleStrategy (internal/modules/esm/translators.js:122:18)
    at async link (internal/modules/esm/module_job.js:42:21)

That's the significance of the change in Node.js 14.8. We can now use the await keyword in top level Node.js code, giving us natural looking async scripts. We no longer have to face this unpleasant error.

On a final node, this feature only works when Node.js is running an ES6 module. If it is executing a CommonJS module, like the first example:

$ node --version
v14.8.0
$ node m1.js
/Volumes/Extra/nodejs/top-level-async-await/m1.js:8
const txt = await fs.readFile('test.txt', 'utf-8');
            ^^^^^

SyntaxError: await is only valid in async function
    at wrapSafe (internal/modules/cjs/loader.js:1167:16)
    at Module._compile (internal/modules/cjs/loader.js:1215:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1272:10)
    at Module.load (internal/modules/cjs/loader.js:1100:32)
    at Function.Module._load (internal/modules/cjs/loader.js:962:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47

This is the same CommonJS example shown at the top, but executed this time with Node.js 14.8. It's the same error in both cases.

That was nice, we can now use async code in a Node.js script. But let's try a more comprehensive example rather than this artificial simple thing.

Async download and processing of images in a Node.js script

The node-fetch package brings the fetch function to Node.js, making it an excellent way to download a file from somewhere. The download is processed as an async invocation, of course. The sharp package is an excellent tool for manipulating images. Let's use the two together to download and process an image.

On the Node.js website (at https://nodejs.org/en/about/resources/) there is available SVG versions of the standard Node.js logo. It's important to properly use the correct trademarked image. Let's write a little script to download the SVG, resize it to 300 pixels, make it grayscale, and write it to a JPG.

First, in a directory with a package.json:

$ npm install sharp node-fetch --save

This installs the required packages. If you prefer Yarn, use that.

Then we write a file, node-logo.mjs, containing:

import { default as fetch } from 'node-fetch';
import { default as sharp } from 'sharp';
import { default as fs } from 'fs';

(await fetch('https://nodejs.org/static/images/logos/nodejs-new-pantone-black.svg'))
        .body.pipe(
            sharp().resize(300).grayscale()
                .toFormat('jpeg')
        )
        .pipe(fs.createWriteStream('node-logo.jpg'));

The fetch function returns a Promise, and we use await to handle its success or failure. I do not understand why the node-fetch documentation tells us to use a .then handler when we can just use await. Used this way it produces a ReadStream that can be piped somewhere.

In the middle portion of the pipeline we have an image processor implemented with Sharp. We simply tell it to resize the image, make it grayscale, then convert to JPG format.

The final portion of the pipeline uses a WriteStream to send the data to the named file.

With Node.js 14.7 we had to write this inside an async function like was shown earlier. With Node.js 14.8 it simply runs like so:

$ node node-logo.mjs 

And it produces this:

Summary

This advancement will unlock a new ease for writing simple scripts in Node.js. There are close to a zillion different one-off tasks we might want to do with Node.js. For example, I was exploring, just a few days ago, that exact image download/resize/etc pipeline for use with my static website generator (AkashaCMS). The example I wrote just last week had to use the async wrapper function, but today that's no longer required.

This should eliminate one of the Node.js adoption hurdles. Instead of having to turn to Python or other languages for simple scripts like this, we can stay with Node.js.