; Date: Tue Jan 21 2020

Tags: Node.JS »»»» JavaScript

With ES6, JavaScript programmers finally gained a standard module format that worked on both browser-side and server-side (Node.js) JavaScript runtimes. Plus, ES6 modules offer many interesting features. But Node.js programmers need to know how to integrate ES6 modules with the CommonJS module format we're accustomed to using. Now that Node.js v14 is coming, and bringing with it full ES6 module support as a standard feature, it is time to update a previous blog post about Dynamic Import on Node.js. The Dynamic Import feature, a.k.a. the import() function, gives JavaScript programmers much more freedom over the standard import statement, plus it lets us import an ES6 module into CommonJS code.

This is an update to a nearly two-year-old blog post going over an early version of the Dynamic Import feature. For the old version, see: Dynamic import lands in Node.js, we can import ES6 modules in CommonJS code

The issue is - how do we access a module from another module? Traditionally Node.js has used the CommonJS module format, since it was the mechanism that Ryan Dahl found for use in Node.js at the time he developed the platform. The CommonJS module specification gave us the require statement and the module.exports object we've been using from the beginning of Node.js.

While CommonJS and ES6 modules are conceptually very similar, they are incompatible in many practical ways. Where CommonJS uses require to load a module, ES6 modules use the import statement. The syntax of ES6 modules is very different, as is the loading process. What this means is:

• A CommonJS module can load other CommonJS modules using the require statement
• Because require is actually a function call, we can use a dynamically computed string to determine at runtime the module to be loaded
• An ES6 module can load either CommonJS or ES6 modules using the import statement
• The import statement takes a static string as the module identifier, and cannot dynamically compute a module name

Until one of the Node.js 13.x releases ES6 modules were an experimental feature requiring the use of a command-line flag to enable. As it stands at Node.js 13.7, using an ES6 module still prints a warning message about an experimental feature, but it can be fully used without being enabled by a command-line flag.

The Dynamic Import feature adds two important capabilities

• Dynamically determining the module to load at run time - because import() is a function just like require()
• Loading an ES6 module into a Node.js/CommonJS module

In this post we will use the file extension .mjs to mark ES6 modules, and .js for CommonJS modules. Node.js directly supports this combination, but the defaults can be changed using various configuration options.

Our goal is exploring how to use import() in both ES6 and CommonJS modules. The topics covered include:

• Using an ES6 module from another ES6 module
• Using an ES6 module from a CommonJS module the right way
• Failing use case in global scope - asynchronous loading
• For a pair of modules w/ the same API, dynamically loading either at runtime

Using an ES6 module with either import or import()

An ES6 module can be imported either with the import statement, in an ES6 module, or via the import() function in either an ES6 or CommonJS module. Let's start with the normal case, an ES6 module loading an ES6 module.

Let's create a simple ES6 module, calling it simple.mjs:

var count = 0;
export function next() { return ++count; }
function squared() { return Math.pow(count, 2); }
export default function() { return count; }
export { squared };

This is a simple counter where the count starts at zero, and is incremented by calling next. The default export returns the current value, and we can get the square of the current value using squared. The functionality doesn't matter a lot, since we just want to demonstrate using an ES6 module.

Using this from another ES6 module is easy. Create a file, demo-simple-1.mjs, containing:

import * as simple from './simple.mjs';

console.log(`${simple.next()} ${simple.squared()}`);
console.log(`${simple.next()} ${simple.squared()}`);
console.log(`${simple.default()} ${simple.squared()}`);

We import the module, and make a few calls while printing out results. Obviously another way to do the import is this:

import { next, squared, default as current } from './simple.mjs';

But either way doesn't make any difference, since the two are equivalent. To run the demo:

$ node demo-simple-1.mjs 
(node:41714) ExperimentalWarning: The ESM module loader is experimental.
1 1
2 4
2 4

We're still warned that this is an experimental feature, but at least we do not have to specify a flag any longer.

Loading an ES6 module in a CommonJS module

Suppose we have a hybrid scenario where some of our code is CommonJS, and some of it is ES6 modules. While that is a suboptimal scenario, in this transitionary phase we may be faced with converting an application piece by piece and therefore be faced with using an ES6 module from a CommonJS module.

Having the import() function in CommonJS modules allows us to use ES6 modules. But it comes with a big caveat, and that is that both import and import() are asynchronous operations. If you think about it, require() is a synchronous operation since it does not return until the module fully loads. However import() returns a Promise, and some time in the future either the module will load or an error will be thrown.

This means - to use an ES6 module means something like this - save it as cjs-import-1.js:

(async () => {
    const simple = await import('./simple.mjs');

    console.log(`${simple.next()} ${simple.squared()}`);
    console.log(`${simple.next()} ${simple.squared()}`);
    console.log(`${simple.default()} ${simple.squared()}`);
})().catch(err => console.error(err));

Here we have an inline async function in the global scope. We use the await keyword to wait for the module to load, before using it. Since async functions return a Promise, we have to use a .catch to capture any possible error and print it out.

It is run as so:

$ node cjs-import-1.js 
(node:41882) ExperimentalWarning: The ESM module loader is experimental.
1 1
2 4
2 4

The output is as expected, and notice that we use a CommonJS module this time.

Also - the exact same syntax works exactly as it is as an ES6 module. That is, the import() function works exactly the same in an ES6 context. To prove this let's copy the demo changing the file name extension to .mjs, so that Node.js interprets it as an ES6 module, and then rerun it:

$ cp cjs-import-1.js cjs-import-mjs.mjs
$ node cjs-import-mjs.mjs 
(node:42229) ExperimentalWarning: The ESM module loader is experimental.
1 1
2 4
2 4

This is the exact same source code. Because Node.js treats files with the .mjs extension as ES6 modules, changing the file name means Node.js is interpreting it differently. As was promised, import() works in an ES6 module.

A variant is - save it as cjs-import-2.js:

import('./simple.mjs')
.then(simple => {
    console.log(`${simple.next()} ${simple.squared()}`);
    console.log(`${simple.next()} ${simple.squared()}`);
    console.log(`${simple.default()} ${simple.squared()}`);
})
.catch(err => {
    console.error(err);
});

Instead of using an async function to handle the Promise returned by import() we handle it using .then and .catch.

$ node cjs-import-2.js 
(node:42004) ExperimentalWarning: The ESM module loader is experimental.
1 1
2 4
2 4

Execution is the same, again using a CommonJS module.

The import() function is asynchronous, and in Node.js' current features that makes it difficult to use an ES6 module as a global scope module identifier.

This is demonstrated by this failure mode - save it as cjs-import-fail-1.js

const simple = await import('./simple.mjs');

console.log(`${simple.next()} ${simple.squared()}`);
console.log(`${simple.next()} ${simple.squared()}`);
console.log(`${simple.default()} ${simple.squared()}`);

The await keyword cannot be used outside of an async function, currently. Therefore running it will fail:

$ node cjs-import-fail-1.js 
/home/david/t/cjs-import-fail-1.js:2
const simple = await import('./simple.mjs');
               ^^^^^

SyntaxError: await is only valid in async function
    at wrapSafe (internal/modules/cjs/loader.js:1067:16)
    at Module._compile (internal/modules/cjs/loader.js:1115:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    at Module.load (internal/modules/cjs/loader.js:1000:32)
    at Function.Module._load (internal/modules/cjs/loader.js:899:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

And - what if we leave out the await keyword? - Save this as cjs-import-fail-2.js

const simple = import('./simple.mjs');
console.log(simple);

console.log(`${simple.next()} ${simple.squared()}`);
console.log(`${simple.next()} ${simple.squared()}`);
console.log(`${simple.default()} ${simple.squared()}`);

We've added a console.log to see the value returned by import().

$ node cjs-import-fail-2.js 
Promise { <pending> }
/Volumes/Extra/ws/techsparx.com/t/cjs-import-fail-2.js:5
console.log(`${simple.next()} ${simple.squared()}`);
                      ^

TypeError: simple.next is not a function
    at Object.<anonymous> (/Volumes/Extra/ws/techsparx.com/t/cjs-import-fail-2.js:5:23)
    at Module._compile (internal/modules/cjs/loader.js:1151:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    at Module.load (internal/modules/cjs/loader.js:1000:32)
    at Function.Module._load (internal/modules/cjs/loader.js:899:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

We see that indeed import() gives us a Promise. As a result, there is no next function on the Promise object, and therefore it fails.

What we've learned so far

It should be no surprise that we can use an ES6 module from an ES6 module using import.

As promised an ES6 module can be used in an ES6 module using import() as well.

As promised an ES6 module can be used in a CommonJS module using import().

When using import() we have no choice but for the loaded module object to land inside a function. The module object cannot land directly in a global scope variable in a CommonJS module, because of the inability to use await in the global scope. Either the module object lands in the .then of a Promise chain, or it is a result obtained via the await keyword inside an async function.

The module object can be assigned to a global scope variable, and therefore be useful to other functions in the module. However, those other functions will have to accommodate two states for that variable, one state where the module object has not finished loading, and the other state when it has finished loading.

Dynamically computing the ES6 module to load at runtime

The primary difference between import and import() is the latter lets us compute the module identifier string. Does this seem like a small difference? Ah, but it is a major difference, and gives us an incredible area of freedom in designing Node.js applications.

Consider an internal API of some kind where you have multiple implementations. For example you might want to store/retrieve files from different cloud-based file sharing services. It will simplify your code to have multiple driver modules, one for each service, all of which have the same API.

In my book, Node.js Web Development, I show a series of modules for storing the same object in several different database systems. The modules all support the same API, but under the covers one module uses SQL commands, another uses Sequelize commands, and another uses MongoDB commands.

With CommonJS modules we can compute the module identifier like so:

const api = require('./api-${process.env.VERSION}.js');

And this just works, no fuss, no muss. But as we saw earlier some care must be taken when using import().

Let's implement a fantastic API that is sure to revolutionize the world -- save it as api-1.mjs:

export function apiFunc1() { console.log(`Function 1`); }
export function apiFunc2() { console.log(`Function 2`); }
export function apiFunc3() { console.log(`Function 3`); }

Then because we need the same API to run against a different service, we implement this - save it as api-2.mjs:

export function apiFunc1() { console.log(`Function 1 - ALTERNATE`); }
export function apiFunc2() { console.log(`Function 2 - ALTERNATE`); }
export function apiFunc3() { console.log(`Function 3 - ALTERNATE`); }

We've got two modules, each exporting the same function names with the same signatures. Take our word for it, please, that this is two implementations of the same API.

To consume one or the other, create api-consume.js containing:

var api;

const loadAPI = async () => {
    if (api) return api;
    api = await import(`./api-${process.env.VERSION}.mjs`);
    return api;
}

(async () => {
    await loadAPI();
    api.apiFunc1();
    api.apiFunc2();
    api.apiFunc3();
})().catch(err => console.error(err));

This is only one of a number of ways to do this. If you need a more concrete example, in an Express app.mjs file (the main of an Express application) we could do:

An alternate implementation is this:

(async () => {
    (await loadAPI()).apiFunc1();
    (await loadAPI()).apiFunc2();
    (await loadAPI()).apiFunc3();
})().catch(err => console.error(err));

These are equivalent, but the 2nd is more succinct. What's happening here is that because loadAPI is an async function, we have to await it before calling any of the functions. In the previous example our code would have to know it had already await'ed and therefore it can use api rather than await loadAPI().

A similar approach is this, which avoids having to call a function but instead deal directly with an object api.

var _api;
export { _api as api };
import(`./api-${process.env.VERSION}.mjs`)
.then(loaded => { _api = loaded; })
.catch(err => { HALT FAIL PRINT ERRORS AND EXIT });

Then any code using that API would use import { api } from 'app.mjs'; to use the dynamically selected module.

UPDATE As pointed out in the comments, this has an issue. The object, api, would have three states: a) undefined, b) an unresolved Promise, c) the loaded module. Hence any code wishing to use api would have to deal primarily with the unresolved Promise. Hence, code using api should use (await api).apiFunc1() to wait for the module to finish loading before executing a function from the module.

import { api } from './that-example-script.mjs';
(async () => {
    (await api).apiFunc1();
    (await api).apiFunc2();
    (await api).apiFunc3();
})().catch(err => console.error(err));

Which is preferable? This may be preferable since it doesn't look like a function call and you have fewer doubts about the performance impacts. But it should be roughly the same performance impact as for the loadAPI function.

/UPDATE

In any case, let's run the demo script to show our point:

$ VERSION=1 node api-consume.js 
(node:42502) ExperimentalWarning: The ESM module loader is experimental.
Function 1
Function 2
Function 3
$ VERSION=2 node api-consume.js 
(node:42507) ExperimentalWarning: The ESM module loader is experimental.
Function 1 - ALTERNATE
Function 2 - ALTERNATE
Function 3 - ALTERNATE
$ cp api-consume.js api-consume-mjs.mjs 
$ VERSION=2 node api-consume-mjs.mjs 
(node:42673) ExperimentalWarning: The ESM module loader is experimental.
Function 1 - ALTERNATE
Function 2 - ALTERNATE
Function 3 - ALTERNATE

And there we have successfully used two versions of our world-renowned API. Further we can use the same technique in either CommonJS or ES6 modules.