Importing an ES6 modules over HTTP/HTTPS in a Node.js

; Date: Sat Feb 26 2022

Tags: Node.JS

In Node.js 17.6.0, an experimental new feature allows us to import modules from an HTTP or HTTPS URL. That will close one of the differences between Node.js and Deno, which is that Deno allows packages to be imported using HTTPS. Further, ES6 modules in the browser allow importing modules over HTTPS. Until now, the Node.js team did not allow this, citing security concerns.

This new ability to import modules over HTTP or HTTPS is a highly experimental feature. There are serious security implications galore around this, and I personally have misgivings. However, the pull request for this feature has a lot of discussion around several sides of the issue. Having not read the whole thread of conversation, I'll have to trust that the Node.js technical team has already considered the issues over which I'm having misgivings.

First, let's explain what HTTP Import is about:

import example from 'https://example.com/path/to/index.mjs';

example('Hello, world!');

In other words, this is about directly importing a Node.js module that is retrieved over HTTPS or HTTP.

The feature was added to Node.js in v17.6.0. The documentation is already available at: https://nodejs.org/api/esm.html#https-and-http-imports

Background and some misgivings

Before we get more deeply into using this feature, let's talk about theory. The relavent pull request is at: https://github.com/nodejs/node/pull/36328

I have two concerns about this idea:

1. The obvious is security - apparently lots of folks agree - and as we'll see, the implementation is very limited apparently because of security concerns. We're accustomed to using the npm/yarn repository, and while there have been security intrusions vectored through that repository, we still use it every day at a huge scale.
2. Discoverability of packages is greatly enhanced by having a common package repository

A comment by wperron has this to say about the security issue:

Importing from npm isn't inherently more secure than importing from a raw URL, so let's work to make the whole ecosystem more secure, regardless of how packages are imported.

An HTTP request is subject to "man in the middle" attacks, where an intruder presents a fake version of a package server. HTTPS is, at least, not subject to that problem. But I'm envisioning an issue with trustability of random stuff you find on web servers.

But, reading wperron's comment, I realize that I hadn't thought through this issue, and that it's probably not accurate. For example, how much difference is there between import foo from 'https://example.com/path/to/index.mjs'; and the following:

$ npm init -y      # Create a blank project
$ npm install https://example.com/path/to --save

Even though npm/yarn were designed primarily to get packages from the npm/yarn repository, there is a long list of possible package URL's it supports. I regularly use GitHub URL's to run code against a test version of my packages, for example. And it's quite possible for someone to advertise an HTTPS URL from which to install packages.

In other words both import .. from 'https://...' and npm install https://... are grabbing Node.js code from an HTTPS server. What's the difference?

The comment by wperron goes on to discuss the various security violations that have been vectored through the npm/yarn repository. Repeatedly, malware has been distributed through the repositories. That is clearly a problem, but it also demonstrates that the npm/yarn repository doesn't give many security guarantees.

What did I mean by package discoverability? While the npm/yarn repository is flawed, it is a great way of discovering what packages are available. You want a Node.js or even front-end JavaScript package? Head to npmjs.com and search away.

If the Node.js community switches away from the npm/yarn repository, then how will we discover what packages are available? Will we be relying on the search engines to find packages? I don't see that as a good solution, and think it is better for there to be a purpose-built website for advertising JavaScript packages for Node.js/Deno/Browser.

Testing HTTPS/HTTP imports with an example

Okay, we need to get our hands into some code after that theory.

I've created a simple package at https://github.com/robogeek/example-es6-nodejs-package which is a pure ES6 module.

This is a small bit of code, with simple functionality, so that we can focus on usability via an HTTP/HTTPS import statement. The first function to present from lib/main.mjs is this:

export default function hello(message) {
    console.log(`main Hello ${message}`);
}

The intent is the typical Hello World example so we can quickly see whether the HTTPS import technique works.

Because this is an experimental feature, it is enabled using a command-line option --experimental-network-imports like this:

$ node --experimental-network-imports example.mjs

Going by the documentation it seems that this client code might work:

import example from 'https://github.com/robogeek/example-es6-nodejs-package/lib/index.mjs';
// ALTERNATE using the raw file
// import example from 'https://raw.githubusercontent.com/robogeek/example-es6-nodejs-package/main/lib/main.mjs';

example('World!');

But, this results in errors:

$ node --experimental-network-imports index.mjs 
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_NETWORK_IMPORT_BAD_RESPONSE]: import 'https://github.com/robogeek/example-es6-nodejs-package/lib/index.mjs' received a bad response: HTTP response returned status code of 404

###### Using the alternate
$ node --experimental-network-imports index.mjs 
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^

RangeError [ERR_UNKNOWN_MODULE_FORMAT]: Unknown module format: null for URL https://raw.githubusercontent.com/robogeek/example-es6-nodejs-package/main/lib/main.mjs

Hmm, not sure what this is about other than a problem with the module which was retrieved.

We can check whether the module source can be retrieved over HTTPS with the curl command:

$ curl -f https://raw.githubusercontent.com/robogeek/example-es6-nodejs-package/main/lib/main.mjs
export default function hello(message) {
    console.log(`main Hello ${message}`);
}

Indeed, the module source is correctly retrieved over HTTPS. You can run this again with the -v option to see the HTTP conversation in detail. One detail to notice is that GitHub uses HTTP/2. The next detail to notice is that the HTTPS Import feature only works over HTTP/1 and not HTTP/2. Support for HTTP/2 and HTTP/3 is in process, according to the documentation.

Therefore we need a different HTTP or HTTPS server. An alternate is to install a local web server, such this:

$ npm install @compodoc/live-server --save
...
$ npx @compodoc/live-server
Serving "/home/david/Projects/nodejs/example-es6-nodejs-package" at http://127.0.0.1:8080
Ready for changes

This starts a local web server that can even be configured for HTTPS and various advanced features. Run in this default mode, it serves files from the local directory over HTTP.

import example from 'http://127.0.0.1:8080/lib/main.mjs';

// THIS FAILS
// import example from 'http://127.0.0.1:8080/package.json';

example('World!');

In other words, switch to the URL on the local server for testing purposes.

$ node --experimental-network-imports index.mjs 
main Hello World!

With this URL, importing the Node.js module over HTTP works great.

Having solved that problem, let's explore a bit further to see what else we might be able to do, or what other errors we might run into.

Can this load a whole package?

Importing a single file is fine and useful, but of course we are almost always using complex packages. These start with a package.json and can easily have a dozen or a hundred individual JavaScript files.

But, the Node.js documentation for HTTPS Import shows a URL which ends in a file name, like http://127.0.0.1:8080/lib/main.mjs. What about all the advanced features we can implement with package.json? Will we be ignoring all that over HTTP/HTTPS?

There is a package.json in the example package. What happens if we change the client to this?

// THIS FAILS
import example from 'http://127.0.0.1:8080/package.json';
// AS DOES THIS
// import example from 'http://127.0.0.1:8080';

example('World!');

Using an HTTP or HTTPS URL, I was wondering how to make it clear to Node.js to retrieve a whole package. Should the URL reference the package.json directly, or should it reference the parent URL of the package.json?

$ node --experimental-network-imports index.mjs 
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^

RangeError [ERR_UNKNOWN_MODULE_FORMAT]: Unknown module format: null for URL http://127.0.0.1:8080/package.json

Unfortunately, that example results in this error. The documentation doesn't say whether there will be support for using a package.json over HTTP/HTTPS to identify a whole package.

Can we load other code from a module loaded over HTTP/HTTPS?

The purpose in mind for using a package.json was whether we could import a complex package over HTTP/HTTPS.

The documentation says that a module loaded over HTTP/HTTPS cannot access modules that are not loaded over HTTP/HTTPS. Presumably the security analysis identified a hole which is closed by this requirement.

To use data or resources loaded from a local file, pass them as data. To see what that means, I've added another function to the example module:

export default function hello(message) {
    console.log(`main Hello ${message}`);
}

export function configure(settings) {
    console.log(`We're being configured with `, settings);
}

Then, in the client application which uses this module, we can create a second file named config.mjs containing:

export const settings = {
    type: 'mysql',
    port: 3360
};

And we can rewrite the client program as so:

import { 
    default as example,
    configure
} from 'http://127.0.0.1:8080/lib/main.mjs';
import { settings } from './config.mjs';

example('World!');

/* const settings = {
    type: 'mysql',
    port: 3360
}; */
configure(settings);

Now, we have a data object which can be either created in-line or loaded from a local module.

Executing the script we get this:

$ node --experimental-network-imports index.mjs 
main Hello World!
We're being configured with  { type: 'mysql', port: 3360 }

That's simple, we can supply some data to a function in the module which was remotely loaded.

Attempting to import a complex Node.js module over HTTP/HTTPS

But, let's try to use a complex Node.js module. Can that remotely loaded module itself load a module that's local to the remotely loaded module? In the example-es6-nodejs-package repository, there is another module file, api.mjs, containing this:

export function echo(message) {
    return message;
}

It's meant to mimic an API. The echo function simply returns the data it is given.

Then in main.mjs add this:

import * as _api from './api.mjs';
export const api = _api;
// ALTERNATIVE:
// export * from './api.mjs';

This is a typical local reference we would make between modules. The echo function is exported as part of an object called api.

In the client program:

import { 
    default as example,
    configure,
    api
} from 'http://127.0.0.1:8080/lib/main.mjs';

...
console.log(api.echo('Hello, world'));

This tries to access the function in the api object. But, this fails with the following:

$ node --experimental-network-imports index.mjs 
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:372:5)
    at onParseError (node:internal/url:563:9)
    at new URL (node:internal/url:643:5)
    at ESMLoader.resolve (node:internal/modules/esm/loader:577:5)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ESMLoader.getModuleJob (node:internal/modules/esm/loader:250:7)
    at async ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:81:21)
    at async Promise.all (index 0)
    at async link (node:internal/modules/esm/module_job:86:9) {
  input: './api.mjs',
  code: 'ERR_INVALID_URL'
}

The module we loaded has an import statement referencing ./api.mjs. The loaded module is executing on our machine, which have access to the remote file system, instead it only has HTTP access. Of course it cannot access this file, but shouldn't it be able to compute an HTTP or HTTPS URL from this? It's a simple computation to go from http://127.0.0.1:8080/lib/main.mjs to http://127.0.0.1:8080/lib/api.mjs in this case. Clearly the current implementation cannot do so.

One solution is, in main.mjs to not load api.mjs at all, then in the client program do this:

import * as api from 'http://127.0.0.1:8080/lib/api.mjs';

The program then works correctly:

$ node --experimental-network-imports index.mjs 
main Hello World!
We're being configured with  { type: 'mysql', port: 3360 }
Hello, world

Another combination which works is for main.mjs to contain this:

import * as _api from 'http://localhost:8080/lib/api.mjs';
export const api = _api;

This requires that main.mjs know that it will be accessed via this URL. With this change the client program can be changed to do this:

import { 
    default as example,
    configure,
    api
} from 'http://127.0.0.1:8080/lib/main.mjs';

// import * as api from 'http://127.0.0.1:8080/lib/api.mjs';

The "api" is then managed by main.mjs rather than in the client program. And, when this is executed it works correctly.

Another experiment is to import a module which has been installed via other means, such as from node_modules. In main.mjs we can add this:

...
import * as mime from 'mime-types';

...
export function mimetype(mt) {
    return mime.lookup(mt);
}

We could use any 3rd party package, this is one I selected at random. This is a reference to a module that in normal circumstances would be imported from node_module. This means we must run the following command:

$ npm install mime-types --save

Then in the client program we make the following change:

import { 
    default as example,
    configure,
    api,
    mimetype
} from 'http://127.0.0.1:8080/lib/main.mjs';

console.log(mimetype('foo.json'));

We've added a new function, mimetype, which turns around and asks the mime-types package for information.

We can try to run the client program, but get this error message:

$ node --experimental-network-imports index.mjs 
node:internal/errors:465
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_URL_SCHEME]: The URL must be of scheme file

In other words, the import .. 'mime-types'; command is disallowed. We refer back to the Node.js documentation which says:

These modules cannot access other modules that are not over http: or https:.

This import for mime-types does not fit that requirement.

Can a remotely loaded Node.js module read a file?

Another security test to make is whether the module imported from an HTTP or HTTPS URL can use internal packages like fs. To test this, let's try to read a file.

In main.mjs add this:

import { promises as fsp, default as fs } from 'fs';
...
export async function readFile(fn) {
    return await fsp.readFile(fn, 'utf8');
}

This uses the fs.readFile function to try and read a file, loading it from the built-in module named fs.

In the client program we can make this change:

import { 
    default as example,
    configure,
    api,
    readFile
} from 'http://127.0.0.1:8080/lib/main.mjs';
...
console.log(readFile('package.json'));

Which uses the readFile function in the remotely loaded code to try and read a file. But, we get the same error: The URL must be of scheme file

It's that same error message. Hey, Node.js team, I don't think this error message gives good advice about what to do.

An even simpler test is to change api.mjs as so:

import * as util from 'util';

export function echo(message) {
    return util.inspect(message);
}

This imports another function from another built-in package. In main.mjs remember to comment out the code related to the fs module. Then, run the client program again and we yet again get the same error: The URL must be of scheme file

Which verifies that this message will occur for any built-in Node.js module. The meaning of the error message has something to do with the URL in the import statement.

Summary

What we've learned is that after enabling --experimental-network-imports we can use an import statement which references an HTTP or HTTPS URL.

It may be puzzling why this article showed so many failures? First, we were exploring both what we can do with this feature, and what we cannot do with it. But, we also must recognize that the limitations we ran into are due to security concerns.

Since I used to work in the Java SE team at Sun Microsystems, I'm reminded of restrictions that can be configured in the Java runtime. In Java, classes are loaded via class loader objects, much like in Node.js modules are loaded by a module loader. In some circumstances, secure class loaders are used to limit the possible source of class files, and to limit the capabilities of code running from a given class loader. Because Java was originally designed to include use in mobile devices or in browser-based Applets, secure class loaders were added to constrain the capabilities of code loaded from one source or another.

A similar model is used in smart phone apps. They often present dialog boxes requesting permission to perform this or that action. That is due to the security model under which those apps are executing.

The HTTP/HTTPS import behavior we just examined indicates some kind of security model in action.

So far, Node.js has had an open security model. Any code we import into node_modules had full access to any built-in module, or any other module in node_modules. To move forward perhaps the security model in Node.js needs to change.