; Date: Sun Apr 15 2018

Tags: Node.JS

It is best to automate all things, and in Node.js applications a convenient tool is the scripts section of package.json. Using the scripts section, npm helps you to easily record the necessary commands to build or maintain an application. However, npm does nothing to help with cross platform command-line compatibility, because it simply hands the command string to Bash (on Linux/macOS/etc) or to CMD.EXE on Windows. For the most part this is okay, but what if you need to set environment variables? The Bash way of setting command variables gives an error message on Windows.

I came across this issue while verifying the code in my latest book, Node.js Web Development 4th Edition, would work correctly on Windows. The book extolls the virtues of the Twelve Factor Application model, and for this there are two aspects:

• Automate all administrative tasks -- for a Node.js app, that includes using npm scripts
• Use environment variables to configure the application -- This is easy on Bash, but the command format is not portable to Windows

What I mean is to do something like this:

  "scripts": {
    "start": "DEBUG=notes:* node --experimental-modules ./bin/www.mjs", 
    "start-fs": "DEBUG=notes:* NOTES_MODEL=fs node --experimental-modules ./bin/www.mjs", 
    "start-level": "DEBUG=notes:* NOTES_MODEL=level node --experimental-modules ./bin/www.mjs", 
    "start-sqlite3": "SQLITE_FILE=chap07.sqlite3 DEBUG=notes:* NOTES_MODEL=sqlite3 node --experimental-modules ./bin/www.mjs",
    "start-sequelize": "SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml NOTES_MODEL=sequelize node --experimental-modules ./bin/www.mjs",
    "start-mongodb": "MONGO_URL=mongodb://localhost/ MONGO_DBNAME=chap07 NOTES_MODEL=mongodb node --experimental-modules ./bin/www.mjs",
    "server1": "NOTES_MODEL=fs PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2": "NOTES_MODEL=fs PORT=3002 node --experimental-modules ./bin/www.mjs",
    "server1-sqlite3": "SQLITE_FILE=chap07.sqlite3 NOTES_MODEL=sqlite3 PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2-sqlite3": "SQLITE_FILE=chap07.sqlite3 NOTES_MODEL=sqlite3 PORT=3002 node --experimental-modules ./bin/www.mjs",
    "server1-sequelize": "SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml NOTES_MODEL=sequelize PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2-sequelize": "SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml NOTES_MODEL=sequelize PORT=3002 node --experimental-modules ./bin/www.mjs",
    "server1-mongodb": "MONGO_URL=mongodb://localhost/ MONGO_DBNAME=chap07 NOTES_MODEL=mongodb PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2-mongodb": "MONGO_URL=mongodb://localhost/ MONGO_DBNAME=chap07 NOTES_MODEL=mongodb PORT=3002 node --experimental-modules ./bin/www.mjs",
    "sqlite3-setup": "sqlite3 chap07.sqlite3 --init models/schema-sqlite3.sql"
  },

What we have here is the same application (bin/www.mjs) being executed in different modes by setting environment variables to configure each mode. For example PORT sets the TCP port the app will listen on, NOTES_MODEL selects the type of storage for the application, and SEQUELIZE_CONNECT is configuration for the Sequelize module to specify the database to be used.

According to the Twelve Factor Application model, environment variables are an incredibly simple mechanism to inject configuration into an application. The application simply queries process.env.ENVIRONMENT_VARIABLE_NAME to pick up the configuration option. No need for baroque parsing of command line options, or yet another configuration file syntax.

The problem with this approach is that this command format only works for Bash. As a Linux and MacOS based software developer, Bash is second nature to me, but there are a significant number of folks using Windows.

Our code needs to be useable by those who use Windows. It's unfortunate that Windows chose such an incompatible usage environment, but it's not up to us to stand in high judgement over Microsoft or those who prefer Microsoft's software.

The problem is - how do we write npm scripts that are portable across Windows and Linux/macOS/etc systems?

If we legitimately have users on a variety of operating systems, we must accommodate them in our software. Otherwise the distribution of our software would be harmed.

It's arguable that since this is such a strong generic problem across all software developers, that the solution should be implemented inside npm rather than the approach I'm about to show. What if npm incorporated something to parse the scripts using Bash-like syntax, for full cross-platform enjoyment?

But that's not what npm does, so we must work with the product we have.

Best solution with current npm capabilities

I came up with several options by searching the npm repository using the phrase "windows script".

The best solution looks to be the cross-env package, see: https://www.npmjs.com/package/cross-env

This adds a command in node_modules/.bin that interpolates the Bash command-line style of setting environment variables.

Installation via: npm install cross-env@5.1.4 --save

Then you simply modify each npm scripts entry to add cross-env to the front of the command line, like so:

  "scripts": {
    "start": "cross-env DEBUG=notes:* node --experimental-modules ./bin/www.mjs", 
    "start-fs": "cross-env DEBUG=notes:* NOTES_MODEL=fs node --experimental-modules ./bin/www.mjs", 
    "start-level": "cross-env DEBUG=notes:* NOTES_MODEL=level node --experimental-modules ./bin/www.mjs", 
    "start-sqlite3": "cross-env SQLITE_FILE=chap07.sqlite3 DEBUG=notes:* NOTES_MODEL=sqlite3 node --experimental-modules ./bin/www.mjs",
    "start-sequelize": "cross-env SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml NOTES_MODEL=sequelize node --experimental-modules ./bin/www.mjs",
    "start-mongodb": "cross-env MONGO_URL=mongodb://localhost/ MONGO_DBNAME=chap07 NOTES_MODEL=mongodb node --experimental-modules ./bin/www.mjs",
    "server1": "cross-env NOTES_MODEL=fs PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2": "cross-env NOTES_MODEL=fs PORT=3002 node --experimental-modules ./bin/www.mjs",
    "server1-sqlite3": "cross-env SQLITE_FILE=chap07.sqlite3 NOTES_MODEL=sqlite3 PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2-sqlite3": "cross-env SQLITE_FILE=chap07.sqlite3 NOTES_MODEL=sqlite3 PORT=3002 node --experimental-modules ./bin/www.mjs",
    "server1-sequelize": "cross-env SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml NOTES_MODEL=sequelize PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2-sequelize": "cross-env SEQUELIZE_CONNECT=models/sequelize-sqlite.yaml NOTES_MODEL=sequelize PORT=3002 node --experimental-modules ./bin/www.mjs",
    "server1-mongodb": "cross-env MONGO_URL=mongodb://localhost/ MONGO_DBNAME=chap07 NOTES_MODEL=mongodb PORT=3001 node --experimental-modules ./bin/www.mjs", 
    "server2-mongodb": "cross-env MONGO_URL=mongodb://localhost/ MONGO_DBNAME=chap07 NOTES_MODEL=mongodb PORT=3002 node --experimental-modules ./bin/www.mjs",
    "sqlite3-setup": "sqlite3 chap07.sqlite3 --init models/schema-sqlite3.sql",
  },

This is simple, you just use the Bash syntax and proceed with life.

Other alternatives

npmcs: https://www.npmjs.com/package/npmcs -- Has you implement a Module similar to the npm scripts section that has sub-sections for each operating system. Like:

//npmcs-scripts.js
module.exports = {
    scripts: {
        //nodemon, build and test commands same regardless of running on windows/linux (remove duplication)
        nodemon: 'nodemon --debug src/app.js',
        build: 'start webpack -d --watch',
        test: 'echo "Error: no test specified" && exit 1'
        /*
            Windows scripts, run on a windows environment
        */
        win: {
            start: 'start npm run dev',
            dev: 'SET NODE_ENV=development && npm run build && npm run nodemon',
            prod: 'SET NODE_ENV=production node src/app.js',
        },
        /*
            Unix scripts, run on a unix environment
        */
        nix: {
            start: 'npm run dev',
            dev: 'export NODE_ENV=development && npm run build && npm run nodemon',
            prod: 'export NODE_ENV=production node src/app.js',
        },
    }
}

cross-os: https://www.npmjs.com/package/cross-os -- Modifies the npm package.json to have subsections for each operating system, like so:

"scripts": {
  "foo": "cross-os bar",
  "bar": {
    "darwin": "echo 'i will only run on Mac'",
    "win32": "echo 'i will only run on Windows'",
    "linux": "echo 'i will only run on Linux'"
  }
}

Windows Subsystem for Linux

Another approach is for Windows developers to install WSL. This is a Linux installation running on Windows - for example, Ubuntu - where you can apt-get to install packages and all the other things you'd do on Linux. The important bit is you have a Bash shell, and the scripts commands will automatically run correctly.

This requires Windows 10 updated with the latest creators pack or whatever it is. I've done this, and the WSL environment is a breath of fresh air. No longer do I feel hampered by CMD.EXE whose usage paradigm was designed in the 1970's. Instead I'm breathing easy in Bash, whose usage paradigm was also designed in the 1970's, but is a paradigm that is far more sensible. Obviously saying "usage paradigm designed in the 1970's" is incorrect, but to me there's something about CMD.EXE that feels crufty and backwards.

We have to recognize WSL may not be the correct solution for all use cases.

It's possible for some to like the CMD.EXE environment, prefer it over Bash, and to be affronted by our suggestion to install WSL.

Some of them will also be affronted by the attitude I'm probably expressing through my word choice. I may be expressing judgmentalism about Windows and Windows users which I'm trying hard to avoid. Windows users are people just like me, and deserve all the respect I deserve.

At the same time, CMD.EXE seems so primitive and I can't imagine why anyone would possibly like it. Having looked into PowerShell, it's really no better. But it's easy to find folks who are ecstatic about PowerShell, and who am I or anyone else to judge them?

Possible modification to npm

What about:

{
  "scripts": {
    "start": {
        "environment": {
            "DEBUG": "notes:*"
        },
        "command": "node --experimental-modules ./bin/www.mjs"
    },
    "start-fs": {
        "environment": {
            "DEBUG": "notes:*",
            "NOTES_MODEL": "fs"
        },
        "command": "node --experimental-modules ./bin/www.mjs"
    },
    "start-level": {
        "environment": {
            "DEBUG": "notes:*",
            "NOTES_MODEL": "level"
        },
        "command": "node --experimental-modules ./bin/www.mjs"
    },
    "start-sqlite3": {
        "environment": {
            "DEBUG": "notes:*",
            "NOTES_MODEL": "sqlite3",
            "SQLITE_FILE": "chap07.sqlite3"
        },
        "command": "node --experimental-modules ./bin/www.mjs"
    },
    "start-sequelize": {
        "environment": {
            "DEBUG": "notes:*",
            "NOTES_MODEL": "sequelize",
            "SEQUELIZE_CONNECT": "models/sequelize-sqlite.yaml"
        },
        "command": "node --experimental-modules ./bin/www.mjs"
    },
    ...
    "sqlite3-setup": "sqlite3 chap07.sqlite3 --init models/schema-sqlite3.sql"
  }
}

In other words the rule is - if the value portion of a scripts entry is a simple string, it will be interpreted as so, but if it is instead an object there are fields to specify the environment as a data structure.

The mechanism could even expose some of the options to the exec function