; Date: Tue Nov 28 2017

Tags: Node.JS »»»» npm

With npm version 5 we gained a lot of welcome new features and performance improvements. I've been happily using npm@5 for several months, but recently discovered a major problem that dramatically affects my workflow. When I'm updating a package, I want to test that package locally WITHOUT pushing changes to the Git repository. To do so, I found it best to install that package into another project to test/run the code. This worked great with npm versions prior to npm@5, but now I have two major problems. First, npm modifies the package.json to insert a "file:" dependency, overwriting the existing dependency, and second it makes a symlink to the package rather than doing a proper installation.

The relavent npm Pull Request is: https://github.com/npm/npm/pull/15900 In that pull request, they wanted to change some behaviors including converting file: dependencies so they behave like link: dependencies. It looks like this Pull Request is what broke my workflow, and it appears others noted the problem in the pull request discussion.

Let me first describe the problem more carefully.

I use AkashaCMS to manage this website, and I am also the author of AkashaCMS. This means I'm frequently testing updates to AkashaCMS code using this website or one of the other websites I build using AkashaCMS. I'll have edited one of the AkashaCMS modules, and then in the project directory for this website I'll type:

$ npm install ~/akasharender/akasharender

This used to automatically do a proper installation of the latest version of that package in the project directory. It's easy -- make some edits -- in the terminal window do the install command -- then in the terminal window, rebuild the website, and see whether it worked or not.

What happens now, with npm@5, is these two malfeatures:

$ grep akasharender package.json 
    "akasharender": ">=0.6.15",
$ npm install ~/akasharender/akasharender

+ akasharender@0.6.15
added 100 packages, removed 2 packages, updated 1 package and moved 63 packages in 8.472s
$ grep akasharender package.json 
    "akasharender": "file:../../akasharender/akasharender",
$ ls -ld node_modules/akasharender
lrwxr-xr-x  1 david  staff  34 Nov 27 22:58 node_modules/akasharender -> ../../../akasharender/akasharender

First - notice how a proper dependency has been rewritten to a file: dependency. This is WRONG and INCORRECT.

Second - the changes introduced by the pull request linked above cause file: dependencies act as link: dependencies. Meaning, that instead of making a proper installation with a local directory and copying the installable things into that directory, it makes a symbolic link over to the named directory.

Third - You can see that symbolic link above.

Observations about the problem

The first problem is because npm@5 introduced a change where npm install always acts according to the old behavior of npm install __package__ --save, meaning that you no longer had to add the --save option. Therefore, npm always updates the package.json dependency to match what was specified in the command-line arguments.

The second problem is due to changing file: installations to act as link: installations, and therefore create the symlink.

What's the problem?

That npm is rewriting the package.json is a major problem. What if I were to neglectfully commit that modified package.json? Breakage would happen elsewhere.

It is 100% incorrect for npm to have modified the package.json this way.

You might be thinking the symbolic link is not a problem, but is actually a blessing. There is a small degree of a blessing because with the symbolic link you can edit the source package all you like without having to rerun npm install each time. In my case there is a problem.

It is because I'll be doing this with multiple dependent packages at the same time. AkashaCMS is made of multiple independent packages. All of them require the akasharender package, but none declare it in their package.json dependencies. The reason for this is rather arcane, and has to do with certain objects (class objects) exported from the akasharender module and for proper use of JavaScript objects there must be one-and-only-one source instance of the class definition.

In any case, what I end up with is two (or more) packages whose dependency is rewritten to a file: URL, and whose entry in node_modules is now a symlink. When the akashacms-base plugin tries to require('akashacms') it fails.

I have made two postings in the npm issue queue:

• https://github.com/npm/npm/issues/18503#issuecomment-346919698 -- describing even more detail about the whole problem
• https://github.com/npm/npm/issues/19240 -- Describes just the issue with rewriting package.json

I see that the npm issue queue is very large - well over 1,000 issues - and that in the npm blog they've announced a plan to sweep away reports in the issue queue that don't get any activity. They claim to be overwhelmed and unable to keep up with their issue queue.

That npm cannot keep up with issue queue is troubling

Isn't it troubling to hear that the npm project is unable to keep up with the issue queue?

This tool is vitally important to the health of the Node.js ecosystem. The npm project must be capable of delivering good quality software that satisfies all our needs. Indeed, generally speaking, npm does an excellent job and serves our needs well. (though - there must be some troubles in paradise, or else the yarn project would never have come into existence)

Therefore, I think it'd be great for some (more) folks to step up and volunteer to help the npm project. No doubt there are folks who are doing so, but the size of the issue queue backlog suggests more folks are required.

The completely disgusting workaround that is completely wrong but does work

There is a workaround that kinda-sorta works in this case. Namely, to commit the local changes to the repository, and then install from the repository in the project directory.

$ git commit -a && git push    # In source directory
$ npm install organization/repository   # In project directory

This is completely disgusting because we should not -- MUST NOT -- commit untested probably broken code to the repository. That's plainly bad software engineering practice.

In a typical cycle of implementing/debugging/testing a change, how many times would you have to push untested code to the repository?

The workaround that actually works

Thanks to some discussion in the pull request above, I was reminded of the npm pack command.

This command creates a tarball of the stuff that would be installed for a named package, or from the current directory if you're in a package directory. Hence, in the source directory I would type this:

$ npm pack
akasharender-0.6.15.tgz

This creates a tarball in the named file. Then in the project directory I type this:

$ npm install ~/akasharender/akasharender/akasharender-0.6.15.tgz 

+ akasharender@0.6.15
updated 1 package in 5.894s

This makes a proper install of the package in the local project directory. No symlinking nonsense here. BUT, it still modifies package.json this way:

$ grep akasharender package.json 
    "akasharender": "file:../../akasharender/akasharender/akasharender-0.6.15.tgz",

Going back to the drawing board I remembered one of the big features in npm@5 -- that it automatically updates package.json as if you had used the --save flag.

Sure enough:

$ npm install ~/akasharender/akasharender/akasharender-0.6.15.tgz  --no-save
npm WARN techsparx.com@1.0.0 No description

+ akasharender@0.6.15
added 69 packages in 8.015s
$ ls -ld node_modules/akasharender
drwxr-xr-x  27 david  staff  918 Nov 27 23:20 node_modules/akasharender
$ grep akasharender package.json 
    "akasharender": ">=0.6.15",

This does the right thing. A combination of npm pack followed by npm install --no-save does the trick.

Configuring npm to not --save

There is a config setting that turns off the new automatic --save feature.

$ npm config set save false
$ npm install ~/akasharender/akasharender

+ akasharender@0.6.15
removed 65 packages and updated 1 package in 3.671s
$ grep akasharender package.json 
    "akasharender": ">=0.6.15",
$ ls -ld node_modules/akasharender
lrwxr-xr-x  1 david  staff  34 Nov 27 23:37 node_modules/akasharender -> ../../../akasharender/akasharender

That was an improvement, the package.json is no longer modified. But it still symlinked the installation.

It does means that combining the npm pack then installing from the resulting tarball does not require adding the --no-save option.

Going by the discussion in npm help 7 config setting the link option to false should turn off the symlinking behavior. But:

$ npm config ls -l | grep link
link = false
bin-links = true
link = false
$ npm install ~/akasharender/akasharender

+ akasharender@0.6.15
added 1 package in 2.676s
$ ls -ld node_modules/akasharender
lrwxr-xr-x  1 david  staff  34 Nov 27 23:48 node_modules/akasharender -> ../../../akasharender/akasharender
$ grep akasharender package.json 
    "akasharender": ">=0.6.15",

Even with link=false npm does the symlinking thing.