By: +David Herron; Date: Mon Nov 27 2017 16:00:00 GMT-0800 (Pacific Standard Time)
With npm version 5 we gained a lot of welcome new features and performance improvements. I've been happily using npm@5 for several months, but recently discovered a major problem that dramatically affects my workflow. When I'm updating a package, I want to test that package locally WITHOUT pushing changes to the Git repository. To do so, I found it best to install that package into another project to test/run the code. This worked great with npm versions prior to npm@5, but now I have two major problems. First, npm modifies the package.json to insert a "file:" dependency, overwriting the existing dependency, and second it makes a symlink to the package rather than doing a proper installation.
The relavent npm Pull Request is:
https://github.com/npm/npm/pull/15900 In that pull request, they wanted to change some behaviors including converting
file: dependencies so they behave like
link: dependencies. It looks like this Pull Request is what broke my workflow, and it appears others noted the problem in the pull request discussion.
Let me first describe the problem more carefully.
I use AkashaCMS to manage this website, and I am also the author of AkashaCMS. This means I'm frequently testing updates to AkashaCMS code using this website or one of the other websites I build using AkashaCMS. I'll have edited one of the AkashaCMS modules, and then in the project directory for this website I'll type:
$ npm install ~/akasharender/akasharender
This used to automatically do a proper installation of the latest version of that package in the project directory. It's easy -- make some edits -- in the terminal window do the install command -- then in the terminal window, rebuild the website, and see whether it worked or not.
What happens now, with
npm@5, is these two malfeatures:
$ grep akasharender package.json "akasharender": ">=0.6.15", $ npm install ~/akasharender/akasharender + email@example.com added 100 packages, removed 2 packages, updated 1 package and moved 63 packages in 8.472s $ grep akasharender package.json "akasharender": "file:../../akasharender/akasharender", $ ls -ld node_modules/akasharender lrwxr-xr-x 1 david staff 34 Nov 27 22:58 node_modules/akasharender -> ../../../akasharender/akasharender
First - notice how a proper dependency has been rewritten to a
file: dependency. This is WRONG and INCORRECT.
Second - the changes introduced by the pull request linked above cause
file: dependencies act as
link: dependencies. Meaning, that instead of making a proper installation with a local directory and copying the installable things into that directory, it makes a symbolic link over to the named directory.
Third - You can see that symbolic link above.
Observations about the problem
The first problem is because
npm@5 introduced a change where
npm install always acts according to the old behavior of
npm install __package__ --save, meaning that you no longer had to add the
--save option. Therefore, npm always updates the
package.json dependency to match what was specified in the command-line arguments.
The second problem is due to changing
file: installations to act as
link: installations, and therefore create the symlink.
What's the problem?
That npm is rewriting the
package.json is a major problem. What if I were to neglectfully commit that modified
package.json? Breakage would happen elsewhere.
It is 100% incorrect for npm to have modified the
package.json this way.
You might be thinking the symbolic link is not a problem, but is actually a blessing. There is a small degree of a blessing because with the symbolic link you can edit the source package all you like without having to rerun
npm install each time. In my case there is a problem.
It is because I'll be doing this with multiple dependent packages at the same time. AkashaCMS is made of multiple independent packages. All of them
require the akasharender package, but none declare it in their
package.json dependencies. The reason for this is rather arcane, and has to do with certain objects (class objects) exported from the
In any case, what I end up with is two (or more) packages whose dependency is rewritten to a
file: URL, and whose entry in
node_modules is now a symlink. When the
akashacms-base plugin tries to
require('akashacms') it fails.
I have made two postings in the npm issue queue:
- https://github.com/npm/npm/issues/18503#issuecomment-346919698 -- describing even more detail about the whole problem
https://github.com/npm/npm/issues/19240 -- Describes just the issue with rewriting
I see that the npm issue queue is very large - well over 1,000 issues - and that in the npm blog they've announced a plan to sweep away reports in the issue queue that don't get any activity. They claim to be overwhelmed and unable to keep up with their issue queue.
That npm cannot keep up with issue queue is troubling
Isn't it troubling to hear that the npm project is unable to keep up with the issue queue?
This tool is vitally important to the health of the Node.js ecosystem. The npm project must be capable of delivering good quality software that satisfies all our needs. Indeed, generally speaking, npm does an excellent job and serves our needs well. (though - there must be some troubles in paradise, or else the yarn project would never have come into existence)
Therefore, I think it'd be great for some (more) folks to step up and volunteer to help the npm project. No doubt there are folks who are doing so, but the size of the issue queue backlog suggests more folks are required.
The completely disgusting workaround that is completely wrong but does work
There is a workaround that kinda-sorta works in this case. Namely, to commit the local changes to the repository, and then install from the repository in the project directory.
$ git commit -a && git push # In source directory $ npm install organization/repository # In project directory
This is completely disgusting because we should not -- MUST NOT -- commit untested probably broken code to the repository. That's plainly bad software engineering practice.
In a typical cycle of implementing/debugging/testing a change, how many times would you have to push untested code to the repository?
The workaround that actually works
Thanks to some discussion in the pull request above, I was reminded of the
npm pack command.
This command creates a tarball of the stuff that would be installed for a named package, or from the current directory if you're in a package directory. Hence, in the source directory I would type this:
$ npm pack akasharender-0.6.15.tgz
This creates a tarball in the named file. Then in the project directory I type this:
$ npm install ~/akasharender/akasharender/akasharender-0.6.15.tgz + firstname.lastname@example.org updated 1 package in 5.894s
This makes a proper install of the package in the local project directory. No symlinking nonsense here. BUT, it still modifies
package.json this way:
$ grep akasharender package.json "akasharender": "file:../../akasharender/akasharender/akasharender-0.6.15.tgz",
Going back to the drawing board I remembered one of the big features in
npm@5 -- that it automatically updates
package.json as if you had used the
$ npm install ~/akasharender/akasharender/akasharender-0.6.15.tgz --no-save npm WARN email@example.com No description + firstname.lastname@example.org added 69 packages in 8.015s $ ls -ld node_modules/akasharender drwxr-xr-x 27 david staff 918 Nov 27 23:20 node_modules/akasharender $ grep akasharender package.json "akasharender": ">=0.6.15",
This does the right thing. A combination of
npm pack followed by
npm install --no-save does the trick.
Configuring npm to not
There is a config setting that turns off the new automatic
$ npm config set save false $ npm install ~/akasharender/akasharender + email@example.com removed 65 packages and updated 1 package in 3.671s $ grep akasharender package.json "akasharender": ">=0.6.15", $ ls -ld node_modules/akasharender lrwxr-xr-x 1 david staff 34 Nov 27 23:37 node_modules/akasharender -> ../../../akasharender/akasharender
That was an improvement, the
package.json is no longer modified. But it still symlinked the installation.
It does means that combining the
npm pack then installing from the resulting tarball does not require adding the
Going by the discussion in npm help 7 config setting the
link option to
false should turn off the symlinking behavior. But:
$ npm config ls -l | grep link link = false bin-links = true link = false $ npm install ~/akasharender/akasharender + firstname.lastname@example.org added 1 package in 2.676s $ ls -ld node_modules/akasharender lrwxr-xr-x 1 david staff 34 Nov 27 23:48 node_modules/akasharender -> ../../../akasharender/akasharender $ grep akasharender package.json "akasharender": ">=0.6.15",
link=false npm does the symlinking thing.