npm version 5 has major usability bug with installing packages locally

By: ( +David Herron; Date: Mon Nov 27 2017 16:00:00 GMT-0800 (Pacific Standard Time)

Tags: Node.JS » npm

With npm version 5 we gained a lot of welcome new features and performance improvements. I've been happily using npm@5 for several months, but recently discovered a major problem that dramatically affects my workflow. When I'm updating a package, I want to test that package locally WITHOUT pushing changes to the Git repository. To do so, I found it best to install that package into another project to test/run the code. This worked great with npm versions prior to npm@5, but now I have two major problems. First, npm modifies the package.json to insert a "file:" dependency, overwriting the existing dependency, and second it makes a symlink to the package rather than doing a proper installation.

The relavent npm Pull Request is: ( In that pull request, they wanted to change some behaviors including converting file: dependencies so they behave like link: dependencies. It looks like this Pull Request is what broke my workflow, and it appears others noted the problem in the pull request discussion.

Let me first describe the problem more carefully.

I use ( AkashaCMS to manage this website, and I am also the author of AkashaCMS. This means I'm frequently testing updates to AkashaCMS code using this website or one of the other websites I build using AkashaCMS. I'll have edited one of the AkashaCMS modules, and then in the project directory for this website I'll type:

$ npm install ~/akasharender/akasharender

This used to automatically do a proper installation of the latest version of that package in the project directory. It's easy -- make some edits -- in the terminal window do the install command -- then in the terminal window, rebuild the website, and see whether it worked or not.

What happens now, with npm@5, is these two malfeatures:

$ grep akasharender package.json 
    "akasharender": ">=0.6.15",
$ npm install ~/akasharender/akasharender

+ akasharender@0.6.15
added 100 packages, removed 2 packages, updated 1 package and moved 63 packages in 8.472s
$ grep akasharender package.json 
    "akasharender": "file:../../akasharender/akasharender",
$ ls -ld node_modules/akasharender
lrwxr-xr-x  1 david  staff  34 Nov 27 22:58 node_modules/akasharender -> ../../../akasharender/akasharender

First - notice how a proper dependency has been rewritten to a file: dependency. This is WRONG and INCORRECT.

Second - the changes introduced by the pull request linked above cause file: dependencies act as link: dependencies. Meaning, that instead of making a proper installation with a local directory and copying the installable things into that directory, it makes a symbolic link over to the named directory.

Third - You can see that symbolic link above.

Observations about the problem

The first problem is because npm@5 introduced a change where npm install always acts according to the old behavior of npm install __package__ --save, meaning that you no longer had to add the --save option. Therefore, npm always updates the package.json dependency to match what was specified in the command-line arguments.

The second problem is due to changing file: installations to act as link: installations, and therefore create the symlink.

What's the problem?

That npm is rewriting the package.json is a major problem. What if I were to neglectfully commit that modified package.json? Breakage would happen elsewhere.

It is 100% incorrect for npm to have modified the package.json this way.

You might be thinking the symbolic link is not a problem, but is actually a blessing. There is a small degree of a blessing because with the symbolic link you can edit the source package all you like without having to rerun npm install each time. In my case there is a problem.

It is because I'll be doing this with multiple dependent packages at the same time. AkashaCMS is made of multiple independent packages. All of them require the akasharender package, but none declare it in their package.json dependencies. The reason for this is rather arcane, and has to do with certain objects (class objects) exported from the akasharender module and for proper use of JavaScript objects there must be one-and-only-one source instance of the class definition.

In any case, what I end up with is two (or more) packages whose dependency is rewritten to a file: URL, and whose entry in node_modules is now a symlink. When the akashacms-base plugin tries to require('akashacms') it fails.

I have made two postings in the npm issue queue:

I see that the npm issue queue is very large - well over 1,000 issues - and that in the npm blog they've announced a plan to sweep away reports in the issue queue that don't get any activity. They claim to be overwhelmed and unable to keep up with their issue queue.

That npm cannot keep up with issue queue is troubling

Isn't it troubling to hear that the npm project is unable to keep up with the issue queue?

This tool is vitally important to the health of the Node.js ecosystem. The npm project must be capable of delivering good quality software that satisfies all our needs. Indeed, generally speaking, npm does an excellent job and serves our needs well. (though - there must be some troubles in paradise, or else the yarn project would never have come into existence)

Therefore, I think it'd be great for some (more) folks to step up and volunteer to help the npm project. No doubt there are folks who are doing so, but the size of the issue queue backlog suggests more folks are required.

The completely disgusting workaround that is completely wrong but does work

There is a workaround that kinda-sorta works in this case. Namely, to commit the local changes to the repository, and then install from the repository in the project directory.

$ git commit -a && git push    # In source directory
$ npm install organization/repository   # In project directory

This is completely disgusting because we should not -- MUST NOT -- commit untested probably broken code to the repository. That's plainly bad software engineering practice.

In a typical cycle of implementing/debugging/testing a change, how many times would you have to push untested code to the repository?

The workaround that actually works

Thanks to some discussion in the pull request above, I was reminded of the npm pack command.

This command creates a tarball of the stuff that would be installed for a named package, or from the current directory if you're in a package directory. Hence, in the source directory I would type this:

$ npm pack

This creates a tarball in the named file. Then in the project directory I type this:

$ npm install ~/akasharender/akasharender/akasharender-0.6.15.tgz 

+ akasharender@0.6.15
updated 1 package in 5.894s

This makes a proper install of the package in the local project directory. No symlinking nonsense here. BUT, it still modifies package.json this way:

$ grep akasharender package.json 
    "akasharender": "file:../../akasharender/akasharender/akasharender-0.6.15.tgz",

Going back to the drawing board I remembered one of the big features in npm@5 -- that it automatically updates package.json as if you had used the --save flag.

Sure enough:

$ npm install ~/akasharender/akasharender/akasharender-0.6.15.tgz  --no-save
npm WARN No description

+ akasharender@0.6.15
added 69 packages in 8.015s
$ ls -ld node_modules/akasharender
drwxr-xr-x  27 david  staff  918 Nov 27 23:20 node_modules/akasharender
$ grep akasharender package.json 
    "akasharender": ">=0.6.15",

This does the right thing. A combination of npm pack followed by npm install --no-save does the trick.

Configuring npm to not --save

There is a config setting that turns off the new automatic --save feature.

$ npm config set save false
$ npm install ~/akasharender/akasharender

+ akasharender@0.6.15
removed 65 packages and updated 1 package in 3.671s
$ grep akasharender package.json 
    "akasharender": ">=0.6.15",
$ ls -ld node_modules/akasharender
lrwxr-xr-x  1 david  staff  34 Nov 27 23:37 node_modules/akasharender -> ../../../akasharender/akasharender

That was an improvement, the package.json is no longer modified. But it still symlinked the installation.

It does means that combining the npm pack then installing from the resulting tarball does not require adding the --no-save option.

Going by the discussion in npm help 7 config setting the link option to false should turn off the symlinking behavior. But:

$ npm config ls -l | grep link
link = false
bin-links = true
link = false
$ npm install ~/akasharender/akasharender

+ akasharender@0.6.15
added 1 package in 2.676s
$ ls -ld node_modules/akasharender
lrwxr-xr-x  1 david  staff  34 Nov 27 23:48 node_modules/akasharender -> ../../../akasharender/akasharender
$ grep akasharender package.json 
    "akasharender": ">=0.6.15",

Even with link=false npm does the symlinking thing.

« A recommended Node.js/Express application folder structure Streamline Mocha/SuperTest REST tests with async functions »
2016 Election 2018 Elections Acer C720 Ad block Affiliate marketing Air Filters Air Quality Air Quality Monitoring AkashaCMS Amazon Amazon Kindle Amazon Web Services America Amiga and Jon Pertwee Android Anti-Fascism AntiVirus Software Apple Apple Flexgate Apple Hardware History Apple Hardware Mistakes Apple iPhone Apple iPhone Hardware April 1st Arduino ARM Compilation Artificial Intelligence Astronomy Astrophotography Asynchronous Programming Authoritarianism Automated Social Posting AWS DynamoDB AWS Lambda Ayo.JS Bells Law Big Brother Big Data Big Finish Big Science Bitcoin Mining Black Holes Blade Runner Blockchain Blogger Blogging Books Botnets Cassette Tapes Cellphones China China Manufacturing Christopher Eccleston Chrome Chrome Apps Chromebook Chromebox ChromeOS CIA CitiCards Citizen Journalism Civil Liberties Climate Change Clinton Cluster Computing Command Line Tools Comment Systems Computer Accessories Computer Hardware Computer Repair Computers Conservatives Cross Compilation Crouton Cryptocurrency Curiosity Rover Currencies Cyber Security Cybermen Cybersecurity Daleks Darth Vader Data backup Data Formats Data Storage Database Database Backup Databases David Tenant DDoS Botnet Department of Defense Department of Justice Detect Adblocker Developers Editors Digital audio Digital Nomad Digital Photography Direct Attach Storage Diskless Booting Disqus DIY DIY Repair DNP3 Do it yourself Docker Docker MAMP Docker Swarm Doctor Who Doctor Who Paradox Doctor Who Review Drobo Drupal Drupal Themes DuckDuckGo DVD E-Books E-Readers Early Computers eGPU Election Hacks Electric Bicycles Electric Vehicles Electron Eliminating Jobs for Human Emdebian Encabulators Energy Efficiency Enterprise Node EPUB ESP8266 Ethical Curation Eurovision Event Driven Asynchronous Express Face Recognition Facebook Fake Advertising Fake News Fedora VirtualBox Fifth Doctor File transfer without iTunes FireFly Flash Flickr Fraud Freedom of Speech Front-end Development G Suite Gallifrey Gig Economy git Github GitKraken Gitlab GMAIL Google Google Adsense Google Chrome Google Gnome Google+ Government Spying Great Britain Green Transportation Hate Speech Heat Loss Hibernate High Technology Hoax Science Home Automation HTTP Security HTTPS Human ID I2C Protocol Image Analysis Image Conversion Image Processing ImageMagick In-memory Computing Incognito Mode InfluxDB Infrared Thermometers Insulation Internet Internet Advertising Internet Law Internet of Things Internet Policy Internet Privacy iOS iOS Devices iPad iPhone iPhone hacking Iron Man iShowU Audio Capture iTunes Janet Fielding Java JavaFX JavaScript JavaScript Injection JDBC John Simms Journalism Joyent jQuery Kaspersky Labs Kext Kindle Kindle Marketplace Large Hadron Collider Lets Encrypt LibreOffice Linux Linux Hints Linux Single Board Computers Logging Mac Mini Mac OS Mac OS X Mac Pro MacBook Pro Machine Learning Machine Readable ID Macintosh macOS macOS High Sierra macOS Kext MacOS X setup Make Money Online Make Money with Gigs March For Our Lives MariaDB Mars Mass Violence Matt Lucas MEADS Anti-Missile Mercurial MERN Stack Michele Gomez Micro Apartments Microsoft Military AI Military Hardware Minification Minimized CSS Minimized HTML Minimized JavaScript Missy Mobile Applications Mobile Computers MODBUS Mondas Monetary System MongoDB Mongoose Monty Python MQTT Music Player Music Streaming MySQL NanoPi Nardole NASA Net Neutrality Network Attached Storage Node Web Development Node.js Node.js Database Node.js Performance Node.js Testing Node.JS Web Development Node.x North Korea npm NSA NVIDIA NY Times Online advertising Online Community Online Fraud Online Journalism Online News Online Photography Online Video Open Media Vault Open Source Open Source and Patents Open Source Governance Open Source Licenses Open Source Software OpenAPI OpenJDK OpenVPN Palmtop PDA Patrick Troughton PayPal Paywalls Personal Flight Peter Capaldi Peter Davison Phishing Photography PHP Plex Plex Media Server Political Protest Politics Postal Service Power Control President Trump Privacy Private E-mail server Production use Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero ReactJS Recaptcha Recycling Refurbished Computers Remote Desktop Removable Storage Renewable Energy Republicans Retro Computing Retro-Technology Reviews RFID Rich Internet Applications Right to Repair River Song Robotics Robots Rocket Ships RSS News Readers rsync Russia Russia Troll Factory Russian Hacking Rust SCADA Scheme Science Fiction SD Cards Search Engine Ranking Search Engines Season 1 Season 10 Season 11 Security Security Cameras Server-side JavaScript Serverless Framework Servers Shell Scripts Silence Simsimi Skype SmugMug Social Media Social Media Networks Social Media Warfare Social Network Management Social Networks Software Development Software Patents Space Flight Space Ship Reuse Space Ships SpaceX Spear Phishing Spring Spring Boot Spy Satellites SQLite3 SSD Drives SSD upgrade SSH SSH Key SSL Stand For Truth Strange Parts Swagger Synchronizing Files Tegan Jovanka Telescopes Terrorism The Cybermen The Daleks The Master Time-Series Database Tom Baker Torchwood Total Information Awareness Trump Trump Administration Trump Campaign Twitter Ubuntu Udemy UDOO US Department of Defense Video editing Virtual Private Networks VirtualBox VLC VNC VOIP Vue.js Walmart Weapons Systems Web Applications Web Developer Resources Web Development Web Development Tools Web Marketing Webpack Website Advertising Website Business Models Weeping Angels WhatsApp William Hartnell Window Insulation Windows Windows Alternatives Wordpress World Wide Web Yahoo YouTube YouTube Adpocalypse YouTube Monetization