Fixing "Maximum call stack size exceeded" in async Node.js code

By: (plus.google.com) +David Herron; Date: 2016-09-09 13:20

Tags: Node.JS » JavaScript

I've happily used the async module for Node.js for years to simplify asynchronous operations over arrays. This is a difficulty in the Node.js paradigm, since the "normal" way to process an array is with a simple for loop. But if the operation is asynchronous the for loop has no way of knowing when to iterate to the next item, or even when processing is finished. Instead the async.eachSeries function does the trick, because your code tells async.eachSeries when to go to the next item, when there's an error, and it knows when the last item is processed, etc. It's been great.

But I just came across this:

RangeError: Maximum call stack size exceeded

The line of code in the stack trace pointed to a next() call that would iterate to the next entry in an async.eachSeries loop.

After some pondering and (stackoverflow.com) reading it seems that under the covers async.eachSeries creates a new level of stack recursion each time next() is called. Whoops. With a large enough array that will inevitably exceed the call stack size.

async.eachSeries(data, (datum, next) => {
        try { var d = new Date(datum["Timestamp"]).toISOString(); } catch (e) { d = undefined; }
        if (d) {
            db.run(`INSERT INTO ${tblnm} ( timestamp, json ) VALUES ( ?, ? )`, [
                d,
                JSON.stringify(datum)
            ],
            err => {
                if (err) next(err);
                else next();
            });
        } else next();
    },
    err => {
        if (err) reject(err);
        else resolve(db);
});

That's the code ...

The solution was simple ... replace the two next() calls with this: setTimeout(() => { next(); });

What this codes is reset the call stack before calling next(). Therefore the call stack will not be exceeded.

Another possibility would be something like this:

return Promise.all(data.map(datum => {
   return new Promise((resolve, reject) => {
       // perform async operation on datum
   });
})

But I did not try this. With async.eachSeries it is guaranteed to do one operation at a time, which isn't guaranteed with Promise.all. Hence, isn't there a risk of blowing up some other limit?

Promise.all is to be given an array of Promise's which it executes and finishes when all Promises have executed. It is another way of neatly executing an operation over an array. The data.map operation creates that array of Promise objects. In theory it would work, but we want to do this without blowing limits.

« Easily import your CSV data into MySQL with Node.js Deprecating buggy npm packages, dealing with deprecations »
2016 Election Acer C720 Ad block AkashaCMS Amazon Amazon Kindle Amiga Android Anti-Fascism AntiVirus Software Apple Apple Hardware History Apple iPhone Apple iPhone Hardware April 1st Arduino ARM Compilation Astronomy Asynchronous Programming Authoritarianism Automated Social Posting Ayo.JS Bells Law Big Brother Big Finish Bitcoin Mining Black Holes Blade Runner Blockchain Blogger Blogging Books Botnet Botnets Cassette Tapes Cellphones Christopher Eccleston Chrome Chrome Apps Chromebook Chromebooks Chromebox ChromeOS CIA CitiCards Citizen Journalism Civil Liberties Clinton Cluster Computing Command Line Tools Computer Hardware Computer Repair Computers Cross Compilation Crouton Cryptocurrency Curiosity Rover Cyber Security Cybermen Daleks Darth Vader Data backup Data Storage Database Database Backup Databases David Tenant DDoS Botnet Detect Adblocker Developers Editors Digital Photography Diskless Booting DIY DIY Repair DNP3 Do it yourself Docker Docker Swarm Doctor Who Doctor Who Paradox Drobo Drupal Drupal Themes DVD E-Books E-Readers Early Computers Election Hacks Electric Bicycles Electric Vehicles Electron Emdebian Encabulators Energy Efficiency Enterprise Node EPUB ESP8266 Ethical Curation Eurovision Event Driven Asynchronous Express Facebook Fake News Fedora VirtualBox File transfer without iTunes FireFly Fraud Freedom of Speech Gallifrey git Gitlab GMAIL Google Google Chrome Google Gnome Google+ Government Spying Great Britain Heat Loss Hibernate Hoax Science Home Automation HTTPS Human ID I2C Protocol Image Analysis Image Conversion Image Processing ImageMagick InfluxDB Infrared Thermometers Insulation Internet Internet Advertising Internet Law Internet of Things Internet Policy Internet Privacy iOS Devices iPad iPhone iPhone hacking Iron Man Iternet of Things iTunes Java JavaScript JavaScript Injection JDBC John Simms Journalism Joyent Kaspersky Labs Kindle Marketplace Lets Encrypt LibreOffice Linux Linux Hints Linux Single Board Computers Logging Mac OS Mac OS X Machine Readable ID MacOS X setup Make Money Online MariaDB Mars Matt Lucas MEADS Anti-Missile Mercurial Michele Gomez Micro Apartments Military Hardware Minification Minimized CSS Minimized HTML Minimized JavaScript Missy Mobile Applications MODBUS Mondas MongoDB Mongoose Monty Python MQTT Music Player Music Streaming MySQL NanoPi Nardole NASA Net Neutrality Node Web Development Node.js Node.js Database Node.js Testing Node.JS Web Development Node.x North Korea Online advertising Online Fraud Online Journalism Online Video Open Media Vault Open Source Governance Open Source Licenses Open Source Software OpenAPI OpenVPN Personal Flight Peter Capaldi Photography PHP Plex Plex Media Server Political Protest Postal Service Power Control Privacy Production use Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero Recycling Remote Desktop Republicans Retro-Technology Reviews Right to Repair River Song Robotics Rocket Ships RSS News Readers rsync Russia Russia Troll Factory Russian Hacking SCADA Scheme Science Fiction Search Engine Ranking Season 1 Season 10 Season 11 Security Security Cameras Server-side JavaScript Shell Scripts Silence Simsimi Skype Social Media Social Media Warfare Social Networks Software Development Space Flight Space Ship Reuse Space Ships SpaceX Spear Phishing Spring Spring Boot SQLite3 SSD Drives SSD upgrade SSH SSH Key SSL Swagger Synchronizing Files Telescopes Terrorism The Cybermen The Daleks The Master Time-Series Database Torchwood Total Information Awareness Trump Trump Administration Trump Campaign Ubuntu UDOO Virtual Private Networks VirtualBox VLC VNC VOIP Web Applications Web Developer Resources Web Development Web Development Tools Web Marketing Website Advertising Weeping Angels WhatsApp Window Insulation Wordpress YouTube YouTube Monetization