Fixing "Maximum call stack size exceeded" in async Node.js code

By: ; Date: 2016-09-09 13:20

Tags: Node.JS » JavaScript

I've happily used the async module for Node.js for years to simplify asynchronous operations over arrays. This is a difficulty in the Node.js paradigm, since the "normal" way to process an array is with a simple for loop. But if the operation is asynchronous the for loop has no way of knowing when to iterate to the next item, or even when processing is finished. Instead the async.eachSeries function does the trick, because your code tells async.eachSeries when to go to the next item, when there's an error, and it knows when the last item is processed, etc. It's been great.

But I just came across this:

RangeError: Maximum call stack size exceeded

The line of code in the stack trace pointed to a next() call that would iterate to the next entry in an async.eachSeries loop.

After some pondering and reading it seems that under the covers async.eachSeries creates a new level of stack recursion each time next() is called. Whoops. With a large enough array that will inevitably exceed the call stack size.

async.eachSeries(data, (datum, next) => {
        try { var d = new Date(datum["Timestamp"]).toISOString(); } catch (e) { d = undefined; }
        if (d) {
            db.run(`INSERT INTO ${tblnm} ( timestamp, json ) VALUES ( ?, ? )`, [
                d,
                JSON.stringify(datum)
            ],
            err => {
                if (err) next(err);
                else next();
            });
        } else next();
    },
    err => {
        if (err) reject(err);
        else resolve(db);
});

That's the code ...

The solution was simple ... replace the two next() calls with this: setTimeout(() => { next(); });

What this codes is reset the call stack before calling next(). Therefore the call stack will not be exceeded.

Another possibility would be something like this:

return Promise.all(data.map(datum => {
   return new Promise((resolve, reject) => {
       // perform async operation on datum
   });
})

But I did not try this. With async.eachSeries it is guaranteed to do one operation at a time, which isn't guaranteed with Promise.all. Hence, isn't there a risk of blowing up some other limit?

Promise.all is to be given an array of Promise's which it executes and finishes when all Promises have executed. It is another way of neatly executing an operation over an array. The data.map operation creates that array of Promise objects. In theory it would work, but we want to do this without blowing limits.

« Easily import your CSV data into MySQL with Node.js Deprecating buggy npm packages, dealing with deprecations »
2016 Election Acer C720 Ad block AkashaCMS Android Apple Apple Hardware History Apple iPhone Hardware April 1st Arduino ARM Compilation Asynchronous Programming Authoritarianism Automated Social Posting Bells Law Big Brother Blade Runner Blogger Blogging Books Botnet Botnets Cassette Tapes Cellphones Christopher Eccleston Chrome Chrome Apps Chromebook Chromebooks Chromebox ChromeOS CIA CitiCards Civil Liberties Clinton Cluster Computing Command Line Tools Computer Hardware Computer Repair Computers Cross Compilation Crouton Cyber Security Cybermen Daleks Darth Vader Data backup Data Storage Database Database Backup Databases David Tenant DDoS Botnet Detect Adblocker Digital Photography DIY DIY Repair Docker Doctor Who Doctor Who Paradox Drobo Drupal Drupal Themes DVD Election Hacks Emdebian Enterprise Node ESP8266 Ethical Curation Eurovision Event Driven Asynchronous Express Facebook Fake News File transfer without iTunes FireFly Fraud Freedom of Speech Gallifrey git Gitlab GMAIL Google Google Chrome Google Gnome Google+ Government Spying Great Britain Home Automation HTTPS I2C Protocol Image Conversion Image Processing ImageMagick InfluxDB Internet Internet Advertising Internet Law Internet of Things Internet Policy Internet Privacy iOS Devices iPad iPhone iPhone hacking Iron Man Iternet of Things iTunes Java JavaScript JavaScript Injection JDBC John Simms Joyent Lets Encrypt LibreOffice Linux Linux Hints Linux Single Board Computers Logging Mac OS Matt Lucas MEADS Anti-Missile Mercurial Michele Gomez Military Hardware Missy Mobile Applications MODBUS Mondas Monty Python MQTT Music Player Music Streaming MySQL NanoPi Nardole Net Neutrality Node Web Development Node.js Node.js Database Node.js Testing Node.JS Web Development Node.x North Korea Online advertising Online Fraud Open Media Vault Open Source Software OpenAPI OpenVPN Personal Flight Peter Capaldi Photography Plex Media Server Political Protest Power Control Privacy Production use Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero Recycling Republicans Retro-Technology Reviews Right to Repair River Song Rocket Ships RSS News Readers rsync Russia Russia Troll Factory Scheme Science Fiction Season 1 Season 10 Season 11 Security Security Cameras Server-side JavaScript Shell Scripts Silence Simsimi Skype Social Media Warfare Social Networks Software Development Space Flight Space Ship Reuse Space Ships SpaceX Spring SQLite3 SSD Drives SSD upgrade SSH SSH Key SSL Swagger Synchronizing Files Terrorism The Cybermen The Daleks The Master Time-Series Database Torchwood Total Information Awareness Trump Trump Administration Ubuntu Virtual Private Networks VirtualBox VLC VOIP Web Applications Web Developer Resources Web Development Web Development Tools Weeping Angels WhatsApp Wordpress