Asynchronous array operations in ES7 JavaScript async/await functions

By: (plus.google.com) +David Herron; Date: 2016-11-26 22:53

Tags: Node.js » Asynchronous Programming

A glaring problem with the most excellent async/await feature in JavaScript is all the existing coding structures that only work with synchronous code. In an async/await function it's trivially easy to write a simple "for" loop that contains whatever async/await calls you want. But as soon as you try "Array.forEach" with an async callback function it will fail miserably. That's because this function, like so many existing interfaces which take a callback function, assumes the callback is synchronous. Instead of waiting for the Promise to resolve, it plows on ahead executing the next iteration.

Let's first do a little review of async functions.

async function foo(bar) {
    var text = await new Promise((resolve, reject) => {
        fs.readFile(bar, 'utf8', (err, text) => {
            if (err) reject(err);
            else resolve(text);
        });
    });
    // act on text
    return text;
}

An async function adds magic to regular functions allowing you to in-line asynchronous operations that produce Promises. That is, with an asynchronous function like fs.readFile the result (either error or data) arrives some undeterminate time in the future. With regular Node.js we had to deal with callback functions and keep from sliding down a slippery slope into callback hell. The await keyword is available inside an async function, and arranges to return the value from the Promise, either its successful resolution, or to throw any error that occurs.

The result is cleaner asynchronous code. It's even cleaner if you use a package like fs-extra-promise which produces a Promise for asynchronous code:

async function foo(bar) {
    var text = await fs.readFileAsync(bar, 'utf8');
    // act on text
    return text;
}

Async functions are a breath of fresh air, and all Node.js programmers should sit up and take a hard look at this new feature.

But ... the magic of async functions is not a magic wand that improves all asynchronous coding. JavaScript has array functions like foreach or map or filter that take a callback function but do not play well with async functions. To take an example, consider this pale cheap attempt to duplicate the Unix "du" command:

async function du(filez) {
    var total = 0;
    await filez.forEach(async (filenm) => {
        var stats = await fs.statAsync(filenm);
        total += stats.size;
    });
    return total;
}

After you start writing async functions it's natural to want to write this. The syntax for an async callback function is as shown here, and each loop iteration will wait for the statAsync function to finish executing. BTW -- this relies on the fs-extra-promise module, which provides functions equivalent to the fs module but producing Promise objects.

If you run this with a suitable array, the value for "total" is as shown here:

$ node --harmony-async-await du access.js 2files.js
0

The problem is what I stated above -- the forEach operation does not know to wait for the async function to finish. It just plows on, and its work finishes long before any of the callback functions execute. Inserting some debugging printout demonstrates that:

$ node --harmony-async-await du access.js 2files.js
0
access.js 523 523
2files.js 331 854

The total was printed before the loop iterations ran.

It's possible to fix this problem by making the code ugly:

async function du(filez) {
    var total = 0;
    await Promise.all(filez.map(async (filenm) => {
        var stats = await fs.statAsync(filenm);
        total += stats.size;
        console.log(`${filenm} ${stats.size} ${total}`);
    }));
    return total;
}

Remember that the async function returns a Promise, so therefore filez.map constructs an array of Promises which works well with Promise.all which can be awaited as shown here.

The attraction of the async/await feature is to make our code beautiful. We have here code that works, but at the cost of uglification.

Enter the arraync package

A newly created package, (www.npmjs.com) https://www.npmjs.com/package/arraync, gives us an excellent solution. It is a Polyfill adding several functions to the Array object.

To install the module:

$ npm install arraync --save

Then in the code add:

require('asyncnc');

Then ... we can write our function as so:

async function du(filez) {
    var total = 0;
    await filez.forEachAsync(async (filenm) => {
        var stats = await fs.statAsync(filenm);
        total += stats.size;
        console.log(`${filenm} ${stats.size} ${total}`);
    });
    return total;
}

This is exactly what we had earlier, with the addition of an "await" keyword before the loop. In other words, this exactly fulfills the code we naturally want to write.

The arraync package provides a full complement of async-ified Array functions. They're automatically attached to the Array prototype for you.

The code shown here is an example out of a soon-to-be-released book on asynchronous programming in JavaScript. The book focuses on Promises, Generators and async/await. Keep an eye on this blog for an announcement.

« New Book: Asynchronous JavaScript with Promises, Generators and async/await Useful reading to understand the Promises, Generators and the async/await feature for Node.js/JavaScript »
2016 Election 2018 Elections Acer C720 Ad block Air Filters Air Quality Air Quality Monitoring AkashaCMS Amazon Amazon Kindle Amazon Web Services America Amiga and Jon Pertwee Android Anti-Fascism AntiVirus Software Apple Apple Hardware History Apple iPhone Apple iPhone Hardware April 1st Arduino ARM Compilation Artificial Intelligence Astronomy Astrophotography Asynchronous Programming Authoritarianism Automated Social Posting AWS DynamoDB AWS Lambda Ayo.JS Bells Law Big Brother Big Data Big Finish Big Science Bitcoin Mining Black Holes Blade Runner Blockchain Blogger Blogging Books Botnets Cassette Tapes Cellphones China China Manufacturing Christopher Eccleston Chrome Chrome Apps Chromebook Chromebox ChromeOS CIA CitiCards Citizen Journalism Civil Liberties Climate Change Clinton Cluster Computing Command Line Tools Comment Systems Computer Accessories Computer Hardware Computer Repair Computers Conservatives Cross Compilation Crouton Cryptocurrency Curiosity Rover Currencies Cyber Security Cybermen Cybersecurity Daleks Darth Vader Data backup Data Formats Data Storage Database Database Backup Databases David Tenant DDoS Botnet Department of Defense Department of Justice Detect Adblocker Developers Editors Digital Nomad Digital Photography Diskless Booting Disqus DIY DIY Repair DNP3 Do it yourself Docker Docker MAMP Docker Swarm Doctor Who Doctor Who Paradox Doctor Who Review Drobo Drupal Drupal Themes DVD E-Books E-Readers Early Computers eGPU Election Hacks Electric Bicycles Electric Vehicles Electron Eliminating Jobs for Human Emdebian Encabulators Energy Efficiency Enterprise Node EPUB ESP8266 Ethical Curation Eurovision Event Driven Asynchronous Express Face Recognition Facebook Fake News Fedora VirtualBox Fifth Doctor File transfer without iTunes FireFly Flash Flickr Fraud Freedom of Speech Front-end Development G Suite Gallifrey Gig Economy git Github GitKraken Gitlab GMAIL Google Google Chrome Google Gnome Google+ Government Spying Great Britain Green Transportation Hate Speech Heat Loss Hibernate High Technology Hoax Science Home Automation HTTP Security HTTPS Human ID I2C Protocol Image Analysis Image Conversion Image Processing ImageMagick In-memory Computing InfluxDB Infrared Thermometers Insulation Internet Internet Advertising Internet Law Internet of Things Internet Policy Internet Privacy iOS iOS Devices iPad iPhone iPhone hacking Iron Man iShowU Audio Capture iTunes Janet Fielding Java JavaFX JavaScript JavaScript Injection JDBC John Simms Journalism Joyent Kaspersky Labs Kext Kindle Kindle Marketplace Large Hadron Collider Lets Encrypt LibreOffice Linux Linux Hints Linux Single Board Computers Logging Mac Mini Mac OS Mac OS X MacBook Pro Machine Learning Machine Readable ID Macintosh macOS macOS High Sierra macOS Kext MacOS X setup Make Money Online Make Money with Gigs March For Our Lives MariaDB Mars Mass Violence Matt Lucas MEADS Anti-Missile Mercurial MERN Stack Michele Gomez Micro Apartments Microsoft Military AI Military Hardware Minification Minimized CSS Minimized HTML Minimized JavaScript Missy Mobile Applications Mobile Computers MODBUS Mondas Monetary System MongoDB Mongoose Monty Python MQTT Music Player Music Streaming MySQL NanoPi Nardole NASA Net Neutrality Network Attached Storage Node Web Development Node.js Node.js Database Node.js Performance Node.js Testing Node.JS Web Development Node.x North Korea npm NVIDIA NY Times Online advertising Online Community Online Fraud Online Journalism Online Photography Online Video Open Media Vault Open Source Open Source and Patents Open Source Governance Open Source Licenses Open Source Software OpenAPI OpenJDK OpenVPN Palmtop PDA Patrick Troughton PayPal Paywalls Personal Flight Peter Capaldi Peter Davison Phishing Photography PHP Plex Plex Media Server Political Protest Politics Postal Service Power Control President Trump Privacy Private E-mail server Production use Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero ReactJS Recaptcha Recycling Refurbished Computers Remote Desktop Removable Storage Republicans Retro Computing Retro-Technology Reviews RFID Rich Internet Applications Right to Repair River Song Robotics Robots Rocket Ships RSS News Readers rsync Russia Russia Troll Factory Russian Hacking Rust SCADA Scheme Science Fiction SD Cards Search Engine Ranking Season 1 Season 10 Season 11 Security Security Cameras Server-side JavaScript Serverless Framework Servers Shell Scripts Silence Simsimi Skype SmugMug Social Media Social Media Networks Social Media Warfare Social Network Management Social Networks Software Development Software Patents Space Flight Space Ship Reuse Space Ships SpaceX Spear Phishing Spring Spring Boot Spy Satellites SQLite3 SSD Drives SSD upgrade SSH SSH Key SSL Stand For Truth Strange Parts Swagger Synchronizing Files Tegan Jovanka Telescopes Terrorism The Cybermen The Daleks The Master Time-Series Database Tom Baker Torchwood Total Information Awareness Trump Trump Administration Trump Campaign Twitter Ubuntu Udemy UDOO US Department of Defense Video editing Virtual Private Networks VirtualBox VLC VNC VOIP Vue.js Walmart Weapons Systems Web Applications Web Developer Resources Web Development Web Development Tools Web Marketing Webpack Website Advertising Weeping Angels WhatsApp William Hartnell Window Insulation Windows Windows Alternatives Wordpress World Wide Web Yahoo YouTube YouTube Monetization