Easily deploy files or directory hierarchies to a server using Grunt

By: ; Date: 2015-03-03 12:36

Tags: Node.JS » Web Development

Something we geeks need to do all the time is deploy files between machines. Such as, deploying a directory hierarchy over to a server for staging or production use. There's a ton of ways to do this. The old-school way is a shell script with carefully crafted rsync commands. In my case I build websites using AkashaCMS and need to deploy them to the destination webserver. Until now I'd added a command to AkashaCMS solely for deployment, but I'm experimenting with Grunt to see how much can be done using the Grunt ecosystem rather than having to maintain code in AkashaCMS.

In AkashaCMS I simply used the Node.js spawn function to run an rsync command with selected command line options, some of which came out of the config file:

$ rsync --archive --delete --verbose --compress localDir/ user@remotehost.com:remoteDir/

One thing I'd like to avoid is that this creates a dependency on Unix/Linux/MacOSX systems that have rsync. Those poor people who have to suffer through using Windows don't have rsync. Let's all feel sorry for them.

Unfortunately the solution I've ended up with doesn't solve the rsync-on-windows problem.

grunt-sftp-deploy

The first tool I tried could possibly solve that problem, but I wasn't able to work out how to get it to run at a decent speed. But here goes anyway. In your Gruntfile.js load the grunt-sftp-deploy plugin as so

grunt.loadNpmTasks('grunt-sftp-deploy');

And make sure it's installed

$ npm install grunt-sftp-deploy

And/Or add it to your package.json to make sure the plugin is always installed.

The next step of course is to configure the plugin.

module.exports = function(grunt) {

    grunt.initConfig({
       'sftp-deploy': {
            deploy: {
                auth: {
                    host: 'example.com',
                    port: 22,
                    authKey: process.env.HOME +"/.sftp-deploy-example.txt"
                },
                cache: 'sftpCache.json',
                src: 'source-directory-name',
                dest: 'destination-directory-name',  // this is on the remote host
                exclusions: [],
                serverSep: '/',
                concurrency: 4,
                progress: true
            }
   });
};

This sets up a copying from a local directory to one on a remote host. The remove host name is in the host parameter, with the remote directory name in the dest parameter, and so on. That part is pretty straight-forward.

The tricky part is the authentication. Since this is a file which gets checked into source control it's a really bad idea to put user names or passwords in the Gruntfile. Fortunately Grunt makes it easy to read stuff in from external files, that you hopefully don't put under source control.

In this case the grunt-sftp-deploy plugin uses this authKey parameter in several ways, one of which is a file name containing a JSON object. It should be like so:

{
"username": "user-name-on-server"
}

Then grunt-sftp-deploy also automatically looks for local SSH keys to use for passwordless authentication.

This works pretty slick and you can then type this command:

$ grunt sftp-deploy:deploy

The only problem is that it takes a long time because it appears to deploy every last file. Supposedly the cache parameter is for a file that's used to keep data to help avoid uploading files which haven't changed. But that didn't work for me.

grunt-ssh, with .zip file

My next thought was to build a .zip file of the directory hierarchy I wanted to deploy, then use the grunt-ssh plugin to upload it, then ssh over a command to unpack the .zip file. This didn't work out because of the time involved in copying the .zip file. But let's take a look at this anyway.

To create the .zip file is something like this (using the archiver module):

var archiver = require('archiver');

module.exports.zipRenderedSite = function(config, done) {

    var archive = archiver('zip');

    var output = fs.createWriteStream(config.root_out +'.zip');

    output.on('close', function() {
        logger.info(archive.pointer() + ' total bytes');
        logger.info('archiver has been finalized and the output file descriptor has closed.');  
        done();
    });

    archive.on('error', function(err) {
      done(err);
    });

    archive.pipe(output);

	archive.directory(config.root_out, ".");

	archive.finalize();
};

That was easy.

Then in the Gruntfile.js:

grunt.loadNpmTasks('grunt-ssh');

To load the plugin tasks.

    deployData: grunt.file.readJSON(process.env.HOME +'/.sftp-deploy-example.json'),
    sftp: {
            deployZip: {
                files: { "./": config.root_out +'.zip' }
            },
            options: {
                host: '<%= deployData.host %>',
                path: '<%= deployData.path %>',
                username: '<%= deployData.username %>',
                privateKey: grunt.file.read(process.env.HOME + "/.ssh/id_rsa"),
                showProgress: true
            }
        }

This again uses a JSON file in the home directory to store info that shouldn't be checked into source code control. The SSH private key is read directly.

This works without any external dependencies. I was going to take the next step to unpack the .zip file on the remote host, but it took too long to upload that file (over 200MB) that this too was a non-starter.

grunt-rsync

That left using an rsync wrapper, since I was running out of options that use a pure JavaScript SSH2 implementation.

grunt.loadNpmTasks("grunt-rsync");

This loads the plugin.

rsync: {
    deploySite: {
        options: {
            args: [ '--verbose', '--archive', '--delete', '--compress' ],
            src: config.root_out +'/',
            dest: "example.com/",
            host: "remote-user-name@example.com"
        }
    }
}

This is a simple wrapper around the rsync command. The args array are the exact same args you'd pass on the command line. The src and dest directories are exactly as you'd do on the command line. You have to take the exact same precise care about when and where to put a trailing / on the directory names, just as in rsync.

With this deployment is now this easy:

$ grunt rsync:deploySite
« Safely detect if a Node.js module is installed before using require() to load it Don't rip your hair out when Vows tells you "Errored callback not fired" - try this one weird trick instead »
2016 Election Acer C720 Ad block AkashaCMS Android Apple Apple Hardware History Apple iPhone Hardware April 1st Arduino ARM Compilation Asynchronous Programming Authoritarianism Automated Social Posting Bells Law Big Brother Blade Runner Blogger Blogging Books Botnet Botnets Cassette Tapes Cellphones Christopher Eccleston Chrome Chrome Apps Chromebook Chromebooks Chromebox ChromeOS CIA CitiCards Civil Liberties Clinton Cluster Computing Command Line Tools Computer Hardware Computer Repair Computers Cross Compilation Crouton Cyber Security Cybermen Daleks Darth Vader Data backup Data Storage Database Database Backup Databases David Tenant DDoS Botnet Detect Adblocker Digital Photography DIY DIY Repair Docker Doctor Who Doctor Who Paradox Drobo Drupal Drupal Themes DVD Election Hacks Emdebian Enterprise Node ESP8266 Ethical Curation Eurovision Event Driven Asynchronous Express Facebook Fake News File transfer without iTunes FireFly Fraud Freedom of Speech Gallifrey git Gitlab GMAIL Google Google Chrome Google Gnome Google+ Government Spying Great Britain Home Automation HTTPS I2C Protocol Image Conversion Image Processing ImageMagick InfluxDB Internet Internet Advertising Internet Law Internet of Things Internet Policy Internet Privacy iOS Devices iPad iPhone iPhone hacking Iron Man Iternet of Things iTunes Java JavaScript JavaScript Injection JDBC John Simms Joyent Lets Encrypt LibreOffice Linux Linux Hints Linux Single Board Computers Logging Mac OS Matt Lucas MEADS Anti-Missile Mercurial Michele Gomez Military Hardware Missy Mobile Applications MODBUS Mondas Monty Python MQTT Music Player Music Streaming MySQL NanoPi Nardole Net Neutrality Node Web Development Node.js Node.js Database Node.js Testing Node.JS Web Development Node.x North Korea Online advertising Online Fraud Open Media Vault Open Source Software OpenAPI OpenVPN Personal Flight Peter Capaldi Photography Plex Media Server Political Protest Power Control Privacy Production use Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero Recycling Republicans Retro-Technology Reviews Right to Repair River Song Rocket Ships RSS News Readers rsync Russia Russia Troll Factory Scheme Science Fiction Season 1 Season 10 Season 11 Security Security Cameras Server-side JavaScript Shell Scripts Silence Simsimi Skype Social Media Warfare Social Networks Software Development Space Flight Space Ship Reuse Space Ships SpaceX Spring SQLite3 SSD Drives SSD upgrade SSH SSH Key SSL Swagger Synchronizing Files Terrorism The Cybermen The Daleks The Master Time-Series Database Torchwood Total Information Awareness Trump Trump Administration Ubuntu Virtual Private Networks VirtualBox VLC VOIP Web Applications Web Developer Resources Web Development Web Development Tools Weeping Angels WhatsApp Wordpress