CiprianDorinCraciun/Notes/Public/Asynchronous

File loading example

We'll take the simple case of loading the contents of a file and passing its whole content to a function that will hopefully do something useful. Although the example is almost meaningless in the real world as the functionality is already available one callback away (see the second code snippet), it is both simple to understand and exposes the issues we might encounter if we are forced to implement similar solutions ourselves. In order to better see the intricacies of asynchronous programming, I'll start with the "high-level" variant, and move backwards towards a low-level implementation, and then to alternative solutions.

First by using the synchronous blocking API --- thus a quick and dirty shortcut, a technique not advisable in all cases:

var fs = require ("fs");

function load (path, onSuccess) {
    var data = fs.readFileSync ("/etc/services");
    onSuccess (data);
}

load ("/etc/services", function (data) { console.log (data.length); })

Then by using the caned solution:

var fs = require ("fs");

function load (path, onSuccess) {
    fs.readFile (path, function (error, data) {
        if (error) throw (error);
        onSuccess (data);
    });
};

load ("/etc/services", function (data) { console.log (data.length); })

Afterwards by using a "nested" callbacks variant with the low-level POSIX API:

var fs = require ("fs");

function load (path, onSuccess) {

    fs.open (path, "r", function (error, file) {
            if (error) throw (error);
            var buffers = [];
            var loop = function () {
                    var buffer = new Buffer (1024);
                    fs.read (file, buffer, 0, buffer.length, null, function (error, read) {
                            if (error) throw (error);
                            if (read == 0) {
                                    fs.close (file, function (error) {
                                            if (error) throw (error);
                                            var data = Buffer.concat (buffers);
                                            onSuccess (data);
                                    });
                            } else {
                                    buffer = buffer.slice (0, read);
                                    buffers.push (buffer);
                                    loop ();
                            }
                    });
            };
            loop ();
    });
}

load ("/etc/services", function (data) { console.log (data.length); })

Now by using a "flattened" callbacks variant with the low-level POSIX API:

var fs = require ("fs");

function load (path, onSuccess) {

    var file;
    var buffers = [];

    function tryOpen () {
        fs.open (path, "r", function (error, file_) {
            if (error) throw (error);
            file = file_;
            tryRead ();
        });
    }

    function tryRead () {
        var buffer = new Buffer (1024);
        fs.read (file, buffer, 0, buffer.length, null, function (error, read) {
            if (error) throw (error);
            if (read == 0) return tryClose ();
            buffer = buffer.slice (0, read);
            buffers.push (buffer);
            tryRead ();
        });
    }

    function tryClose () {
        fs.close (file, function (error) {
            if (error) throw (error);
            done ();
        });
    }

    function done () {
        var data = Buffer.concat (buffers);
        onSuccess (data);
    }

    tryOpen ();
}

load ("/etc/services", function (data) { console.log (data.length); })

And finally by using the event-based streaming API that some libraries (especially the I/O ones) expose:

var fs = require ("fs");

function load (path, onSuccess) {

    var file = fs.createReadStream (path);
    var buffers = [];

    file.on ("data", function (buffer) {
            buffers.push (buffer);
    });

    file.on ("end", function () {
            file.destroy ();
    });

    file.on ("close", function () {
            var data = Buffer.concat (buffers);
            onSuccess (data);
    });

    file.on ("error", function (error) {
            throw (error);
    });
}

load ("/etc/services", function (data) { console.log (data.length); })

Observations:

• the function onSuccess does not respect the the "callback" pattern that the NodeJs environment expects; as such it should have actually been called callback and had called like: callback (null, data) or callback (error) (here the value of data will be implicitly set to undefined);
• in all cases any error encountered is "handled" by just throwing it away and crashing the process; but this isn't necessarily so, because with a simple modification errors can be gracefully handled and propagated to the callback function;
• of the all solutions I would guess the second one (the "canned" one) would be preferred, or the last one (the "stream" one);
• but in general when not having either a "canned" solution, or the API isn't a stream based one, I would choose the "flattened" variant if the nesting is more than two levels deep;

Another important observation is that although the code is written in an asynchronous manner, its logic is still linear, thus synchronous or serial in effect --- an observation made also by Mark in one of his replies (from Mark ?), --- as we do not take advantage of the full potential available, for example by issuing multiple read requests in batches and always keeping the downstream system busy. Now in this case involving access to the file-system, unless the backing drive has multiple spindles (which most likely implies a RAID system), such multiple pending requests would just put pressure on the drive, unless we know low-level details like file-system block size, stripe size. But in other cases, like fetching some content from a NoSQL database or from the Internet, having such batches might make a difference by reducing the completion time of the overall task. Unfortunately however the code gets more complicated...

Non-trivial examples

In one of his replies to the NodeJs mailing list, (from Isaac Schlueter, 1), Isaac complains that most of the examples used to lure developers into one library or another are trivial and non-realistic, and that maybe proponents should try to prove their point by using more complex, real-world examples, like his rimarf tool (an equivalent of rm -rf written in NodeJs): rimraf.