To explain the problem, we will create a little web spider, a command-line application that takes in a web URL as input and downloads its contents locally into a file. In the code presented in this chapter, we are going to use a few npm dependencies:

Also, we will often refer to a local module named ./utilities, which contains some helpers which we will be using in our application. We omit the contents of this file for brevity, but you can find the full implementation, along with a package.json file containing the full list of dependencies, in the download pack for this book available at:

  http://www.packtpub.com .

The core functionality of our application is contained inside a module named spider.js. Let's see how it looks. To start with, let's load all the dependencies that we are going to use:

const request = require('request'); 
const fs = require('fs'); 
const mkdirp = require('mkdirp'); 
const path = require('path'); 
const utilities = require('./utilities'); 

Next, we create a new function named spider(), which takes in the URL to download and a callback function that will be invoked when the download process completes:

function spider(url, callback) { 
  const filename = utilities.urlToFilename(url); 
  fs.exists(filename, exists => {                    //[1] 
    if(!exists) { 
      console.log(`Downloading ${url}`); 
      request(url, (err, response, body) => {        //[2] 
        if(err) { 
          callback(err); 
        } else { 
          mkdirp(path.dirname(filename), err => {    //[3] 
            if(err) { 
              callback(err); 
            } else { 
              fs.writeFile(filename, body, err => {  //[4] 
                if(err) { 
                  callback(err); 
                } else { 
                  callback(null, filename, true); 
                } 
              }); 
            } 
          }); 
        } 
      }); 
    } else { 
      callback(null, filename, false); 
    } 
  }); 
} 

The preceding function executes the following tasks:

To complete our web spider application, we just need to invoke the spider() function by providing a URL as an input (in our case, we read it from the command-line arguments):

spider(process.argv[2], (err, filename, downloaded) => { 
  if(err) { 
    console.log(err); 
  } else if(downloaded){ 
    console.log(`Completed the download of "${filename}"`); 
  } else { 
    console.log(`"${filename}" was already downloaded`); 
  } 
}); 

Now we are ready to try our web spider application, but first, make sure you have the utilities.js module and the package.json containing the full list of dependencies in your project directory. Then, install all the dependencies by running the following command:

npm install

Next, we can execute the spider module to download the contents of a web page, with a command like this:

node spider http://www.example.com

Looking at the spider() function we defined earlier, we can surely notice that even though the algorithm we implemented is really straightforward, the resulting code has several levels of indentation and is very hard to read. Implementing a similar function with a direct style blocking API would be straightforward, and there would be very few chances to make it look so wrong. However, using asynchronous CPS is another story, and making bad use of closures can lead to incredibly bad code.

The situation where the abundance of closures and in-place callback definitions transform the code into an unreadable and unmanageable blob is known as callback hell. It's one of the most well recognized and severe anti-patterns in Node.js and JavaScript in general. The typical structure of a code affected by this problem looks like the following:

asyncFoo( err => { 
  asyncBar( err => { 
    asyncFooBar( err => { 
      //... 
    }); 
  }); 
}); 

We can see how code written in this way assumes the shape of a pyramid due to the deep nesting and that's why it is also colloquially known as the pyramid of doom.

The most evident problem with code such as the preceding snippet, is the poor readability. Due to the nesting being too deep, it's almost impossible to keep track of where a function ends and where another one begins.

Another issue is caused by the overlapping of the variable names used in each scope. Often, we have to use similar or even identical names to describe the content of a variable. The best example is the error argument received by each callback. Some people often try to use variations of the same name to differentiate the object in each scope, for example, err, error, err1, err2, and so on; others prefer to just hide the variable defined in the scope by always using the same name; for example, err. Both alternatives are far from perfect, and cause confusion and increase the probability of introducing defects.

Also, we have to keep in mind that closures come at a small price in terms of performance and memory consumption. In addition, they can create memory leaks that are not so easy to identify because we shouldn't forget that any context referenced by an active closure is retained from garbage collection.

If we look at our spider() function, we will notice that it clearly represents a callback hell situation and has all the problems we just described. That's exactly what we are going to fix with the patterns and techniques we will learn in this chapter.