The foreach() function^[62] executes an arbitrary R expression across an input. foreach()’s strength is that it can execute in parallel with the help of a supplied parallel backend. The doRedis package provides such a backend, using the Redis datastore^[63] as a job queue.

doRedis can work locally to take advantage of multicore systems, and also farm tasks out to remote R instances (“workers”). It’s straightforward to add or remove workers at runtime—even in mid-job—to adapt to changing work conditions or speed up job processing. Similar to Hadoop, doRedis is fault-tolerant in that failed tasks are automatically resubmitted to their job queue.

doRedis supports Linux, Mac OS X, and Windows systems.

Revolution Analytics is a company that provides R tools, support, and training. They have two products of note.

First up is the commercial Revolution R Enterprise. The current beta release includes RevoScaleR (RSR), which brings distributed computing to R. When you use the special XDF data format, RSR functions know to work on that data one chunk at a time, which addresses R’s memory limitations. (This is not unlike Hadoop+HDFS.) To address R’s CPU limitations, RSR includes functions to run code across several local cores, or across a cluster of machines running using MS Windows HPC Server.^[64]

Second, and more recently, the Revolution gang released the open-source RHadoop packages (also known as RevoConnectR) to marry Hadoop and R: rmr provides the core MapReduce functionality; rhdfs routines let you manage data in HDFS; and rhbase talks to HBase, the Hadoop-backed database. We’re especially interested in rmr, which strives to be a clean, intuitive way to access Hadoop power without leaving the R comfort zone. RHadoop is still young, but we think it has strong potential.

Both RSR and rmr fold into your typical data analysis work: you use special functions and constructs to get the essence of a larger dataset, then pass those results to standard R functions for plotting and further analysis.

cloudnumbers.com is a platform for on-demand distributed, parallel computing. It provides out-of-the-box support for R as well as C/C++ and Python. We see cloudnumbers.com as a cousin of Amazon’s EC2, but specialized for scientific HPC work.

That said, cloudnumbers.com is an infrastructure, not a packaged parallelism strategy. It’s up to the researcher to choose and set up their tools—perhaps some of the topics we cover in this book—to take advantage of the hardware. We nonetheless feel this is worth mention because it is closely related to this book’s topic. You can find out more at http://cloudnumbers.com/.