Chapter 4. Improving Performance with Parallel Programming

In this chapter, we will cover the following topics:

  • Parallelizing processing with pmap
  • Parallelizing processing with Incanter
  • Partitioning Monte Carlo simulations for better pmap performance
  • Finding the optimal partition size with simulated annealing
  • Combining function calls with reducers
  • Parallelizing with reducers
  • Generating online summary statistics for data streams with reducers
  • Using type hints
  • Benchmarking with Criterium

Introduction

If concurrent processing has performance implications when structuring programs, parallel processing is a way to get better performance that has implications on how we structure our programs. Although they are often conflated, concurrent processing and parallel processing are different solutions to different problems. Concurrency is good for expressing programs that involve different tasks that can be, or must be, carried out at the same time. Parallelization is a good option if you want to perform the same task many times, all at once. Parallelization is not necessary, but it can help tremendously with your program's performance.

Earlier, the easiest, and often best, strategy to improve performance was to go on a vacation. Moore's law implies that processor speed will double approximately every 18 months, so in the 1990s, we could go on vacation, return, buy a new computer, and our programs were faster. This was magic.

Today, we're no longer under Moore's law, instead, as the saying goes, the free lunch is over. Now, processor speeds have plateaued or even declined. Instead, computers are made faster by packing more processors into them. To make use of these processors, we have to employ parallel programming.

Of course, the processor isn't always the slowest part of the program (that is, our programs aren't always CPU bound). Sometimes, it's the disk or network that limits how fast our programs run. If that's the case, we have to read from multiple disks or network connections simultaneously in order to see any improvement in speed. For example, reading from a single file from different processors might even be slower, but if you can copy the file onto different disks and read each file from a separate processor, it will be faster in all likelihood.

The recipes in this chapter focus on leveraging multiple cores by showing different ways to parallelize Clojure programs. It also includes a few recipes on related topics. For instance, consider the Using type hints recipe, which talks about how to optimize our code, and the Benchmarking with Criterium recipe, which discusses how to get good data for optimizing our code.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.145.52.188