How it works...

We build a list of numbers stored in number_list:

number_list = list(range(1, 11))

For each element in the list, we operate the counting procedure until we reach 100000000 iterations, and then multiply the resulting value for 100000000:

def count(number) : 
for i in range(0, 100000000):
i=i+1
return i*number

def evaluate_item(x):
result_item = count(x)

In the main program, we execute the same task in sequential mode:

if __name__ == "__main__":
for item in number_list:
evaluate_item(item)

Then, in parallel mode, use the concurrent.futures pooling capabilities for a thread pool: 

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
for item in number_list:
executor.submit(evaluate, item)

And do the same for a process pool:  

with concurrent.futures.ProcessPoolExecutor(max_workers=5) as executor:
for item in number_list:
executor.submit(evaluate, item)
Note that both the thread and process pools are set with max_workers=5; moreover, if max_workers is equal to None, it will default to the number of processors on the machine.

To run this example, open Command Prompt and, in the same folder where the example is contained, type the following:

> python concurrent_futures_pooling.py

By executing the preceding example, we can see the execution of the three execution models with relative times:

Item 1, result 10000000
Item 2, result 20000000
Item 3, result 30000000
Item 4, result 40000000
Item 5, result 50000000
Item 6, result 60000000
Item 7, result 70000000
Item 8, result 80000000
Item 9, result 90000000
Item 10, result 100000000
Sequential Execution in 6.8109448 seconds
Item 2, result 20000000
Item 1, result 10000000
Item 4, result 40000000
Item 5, result 50000000
Item 3, result 30000000
Item 8, result 80000000
Item 7, result 70000000
Item 6, result 60000000
Item 10, result 100000000
Item 9, result 90000000
Thread Pool Execution in 6.805766899999999 seconds
Item 1, result 10000000
Item 4, result 40000000
Item 2, result 20000000
Item 3, result 30000000
Item 5, result 50000000
Item 6, result 60000000
Item 7, result 70000000
Item 9, result 90000000
Item 8, result 80000000
Item 10, result 100000000
Process Pool Execution in 4.166398899999999 seconds

It should be noted that although the example is not expensive in computational terms, sequential and thread pool execution are comparable in terms of time. Using a process pool gives us the fastest execution time. 

The pool then distributes the processes (in this case, five processes) between the available cores (for this example, a machine with four cores was used) in FIFO (short for first in, first out) mode.

So, for each core, the assigned process runs in series. Only after the I/O operation is performed does the pool schedule the execution of another process. Of course, the execution mechanism is the same if you use a thread pool.

The computational times, which are lower in the case of the process pool, must be traced back to the fact that I/O operations are not significant. This allows the pool of processes to be faster because, unlike threads, they do not require any synchronization mechanisms (as explained in Chapter 1Getting Started with Parallel Computing and Python, in the Introducing parallel programming recipe).

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
13.59.82.167