From version 3.2 onward, it is possible to execute Python code in parallel using the Executor interface provided in the concurrent.futures module. We already saw the Executor interface in action in the previous chapter, when we used ThreadPoolExecutor to perform multiple tasks concurrently. In this subsection, we'll demonstrate the usage of the ProcessPoolExecutor class.

ProcessPoolExecutor exposes a very lean interface, at least when compared to the more featureful multiprocessing.Pool. A ProcessPoolExecutor can be instantiated, similar to ThreadPoolExecutor, by passing a number of worker threads using the max_workers argument (by default, max_workers will be the number of CPU cores available). The main methods available to the ProcessPoolExecutor are submit and map.

The submit method will take a function and return a Future (see the last chapter) that will keep track of the execution of the submitted function. The method map works similarly to the Pool.map function, except that it returns an iterator rather than a list:

    from concurrent.futures import ProcessPoolExecutor

    executor = ProcessPoolExecutor(max_workers=4)
    fut = executor.submit(square, 2)
    # Result:
    # <Future at 0x7f5b5c030940 state=running>

    result = executor.map(square, [0, 1, 2, 3, 4])
    list(result)
    # Result:
    # [0, 1, 4, 9, 16]

To extract the result from one or more Future instances, you can use the concurrent.futures.wait and concurrent.futures.as_completed functions. The wait function accepts a list of future and will block the execution of the programs until all the futures have completed their execution. The result can then be extracted using the Future.result method. The as_completed function also accepts a function but will, instead, return an iterator over the results:

    from concurrent.futures import wait, as_completed

    fut1 = executor.submit(square, 2)
    fut2 = executor.submit(square, 3)
    wait([fut1, fut2])
    # Then you can extract the results using fut1.result() and fut2.result()

    results = as_completed([fut1, fut2])
    list(results)
    # Result:
    # [4, 9]

Alternatively, you can generate futures using the asyncio.run_in_executor function and manipulate the results using all the tools and syntax provided by the asyncio libraries so that you can achieve concurrency and parallelism at the same time.