Before we continue, let's briefly do a review of two functions available in Python for functional programming—map and reduce. These are both considered to be functional because they both act on functions for their operation. We find these interesting because these both correspond to common design patterns in programming, so we can swap out different functions in the input to get a multitude of different (and useful) operations.

Let's first recall the lambda keyword in Python. This allows us to define an anonymous function—in most cases, these can be thought of as a throwaway function that we may only wish to use once, or functions that are able to be defined on a single line. Let's open up IPython right now and define a little function that squares a number as such—pow2 = lambda x : x**2. Let's test it out on a few numbers:

Let's recall that map acts on two input values: a function and a list of objects that the given function can act on. map outputs a list of the function's output for each element in the original list. Let's now define our squaring operation as an anonymous function which we input into map, and a list of the last few numbers we checked with the following—map(lambda x : x**2, [2,3,4]):

We see that map acts as ElementwiseKernel! This is actually a standard design pattern in functional programming. Now, let's look at reduce; rather than taking in a list and outputting a directly corresponding list, reduce takes in a list, performs a recursive binary operation on it, and outputs a singleton. Let's get a notion of this design pattern by typing reduce(lambda x, y : x + y, [1,2,3,4]). When we type this in IPython, we will see that this will output a single number, 10, which is indeed the sum of 1+2+3+4. You can try replacing the summation above with multiplication, and seeing that this indeed works for recursively multiplying a long list of numbers together. Generally speaking, we use reduce operations with associative binary operations; this means that, no matter the order we perform our operation between sequential elements of the list, will always invariably give the same result, provided that the list is kept in order. (This is not to be confused with the commutative property.)

We will now see how PyCUDA handles programming patterns akin to reduce—with parallel scan and reduction kernels.