Generators are Python expressions that allow you to create functions that act as iterables; rather than returning all of the data, they yield data one "part" at a time in a memory-efficient iteration context. As our datasets get larger, it's useful to use generators to perform filtering on demand and clean data as you read it.

def dataset(path):
    with open(path, 'r') as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:
            yield row

Also, take note of the with open(path, 'r') as csvfile statement. This statement ensures that the CSV file is closed when the with block is exited, even (or especially) if there is an exception. Python with blocks replace the try, except, and finally statements, and are syntactically brief while semantically more correct programming constructs.

print min(set([int(row["Year"]) for row in dataset(data_file)]))
1875
print max(set([int(row["Year"]) for row in dataset(data_file)]))2010

In both of these previous examples, we used a Python list comprehension to generate a set. A comprehension is a concise statement that generates an iterable, much like the earlier memory-safe generators. The output variable (or variables) is specified, along with the for keyword, and the iterable to express the variable, along with an optional if condition. In Python 2.7, set and dictionary comprehensions also exist. The previous country set could also be expressed as follows:

{row["Country"] for row in dataset(data_file)}set(['Canada', 'Italy', 'France', 'Netherlands', 'Ireland', ...])

filter(lambda row: row["Country"] == "United States",
           dataset(data_file))

The Python filter function creates a list from all of the values of a sequence or iterable (the second parameter) that make the function specified by the first parameter true. In this case, we use an anonymous function (a lambda function) to check whether the value in the specified row's Country column is equal to United States.

import csv
import numpy as np
import matplotlib.pyplot as plt

def dataset(path, filter_field=None, filter_value=None):
    with open(path, 'r') as csvfile:
        reader = csv.DictReader(csvfile)
        if filter_field:
            for row in filter(lambda row: row[filter_field]==filter_value, reader):
                yield row
        else:
            for row in reader:
                yield row

def main(path):
    data  = [(row["Year"], float(row["Average income per tax unit"]))
             for row in dataset(path, "Country", "United States")]

    width = 0.35
    ind   = np.arange(len(data))
    fig   = plt.figure()
    ax    = plt.subplot(111)

    ax.bar(ind, list(d[1] for d in data))
    ax.set_xticks(np.arange(0, len(data), 4))
    ax.set_xticklabels(list(d[0] for d in data)[0::4], rotation=45)
    ax.set_ylabel("Income in USD")
    plt.title("U.S. Average Income 1913-2008")

    plt.show()

if __name__ == "__main__":
    main("income_dist.csv")

The preceding snippet will give us the following output:

The preceding example of data exploration with Python should seem familiar from many of the R chapters. Loading the dataset, filtering, and computing ranges required a few more lines of code and specific typecasting, but we quickly created analyses in a memory-safe fashion.

Have a look at the following snippet:

import numpy as np
dataset = np.recfromcsv(data_file, skip_header=1)
dataset
array([[             nan,   1.93200000e+03,              nan, ...,
                 nan,   1.65900000e+00,   2.51700000e+00],
   [             nan,   1.93300000e+03,              nan, ...,
                 nan,   1.67400000e+00,   2.48400000e+00],
   [             nan,   1.93400000e+03,              nan, ...,
                 nan,   1.65200000e+00,   2.53400000e+00],
   ..., 
   [             nan,   2.00600000e+03,   4.52600000e+01, ...,
      1.11936337e+07,   1.54600000e+00,   2.83000000e+00],
   [             nan,   2.00700000e+03,   4.55100000e+01, ...,
      1.19172976e+07,   1.53000000e+00,   2.88500000e+00],
   [             nan,   2.00800000e+03,   4.56000000e+01, ...,
      9.14119000e+06,   1.55500000e+00,   2.80300000e+00]])

The first argument to the function should be the data source. It should be either a string that points to a local or remote file or a file-like object with a read method. URLs will be downloaded to the current working directory before they are loaded. Additionally, the input can be either text or a compressed file. The function recognizes gzip and bzip2. These files need to have the .gz or .bz2 extensions to be readable. Notable optional arguments to genfromtext include the delimiter, , (comma) by default in recfromcsv; skip_header and skip_footer, which take an optional number of lines to skip from the top or bottom respectively; and dtype, which specifies the datatype of the cells. By default, the dtype is None, and NumPy will attempt to detect the correct format.

dataset.size
771720
dataset.shape
(2179, 354)

The size property on ndarray returns the number of elements in the matrix. The shape property returns a tuple of the dimensions in our array. CSVs are naturally two-dimensional, therefore the (m, n) tuple indicates the number of rows and columns, respectively.

However, there are a couple of gotchas with using this method. First, note that we had to skip our header line; genfromtxt does allow named columns by setting the keyword argument names to True (and in this case, you won't set skip_headers=1). Unfortunately, in this particular dataset, the column names might contain commas. The CSV reader deals with this correctly since the strings that contain commas are quoted, but genfromtxt is not a CSV reader in general. To fix this, either the headers have to be fixed, or some other names need to be added. Secondly, the Country column has been reduced to NaN, and the Year column has been turned into a floating point integer, which is not ideal.

names = ["country", "year"]
names.extend(["col%i" % (idx+1) for idx in xrange(352)])
dtype = "S64,i4," + ",".join(["f18" for idx in xrange(352)])

dataset = np.genfromtxt(data_file, dtype=dtype, names=names, delimiter=",", skip_header=1, autostrip=2)

We name the first two columns country and year, respectively, and assign them datatypes of S64 or string-64, then assign the year column as i4 or integer-4. For the rest of the columns, we assign them the name coln, where n is an integer from 1 to 352, and the datatype is f18 or float-18. These character lengths allow us to capture as much data as possible, including exponential floating point representations.

Unfortunately, as we look through the data, we can see a lot of nan values that represent Not a Number, a fixture in floating point arithmetic used to represent values that are not numbers nor are equivalent to infinity. Missing data is a very common issue in the data wrangling and cleaning stage of the pipeline. It appears that the dataset contains many missing or invalid entries, which makes sense given the historical data, and countries that may not have had effective data collection for given columns.