Writing Algorithms That Use the File-Reading Techniques

There are several common ways to organize information in files. The rest of this chapter will show how to apply the various file-reading techniques to these situations and how to develop some algorithms to help with this.

Skipping the Header

Many data files begin with a header. As described in ​The Readline Technique​, TSDL files begin with a one-line description followed by comments in lines beginning with a #, and the Readline technique can be used to skip that header. The technique ends when we read the first real piece of data, which will be the first line after the description that doesn’t start with a #.

In English, we might try this algorithm to process this kind of a file:

The problem with this approach is that we can’t tell whether a line is a comment line until we’ve read it, but we can read a line from a file only once—there’s no simple way to “back up” in the file. An alternative approach is to read the line, skip it if it’s a comment, and process it if it’s not. Once we’ve processed the first line of data, we process the remaining lines:

The thing to notice about this algorithm is that it processes lines in two places: once when it finds the first “interesting” line in the file and once when it handles all of the following lines:

In skip_header, we return the first line of read data, because once we’ve found it, we can’t read it again (we can go forward but not backward). We’ll want to use skip_header in all of the file-processing functions in this section. Rather than copying the code each time we want to use it, we can put the function in a file called time_series.py (for Time Series Data Library) and use it in other programs using import time_series, as shown in the next example. This allows us to reuse the skip_header code, and if it needs to be modified, then there is only one copy of the function to edit.

This program processes the Hopedale data set to find the smallest number of fox pelts produced in any year. As we progress through the file, we keep the smallest value seen so far in a variable called smallest. That variable is initially set to the value on the first line, since it’s the smallest (and only) value seen so far:

As with any algorithm, there are other ways to write this; for example, we can replace the if statement with this single line:

Dealing with Missing Values in Data

We also have data for colored fox fur production in Hebron, Labrador:

The hyphen indicates that data for the year 1836 is missing. Unfortunately, calling read_smallest on the Hebron data produces this error:

The problem is that ’-’ isn’t an integer, so calling int(’-’) fails. This isn’t an isolated problem. In general, we will often need to skip blank lines, comments, or lines containing other “nonvalues” in our data. Real data sets often contain omissions or contradictions; dealing with them is just a fact of scientific life.

For the development of this algorithm, we assume that the first value is an integer, because otherwise the time series would simply start at the second value.

To fix our code, we must add a check inside the loop that processes a line only if it contains a real value. We will assume that the first value is never a hyphen because in the TSDL data sets, missing entries are always marked with hyphens. So we just need to check for that before trying to convert the string we have read to an integer:

Notice that the update to smallest is nested inside the check for hyphens.

Processing Whitespace-Delimited Data

The file at http://robjhyndman.com/tsdldata/ecology1/lynx.dat (Time Series Data Library [Hyn06]) contains information about lynx pelts in the years 1821–1934. All data values are integers, each line contains many values, the values are separated by whitespace, and for reasons best known to the file’s author, each value ends with a period. (Note that author M. J. Campbell’s name below is misspelled in the original file.)

Now we’ll develop a program to find the largest value. To process the file, we will break each line into pieces and strip off the periods. Our algorithm is the same as it was for the fox pelt data: find and process the first line of data in the file, and then process each of the subsequent lines. However, the notion of “processing a line” needs to be examined further because there are many values per line. Our refined algorithm, shown next, uses nested loops to handle the notion of “for each line and for each value on that line”:

Once again we are processing lines in two different places. That is a strong hint that we should write a helper function to avoid duplicate code. Rewriting our algorithm and making it specific to the problem of finding the largest value makes this clearer:

The helper function required is one that finds the largest value in a line, and it must split up the line. String method split will split around the whitespace, but we still have to remove the periods at the ends of the values.

We can also simplify our code by initializing largest to -1, since that value is guaranteed to be smaller than any of the (positive) values in the file. That way, no matter what the first real value is, it’ll be larger than the “previous” value (our -1) and replace it.

We now face the same choice as with skip_header: we can put find_largest in a module (possibly time_series), or we can include it in the same file as the rest of the code. We choose the latter this time because the code is specific to this particular data set and problem:

Notice how simple the code in process_file looks! This happened only because we decided to write helper functions. To show you how much clearer this is, here is the same code without using time_series.skip_header and find_largest as helper methods: