The string module is somewhat of a historical anomaly. If Python were being designed today, the string module would not exist—it is mostly a remnant of a less civilized age before everything was a first-class object. Nowadays, string objects have methods like split and join, which replace the functions that are still defined in the string module. The string module does define a convenient function, maketrans, used to automatically do string “mapping” operations with the translate method of string objects. maketrans/translate is useful when you want to translate several characters in a string at once. For example, if you want to replace all occurrences of the space character with an underscore, change underscores to minus signs, and change minus signs to plus signs. Doing so with repeated .replace( ) operations is in fact quite tricky, but doing it with maketrans is trivial:

>>> import string
>>> conversion = string.maketrans(" _-", "_-+")
>>> input_string = "This is a two_part - one_part"
>>> input_string.translate(conversion)
'This_is_a_two-part_+_one-part'

In addition, the string module defines a few useful constants, which haven’t been implemented as string attributes yet. These are shown in Table 27-2.

The constants in Table 27-2 are useful to test whether specific characters fit a criterion—for example, x in string.whitespace returns true only if x is one of the whitespace characters. Note that the values given above aren’t always the values you’ll find—for example, the definition of 'uppercase' depends on the locale: if you’re running a French operating system, string.lowercase will include ç and ê.

If strings and their methods aren’t enough (and they do get clumsy in many perfectly normal use cases), Python provides a specialized string-processing tool in the form of a regular expression engine.

Regular expressions are strings that let you define complicated pattern matching and replacement rules for strings. The syntax for regular expressions emphasizes compact notation over mnemonic value. For example, the single character . means “match any single character.” The character + means “one or more of what just preceded me.” Table 27-3 lists some of the most commonly used regular expression symbols and their meanings in English. Describing the full set of regular expression tokens and their meaning would take quite a few pages—instead, we’ll cover a simple use case and walk through how to solve the problem using regular expressions.

This is a paragraph that mentions bell peppers multiple times. For
one, here is a red pepper and dried tomato salad recipe. I don't like
to use green peppers in my salads as much because they have a harsher
flavor.

This second paragraph mentions red peppers and green peppers but not
the "s" word (s-a-l-a-d), so no bells should show up.

This third paragraph mentions red peppercorns and green peppercorns,
which aren't vegetables but spices (by the way, bell peppers really
aren't peppers, they're chilies, but would you rather have a good cook
or a good botanist prepare your salad?).

file = open('pepper.txt')
text = file.read(  )

paragraphs = text.split('

')

import re
matchstr = re.compile(
                     r"""(red|green)      # 'red' or 'green' starting new words
        (s+               # followed by whitespace
         pepper            # The word 'pepper',
         (?!corn)          # if not followed immediately by 'corn'
         (?=.*salad))""",  # and if followed at some point by 'salad',
      re.IGNORECASE |      # allow pepper, Pepper, PEPPER, etc.
      re.DOTALL |          # Allow dots to match newlines as well.
      re.VERBOSE)          # This allows the comments and the newlines above.
for paragraph in paragraphs:
    fixed_paragraph = matchstr.sub(r'bell2', paragraph)
    print fixed_paragraph+'
'

matchstr = re.compile(r"(red|green)(s+pepper(?!corn)(?=.*salad))", re.I | re.S)

fixed_paragraph = matchstr.sub(r'bell2', paragraph)

                     /home/David/book$ python pepper.py
This is a paragraph that mentions bell peppers multiple times. For
one, here is a bell pepper and dried tomato salad recipe. I don't like
to use bell peppers in my salads as much because they have a harsher
flavor.

This second paragraph mentions red peppers and green peppers but not
the "s" word (s-a-l-a-d), so no bells should show up.

This third paragraph mentions red peppercorns and green peppercorns,
which aren't vegetables but spices (by the way, bell peppers really
aren't peppers, they're chilies, but would you rather have a good cook
or a good botanist prepare your salad?).

>>> import regex
__main__:1: DeprecationWarning: the regex module is deprecated; please use the re 
module

Making copies of objects is a reasonable task in many programming contexts. Often, the only kind of copy that’s needed is just another reference to an object, as in:

x = 'tomato'
y = x                   # y is now 'tomato'.
x = x + ' and cucumber' # x is now 'tomato and cucumber', but y is unchanged.

Due to Python’s reference management scheme, the statement a = b doesn’t make a copy of the object referenced by b; instead, it makes a new reference to that same object. When the object being copied is an immutable object (e.g., a string), there is no real difference. When dealing with mutable objects like lists and dictionaries, however, sometimes a real, new copy of the object, not just a shared reference, is needed. How to do this depends on the type of the object in question. The simplest way of making a copy is to use the list( ) or tuple( ) constructors:

newList = list(myList)
newTuple = tuple(myTuple)

As opposed to the simplest, the most common way to make copies of sequences like lists and tuples is somewhat odd. If myList is a list, then to make a copy of it, you can use:

newList = myList[:]

which you can read as “slice from beginning to end,” since the default index for the start of a slice is the beginning of the sequence (0), and the default index for the end of a slice is the end of sequence (see Chapter 3). Since tuples support the same slicing operation as lists, this same technique can also be applied to tuples, except that if x is a tuple, then x[:] is the same object as x, since tuples are immutable. Dictionaries, on the other hand, don’t support slicing. To make a copy of a dictionary myDict, you can either use:

newDict = myDict.copy(  )

or the dict( ) constructor:

newDict = dict(myDict)

For a different kind of copying, if you have a dictionary oneDict, and want to update it with the contents of a different dictionary otherDict, simply type oneDict.update(otherDict). This is the equivalent of:

for key in otherDict.keys(  ):
    oneDict[key] = otherDict[key]

If oneDict shared some keys with otherDict before the update( ) operation, the old values associated with the keys in oneDict are obliterated by the update. This may be what you want to do (it usually is). If it isn’t, the right thing to do might be to raise an exception. To do this, make a copy of one dictionary, then look over each entry in the second. If we find shared keys, we raise an exception, if not, we just add the key-value mapping to the new dictionary.

def mergeWithoutOverlap(oneDict, otherDict):
    newDict = oneDict.copy(  )
    for key in otherDict:
        if key in oneDict:
            raise ValueError, "the two dictionaries share keys!"
        newDict[key] = otherDict[key]
    return newDict

or, alternatively, combine the values of the two dictionaries, with a tuple, for example. Using the same logic as in mergeWithoutOverlap, but combining the values instead of throwing an exception:

def mergeWithOverlap(oneDict, otherDict):
    newDict = oneDict.copy(  )
    for key in otherDict:
        if key in oneDict:
            newDict[key] = oneDict[key], otherDict[key]
        else:
            newDict[key] = otherDict[key]
    return newDict

To illustrate the differences between the preceding three algorithms, consider the following two dictionaries:

phoneBook1 = {'michael': '555-1212', 'mark': '554-1121', 'emily': '556-0091'}
phoneBook2 = {'latoya': '555-1255', 'emily': '667-1234'}

If phoneBook1 is possibly out of date, and phoneBook2 is more up to date but less complete, the right usage is probably phoneBook1.update(phoneBook2). If the two phoneBooks are supposed to have nonoverlapping sets of keys, using newBook = mergeWithoutOverlap(phoneBook1, phoneBook2) lets you know if that assumption is wrong. Finally, if one is a set of home phone numbers and the other a set of office phone numbers, chances are newBook = mergeWithOverlap(phoneBook1, phoneBook2) is what you want, as long as the subsequent code that uses newBook can deal with the fact that newBook['emily'] is the tuple ('556-0091', '667-1234').

>>> import copy
>>> listOne = [{"name": "Willie", "city": "Providence, RI"}, 1, "tomato", 3.0]
>>> listTwo = listOne[:]                   # Or listTwo=copy.copy(listOne)
>>> listThree = copy.deepcopy(listOne)
>>> listOne.append("kid")
>>> listOne[0]["city"] = "San Francisco, CA"
>>> print listOne, listTwo, listThree
[{'name': 'Willie', 'city': 'San Francisco, CA'}, 1, 'tomato', 3.0, 'kid']
[{'name': 'Willie', 'city': 'San Francisco, CA'}, 1, 'tomato', 3.0]
[{'name': 'Willie', 'city': 'Providence, RI'}, 1, 'tomato', 3.0]

Lists have a sort method that does an in-place sort. Sometimes you want to iterate over the sorted contents of a list, without disturbing the contents of this list. Or you may want to list the sorted contents of a tuple. Because tuples are immutable, an operation such as sort, which modifies it in place, is not allowed. The only solution is to make a list copy of the elements, sort the list copy, and work with the sorted copy, as in:

listCopy = list(myTuple)
listCopy.sort(  )
for item in listCopy:
    print item                             # Or whatever needs doing

This solution is also the way to deal with data structures that have no inherent order, such as dictionaries. One of the reasons that dictionaries are so fast is that the implementation reserves the right to change the order of the keys in the dictionary. It’s really not a problem, however, given that you can iterate over the keys of a dictionary using an intermediate copy of the keys of the dictionary:

keys = myDict.keys(  )                          # Returns an unsorted list of
                                           # the keys in the dict.
keys.sort(  )
for key in keys:                           # Print key/value pairs 
    print key, myDict[key]                 # sorted by key.

The sort method on lists uses the standard Python comparison scheme. Sometimes, however, that scheme isn’t what’s needed, and you need to sort according to some other procedure. For example, when sorting a list of words, case (lower versus UPPER) may not be significant. The standard comparison of text strings, however, says that all uppercase letters come before all lowercase letters, so 'Baby' is less than 'apple', but 'baby' is greater than 'apple‘. In order to do a case-independent sort, you need to define a comparison function that takes two arguments, and returns -1, 0, or 1 depending on whether the first argument is smaller than, equal to, or greater than the second argument. So, for case-independent sorting, you can use:

>>> def caseIndependentSort(something, other):
...    something, other = something.lower(  ), other.lower(  )
...    return cmp(something, other)
... 
>>> testList = ['this', 'is', 'A', 'sorted', 'List']
>>> testList.sort(  )
>>> print testList
['A', 'List', 'is', 'sorted', 'this']
>>> testList.sort(caseIndependentSort)
>>> print testList
['A', 'is', 'List', 'sorted', 'this']

We’re using the built-in function cmp, which does the hard part of figuring out that 'a' comes before 'b', 'b' before 'c', etc. Our sort function simply converts both items to lowercase and compares the lowercase versions. Also note that the conversion to lowercase is local to the comparison function, so the elements in the list aren’t modified by the sort.

What about randomizing a sequence, such as a list of lines? The easiest way to randomize a sequence is to call the shuffle function in the random module, which randomizes a sequence in-place:^[5]

random.shuffle(myList)

If you need to shuffle a nonlist object, it’s usually easiest to convert that object to a list and shuffle the list version of the same data, rather than come up with a new strategy for each data type. This might seem a wasteful strategy, given that it involves building intermediate lists that might be quite large. In general, however, what seems large to you probably won’t seem so to the computer, thanks to the reference system. Also, consider the time saved by not having to come up with a different strategy for each data type! Python is designed to save programmer time; if that means running a slightly slower or bigger program, so be it. If you’re handling enormous amounts of data, it may be worthwhile to optimize. But never optimize until the need for optimization is clear; that would be a waste of your time.

This chapter emphasizes the silliness involved in reinventing wheels. This point is especially important when it comes to data structures. For example, Python lists and dictionaries might not be the lists and dictionaries or mappings you’re used to, but you should avoid designing your own data structure if these structures will suffice. The algorithms they use have been tested under wide ranges of conditions, and they’re fast and stable. Sometimes, however, the interface to these algorithms isn’t convenient for a particular task.

For example, computer science textbooks often describe algorithms in terms of other data structures such as queues and stacks. To use these algorithms, it may make sense to come up with a data structure that has the same methods as these data structures (such as pop and push for stacks or enqueue/dequeue for queues). However, it also makes sense to reuse the built-in list type in the implementation of a stack. In other words, you need something that acts like a stack but is based on a list. A simple solution is to use a class wrapper around a list. For a minimal stack implementation, you can do this:

class Stack:
    def __init__(self, data):
        self._data = list(data)
        self.push = self._data.append
        self.pop = self._data.pop

The following is simple to write, to understand, to read, and to use:

>>> thingsToDo = Stack(['write to mom', 'invite friend over', 'wash the kid'])
>>> thingsToDo.push('do the dishes')
>>> print thingsToDo.pop(  )
do the dishes
>>> print thingsToDo.pop(  )
wash the kid

Two standard Python naming conventions are used in the Stack class above. The first is that class names start with an uppercase letter, to distinguish them from functions. The other is that the _data attribute starts with an underscore. This is a half-way point between public attributes (which don’t start with an underscore), private attributes (which start with two underscores; see Chapter 7), and Python-reserved identifiers (which both start and end with two underscores). What it means is that _data is an attribute of the class that shouldn’t be needed by clients of the class. The class designer expects such pseudo-private attributes to be used only by the class methods and by the methods of any eventual subclass.

The Stack class presented earlier does its minimal job just fine. It assumes a fairly minimal definition of what a stack is, specifically, something that supports just two operations, a push and a pop. Some of the features of lists would be nice to use, such as the ability to iterate over all the elements using the for . . . in . . . construct. While you could continue in the style of the previous class and delegate to the “inner” list object, at some point it makes more sense to simply reuse the implementation of list objects directly, through subclassing. In this case, you should derive a class from the list base class. The dict base class can also be used to create dictionary-like classes.

# Subclass the list class.
class Stack(list):
    push = list.append

This Stack is a subclass of the list class. The list class implements the pop methods among others. You don’t need to define your own __init__ method because list defines a perfectly good default. The push method is defined just by saying that it’s the same as list’s append method. Now we can do list-like things as well as stack-like things:

>>> thingsToDo = Stack(['write to mom', 'invite friend over', 'wash the kid'])
>>> print thingsToDo                  # Inherited from list base class
['write to mom', 'invite friend over', 'wash the kid']
>>> thingsToDo.pop(  )        
'wash the kid'
>>> thingsToDo.push('change the oil')
>>> for chore in thingsToDo:          # We can also iterate over the contents.
...    print chore 
...
write to mom
invite friend over
change the oil

The os module provides a generic interface to the operating system’s most basic set of tools. Different operating systems have different behaviors. This is true at the programming interface as well. This makes it hard to write so-called “portable” programs, which run well regardless of the operating system. Having generic interfaces independent of the operating system helps, as does using an interpreted language like Python. The specific set of calls it defines depend on which platform you use. (For example, the permission-related calls are available only on platforms that support them, such as Unix and Windows.) Nevertheless, it’s recommended that you always use the os module, instead of the platform-specific versions of the module (called by such names as posix, nt, and mac). Table 27-4 lists some of the most often used functions in the os module. When referring to files in the context of the os module, one is referring to filenames, not file objects.

>>> print os.getcwd(  )
h:Davidook

>>> os.listdir(os.getcwd(  ))
['preface.doc', 'part1.doc', 'part2.doc']

>>> os.mkdir('newdir')

>>> os.makedirs('newdir/newsubdir/newsubsubdir')

>>> os.stat('TODO.txt')
                                  
# It returns something like a tuple.
(33206, 0L, 3, 1, 0, 0, 1753L, 1042186004, 
1042186004, 1042175785)
>>> os.stat('TODO.txt').st_size
                                  
# Just look at the size.
1753L
>>> time.asctime(time.localtime
              (os.stat('TODO.txt').st_mtime))
'Fri Jan 10 00:06:44 2003'

dirpath, dirnames, filenames

With just these modules, you can find out a lot about the current state of the filesystem, as well as modify it:

>>> print os.getcwd(  )        # Where am I?
C:Python22
>>> print os.listdir('.')    # What's here?
['DLLs', 'Doc', 'include', 'Lib', 'libs', 'License.txt', ...]
>>> os.chdir('Lib')          # Let's go explore the library.
>>> print os.listdir('.')    # What's here?
['aifc.py', 'anydbm.py', 'anydbm.pyc', 'asynchat.py',
'asyncore.py', 'atexit.py', 'atexit.pyc', 'atexit.pyo',
'audiodev.py', 'base64.py', ...]
>>> os.remove('atexit.pyc')  # We can remove .pyc files safely.
>>>

There are many other functions in the os module; in fact, just about any function that’s part of the POSIX standard and widely available on most Unix and Unix-like platforms is supported by Python on Unix. The interfaces to these routines follow the POSIX conventions. You can retrieve and set UIDs, PIDs, and process groups; control nice levels; create pipes; manipulate file descriptors; fork processes; wait for child processes; send signals to processes; use the execv variants; etc (if you don’t know what half of the words in this paragraph mean, don’t worry, you probably don’t need to).

The os module also defines some important attributes that aren’t functions:

The os module also includes a set of strings that define portable ways to refer to directory-related parts of filename syntax, as shown in Table 27-5.

These strings are used by the functions in the os.path module, which manipulate file paths in portable ways (see Table 27-6). Note that the os.path module is an attribute of the os module, not a sub-module of an os package; it’s imported automatically when the os module is loaded, and (unlike packages) you don’t need to import it explicitly. The outputs of the examples in Table 27-6 correspond to code run on a Windows or DOS machine. On another platform, the appropriate path separators would be used instead. A useful relevant bit of knowledge is that the forward slash (/) can be used safely in Windows to indicate directory traversal, even though the native separator is the backwards slash ()—Python and Windows both do the right thing with it.

>>> os.path.split("h:/David/book/part2.doc"
('h:/David/book', 'part2.doc')

>>> os.path.splitdrive(r"C:fooar.txt")
('C:', '\foo\bar.txt')

>>> os.path.splitext(r"C:fooar.txt")
('C:\foo\bar', '.txt')

>>> os.path.splitunc(r"\machinemountdirectory
                      file.txt")
('\\machine\mount', '\directory\file.txt')

>>> print os.path.join(os.getcwd(  ),
                                 ... os.pardir, 'backup', 'part2.doc')
h:Davidook..ackuppart2.doc

>>> print os.path.expanduser('~/mydir')
h:Davidmydir

>>> print os.path.expandvars('$TMP')
C:TEMP

>>> print os.path.normpath("/foo/bar\../tmp")
foo	mp

>>> print os.path.normcase(r'c:/fooBAR.txt')
c:fooar.txt

>>> def test_walk(arg, dirname, names):
                                 ...     print arg, dirname, names
                                 ...
>>> os.path.walk('..', test_walk, 'show')
show ..logs ['errors.log', 'access.log']
show ..cgi-bin ['test.cgi']
...

The keen-eyed reader might have noticed that the os module, while it provides lots of file-related functions, doesn’t include a copy function. In DOS, copying a file is basically the same thing as opening one file in read/binary mode, reading all its data, opening a second file in write/binary mode, and writing the data to the second file. On Unix and Windows, making that kind of copy fails to copy the stat bits (permissions, modification times, etc.) associated with the file. On the Mac, that operation won’t copy the resource fork, which contains data such as icons and dialog boxes. In other words, copying files is just more complicated than one could reasonably believe. Nevertheless, often you can get away with a fairly simple function that works on Windows, DOS, Unix, and Mac, as long as you’re manipulating just data files with no resource forks. That function, called copyfile, lives in the shutil module. This module includes a few generally useful functions, shown in Table 27-7.

While the previous section lists common functions for working with files, many tasks require more than a single function call.

Let’s take a typical example: you have lots of files, all of which have a space in their name, and you’d like to replace the spaces with underscores. All you need is the os.curdir attribute (which returns an operating-system specific string that corresponds to the current directory), the os.listdir function (which returns the list of filenames in a specified directory), and the os.rename function:

import os
if len(sys.argv) == 1:                     # If no filenames are specified,
    filenames = os.listdir(os.curdir)      # use current dir;
else:                                      # otherwise, use files specified
    filenames = sys.argv[1:]               # on the command line.
for filename in filenames:
    if ' ' in filename:
        newfilename = filename.replace(' ', '_')
        print "Renaming", filename, "to", newfilename, "..."
        os.rename(filename, newfilename)

This program works fine, but it reveals a certain Unix-centrism. That is, if you call it with wildcards, such as:

python despacify.py *.txt

you find that on Unix machines, it renames all the files with names with spaces in them and that end with .txt. In a DOS-style shell, however, this won’t work because the shell normally used in DOS and Windows doesn’t convert from *.txt to the list of filenames; it expects the program to do it. This is called globbing, because the * is said to match a glob of characters. Luckily, Python helps us make the code portable.

The glob module exports a single function, also called glob, which takes a filename pattern and returns a list of all the filenames that match that pattern (in the current working directory):

import sys, glob
print sys.argv[1:]
sys.argv = [item for arg in sys.argv for item in glob.glob(arg)]
print sys.argv[1:]

Running this on Unix and DOS shows that on Unix, the Python glob didn’t do anything because the globbing was done by the Unix shell before Python was invoked, and in DOS, Python’s globbing came up with the same answer:

/usr/python/book$ python showglob.py *.py
['countlines.py', 'mygrep.py', 'retest.py', 'showglob.py', 'testglob.py']
['countlines.py', 'mygrep.py', 'retest.py', 'showglob.py', 'testglob.py']

C:pythonook> python showglob.py *.py
['*.py']
['countlines.py', 'mygrep.py', 'retest.py', 'showglob.py', 'testglob.py']

It’s worth looking at the bold line in showglob.py and understanding exactly what happens there, especially if you’re new to the list comprehension concept (discussed in Chapter 14).

If you’ve ever written a shell script and needed to use intermediary files for storing the results of some intermediate stages of processing, you probably suffered from directory litter. You started out with 20 files called log_001.txt, log_002.txt, etc., and all you wanted was one summary file called log_sum.txt. In addition, you had a whole bunch of log_001.tmp, log_001.tm2, etc. files that, while they were labeled temporary, stuck around. To put order back into your directories, use temporary files in specific directories and clean them up afterwards.

To help in this temporary file management problem, Python provides a nice little module called tempfile that publishes two functions: mktemp( ) and TemporaryFile( ). The former returns the name of a file not currently in use in a directory on your computer reserved for temporary files (such as /tmp on Unix or C:TEMP on Windows). The latter returns a new file object directly. For example:

# Read input file
inputFile = open('input.txt', 'r')

import tempfile
# Create temporary file
tempFile = tempfile.TemporaryFile(  )                   # We don't even need to 
first_process(input = inputFile, output = tempFile)   # know the filename...

# Create final output file
outputFile = open('output.txt', 'w')
second_process(input = tempFile, output = outputFile)

Using tempfile.TemporaryFile( ) works well in cases where the intermediate steps manipulate file objects. One of its nice features is that when the file object is deleted, it automatically deletes the file it created on disk, thus cleaning up after itself. One important use of temporary files, however, is in conjunction with the os.system call, which means using a shell, hence using filenames, not file objects. For example, let’s look at a program that creates form letters and mails them to a list of email addresses (on Unix only):

formletter = """Dear %s,
I'm writing to you to suggest that ..."""    # etc. 
myDatabase = [('Michael Jackson', '[email protected]'),
              ('Bill Gates', '[email protected]'),
              ('Bob', '[email protected]')]
for name, email in myDatabase:
    specificLetter = formletter % name
    tempfilename = tempfile.mktemp(  )
    tempfile = open(tempfilename, 'w')
    tempfile.write(specificLetter)
    tempfile.close(  )
    os.system('/usr/bin/mail %(email)s -s "Urgent!" < %(tempfilename)s' % vars(  )) 
    os.remove(tempfilename)

The first line in the for loop returns a customized version of the form letter based on the name it’s given. That text is then written to a temporary file that’s emailed to the appropriate email address using the os.system call. Finally, to clean up, the temporary file is removed.

The vars( ) function is a built-in function that returns a dictionary corresponding to the variables defined in the current local namespace. The keys of the dictionary are the variable names, and the values of the dictionary are the variable values. vars( ) comes in quite handy for exploring namespaces. It can also be called with an object as an argument (such as a module, a class, or an instance), and it will return the namespace of that object. Two other built-ins, locals( ) and globals( ), return the local and global namespaces, respectively. In all three cases, modifying the returned dictionaries doesn’t guarantee any effect on the namespace in question, so view these as read-only and you won’t be surprised. You can see that the vars( ) call creates a dictionary that is used by the string interpolation mechanism; it’s thus important that the names inside the %(...)s bits in the string match the variable names in the program.

The argv attribute of the sys module holds one set of inputs to the current program—the command-line arguments, more precisely a list of the words input on the command line, excluding the reference to Python itself if it exists. In other words, if you type at the shell:

csh> python run.py a x=3 foo

then when run.py starts, the value of the sys.argv attribute is ['run.py', 'a', 'x=3', 'foo']. The sys.argv attribute is mutable (after all, it’s just a list). Common usage involves iterating over the arguments of the Python program, that is, sys.argv[1:]; slicing from index 1 till the end gives all of the arguments to the program itself, but doesn’t include the name of the program (module) stored in sys.argv[0]. There are two modules that help you process command line options. The first, an older module called getopt, is replaced in Python 2.3 by a similar but more powerful module called optparse. Check the library reference for further details on how to use them.

Experienced programmers will know that there are other inputs to a program, especially the standard input stream, with siblings for output and error messages. Python lets the programmer access and modify these through three file attributes in the sys module: sys.stdin, sys.stdout, and sys.stderr. Standard input is generally associated by the operating system with the user’s keyboard; standard output and standard error are usually associated with the console. The print statement in Python outputs to standard output (sys.stdout), while error messages such as exceptions are output on the standard error stream (sys.stderr). Python lets you modify these on the fly: you can redirect the output of a Python program to a file simply by assigning to sys.stdout:

sys.stdout = open('log.out', 'w')

After this line, any output will be written to the file log.out instead of showing up on the console. Note that if you don’t save it first, the reference to the “original” standard out stream is lost. It’s generally a good idea to save a reference before reallocating any of the standard streams, as in:

old_stdout = sys.stdout
sys.stdout = open('log.out', 'w')

Why have a standard input stream? After all, it’s not that hard to type open('input.txt') in the program. The major argument for reading and writing with standard streams is that you can chain programs so that the standard output from one becomes the standard input of the next, with no file used in the transfer. This facility, known as piping , is at the heart of the Unix philosophy. Using standard I/O this way means that you can write a program to do a specific task once, and then use it to process files or the intermediate results of other programs at any time in the future. As an example, a simple program that counts the number of lines in a file could be written as:

import sys
data = sys.stdin.readlines(  )
print "Counted", len(data), "lines."

On Unix, you could test it by doing something like:

% cat countlines.py | python countlines.py 
Counted 3 lines.

On Windows or DOS, you’d do:

C:> type countlines.py | python countlines.py 
Counted 3 lines.

You can get each line in a file simply by iterating over a file object. This comes in very handy when implementing simple filter operations. Here are a few examples of such filter operations.

# Show comment lines (lines that start with a #, like this one).
import sys
for line in sys.stdin:
    if line[0] == '#':
        print line,

C:> type commentfinder.py | python commentfinder.py | python countlines.py
Counted 1 lines.

import sys
for line in sys.stdin:
    words = line.split(  ) 
    if len(words) >= 4:
        print words[3]

try:
    print words[3]
except IndexError:                     # There aren't enough words.
    pass

import sys, string
for line in sys.stdin:
    words = line.split(':') 
    if len(words) >= 4:
        print words[3].lower(  )

import sys
lines = sys.stdin.readlines(  )
sys.stdout.writelines(lines[:10])          # First 10 lines
sys.stdout.writelines(lines[-10:])         # Last 10 lines
for lineIndex in range(0, len(lines), 2):  # Get 0, 2, 4, ...
    sys.stdout.write(lines[lineIndex])     # Get the indexed line.

text = open(fname).read(  )
print text.count('Python')

Name:   Willie   Mark   Guido   Mary  Rachel   Ahmed
Level:    5       4      3       1     6        4
Tag#:    1234   4451   5515    5124   1881    5132

Name:  Level:  Tag#:
Willie 5       1234
Mark   4       4451
...

import sys
lines = sys.stdin.readlines(  )
wordlists = [line.split(  ) for line in lines]
for row in zip(*wordlists):
    print '	'.join(row)

# Read character by character.
while 1:
    next = sys.stdin.read(1)            # Read a one-character string
    if not next:                        # or an empty string at EOF.
        break
    # Process character 'next'.

# Read line by line.
while 1:
    next = sys.stdin.readline(  )            # Read a one-line string
    if not next:                        # or an empty string at EOF.
        break
    # Process line 'next'.

Being able to read stdin is a great feature; it’s the foundation of the Unix toolset. However, one input is not always enough: many tasks need to be performed on sets of files. This is usually done by having the Python program parse the list of arguments sent to the script as command-line options. For example, if you type:

% python myScript.py input1.txt input2.txt input3.txt output.txt

you might think that myScript.py wants to do something with the first three input files and write a new file, called output.txt. Let’s see what the beginning of such a program could look like:

import sys
inputfilenames, outputfilename = sys.argv[1:-1], sys.argv[-1]
for inputfilename in inputfilenames:
    inputfile = open(inputfilename, "r")
    do_something_with_input(inputfile)
inputfile.close(  )
outputfile = open(outputfilename, "w")
write_results(outputfile)
outputfile.close(  )

The second line extracts parts of the argv attribute of the sys module. Recall that it’s a list of the words on the command line that called the current program. It starts with the name of the script. So, in the example above, the value of sys.argv is:

['myScript.py', 'input1.txt', 'input2.txt', 'input3.txt', 'output.txt'].

The script assumes that the command line consists of one or more input files and one output file. So the slicing of the input file names starts at 1 (to skip the name of the script, which isn’t an input to the script in most cases), and stops before the last word on the command line, which is the name of the output file. The rest of the script should be pretty easy to understand (but won’t work until you provide the do_something_with_input( ) and write_results( ) functions).

Note that the preceding script doesn’t actually read in the data from the files, but passes the file object down to a function to do the real work. A generic version of do_something_with_input( ) is:

def do_something_with_input(inputfile):
    for line in inputfile:
        process(line)

The combination of this idiom with the preceding one regarding opening each file in the sys.argv[1:] list is so common that there is a module, fileinput, to do just this task:

import fileinput
for line in fileinput.input(  ):
    process(line)

The fileinput.input( ) call parses the arguments on the command line, and if there are no arguments to the script, uses sys.stdin instead. It also provides several useful functions that let you know which file and line number you’re currently manipulating, as we can see in the following script:

import fileinput, sys
# Take the first argument out of sys.argv and assign it to searchterm.
searchterm, sys.argv[1:] = sys.argv[1], sys.argv[2:]
for line in fileinput.input(  ):
   num_matches = line.count(searchterm)
   if num_matches:                     # A nonzero count means there was a match.
       print "found '%s' %d times in %s on line %d." % (searchterm, num_matches, 
           fileinput.filename(  ), fileinput.filelineno(  ))

Running mygrep.py on a few Python files produces:

% python mygrep.py in *.py
found 'in' 2 times in countlines.py on line 2.
found 'in' 2 times in countlines.py on line 3.
found 'in' 2 times in mygrep.py on line 1.
found 'in' 4 times in mygrep.py on line 4.
found 'in' 2 times in mygrep.py on line 5.
found 'in' 2 times in mygrep.py on line 7.
found 'in' 3 times in mygrep.py on line 8.
found 'in' 3 times in mygrep.py on line 12.

A file is considered a binary file if it’s not a text file or a file written in a format based on text, such as HTML and XML. Image and sound files are prototypical examples of binary files. A frequent question about file manipulation is “How do I process binary files in Python?” The answer to that question usually involves the struct module. It has a simple interface, since it exports just three functions: pack, unpack, and calcsize.

Let’s start with the task of decoding a binary file. Imagine a binary file bindat.dat that contains data in a specific format: first there’s a float corresponding to a version number, then a long integer corresponding to the size of the data, and then the number of unsigned bytes corresponding to the actual data. The key to using the struct module is to define a format string, which corresponds to the format of the data you wish to read, and find out which subset of the file corresponds to that data. For example:

import struct
data = open('bindat.dat').read(  )
start, stop = 0, struct.calcsize('fl')
version_number, num_bytes = struct.unpack('fl', data[start:stop])
start, stop = stop, start + struct.calcsize('B'*num_bytes)
bytes = struct.unpack('B'*num_bytes, data[start:stop])

'f' is a format string for a single floating-point number (a C float, to be precise), 'l' is for a long integer, and 'B' is a format string for an unsigned char. The available unpack format strings are listed in Table 27-8. Consult the library reference manual for usage details.

At this point, bytes is a tuple of num_bytes Python integers. If we know that the data is in fact storing characters, we could use chars = map(chr, bytes). To be more efficient, we could change the last unpack to use 'c' instead of 'B', which would do the conversion and return a tuple of num_bytes single-character strings. More efficiently still, we could use a format string that specifies a string of characters of a specified length, such as:

chars = struct.unpack(str(num_bytes)+'s', data[start:stop])

The packing operation (struct.pack) is the exact converse; instead of taking a format string and a data string, and returning a tuple of unpacked values, it takes a format string and a variable number of arguments and packs those arguments using that format string into a new packed string.

Note that the struct module can process data that’s encoded with either kind of byte-ordering,^[6] thus allowing you to write platform-independent binary file manipulation code. For large files, also consider using the array module.

Python programs often process forms from web pages. To make this task easy, the standard Python distribution includes a module called cgi. Chapter 28 includes an example of a Python script that uses the CGI, so we won’t cover it any further here.

Universal resource locators are strings such as http://www.python.org that are now ubiquitous. Three modules—urllib, urllib2, and urlparse—provide tools for processing URLs.

The urllib module defines a few functions for writing programs that must be active users of the Web (robots, agents, etc.). These are listed in Table 27-9.

>>> page = urlopen('http://www.python.org')
>>> page.readline(  )
'<HTML>12'
>>> page.readline(  )
'<!-- THIS PAGE IS AUTOMATICALLY GENERATED.DO NOT EDIT. -->12'

>>> urllib.urlretrieve('http://www.python.org/', 
'wwwpython.html')

>>> quote('this & that @ home')
'this%20%26%20that%20%40%20home'

>>> unquote('this%20%26%20that%20%40%20home')
'this & that @ home'

>>> locals(  )
{'urllib': <module 'urllib'>, '__doc__': None, 'x':
3, '__name__': '__main__', '__builtins__': <module
'__builtin__'>}
>>> urllib.urlencode(locals(  ))
'urllib=%3cmodule+%27urllib%27%3e&__doc__=None&x=3&
__name__=__main__&__builtins__=%3cmodule+%27
__builtin__%27%3e'

The module urllib2 focuses on the tasks of opening URLs that the simpler urllib doesn’t know how to deal with, and provides an extensible framework for new kinds of URLs and protocols. It is what you should use if you want to deal with passwords, digest authentication, proxies, HTTPS URLs, and other fancy URLs.

The module urlparse defines a few functions that simplify taking URLs apart and putting new URLs together. These are listed in Table 27-10.

>>> urlparse('http://www.python.org/
FAQ.html')
('http', 'www.python.org', '/FAQ.html', '', '', '')

>>> urljoin('http://www.python.org', 
'doc/lib')
'http://www.python.org/doc/lib'

The most commonly used protocols built on top of TCP/IP are supported with modules named after them. The telnetlib module lets you act like a Telnet client. The httplib module lets you talk to web servers with the HTTP protocol. The ftplib module is for transferring files using the FTP protocol. The gopherlib module is for browsing Gopher servers (now fairly rare). In the domains of mail and news, you can use the poplib and imaplib modules for reading mail files on POP3 and IMAP servers, respectively and the smtplib module for sending mail, and the nntplib module for reading and posting Usenet news from NNTP servers.

There are also modules that can build Internet servers, specifically a generic socket-based IP server (SocketServer), a simple web server (SimpleHTTPServer), a CGI-compliant HTTP server (CGIHTTPSserver), and a module for building asynchronous socket handling services (asyncore).

Support for web services currently consists of a core library to process XML-RPC client-side calls (xmlrpclib), as well as a simple XML-RPC server implementation (SimpleXMLRPCServer). Support for SOAP is likely to be added when the SOAP standard becomes more stable.

Once you use an Internet protocol to obtain files from the Internet (or before you serve them to the Internet), you often must process these files. They come in many different formats. Table 27-11 lists each module in the standard library that processes a specific kind of Internet-related file format (there are others for sound and image format processing; see the library reference manual).

Python comes with a rich set of XML-processing tools. These include parsers, DOM interfaces, SAX interfaces, and more, as shown in Table 27-12.

See the standard library reference for details, or the Python Cookbook (O’Reilly) for example tasks easily solved using the standard XML libraries. The XML facilities are developed by the XML Special Interest Group, which publishes versions of the XML package in-between Python releases. See http://www.python.org/topics/xml for details and the latest version of the code. For expanded coverage, consider Python and XML, by Christopher A. Jones and Fred L. Drake, Jr. (O’Reilly).

The first task is, not surprisingly, debugging. Python’s standard distribution includes a debugger called pdb. Using pdb is fairly straightforward. You import the pdb module and call its run method with the Python code the debugger should execute. For example, if you’re debugging the program in spam.py, do this:

>>> import spam                        # Import the module we want to debug.
>>> import pdb                         # Import pdb.
>>> pdb.run('instance = spam.Spam(  )') # Start pdb with a statement to run.
> <string>(0)?(  )
(Pdb) break spam.Spam.__init__                # We can set break points.
(Pdb) next
>        <string>(1)?(  )
(Pdb) n                                        # 'n' is short for 'next'.
> spam.py(3)__init__(  )
-> def __init__(self):
(Pdb) n
> spam.py(4)__init__(  )
-> Spam.numInstances = Spam.numInstances + 1
(Pdb) list                                     # Show the source code listing.
  1    class Spam:
  2        numInstances = 0
  3 B      def __init__(self):                  # Note the B for Breakpoint.
  4  ->        Spam.numInstances = Spam.numInstances + 1  # Where we are
  5        def printNumInstances(self):
  6            print "Number of instances created: ", Spam.numInstances
  7
[EOF]
(Pdb) where                                    # Show the calling stack.
<string>(1)?(  )
> spam.py(4)__init__(  )
-> Spam.numInstances = Spam.numInstances + 1
(Pdb) Spam.numInstances = 10          # Note that we can modify variables
(Pdb) print Spam.numInstances         # while the program is being debugged.
10
(Pdb) continue                        # This continues until the next break-
--Return--                            # point, but there is none, so we're
-> <string>(1)?(  )->None                 # done.
(Pdb) c                               # This ends up quitting Pdb.
<spam.Spam instance at 80ee60>        # This is the returned instance.
>>> instance.numInstances             # Note that the change to numInstance
11                                    # was before the increment op.

As the session above shows, with pdb you can list the current code being debugged (with an arrow pointing to the line about to be executed), examine variables, modify variables, and set breakpoints. Chapter 9 in the Library Reference covers the debugger in detail. Alternative debuggers abound, from the one in IDLE, to the more full-featured debuggers you’ll find in commercial IDEs for Python.

Testing software is, in the general case, a very hard problem. For software that takes user input or more generally interacts with the outside world, doing comprehensive testing of any medium-sized program quickly becomes hard to do completely. Luckily, one can get many benefits from doing nonexhaustive testing. The easiest kind of testing to do is called unit testing, and it is supported in Python by the module unittest. In unit testing, one writes very small scripts that test one fact about the program being tested at a time. The trick is to write lots of these simple tests, learn how to write useful unit tests as opposed to silly tests, and to run these tests in between every change to the program. If you have a test suite with good coverage, you’ll gain confidence that each change you make is not going to break another part of the system.

unittest is documented as part of the standard library, as well as on the PyUnit web site (http://pyunit.sourceforge.net).

Even when a program is working, it can sometimes be too slow. If you know what the bottleneck in your program is, and you know of alternative ways to code the same algorithm, then you might time the various alternative methods to find out which is fastest. The time module, which is part of the standard distribution, provides many time-manipulation routines. We’ll use just one, which returns the time since a fixed epoch with the highest precision available on your machine. As we’ll use just relative times to compare algorithms, the precision isn’t all that important. Here’s two different ways to create a list of 10,000 zeros:

def lots_of_appends(  ):
    zeros = [  ]
    for i in range(10000):
        zeros.append(0)

def one_multiply(  ):
    zeros = [0] * 10000

How can we time these two solutions? Here’s a simple way:

import time, makezeros
def do_timing(num_times, *funcs):
    totals = {  }
    for func in funcs:
        totals[func] = 0.0
        starttime = time.clock(  )        # Record starting time.
        for x in range(num_times):
            for func in funcs:
                apply(func)
        stoptime = time.clock (  )         # Record ending time.
        elapsed = stoptime--starttime       # Difference yields time elapsed
        totals[func] = totals[func] + elapsed
    for func in funcs:
        print "Running %s %d times took %.3f seconds" % (func.__name__, num_times 
totals[func])

do_timing(100, (makezeros.lots_of_appends, makezeros.one_multiply))

And running this program yields:

csh> python timings.py
Running lots_of_appends 100 times took 7.891 seconds
Running one_multiply 100 times took 0.120 seconds

As you might have suspected, a single list multiplication is much faster than lots of appends. Note that in timings, it’s always a good idea to compare lots of runs of functions instead of just one. Otherwise, the timings are likely to be heavily influenced by things that have nothing to do with the algorithm, such as network traffic on the computer or GUI events. Python 2.3 introduces a new module called timeit that provides a very simple way to do precise code timing correctly.

What if you’ve written a complex program, and it’s running slower than you’d like, but you’re not sure where the problem spot is? In that case, what you need to do is profile the program: determine which parts of the program are the time-sinks and see if they can be optimized, or if the program structure can be modified to even out the bottlenecks. The Python distribution includes just the right tools for that, the profile module, documented in the Library Reference, and another module, hotshot, which is unfortunately not well documented as of this writing. Assuming that you want to profile a given function in the current namespace, do this:

>>> from timings import *
>>> from makezeros import *
>>> profile.run('do_timing(100, (lots_of_appends, one_multiply))')
Running lots_of_appends 100 times took 8.773 seconds
Running one_multiply 100 times took 0.090 seconds
203 function calls in 8.823 CPU seconds
Ordered by: standard name
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      100   8.574   0.086   8.574  0.086 makezeros.py:1(lots_of_appends)
      100   0.101   0.001   0.101  0.001 makezeros.py:6(one_multiply)
        1   0.001   0.001   8.823  8.823 profile:0(do_timing(100, 
(lots_of_appends, one_multiply)))
        0   0.000           0.000        profile:0(profiler)
        1   0.000   0.000   8.821  8.821 python:0(194.C.2)
        1   0.147   0.147   8.821  8.821 timings.py:2(do_timing)

As you can see, this gives a fairly complicated listing, which includes such things as per-call time spent in each function and the number of calls made to each function. In complex programs, the profiler can help find surprising inefficiencies. Optimizing Python programs is beyond the scope of this book; if you’re interested, however, check the Python newsgroup: periodically, a user asks for help speeding up a program and a spontaneous contest starts up, with interesting advice from expert users.