As usual, let’s start an interactive session to explore tuples at work. Notice in Table 9-1 that tuples have no methods (e.g., an append call won’t work here). They do, however, support the usual sequence operations that we saw for strings and lists:

>>> (1, 2) + (3, 4)# Concatenation
(1, 2, 3, 4)

>>> (1, 2) * 4# Repetition
(1, 2, 1, 2, 1, 2, 1, 2)

>>> T = (1, 2, 3, 4)# Indexing, slicing
>>> T[0], T[1:3]
(1, (2, 3))

>>> x = (40)# An integer
>>> x
40
>>> y = (40,)# A tuple containing an integer
>>> y
(40,)

>>> T = ('cc', 'aa', 'dd', 'bb')
>>> tmp = list(T)# Make a list from a tuple's items
>>> tmp.sort(  )# Sort the list
>>> tmp
['aa', 'bb', 'cc', 'dd']
>>> T = tuple(tmp)# Make a tuple from the list's items
>>> T
('aa', 'bb', 'cc', 'dd')

>>> T = (1, 2, 3, 4, 5)
>>> L = [x + 20 for x in T]
>>> L
[21, 22, 23, 24, 25]

>>> T = (1, [2, 3], 4)
>>> T[1] = 'spam'# This fails: can't change tuple itself
TypeError: object doesn't support item assignment

>>> T[1][0] = 'spam'# This works: can change mutables inside
>>> T
(1, ['spam', 3], 4)

This seems to be the first question that always comes up when teaching beginners about tuples: why do we need tuples if we have lists? Some of the reasoning may be historic; Python’s creator is a mathematician by training, and has been quoted as seeing a tuple as a simple association of objects, and a list as a data structure that changes over time.

The best answer, however, seems to be that the immutability of tuples provides some integrity—you can be sure a tuple won’t be changed through another reference elsewhere in a program, but there’s no such guarantee for lists. Tuples, therefore, serve a similar role to “constant” declarations in other languages, though the notion of constant-ness is associated with objects in Python, not variables.

Tuples can also be used in places that lists cannot—for example, as dictionary keys (see the sparse matrix example in Chapter 8). Some built-in operations may also require or imply tuples, not lists. As a rule of thumb, lists are the tool of choice for ordered collections that might need to change; tuples can handle the other cases of fixed associations.

Table 9-2 summarizes common file operations. To open a file, a program calls the built-in open function, with the external name first, followed by a processing mode. The mode is typically the string 'r' to open for input (the default), 'w' to create and open for output, or 'a' to open for appending to the end; there are other options, but we’ll omit them here. For advanced roles, adding a b to the end of the mode string allows for binary data (end-of-line translations are turned off), and adding a + means the file is opened for both input and output (i.e., we can both read and write to the same object).

Both arguments to open must be Python strings, and an optional third argument can be used to control output buffering—passing a zero means that output is unbuffered (it is transferred to the external file immediately on a write method call). The external filename argument may include a platform-specific and absolute or relative directory path prefix; without a directory path, the file is assumed to exist in the current working directory (i.e., where the script runs).

Once you have a file object, you can call its methods to read from or write to the associated external file. In all cases, file text takes the form of strings in Python programs; reading a file returns its text in strings, and text is passed to the write methods as strings. Reading and writing methods come in multiple flavors; Table 9-2 lists the most common.

Though the reading and writing methods in the table are common, keep in mind that probably the best way to read lines from a text file today is not to read the file at all—as we’ll see in Chapter 13, files also have an iterator that automatically reads one line at a time in the context of a for loop, list comprehension, or other iteration context.

Notice that data read from a file always comes back to your script as a string, so you’ll have to convert it to a different type of Python object if a string is not what you need. Similarly, unlike with the print statement, Python does not convert objects to strings automatically when you write data to a file—you must send an already formatted string. Therefore, the tools we have already met to convert strings to and from numbers come in handy when dealing with files (int, float, str, and string formatting expressions, for instance). Python also includes advanced standard library tools for handling generic object storage (such as the pickle module) and for dealing with packed binary data in files (such as the struct module). We’ll see both of these at work later in this chapter.

Calling the file close method terminates your connection to the external file. As discussed in Chapter 6, in Python, an object’s memory space is automatically reclaimed as soon as the object is no longer referenced anywhere in the program. When file objects are reclaimed, Python also automatically closes the files if needed. This means you don’t always need to manually close your files, especially in simple scripts that don’t run for long. On the other hand, manual close calls can’t hurt, and are usually a good idea in larger systems. Also, strictly speaking, this auto-close-on-collection feature of files is not part of the language definition, and it may change over time. Consequently, manually issuing file close method calls is a good habit to form.

Let’s work through a simple example that demonstrates file-processing basics. It first opens a new file for output, writes a string (terminated with a newline marker, ), and closes the file. Later, the example opens the same file again in input mode and reads the line back. Notice that the second readline call returns an empty string; this is how Python file methods tell you that you’ve reached the end of the file (empty lines in the file come back as strings containing just a newline character, not as empty strings). Here’s the complete interaction:

>>> myfile = open('myfile', 'w')# Open for output (creates file)
>>> myfile.write('hello text file
')# Write a line of text
>>> myfile.close(  )# Flush output buffers to disk

>>> myfile = open('myfile')# Open for input: 'r' is default
>>> myfile.readline(  )# Read the line back
'hello text file
'
>>> myfile.readline(  )# Empty string: end of file
''

This example writes a single line of text as a string, including its end-of-line terminator, ; write methods don’t add the end-of-line character for us, so we must include it to properly terminate our line (otherwise the next write will simply extend the current line in the file).

>>> X, Y, Z = 43, 44, 45# Native Python objects
>>> S = 'Spam'# Must be strings to store in file
>>> D = {'a': 1, 'b': 2}
>>> L = [1, 2, 3]
>>>
>>> F = open('datafile.txt', 'w')# Create output file
>>> F.write(S + '
')# Terminate lines with 

>>> F.write('%s,%s,%s
' % (X, Y, Z))# Convert numbers to strings
>>> F.write(str(L) + '$' + str(D) + '
')# Convert and separate with $
>>> F.close(  )

>>> bytes = open('datafile.txt').read(  )# Raw bytes display
>>> bytes
"Spam
43,44,45
[1, 2, 3]${'a': 1, 'b': 2}
"
>>> print bytes# User-friendly display
Spam
43,44,45
[1, 2, 3]${'a': 1, 'b': 2}

>>> F = open('datafile.txt')# Open again
>>> line = F.readline(  )# Read one line
>>> line
'Spam
'
>>> line.rstrip(  )# Remove end-of-line
'Spam'

>>> line = F.readline(  )# Next line from file
>>> line# It's a string here
'43,44,45
'
>>> parts = line.split(',')# Split (parse) on commas
>>> parts
['43', '44', '45
']

>>> int(parts[1])# Convert from string to int
44
>>> numbers = [int(P) for P in parts]# Convert all in list at once
>>> numbers
[43, 44, 45]

>>> line = F.readline(  )
>>> line
"[1, 2, 3]${'a': 1, 'b': 2}
"
>>> parts = line.split('$')# Split (parse) on $
>>> parts
['[1, 2, 3]', "{'a': 1, 'b': 2}
"]
>>> eval(parts[0])# Convert to any object type
[1, 2, 3]
>>> objects = [eval(P) for P in parts]# Do same for all in list
>>> objects
[[1, 2, 3], {'a': 1, 'b': 2}]

>>> F = open('datafile.txt', 'w')
>>> import pickle
>>> pickle.dump(D, F)# Pickle any object to file
>>> F.close(  )

>>> F = open('datafile.txt')
>>> E = pickle.load(F)# Load any object from file
>>> E
{'a': 1, 'b': 2}

>>> open('datafile.txt').read(  )
"(dp0
S'a'
p1
I1
sS'b'
p2
I2
s."

>>> F = open('data.bin', 'wb')# Open binary output file
>>> import struct
>>> bytes = struct.pack('>i4sh', 7, 'spam', 8)# Make packed binary data
>>> bytes
'x00x00x00x07spamx00x08'
>>> F.write(bytes)# Write byte string
>>> F.close(  )

>>> F = open('data.bin', 'rb')
>>> data = F.read(  )# Get packed binary data
>>> data
'x00x00x00x07spamx00x08'
>>> values = struct.unpack('>i4sh', data)# Convert to Python objects
>>> values
(7, 'spam', 8)

There are additional, more advanced file methods shown in Table 9-2, and even more that are not in the table. For instance, seek resets your current position in a file (the next read or write happens at that position), flush forces buffered output to be written out to disk (by default, files are always buffered), and so on.

The Python standard library manual and the reference books described in the Preface provide complete lists of file methods; for a quick look, run a dir or help call interactively, passing in the word file (the name of the file object type). For more file-processing examples, watch for the sidebar "Why You Will Care: File Scanners" in Chapter 13. It sketches common file-scanning loop code patterns with statements we have not covered enough yet to use here.

Also, note that although the open function and the file objects it returns are your main interface to external files in a Python script, there are additional file-like tools in the Python toolset. Also available are descriptor files (integer file handles that support lower-level tools such as file locking), found in the os module; sockets, pipes, and FIFO files (file-like objects used to synchronize processes or communicate over networks); access-by-key files known as “shelves” (used to store unaltered Python objects directly, by key); and more. See more advanced Python texts for additional information on these file-like tools.

Notice that the three values in the tuple returned in the last example of the prior section represent true and false values. They print as the words True and False, but now that we’re using logical tests like this one in earnest, I should be a bit more formal about what these names really mean.

In Python, as in most programming languages, an integer 0 represents false, and an integer 1 represents true. In addition, though, Python recognizes any empty data structure as false, and any nonempty data structure as true. More generally, the notions of true and false are intrinsic properties of every object in Python—each object is either true or false, as follows:

Table 9-4 gives examples of true and false objects in Python.

Python also provides a special object called None (the last item in Table 9-4), which is always considered to be false. None was introduced in Chapter 4; it is the only value of a special data type in Python, and typically serves as an empty placeholder, much like a NULL pointer in C.

For example, recall that for lists you cannot assign to an offset unless that offset already exists (the list does not magically grow if you make an out-of-bounds assignment). To preallocate a 100-item list such that you can add to any of the 100 offsets, you can fill one with None objects:

>>> L = [None] * 100
>>>
>>> L
[None, None, None, None, None, None, None, ... ]

The Python Boolean type bool (introduced in Chapter 5) simply augments the notion of true and false in Python. You may have noticed that the results of tests in this chapter display as the words True and False. As we learned in Chapter 5, these are just customized versions of the integers 1 and 0. Because of the way this new type is implemented, this is really just a minor extension to the notions of true and false already described, designed to make truth values more explicit. When used explicitly in truth tests, the words True and False evaluate to true and false because they truly are just specialized versions of the integers 1 and 0. Results of tests also print as the words True and False, instead of 1 and 0.

You are not required to use only Boolean types in logical statements such as if; all objects are still inherently true or false, and all the Boolean concepts mentioned in this chapter still work as described if you use other types. We’ll explore Booleans further when we study logical statements in Chapter 12.

Because this is such a central concept, I’ll mention it again: you need to understand what’s going on with shared references in your program. For instance, in the following example, the list object assigned to the name L is referenced from L and from inside the list assigned to the name M. Changing L in-place changes what M references, too:

>>> L = [1, 2, 3]
>>> M = ['X', L, 'Y']# Embed a reference to L
>>> M
['X', [1, 2, 3], 'Y']

>>> L[1] = 0# Changes M too
>>> M
['X', [1, 0, 3], 'Y']

This effect usually becomes important only in larger programs, and shared references are often exactly what you want. If they’re not, you can avoid sharing objects by copying them explicitly. For lists, you can always make a top-level copy by using an empty-limits slice:

>>> L = [1, 2, 3]
>>> M = ['X', L[:], 'Y']# Embed a copy of L
>>> L[1] = 0# Changes only L, not M
>>> L
[1, 0, 3]
>>> M
['X', [1, 2, 3], 'Y']

Remember, slice limits default to 0, and the length of the sequence being sliced; if both are omitted, the slice extracts every item in the sequence, and so makes a top-level copy (a new, unshared object).

Sequence repetition is like adding a sequence to itself a number of times. That’s true, but when mutable sequences are nested, the effect might not always be what you expect. For instance, in the following example X is assigned to L repeated four times, whereas Y is assigned to a list containing L repeated four times:

>>> L = [4, 5, 6]
>>> X = L * 4# Like [4, 5, 6] + [4, 5, 6] + ...
>>> Y = [L] * 4# [L] + [L] + ... = [L, L,...]

>>> X
[4, 5, 6, 4, 5, 6, 4, 5, 6, 4, 5, 6]
>>> Y
[[4, 5, 6], [4, 5, 6], [4, 5, 6], [4, 5, 6]]

Because L was nested in the second repetition, Y winds up embedding references back to the original list assigned to L, and so is open to the same sorts of side effects noted in the last section:

>>> L[1] = 0# Impacts Y but not X
>>> X
[4, 5, 6, 4, 5, 6, 4, 5, 6, 4, 5, 6]
>>> Y
[[4, 0, 6], [4, 0, 6], [4, 0, 6], [4, 0, 6]]

The same solutions to this problem apply here as in the previous section, as this is really just another way to create the shared mutable object reference case. If you remember that repetition, concatenation, and slicing copy only the top level of their operand objects, these sorts of cases make much more sense.

We actually encountered this concept in a prior exercise: if a collection object contains a reference to itself, it’s called a cyclic object. Python prints a [...] whenever it detects a cycle in the object, rather than getting stuck in an infinite loop:

>>> L = ['grail']# Append reference to same object
>>> L.append(L)# Generates cycle in object: [...]
>>> L
['grail', [...]]

Besides understanding that the three dots in square brackets represent a cycle in the object, this case is worth knowing about because it can lead to gotchas—cyclic structures may cause code of your own to fall into unexpected loops if you don’t anticipate them. For instance, some programs keep a list or dictionary of items already visited, and check it to determine whether they’re in a cycle. See the solutions to "Part I Exercises" in Chapter 3 for more on this problem, and check out the reloadall.py program at the end of Chapter 21 for a solution.

Don’t use a cyclic reference unless you need to. There are good reasons to create cycles, but unless you have code that knows how to handle them, you probably won’t want to make your objects reference themselves very often in practice.

Finally, you can’t change an immutable object in-place. Instead, construct a new object with slicing, concatenation, and so on, and assign it back to the original reference, if needed:

T = (1, 2, 3)

T[2] = 4             # Error!

T = T[:2] + (4,)     # OK: (1, 2, 4)

That might seem like extra coding work, but the upside is that the previous gotchas can’t happen when you’re using immutable objects such as tuples and strings; because they can’t be changed in-place, they are not open to the sorts of side effects that lists are.

This session asks you to get your feet wet with built-in object fundamentals. As before, a few new ideas may pop up along the way, so be sure to flip to the answers in Appendix B when you’re done (and even when you’re not). If you have limited time, I suggest starting with exercises 10 and 11 (the most practical of the bunch), and then working from first to last as time allows. This is all fundamental material, though, so try to do as many of these as you can.