Start by creating a file with some simple text. To create a new file on your system, create a file object, and tell Python you want to write to it. A file object represents a connection to a file, not the file itself, but if you open a file for writing that doesn't exist, Python creates the file automatically. Enter the following:

>>> def make_text_file():
  a=open('test.txt',"w")
  a.write("This is how you create a new text file")
 a.close()

You start off by creating a new function called make_text_file(). You then tell Python to open a file named test.txt. Because Python does not find this file, it creates it for you. (Note: if the file did exist, Python would have deleted it and created a new one, so be careful when using this technique! In a moment, you learn to check to see if a file exists prior to creating one.) The "w" argument tells Python that you intend to write to the file; without it, Python would assume you intend to read from the file and would raise an exception when it found that the file didn't exist. Next, you add a line of text to the file, namely: "This is how you create a new text file".

Take a moment to navigate to your Python directory, which should be located somewhere such as C://Python31. You will notice that a new file named test.txt has been created. If you double-click it, you will see the text you added in the preceding example. Congratulations, you have created your first file!

Now that you have created a file with the preceding technique, create a program that first checks to see if the file name exists; if so, it will give you an error message; if not, it will create the file. Type in the following code:

>>> import os
>>> def make_another_file():
  if os.path.isfile('test.txt'):
    print("You are trying to create a file that already exists!")
  else:
    f=open('test.txt',"w")
    f.write("This is how you create a new text file")
...
>>> make_another_file()
"You are trying to create a file that already exists!"

When opening a file, and with all the other file-manipulation functions discussed in this chapter, you can specify either a relative path (a path relative to the current directory, the directory in which your program or Python was run) or an absolute path (a path starting at the root of the drive or file system). For example, /tmp/sample.txt is an absolute path, whereas just sample.txt, without the specification of what directory is above it, is a relative path.

Appending text to a file is a pretty simple to do. Instead of using the write method ("w"), you use append instead ("a"). By doing so, you ensure that the data in the existing file is not overwritten, but instead, any new text is appended to the end of the file. Try out the following code:

>>> def add_some_text():
    a=open('test.txt',"a")
    a.write("Here is some additional text!")
...
>>> add_some_text()

>>> def even_more_text():
a=open('test.txt',"a")
a.write("""
here is
more
text""")
...
>>> even_more_text()

Reading from a file is similar. First, open the file by creating a file object. This time, use "r" to tell Python you intend to read from the file. It's the default, so you can omit the second argument altogether if you want.

a=open("test.txt","r")

Make sure you use the path to the file you created earlier, or use the path to some other file you want to read. If the file doesn't exist, Python will raise an exception.

You can read a line from the file using the readline method. The first time you call this method on a file object, it will return the first line of text in the file:

>>> a.readline()
'This is how you create a new text file
'

Notice that readline includes the newline character at the end of the string it returns. To read the contents of the file one line at a time, call readline repeatedly.

You can also read the rest of the file all at once with the read method. This method returns any text in the file that you haven't read yet. (If you call read as soon as you open a file, it will return the entire contents of the file, as one long string.)

>>> f=open("test.txt","r")
>>> text=a.read()
>>> print(text)
This is how you create a new text file
Here is some additional text
here is
more
text

Because you've used print to print the text, Python shows newline characters as actual line breaks, instead of as .

When you're done reading the file, close the file by deleting the file object and closing the file:

>>> del a
>>> a.close()

It's convenient to have Python break a text file into lines, but it's nice to be able to get all the lines at one time — for instance, to use in a loop. The readlines method does exactly that: It returns the remaining lines in the file as a list of strings. Suppose, for instance, that you want to print out the length of each line in a file. This function will do that:

def print_line_lengths():
  a=open("test.txt","r")
  text=a.readlines()
  for line in text:
    print(len(line))

>>> print_line_lengths()
38
28
7
4
4

Because your Python program does not have exclusive control of the computer's file system, it must be prepared to handle unexpected errors when accessing files. When Python encounters a problem performing a file operation, it raises an IOError exception. (Exceptions are described in Chapter 4.) The string representation of the exception will describe the problem.

Many circumstances exist in which you can get an IOError, including the following:

If you want your program to react gracefully when errors occur, you must handle these exceptions. What to do when you receive an exception depends on what your program does. In some cases, you may want to try a different file, perhaps after printing a warning message. In other cases, you may have to ask the user what to do next or simply exit if recovery is not possible.

The os module contains another module, os.path, which provides functions for manipulating paths. Because paths are strings, you could use ordinary string manipulation to assemble and disassemble file paths. Your code would not be as easily portable, however, and would probably not handle special cases that os.path knows about. Use os.path to manipulate paths, and your programs will be better for it.

To assemble directory names into a path, use os.path.join. Python uses the path separator appropriate for your operating system. Don't forget to import the os.path module before you use it. For example, on Windows, enter the following:

>>> import os.path
>>> os.path.join("snakes", "Python")
'snakes\Python'

On Linux, however, using the same parameters to os.path.join gives you the following, different, result:

>>> import os.path
>>> os.path.join("snakes", "Python")
'snakes/Python'

You can specify more than two components as well.

The inverse function is os.path.split, which splits off the last component of a path. It returns a tuple of two items: the path of the parent directory and the last path component. Here's an example:

>>> os.path.split("C:\Program Files\Python30\Lib")
('C:\Program Files\Python30', 'Lib')

On UNIX or Linux, it would look like this:

>>> os.path.split("/usr/bin/python")
('/usr/bin', 'python')

Automatic unpacking of sequences comes in handy here. What happens is that when os.path.split returns a tuple, the tuple can be broken up into the elements on the left-hand side of the equals sign:

>>> parent_path, name = os.path.split("C:\Program Files\Python30\Lib")
>>> print(parent_path)
C:Program FilesPython30
>>> print(name)
Lib

Although os.path.split only splits off the last path component, sometimes you might want to split a path completely into directory names. Writing a function to do this is not difficult; what you want to do is call os.path.split on the path, and then call os.path.split on the parent directory path, and so forth, until you get all the way to the root directory. An elegant way to do this is with a recursive function, which is a function that calls itself. It might look like this:

def split_fully(path):
    parent_path, name = os.path.split(path)
    if name == "":
        return (parent_path, )
    else:
        return split_fully(parent_path) + (name, )

The key line is the last line, where the function calls itself to split the parent path into components. The last component of the path, name, is then attached to the end of the fully split parent path. The lines in the middle of split_fully prevent the function from calling itself infinitely. When os.path.split can't split a path any further, it returns an empty string for the second component; split_fully notices this and returns the parent path without calling itself again.

A function can call itself safely because Python keeps track of the arguments and local variables in each running instance of the function, even if one is called from another. In this case, when split_fully calls itself, the outer (first) instance doesn't lose its value of name even though the inner (second) instance assigns a different value to it, because each has its own copy of the variable name. When the inner instance returns, the outer instance continues with the same variable values it had when it made the recursive call.

When you write a recursive function, make sure that it never calls itself infinitely, which would be bad because it would never return. (Actually, Python would run out of space in which to keep track of all the calls, and would raise an exception.) The function split_fully won't call itself infinitely, because eventually path is short enough that name is an empty string, and the function returns without calling itself again.

Notice in this function the two uses of single-element tuples, which must include a comma in the parentheses. Without the comma, Python would interpret the parentheses as ordinary grouping parentheses, as in a mathematical expression: (name, ) is a tuple with one element; (name) is the same as name.

Here's the function in action:

>>> split_fully("C:\Program Files\Python31\Lib")
('C:', 'Program Files', 'Python31', 'Lib')

After you have the name of a file, you can split off its extension with os.path.splitext:

>>> os.path.splitext("image.jpg")
('image', '.jpg')

The call to splitext returns a two-element tuple, so you can extract just the extension as shown here:

>>> parts = os.path.splitext("image.jpg")
>>> extension = parts[1]

You don't actually need the variable parts at all. You can extract the second component, the extension, directly from the return value of splitext:

>>> extension = os.path.splitext("image.jpg")[1]

Also handy is os.path.normpath, which normalizes or "cleans up" a path:

>>> print(os.path.normpath(r"C:\Program FilesPerl..Python30"))
C:Program FilesPython30

Notice how the ".." was eliminated by backing up one directory component, and the double separator was fixed. Similar to this is os.path.abspath, which converts a relative path (a path relative to the current directory) to an absolute path (a path starting at the root of the drive or file system):

>>> print(os.path.abspath("other_stuff"))
C:Program FilesPython30other_stuff

Your output will depend on your current directory when you call abspath. As you may have noticed, this works even though you don't have an actual file or directory named other_stuff in your Python directory. None of the path manipulation functions in os.path check whether the path you are manipulating actually exists.

If you want to know whether a path actually does exist, use os.path.exists. It simply returns True or False:

>>> os.path.exists("C:\Windows")
True
>>> os.path.exists("C:\Windows\reptiles")
False

Of course, if you're not using Windows, or your Windows is installed in another directory (like C:WinNT), both of these will return False!

Now you know how to construct arbitrary paths and take them apart. But how can you find out what's actually on your disk? The os.listdir module tells you, by returning a list of the name entries in a directory — the files, subdirectories, and so on that it contains.

>>> os.listdir("C:\Python31")
['Chapter 5', 'Chapter 6', 'Chapter 7', 'DLLs', 'Doc', 'ham', 'include',
'Lib', 'libs', 'LICENSE.txt', 'maybe', 'NEWS.txt', 'python.exe', 'pythonw.exe',
'README.txt', 'tcl', 'Test', 'Test.py', 'test.txt', 'test2.txt', 'test6.txt',
'tester.py', 'test.txt', 'Tools', 'w9xpopen.exe']

def print_dir(dir_path):
    for name in os.listdir(dir_path):
       print(os.path.join(dir_path, name))

>>> print_dir("C:\Python30")
C:Python31DLLs
C:Python31Doc
C:Python31ham
C:Python31include
C:Python31Lib
C:Python31libs
C:Python31LICENSE.txt...

>>> sorted(os.listdir("C:\Python31"))
['DLLs', 'Doc', 'LICENSE.txt', 'Lib', 'NEWS.txt', 'README.txt',
'Removepywin32.exe', 'Scripts', 'Tools', 'include', 'libs', 'py.ico',
'pyc.ico', 'python.exe', 'pythonw.exe', 'pywin32-wininst.log', 'tcl',
'w9xpopen.exe']

You can easily determine whether a path refers to a file or to a directory. If it's a file, os.path.isfile will return True; if it's a directory, os.path.isdir will return True. Both return False if the path does not exist at all:

>>> os.path.isfile("C:\Windows")
False
>>> os.path.isdir("C:\Windows")
True

def print_tree(dir_path):
    for name in os.listdir(dir_path):
        full_path = os.path.join(dir_path, name)
        print(full_path)
        if os.path.isdir(full_path):
            print_tree(full_path)

>>> import time
>>> mod_time = os.path.getmtime("C:\Python30")
>>> print(time.ctime(mod_time))
Thu Mar 15 01:36:26 2009

def print_dir_info(dir_path):
    for name in os.listdir(dir_path):
        full_path = os.path.join(dir_path, name)
        file_size = os.path.getsize(full_path)
        mod_time = time.ctime(os.path.getmtime(full_path))
        print("%−32s: %8d bytes, modified %s" % (name, file_size, mod_time))

The shutil module contains functions for operating on files. You can use the function shutil.move to rename a file:

>>> import shutil
>>> shutil.move("server.log", "server.log.backup")

Alternatively, you can use it to move a file to another directory:

>>> shutil.move("old mail.txt", "C:\data\archive\")

You might have noticed that os also contains a function for renaming or moving files, os.rename. You should generally use shutil.move instead, because with os.rename, you may not specify a directory name as the destination and on some systems os.rename cannot move a file to another disk or file system.

The shutil module also provides the copy function to copy a file to a new name or directory. You can simply use the following:

>>> shutil.copy("important.dat", "C:\backups")

Deleting a file is easiest of all. Just call os.remove:

>>> os.remove("junk.dat")

If you're an old-school UNIX hacker (or want to pass yourself off as one), you may prefer os.unlink, which does the same thing.

In this example, you tackle a more difficult real-world file management task. Suppose that you need to keep old versions of a file around. For instance, system administrators will keep old versions of system log files. Often, older versions of a file are named with a numerical suffix — for instance, web.log.1, web.log.2, and so on — in which a larger number indicates an older version. To make room for a new version of the file, the old versions are rotated: The current version of web.log becomes version web.log.1, web.log.1 becomes web.log.2, and so on.

This is clearly tedious to do by hand, but Python can make quick work of it. You have a few tricky points to consider, however. First, the current version of the file is named differently than old versions; whereas old versions have a numerical suffix, the current version does not. One way to get around this is to treat the current version as version zero. A short function, make_version_path, constructs the right path for both current and old versions.

The other subtle point is that you must make sure to rename the oldest version first. For instance, if you rename web.log.1 to web.log.2 before renaming web.log.2, the latter will be overwritten and its contents lost before you get to it, which isn't what you want. Once again, a recursive function will save you. The function can call itself to rotate the next-older version of the log file before it gets overwritten:

import os
import shutil

def make_version_path(path, version):
    if version == 0:
        # No suffix for version 0, the current version.
        return path
    else:
        # Append a suffix to indicate the older version.
        return path + "." + str(version)

def rotate(path, version=0):
    # Construct the name of the version we're rotating.
    old_path = make_version_path(path, version)
    if not os.path.exists(old_path):
        # It doesn't exist, so complain.
        raise IOError("'%s' doesn't exist" % path)
    # Construct the new version name for this file.
    new_path = make_version_path(path, version + 1)
    # Is there already a version with this name?
    if os.path.exists(new_path):
        # Yes.  Rotate it out of the way first!
        rotate(path, version + 1)
    # Now we can rename the version safely.
    shutil.move(old_path, new_path)

Take a few minutes to study this code and the comments. The rotate function uses a technique common in recursive functions: a second argument for handing recursive cases — in this case, the version number of the file being rotated. The argument has a default value, zero, which indicates the current version of the file. When you call the function (as opposed to when the function is calling itself) you don't specify a value for this argument. For example, you can just call rotate("web.log").

You may have noticed that the function checks to make sure that the file being rotated actually exists and raises an exception if it doesn't. But suppose you want to rotate a system log file that may or may not exist. One way to handle this is to create an empty log file whenever it's missing. Remember that when you open a file that doesn't exist for writing, Python creates the file automatically. If you don't actually write anything to the new file, it will be empty. Here's a function that rotates a log file that may or may not exist, creating it first if it doesn't. It uses the rotate function you wrote previously.

def rotate_log_file(path):
    if not os.path.exists(path):
        # The file is missing, so create it.
        new_file = file(path, "w")
        # Close the new file immediately, which leaves it empty.
        del new_file
    # Now rotate it.
    rotate(path)

Creating an empty directory is even easier than creating a file. Just call os.mkdir. The parent directory must exist, however. The following will raise an exception if the parent directory C:photoszoo does not exist:

>>> os.mkdir("C:\photos\zoo\snakes")

You can create the parent directory itself using os.mkdir, but the easy way out is instead to use os.makedirs, which creates missing parent directories. For example, the following will create C:photos and C:photoszoo, if necessary:

>>> os.makedirs("C:\photos\zoo\snakes")

Remove a directory with os.rmdir. This works only for empty directories; if the directory is not empty, you'll have to remove its contents first:

>>> os.rmdir("C:\photos\zoo\snakes")

This removes only the snakes subdirectory.

There is a way to remove a directory even when it contains other files and subdirectories. The function shutil.rmtree does this. Be careful, however; if you make a programming or typing mistake and pass the wrong path to this function, you could delete a whole bunch of files before you even know what's going on! For instance, this will delete your entire photo collection — zoo, snakes, and all:

>>> shutil.rmtree("C:\photos")

If you have used the command prompt on Windows, or a shell command line on GNU/Linux, UNIX, or Mac OS X, you probably have encountered wildcard patterns before. These are the special characters, such as * and ?, which you use to match many files with similar names. For example, you may have used the pattern P* to match all files that start with P, or *.txt to match all files with the extension .txt.

Globbing is hackers' jargon for expanding wildcards in file name patterns. Python provides a function glob, in the module also named glob, which implements globbing of directory contents. The glob.glob function takes a glob pattern and returns a list of matching file names or paths, similar to os.listdir.

For example, try the following command to list entries in your C:Program Files directory that start with M:

>>> import glob
>>> glob.glob("C:\Program Files\M*")
['C:\Program Files\Messenger', 'C:\Program Files\Microsoft Office',
'C:\Program Files\Mozilla Firefox']

Your computer's output will vary depending on what software you have installed. Observe that glob.glob returns paths containing drive letters and directory names if the pattern includes them, unlike os.listdir, which only returns the names in the specified directory.

The following table lists the wildcards you can use in glob patterns. These wildcards are not necessarily the same as those available in the command shell of your operating system, but Python's glob module uses the same syntax on all platforms. Note that the syntax for glob patterns resembles but is not the same as the syntax for regular expressions.

You can also use a range of characters in square brackets. For example, [m-p] matches any one of the letters m, n, o, or p, and [!0-9] matches any character other than a digit.

Globbing is a handy way of selecting a group of similar files for a file operation. For instance, deleting all backup files with the extension .bak in the directory C:source is as easy as these two lines:

>>> for path in glob.glob("C:\source\*.bak"):
...     os.remove(path)

Globbing is considerably more powerful than os.listdir, because you can specify wildcards in directory and subdirectory names. For patterns like this, glob.glob can return paths in more than one directory. For instance, the following code returns all files with the extension .txt in subdirectories of the current directory:

>>> glob.glob("*\*.txt")