Let’s turn to an example to make this nesting concept more concrete. The following module, file nestexc.py, defines two functions; action2 is coded to trigger an exception (you can’t add numbers and sequences), and action1 wraps a call to action2 in a try handler, to catch the exception:

def action2(  ):
    print 1 + [  ]           # Generate TypeError.

def action1(  ):
    try:
        action2(  )
    except TypeError:        # Most recent matching try
        print 'inner try'

try:
    action1(  )
except TypeError:          # Here, only if action1 reraises.
    print 'outer try'

% python nestexc.py
inner try

Notice, though, that the top-level module code at the bottom of the file wraps a call to action1 in a try handler too. When action2 triggers the TypeError exception, there will be two active try statements—the one in action1, and the one at the top level of the module. Python picks and runs just the most recent with a matching except, which in this case is the try inside action1.

In general, the place where an exception winds up jumping to depends on the control flow through a program at runtime. In other words, to know where you will go, you need to know where you’ve been—exceptions are more a function of control flow than statement syntax.

It’s also possible to nest try statements syntactically:

try:
    try:
        action2(  )
    except TypeError:      # Most recent matching try
        print 'inner try'
except TypeError:          # Here, only if nested handler reraises.
    print 'outer try'

But really, this just sets up the same handler nesting structure as, and behaves identical to, the prior example. In fact, syntactic nesting works just like the cases we sketched in Figures Figure 26-1 and Figure 26-2. The only difference is that the nested handlers are physically embedded in a try block, not coded in a called function elsewhere. For example, nested finally handlers all fire on an exception, whether they are nested syntactically, or by the runtime flow through physically separated parts of your code:

>>> try:
...     try:
...         raise IndexError
...     finally:
...         print 'spam'
... finally:
...     print 'SPAM'
...
spam
SPAM
Traceback (most recent call last):
  File "<stdin>", line 3, in ?
IndexError

See Figure 26-2 for a graphic illustration of this code’s operation; it’s the same effect, but function logic has been inlined as nested statements here. For a more useful example of syntactic nesting at work, consider the following file, except-finally.py:

def raise1(  ):  raise IndexError
def noraise(  ): return
def raise2(  ):  raise SyntaxError

for func in (raise1, noraise, raise2):
    print '
', func
    try:
        try:
            func(  )
        except IndexError:
            print 'caught IndexError'
    finally:
        print 'finally run'

This code catches an exception if it is raised, and performs a finally termination time action regardless of whether any exception occurred or not. This takes a few moments to digest, but the effect is much like combining an except and finally clause in a single try statement, even though such a combination would be syntactically illegal (they are mutually exclusive):

% python except-finally.py

<function raise1 at 0x00867DF8>
caught IndexError
finally run

<function noraise at 0x00868EB8>
finally run

<function raise2 at 0x00875B80>
finally run
Traceback (most recent call last):
  File "except-finally.py", line 9, in ?
    func(  )
  File "except-finally.py", line 3, in raise2
    def raise2(  ): raise SyntaxError
SyntaxError

In Python, all errors are exceptions, but not all exceptions are errors. For instance, we saw in Chapter 7 that file object read methods return empty strings at the end of a file. The built-in raw_input function that we first met in Chapter 3, and deployed in an interactive loop in Chapter 10, reads a line of text from the standard input stream (sys.stdin). Unlike file methods, raw_input raises the built-in EOFError at end of file, instead of returning an empty string (an empty string from raw_input means an empty line).

Despite its name, the EOFError exception is just a signal in this context, not an error. Because of this behavior, unless end-of-file should terminate a script, raw_input often appears wrapped in a try handler and nested in a loop, as in the following code.

while 1:
    try:
        line = raw_input(  )     # Read line from stdin.
    except EOFError:
        break                    # Exit loop at end of file
    else:
        ...process next line here...

Other built-in exceptions are similarly signals, not errors. Python also has a set of built-in exceptions that represent warnings, rather than errors. Some of these are used to signal use of deprecated (phased out) language features. See the standard library manual’s description of built-in exceptions and the warnings module for more on warnings.

User-defined exceptions can also signal nonerror conditions. For instance, a search routine can be coded to raise an exception when a match is found, instead of returning a status flag that must be interpreted by the caller. In the following, the try/except/else exception handler does the work of an if/else return value tester:

Found = "Item found"

def searcher(  ):
    if ...success...:
        raise Found
    else:
        return

try:
    searcher(  )
except Found:              # Exception if item was found
    ...success...
else:                      # else returned: not found
    ...failure...

More generally, such a coding structure may also be useful for any function that cannot return a sentinel value to designate success or failure. For instance, if all objects are potentially valid return values, it’s impossible for any return value to signal unusual conditions. Exceptions provide a way to signal results without a return value:

failure = "not found"

def searcher(  ):
    if ...success...:
        return ...founditem...
    else:
        raise failure

try:
    item = searcher(  )
except failure:
    ...report...
else:
    ...use item here...

Because Python is dynamically typed and polymorphic to the core, exceptions, rather than sentinel return values, are the generally preferred way to signal conditions.

You can also make use of exception handlers to replace Python’s default top-level exception-handling behavior. By wrapping an entire program (or a call to it) in an outer try in your top-level code, you can catch any exception that may occur while your program runs, thereby subverting the default program termination.

In the following, the empty except clause catches any uncaught exception raised while the program runs. To get hold of the actual exception that occurred, fetch the exc_type and exc_value attributes from the built-in sys module; they’re automatically set to the current exception’s name and extra data:^[1]

try:
    ...run program...
except:                  # All uncaught exceptions come here.
    import sys
    print 'uncaught!', sys.exc_type, sys.exc_value

This structure is commonly used during development, to keep your program active even after errors occur—you can run additional tests without having to restart. It’s also used when testing code, as described in the next section.

You might combine some of these coding patterns in a test-driver application, which tests other code within the same process:

import sys
log = open('testlog', 'a')
from testapi import moreTests, runNextTest, testName

def testdriver(  ):
    while moreTests(  ):
        try:
            runNextTest(  )
        except:
            print >> log, 'FAILED', testName(  ), sys.exc_type
        else:
            print >> log, 'PASSED', testName(  )

testdriver(  )

The testdriver function here cycles through a series of test calls (module testapi is left abstract in this example). Because an uncaught exception in a test case would normally kill this test driver, we need to wrap test case calls in a try if we want to continue the testing process after a test fails. As usual, the empty except catches any uncaught exception generated by a test case, and uses sys.exc_type to log the exception to a file; the else clause is run when no exception occurs—the test success case.

Such boilerplate code is typical of systems that test functions, modules, and classes, by running them in the same process as the test driver. In practice, testing can be much more sophisticated than this. For instance, to test external programs, we would instead check status codes or outputs generated by program launching tools such as os.system and os.popen, covered in the standard library manual; such tools do not generally raise exceptions for errors in the external program (in fact, the test cases may run in parallel with the test driver). At the end of this chapter, we’ll also meet more complete testing frameworks provided by Python, such as doctest and PyUnit, which provide tools for comparing expected outputs with actual results.

doStuff(  )
{                                    # C program:
     if (doFirstThing(  ) == ERROR)  # Detect errors everywhere
          return ERROR;              # even if not handled here.
     if (doNextThing(  ) == ERROR)
          return ERROR;
     ...
     return doLastThing(  );
}

main(  )
{
     if (doStuff(  ) == ERROR)
          badEnding(  );
     else
          goodEnding(  );
}

def doStuff(  ):       # Python code
    doFirstThing(  )   # We don't care about exceptions here,
    doNextThing(  )    # so we don't need to detect them here.
    ...
    doLastThing(  )

if__name__ == '__main__':
    try:
        doStuff(  )    # This is where we care about results,
    except:            # so it's the only place we must check.
        badEnding(  )
    else:
        goodEnding(  )

In principle, you could wrap every statement in your script in its own try, but that would just be silly (the try statements would then need to be wrapped in try statements!). This is really a design issue that goes beyond the language itself, and becomes more apparent with use. But here are a few rules of thumb:

On to the issue of handler generality. Because Python lets you pick and choose which exceptions to catch, you sometimes have to be careful to not be too inclusive. For example, you’ve seen that an empty except clause catches every exception that might be raised while the code in the try block runs.

That’s easy to code and sometimes desirable, but you may also wind up intercepting an error that’s expected by a try handler higher up in the exception nesting structure. For example, an exception handler such as the following catches and stops every exception that reaches it—whether or not another handler is waiting for it:

def func(  ):
    try:
        ...                # IndexError is raised in here.
    except:
        ...                # But everything comes here and dies!

try:
    func(  )
except IndexError:         # Needed here
    ...

Perhaps worse, such code might also catch system exceptions. Even things like memory errors, programming mistakes, iteration stops, and system exits raise exceptions in Python. Such exceptions should not usually be intercepted.

For example, scripts normally exit when control falls off the end of the top-level file. However, Python also provides a built-in sys.exit call to allow early terminations. This actually works by raising a built-in SystemExit exception to end the program, so that try/finally handlers run on the way out, and special types of programs can intercept the event.^[2] Because of this, a try with an empty except might unknowingly prevent a crucial exit, as in file exiter.py:

import sys

def bye(  ):
    sys.exit(40)             # Crucial error: abort now!

try:
    bye(  )
except:
    print 'got it'           # Oops--we ignored the exit
print 'continuing...'


% python exiter.py
got it
continuing...

You simply might not expect all the kinds of exceptions that could occur during an operation. In fact, an empty except will also catch genuine programming errors, which should also be allowed to pass most of the time:

mydictionary = {...}
...
try:
    x = myditctionary['spam']     # Oops: misspelled 
except:
    x = None                      # Assume we got KeyError.
...continue here...

The coder here assumes the only sort of error that can happen when indexing a dictionary is a key error. But because the name myditctionary is misspelled (it should say mydictionary), Python raises a NameError instead for the undefined name reference, which will be silently caught and ignored by the handler. The event will incorrectly fill in a default for the dictionary access, masking the program error. If this happens in code that is far removed from the place where the fetched values are used, it might make for a very interesting debugging task.

As a rule of thumb, be specific in your handlers—empty except clauses are handy, but potentially error-prone. In the last example, for instance, you should usually say except KeyError: to make your intentions explicit, and avoid intercepting unrelated events. In simpler scripts, the potential for problems might not be significant enough to outweigh the convenience of a catch-all. But in general, general handlers are generally trouble.

Conversely, handlers also shouldn’t be too specific. When listing specific exceptions in a try, you catch only what you actually list. This isn’t necessarily a bad thing either, but if a system evolves to raise other exceptions in the future, you may need to go back and add them to exception lists elsewhere in the code.

For instance, the following handler is written to treat myerror1 and myerror2 as normal cases, and treat everything else as an error. If a myerror3 is added in the future, it is processed as an error unless you update the exception list:

try:
    ...
except (myerror1, myerror2):    # What if I add a myerror3?
    ...                         # Nonerrors
else:
    ...                         # Assumed to be an error

Careful use of class-based exceptions can make this trap go away completely. As we learned in the prior chapter, if you catch a general superclass, you can add and raise more specific subclasses in the future without having to extend except clause lists manually:

try:
    ...
except SuccessCategoryName:     # What if I add a myerror3?
    ...                         # Nonerrors
else:
    ...                         # Assumed to be an error

Whether you use classes here or not, a little design goes a long way. The moral of the story is that you have to be careful not to be too general or too specific in exception handlers, and have to pick the granularity of your try statement wrapping wisely. Especially in larger systems, exception policies should be a part of the overall design.

When an exception is raised (by you or by Python itself), Python searches for the most recently entered try statement with a matching except clause, where matching means the same string object, the same class, or a superclass of the raised class. It’s important to notice that matching is performed by identity, not equality. For instance, suppose we define two string objects we want to raise as exceptions:

>>> ex1 = 'Error: Spam Exception'
>>> ex2 = 'Error: Spam Exception'
>>>
>>> ex1 == ex2, ex1 is ex2
(1, 0)

Applying the == test returns true (1) because they have equal values, but is returns false (0) since they are two distinct string objects in memory. Now, an except clause that names the same string object will always match:

>>> try:
...    raise ex1
... except ex1:
...    print 'got it'
...
got it

But one that lists an equal value, but not an identical object, will fail (assuming the string values are long enough to defeat Python’s string object caching mechanism, which is described in Chapter 4 and Chapter 7:

>>> try:
...    raise ex1
... except ex2:
...    print 'Got it'
...
Traceback (innermost last):
  File "<stdin>", line 2, in ?
    raise ex1
Error: Spam Exception

Here, the exception isn’t caught, so Python climbs to the top level of the process and prints a stack trace and the exception’s text automatically. For strings, be sure to use the same object in the raise and the try. For class exceptions, the behavior is similar, but Python generalizes the notion of exception matching to include superclass relationships.

Perhaps the most common gotchas related to exceptions involve the design guidelines of the prior section. Remember, try to avoid empty except clauses (or you may catch things like system exits), and overly-specific except clauses (use superclass categories instead, to avoid maintenance issues in the future).

From this point forward, your future Python career will largely consist of becoming proficient with the toolset available for application-level Python programmming. You’ll find this to be an ongoing task. The standard library, for example, contains some 200 modules and the public domain offers more tools still. Because new tools appear constantly, it’s possible to spend a decade or more becoming proficient in all these tools. We speak from personal experience here.

In general, Python provides a hierarchy of tool sets:

Because Python layers its tool sets, you can decide how deeply your programs need to delve into this hierarchy for any given task—use built-ins for simple scripts, add Python-coded extensions for larger systems, and code C extensions for advanced work. You’ve covered the first two of these categories above in this book already, and that’s plenty to do substantial programming in Python.

The next part of this book takes you on a tour of standard modules and common tasks in Python. Table 26-1 summarizes some of the sources of built-in or existing functionality available to Python programmers, and topics you’ll explore in the remainder of this book. Up until now, most of our examples have been very small and self-contained. We wrote them that way on purpose to help you master the basics. But now that you know all about the core language, it’s time to start learning how to use Python’s built-in interfaces to do real work. You’ll find that with a simple language like Python, common tasks are often much easier than you might expect.

Finally, once you’ve mastered the basics, you’ll find that your Python programs become substantially larger than the examples you’ve experimented with so far. For developing larger systems, a set of development tools is available in both Python and the public domain. You’ve seen some of these in action, and we’ve talked about others. To help you on your way, here is a summary of some of the most commonly used tools in this domain, many of which you’ve already seen:

For additional large-scale Python development tools available in the public domain, be sure to also browse the pages at the Vaults of Parnassus web site.