Let’s turn to an example to make this nesting concept more concrete. The following module file, nestexc.py, defines two functions. action2 is coded to trigger an exception (you can’t add numbers and sequences), and action1 wraps a call to action2 in a try handler, to catch the exception:

def action2(  ):
    print 1 + [  ]           # Generate TypeError

def action1(  ):
    try:
        action2(  )
    except TypeError:        # Most recent matching try
        print 'inner try'

try:
    action1(  )
except TypeError:            # Here, only if action1 re-raises
    print 'outer try'
% python nestexc.py
inner try

Notice, though, that the top-level module code at the bottom of the file wraps a call to action1 in a try handler, too. When action2 triggers the TypeError exception, there will be two active try statements—the one in action1, and the one at the top level of the module file. Python picks and runs just the most recent try with a matching except, which in this case is the try inside action1.

As I’ve mentioned, the place where an exception winds up jumping to depends on the control flow through the program at runtime. Because of this, to know where you will go, you need to know where you’ve been. In this case, where exceptions are handled is more a function of control flow than of statement syntax. However, we can also nest exception handlers syntactically—an equivalent case we’ll look at next.

As I mentioned when we looked at the new unified try/except/finally statement in Chapter 27, it is possible to nest try statements syntactically by their position in your source code:

try:
    try:
        action2(  )
    except TypeError:        # Most recent matching try
        print 'inner try'
except TypeError:            # Here, only if nested handler re-raises
    print 'outer try'

Really, this code just sets up the same handler-nesting structure as (and behaves identically to) the prior example. In fact, syntactic nesting works just like the cases sketched in Figure 29-1 and Figure 29-2; the only difference is that the nested handlers are physically embedded in a try block, not coded in functions called elsewhere. For example, nested finally handlers all fire on an exception, whether they are nested syntactically, or by means of the runtime flow through physically separated parts of your code:

>>> try:
...     try:
...         raise IndexError
...     finally:
...         print 'spam'
... finally:
...     print 'SPAM'
...
spam
SPAM
Traceback (most recent call last):
  File "<stdin>", line 3, in ?
IndexError

See Figure 29-2 for a graphic illustration of this code’s operation; the effect is the same, but the function logic has been inlined as nested statements here. For a more useful example of syntactic nesting at work, consider the following file, except-finally.py:

def raise1(  ):  raise IndexError
def noraise(  ): return
def raise2(  ):  raise SyntaxError
for func in (raise1, noraise, raise2):
    print '
', func
    try:
        try:
            func(  )
        except IndexError:
            print 'caught IndexError'
    finally:
        print 'finally run'

This code catches an exception if one is raised, and performs a finally termination-time action regardless of whether an exception occurred. This may take a few moments to digest, but the effect is much like combining an except and a finally clause in a single try statement today (recall that such combinations were syntactically illegal until Python 2.5):

% python except-finally.py
<function raise1 at 0x00BA2770>
caught IndexError
finally run
<function noraise at 0x00BB47F0>
finally run

<function raise2 at 0x00BB4830>
finally run

Traceback (most recent call last):
  File "C:/Python25/except-finally.py", line 9, in <module>
    func(  )
  File "C:/Python25/except-finally.py", line 3, in raise2
    def raise2(  ):  raise SyntaxError
SyntaxError: None

As we saw in Chapter 27, as of Python 2.5, except and finally clauses can be mixed in the same try statement. This makes some of the syntactic nesting described in this section unnecessary, though it still works, may appear in code written prior to Python 2.5, and can be used as a technique for implementing alternative exception-handling behaviors.

In Python, all errors are exceptions, but not all exceptions are errors. For instance, we saw in Chapter 9 that file object read methods return an empty string at the end of a file. In contrast, the built-in raw_input function (which we first met in Chapter 3, and deployed in an interactive loop in Chapter 10) reads a line of text from the standard input stream, sys.stdin, at each call, and raises the built-in EOFError at end-of-file. Unlike file methods, this function does not return an empty string—an empty string from raw_input means an empty line. Despite its name, the EOFError exception is just a signal in this context, not an error. Because of this behavior, unless end-of-file should terminate a script, raw_input often appears wrapped in a try handler and nested in a loop, as in the following code:

while 1:
    try:
        line = raw_input(  )       # Read line from stdin
    except EOFError:
        break                      # Exit loop at end-of-file
    else:
        ...process next line here...

Other built-in exceptions are similarly signals, not errors. Python also has a set of built-in exceptions that represent warnings rather than errors. Some of these are used to signal use of deprecated (phased out) language features. See the standard library manual’s description of built-in exceptions, and the warnings module for more on warnings.

User-defined exceptions can also signal nonerror conditions. For instance, a search routine can be coded to raise an exception when a match is found, instead of returning a status flag for the caller to interpret. In the following, the try/except/else exception handler does the work of an if/else return-value tester:

class Found(Exception): pass

def searcher(  ):
    if ...success...:
        raise Found(  )
    else:
        return

try:
    searcher(  )
except Found:                    # Exception if item was found
    ...success...
else:                            # else returned: not found
    ...failure...

More generally, such a coding structure may also be useful for any function that cannot return a sentinel value to designate success or failure. For instance, if all objects are potentially valid return values, it’s impossible for any return value to signal unusual conditions. Exceptions provide a way to signal results without a return value:

class Failure(Exception): pass
def searcher(  ):
    if ...success...:
        return ...founditem...
    else:
        raise Failure(  )

try:
    item = searcher(  )
except Failure:
    ...report...
else:
    ...use item here...

Because Python is dynamically typed and polymorphic to the core, exceptions, rather than sentinel return values, are the generally preferred way to signal such conditions.

You can also make use of exception handlers to replace Python’s default top-level exception-handling behavior. By wrapping an entire program (or a call to it) in an outer try in your top-level code, you can catch any exception that may occur while your program runs, thereby subverting the default program termination.

In the following, the empty except clause catches any uncaught exception raised while the program runs. To get hold of the actual exception that occurred, fetch the sys.exc_info function call result from the built-in sys module; it returns a tuple, whose first two items automatically contain the current exception’s name, and its associated extra data (if any). For class-based exceptions, these two items are the exception class and the raised instance of the class, respectively (more on sys.exc_info in a moment):

try:
    ...run program...
except:                         # All uncaught exceptions come here
    import sys
    print 'uncaught!', sys.exc_info(  )[0], sys.exc_info(  )[1]

This structure is commonly used during development, to keep programs active even after errors occur—it allows you to run additional tests without having to restart. It’s also used when testing other program code, as described in the next section.

You might combine some of the coding patterns we’ve just looked at in a test-driver application, which tests other code within the same process:

import sys
log = open('testlog', 'a')
from testapi import moreTests, runNextTest, testName
def testdriver(  ):
    while moreTests(  ):
        try:
            runNextTest(  )
        except:
            print >> log, 'FAILED', testName(  ), sys.exc_info(  )[:2]
        else:
            print >> log, 'PASSED', testName(  )
testdriver(  )

The testdriver function here cycles through a series of test calls (the module testapi is left abstract in this example). Because an uncaught exception in a test case would normally kill this test driver, you need to wrap test case calls in a try if you want to continue the testing process after a test fails. As usual, the empty except catches any uncaught exception generated by a test case, and it uses sys.exc_info to log the exception to a file. The else clause is run when no exception occurs—the test success case.

Such boilerplate code is typical of systems that test functions, modules, and classes by running them in the same process as the test driver. In practice, however, testing can be much more sophisticated than this. For instance, to test external programs, you could instead check status codes or outputs generated by program-launching tools such as os.system and os.popen, covered in the standard library manual (such tools do not generally raise exceptions for errors in the external programs—in fact, the test cases may run in parallel with the test driver).

At the end of this chapter, we’ll also meet some more complete testing frameworks provided by Python, such as Doctest and PyUnit, which provide tools for comparing expected outputs with actual results.

The sys.exc_info result used in the last two sections is the preferred way to gain access to the most recently raised exception generically. If no exception is being handled, it returns a tuple containing three None values; otherwise, the values returned are (type, value, traceback), where:

The older tools sys.exc_type and sys.exc_value still work to fetch the most recent exception type and value, but they can manage only a single, global exception for the entire process. The preferred sys.exc_info instead keeps track of each thread’s exception information, and so is thread-specific. Of course, this distinction matters only when using multiple threads in Python programs (a subject beyond this book’s scope). See the Python library manual or follow-up texts for more details.

In principle, you could wrap every statement in your script in its own try, but that would just be silly (the try statements would then need to be wrapped in try statements!). This is really a design issue that goes beyond the language itself, and becomes more apparent with use. But here are a few rules of thumb:

The types of programs you write will probably influence the amount of exception handling you code as well. Servers, for instance, must generally keep running persistently, and so will likely require try statements to catch and recover from exceptions. In-process testing programs of the kind we saw in this chapter will probably handle exceptions as well. Simpler one-shot scripts, though, will often ignore exception handling completely because failure at any step requires script shutdown.

On to the issue of handler generality. Python lets you pick and choose which exceptions to catch, but you sometimes have to be careful to not be too inclusive. For example, you’ve seen that an empty except clause catches every exception that might be raised while the code in the try block runs.

That’s easy to code, and sometimes desirable, but you may also wind up intercepting an error that’s expected by a try handler higher up in the exception nesting structure. For example, an exception handler such as the following catches and stops every exception that reaches it, regardless of whether another handler is waiting for it:

def func(  ):
    try:
        ...                      # IndexError is raised in here
    except:
        ...                      # But everything comes here and dies!
try:
    func(  )
except IndexError:               # Exception should be processed here
    ...

Perhaps worse, such code might also catch unrelated system exceptions. Even things like memory errors, genuine programming mistakes, iteration stops, and system exits raise exceptions in Python. Such exceptions should not usually be intercepted.

For example, scripts normally exit when control falls off the end of the top-level file. However, Python also provides a built-in sys.exit(statuscode) call to allow early terminations. This actually works by raising a built-in SystemExit exception to end the program, so that try/finally handlers run on the way out, and special types of programs can intercept the event.^[80] Because of this, a try with an empty except might unknowingly prevent a crucial exit, as in the following file (exiter.py):

import sys
def bye(  ):
    sys.exit(40)                 # Crucial error: abort now!
try:
    bye(  )
except:
    print 'got it'               # Oops--we ignored the exit
print 'continuing...'
% python exiter.py
got it
continuing...

You simply might not expect all the kinds of exceptions that could occur during an operation.

Probably worst of all, an empty except will also catch genuine programming errors, which should be allowed to pass most of the time. In fact, empty except clauses can effectively turn off Python’s error-reporting machinery, making it difficult to notice mistakes in your code. Consider this code, for example:

mydictionary = {...}
...
try:
    x = myditctionary['spam']    # Oops: misspelled
except:
    x = None                     # Assume we got KeyError
...continue here with x...

The coder here assumes that the only sort of error that can happen when indexing a dictionary is a missing key error. But because the name myditctionary is misspelled (it should say mydictionary), Python raises a NameError instead for the undefined name reference, which the handler will silently catch and ignore. The event handler will incorrectly fill in a default for the dictionary access, masking the program error. If this happens in code that is far removed from the place where the fetched values are used, it might make for a very interesting debugging task!

As a rule of thumb, be as specific in your handlers as you can be—empty except clauses are handy, but potentially error-prone. In the last example, for instance, you would be better off saying except KeyError: to make your intentions explicit and avoid intercepting unrelated events. In simpler scripts, the potential for problems might not be significant enough to outweigh the convenience of a catchall, but in general, general handlers are generally trouble.

On the other hand, neither should handlers be too specific. When you list specific exceptions in a try, you catch only what you actually list. This isn’t necessarily a bad thing, but if a system evolves to raise other exceptions in the future, you may need to go back and add them to exception lists elsewhere in your code.

For instance, the following handler is written to treat myerror1 and myerror2 as normal cases, and treat everything else as an error. If a myerror3 is added in the future, it is processed as an error, unless you update the exception list:

try:
    ...
except (myerror1, myerror2):     # Breaks if I add a myerror3
    ...                          # Nonerrors
else:
    ...                          # Assumed to be an error

Luckily, careful use of the class-based exceptions we discussed in Chapter 28 can make this trap go away completely. As we saw, if you catch a general superclass, you can add and raise more specific subclasses in the future without having to extend except clause lists manually—the superclass becomes an extendible exceptions category:

try:
    ...
except SuccessCategoryName:      # OK if I add a myerror3 subclass
    ...                          # Nonerrors
else:
    ...                          # Assumed to be an error

Whether you use class-based exception category hierarchies, a little design goes a long way. The moral of the story is to be careful to not be too general or too specific in exception handlers, and to pick the granularity of your try statement wrappings wisely. Especially in larger systems, exception policies should be a part of the overall design.

Now that exceptions are supposed to be identified with classes instead of strings, this gotcha is something of a legacy issue; however, as you’re likely to encounter string-based exceptions in existing code, it’s worth knowing about. When an exception is raised (by you or by Python itself), Python searches for the most recently entered try statement with a matching except clause—i.e., an except clause that names the same string object (for string-based exceptions), or the same class or a superclass of it (for class-based exceptions). For string exceptions, it’s important to be aware that matching is performed by identity, not equality.

For instance, suppose we define two string objects we want to raise as exceptions:

>>> ex1 = 'The Spanish Inquisition'
>>> ex2 = 'The Spanish Inquisition'

>>> ex1 == ex2, ex1 is ex2
(True, False)

Applying the == test returns True because they have equal values, but is returns False because they are two distinct string objects in memory (assuming they are long enough to defeat the internal caching mechanism for Python strings, described in Chapter 6). Thus, an except clause that names the same string object will always match:

>>> try:
...    raise ex1
... except ex1:
...    print 'got it'
...
got it

but one that lists an equal-value but not identical object will fail:

>>> try:
...    raise ex1
... except ex2:
...    print 'Got it'
...
Traceback (most recent call last):
  File "<pyshell#43>", line 2, in <module>
    raise ex1
The Spanish Inquisition

Here, the exception isn’t caught because there is no match on object identity, so Python climbs to the top level of the process, and prints a stack trace and the exception’s text automatically. As I’ve hinted, however, some different string objects that happen to have the same values will in fact match because Python caches and reuses small strings (as described in Chapter 6):

>>> try:
...     raise 'spam'
... except 'spam':
...     print 'got it'
...
got it

This works because the two strings are mapped to the same object by caching. In contrast, the strings in the following example are long enough to be outside the cache’s scope:

>>> try:
...     raise 'spam' * 8
... except 'spam' * 8:
...     print 'got it'
...
Traceback (most recent call last):
  File "<pyshell#58>", line 2, in <module>
    raise 'spam' * 8
spamspamspamspamspamspamspamspam

If this seems obscure, all you need to remember is this: for strings, be sure to use the same object in the raise and the try, not the same value.

For class exceptions (the preferred technique today), the overall behavior is similar, but Python generalizes the notion of exception matching to include superclass relationships, so this gotcha doesn’t apply—yet another reason to use class-based exceptions today!

Perhaps the most common gotchas related to exceptions involve the design guidelines discussed in the prior section. Remember, try to avoid empty except clauses (or you may catch things like system exits) and overly specific except clauses (use superclass categories instead to avoid maintenance issues in the future as new exceptions are added).

From this point forward, your future Python career will largely consist of becoming proficient with the toolset available for application-level Python programming. You’ll find this to be an ongoing task. The standard library, for example, contains hundreds of modules, and the public domain offers still more tools. It’s possible to spend a decade or more seeking proficiency with all these tools, especially as new ones are constantly appearing (trust me on this!).

Speaking generally, Python provides a hierarchy of toolsets:

Because Python layers its toolsets, you can decide how deeply your programs need to delve into this hierarchy for any given task—you can use built-ins for simple scripts, add Python-coded extensions for larger systems, and code compiled extensions for advanced work. We’ve covered the first two of these categories in this book, and that’s plenty to get you started doing substantial programming in Python.

Table 29-1 summarizes some of the sources of built-in or existing functionality available to Python programmers, and some topics you’ll probably be busy exploring for the remainder of your Python career. Up until now, most of our examples have been very small and self-contained. They were written that way on purpose, to help you master the basics. But now that you know all about the core language, it’s time to start learning how to use Python’s built-in interfaces to do real work. You’ll find that with a simple language like Python, common tasks are often much easier than you might expect.

Once you’ve mastered the basics, you’ll find your Python programs becoming substantially larger than the examples you’ve experimented with so far. For developing larger systems, a set of development tools is available in Python and the public domain. You’ve seen some of these in action, and I’ve mentioned a few others. To help you on your way, here is a summary of some of the most commonly used tools in this domain:

To learn about other large-scale Python development tools available in the public domain, be sure to also browse the pages at the Vaults of Parnassus web site.

As we’ve reached the end of this part of the book, it’s time for a few exception exercises to give you a chance to practice the basics. Exceptions really are simple tools; if you get these, you’ve got them mastered.

See "Part VII, Exceptions and Tools" in Appendix B for the solutions.