In this chapter, we will cover errors and exceptions and how to find and fix them. Handling exceptions is an important part of writing reliable and usable code. We will introduce basic built-in exceptions and show how to use and treat exceptions. We'll introduce debugging and show you how to use the built-in Python debugger.

In this chapter, we cover the following topics:

• What are exceptions?
• Finding errors: debugging

12.1 What are exceptions?

The first error that programmers (even experienced ones) are confronted with is when the code has incorrect syntax, meaning that the code instructions are not correctly formatted.

Consider this example of a syntax error:

>>> for i in range(10)
  File “<stdin>”, line 1
    for i in range(10)
                      ^
SyntaxError: invalid syntax

The error occurs because of a missing colon at the end of the for declaration. This is an example of an exception being raised. In the case of SyntaxError, it tells the programmer that the code has incorrect syntax and also prints the line where the error occurred, with an arrow pointing to where in that line the problem is.

Exceptions in Python are derived (inherited) from a base class called Exception. Python comes with a number of built-in exceptions. Some common exception types are listed in Table 12.1.

Here are two common examples of exceptions. As you might expect, ZeroDivisionError is raised when you try to divide by zero:

def f(x):
    return 1/x

>>> f(2.5)
0.4 
>>> f(0)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "exception_tests.py", line 3, in f
    return 1/x
ZeroDivisionError: integer division or modulo by zero

A division by zero raises ZeroDivisionError and prints out the file name, the line, and the function name where the error occurred.

As we have seen before, arrays can only contain elements of the same datatype. If you try to assign a value of an incompatible type, ValueError is raised. An example of a value error is:

>>> a = arange(8.0) 
>>> a 
array([ 0., 1., 2., 3., 4., 5., 6., 7.]) 
>>> a[3] = 'string'
Traceback (most recent call last): 
  File "<stdin>", line 1, in <module>
ValueError: could not convert string to float: string

Here, ValueError is raised because the array contains floats and an element cannot be assigned a string value.

12.1.1 Basic principles

Let's look at the basic principles on how to use exceptions by raising them with raise and catching them with try statements.

Raising exceptions

Creating an error is referred to as raising an exception. You saw some examples of exceptions in the previous section. You can also define your own exceptions of a predefined type or use an exception of an unspecified type. Raising an exception is done with a command like this:

raise Exception("Something went wrong")

Here an exception of an unspecified type was raised.

It might be tempting to print out error messages when something goes wrong, for example, like this:

print("The algorithm did not converge.")

This is not recommended for a number of reasons. Firstly, printouts are easy to miss, especially if the message is buried in many other messages being printed to your console. Secondly, and more importantly, it renders your code unusable by other code. The calling code will not read what you printed and will not have a way of knowing that an error occurred and therefore has no way of taking care of it.

For these reasons, it is always better to raise an exception instead. Exceptions should always contain a descriptive message, for example:

raise Exception("The algorithm did not converge.")

This message will stand out clearly to the user. It also gives the opportunity for the calling code to know that an error occurred, and to possibly find a remedy.

Here is a typical example of checking the input inside a function to make sure it is usable before continuing. A simple check for negative values and the correct datatype ensures the intended input of a function to compute factorials:

def factorial(n):
    if not (isinstance(n, (int, int32, int64))):
       raise TypeError("An integer is expected")
    if not (n >=0): 
       raise ValueError("A positive number is expected")

The user of the function will immediately know what the error is, if an incorrect input is given, and it is the user's responsibility to handle the exception. Note the use of the exception name when raising a predefined exception type, in this case, ValueError followed by the message. By specifying the type of the exception, the calling code can decide to handle errors differently depending on what type of error is raised.

Summing up, it is always better to raise exceptions than to print error messages.

Catching exceptions

Dealing with an exception is referred to as catching an exception. Checking for exceptions is done with the commands try and except.

An exception stops the program execution flow and looks for the closest try enclosing block. If the exception is not caught, the program unit is left and it continues searching for the next enclosing try block in a program unit higher up in the calling stack. If no block is found and the exception is not handled, execution stops entirely and the standard traceback information is displayed.

Let's look at the factorial example from previously and use it with the try statement:

n=-3
try:
    print(factorial(n))
except ValueError:
    print(factorial(-n))   # Here we catch the error

In this case, if the code inside the try block raises an error of type ValueError, the exception will be caught and the action in the except block is taken. If no exception occurs inside the try block, the except block is skipped entirely and execution continues.

The except statement can catch multiple exceptions. This is done by simply grouping them in a tuple, like this:

except (RuntimeError, ValueError, IOError):

The try block can also have multiple except statements. This makes it possible to handle exceptions differently depending on the type. Let's see another example of multiple exception types:

try: 
    f = open('data.txt', 'r') 
    data = f.readline() 
    value = float(data) 
except FileNotFoundError as FnF: 
    print(f'{FnF.strerror}: {FnF.filename}') 
except ValueError: 
    print("Could not convert data to float.")

Here, FileNotFoundError will be caught if, for example, the file does not exist; and ValueError will be caught if, for example, the data in the first line of the file is not compatible with the float data type.

In this example, we assigned FileNotFoundError to a variable FnF by the keyword as. This allows access to more details when handling this exception. Here we printed the error string FnF.strerror and the name of the related file FnF.filename. Each error type can have its own set of attributes depending on the type. If the file with the name data.txt does not exist, in the preceding example, the message is:

No such file or directory: data.txt

This is a useful way to format the output when catching exceptions.

The try-except combination can be extended with optional else and finally blocks.

An example of using else can be seen in Section 15.2.1: Testing the bisection algorithm. Combining try with finally gives a useful construction when cleanup work needs to happen at the end. This is illustrated by an example for making sure a file is closed properly is:

try:
    f = open('data.txt', 'r')
    # some function that does something with the file
    process_file_data(f) 
except: 
    ... 
finally:
    f.close()

This will make sure that the file is closed at the end no matter what exceptions are thrown while processing the file data. Exceptions that are not handled inside the try statement are saved and raised after the finally block. This combination is used in the with statement; see Section 12.1.3: Context managers – the with statement.

12.1.2 User-defined exceptions

Besides the built-in Python exceptions, it is also possible to define your own exceptions. Such user-defined exceptions should inherit from the base class Exception. This can be useful when you define your own classes such as the polynomial class in Section 19.1.

Take a look at this small example of a simple user-defined exception:

class MyError(Exception):
    def __init__(self, expr):
        self.expr = expr
    def __str__(self):
        return str(self.expr)

try:
   x = random.rand()
   if x < 0.5:
      raise MyError(x)
except MyError as e:
   print("Random number too small", e.expr)
else:
   print(x)

A random number is generated. If the number is below 0.5, an exception is thrown and a message that the value is too small is printed. If no exception is raised, the number is printed.

In this example, you also saw a case of using else in a try statement. The block under else will be executed if no exception occurs.

It is recommended that you define your exceptions with names that end in Error, like the naming of the standard built-in exceptions.

12.1.3 Context managers – the with statement

There is a very useful construction in Python for simplifying exception handling when working with contexts such as files or databases. The statement encapsulates the structure try ... finally in one simple command. Here is an example of using with to read a file:

with open('data.txt', 'w') as f:
    process_file_data(f)

This will try to open the file, run the specified operations on the file (for example, reading), and close the file. If anything goes wrong during the execution of process_file_data, the file is closed properly and then the exception is raised. This is equivalent to:

f = open('data.txt', 'w')
try: 
    # some function that does something with the file 
    process_file_data(f) 
except:
    ... 
finally:
    f.close()

We will use this option in Section 14.1: File handling, when reading and writing files.

The preceding file-reading example is an example of using context managers. Context managers are Python objects with two special methods, __enter__ and __exit__. Any object of a class that implements these two methods can be used as a context manager. In this example, the file object f is a context manager as there are the methods f.__enter__ and f.__exit__.

The method __enter__ should implement the initialization instructions, for example, opening a file or a database connection. If this method has a return statement, the returned object is accessed using the construct as. Otherwise, the keyword as is omitted. The method __exit__ contains the cleanup instructions, for example, closing a file or committing transactions and closing a database connection. For more explanations and an example of a self-written context manager, see Section 15.3.3: Timing with a context manager.

There are NumPy functions that can be used as context managers. For example, the function load supports the context manager for some file formats. NumPy's function errstate can be used as a context manager to specify floating-point error handling behavior within a block of code.

Here is an example of working with errstate and a context manager:

import numpy as np      # note, sqrt in NumPy and SciPy 
                        # behave differently in that example
with errstate(invalid='ignore'):
    print(np.sqrt(-1)) # prints 'nan'

with errstate(invalid='warn'):
    print(np.sqrt(-1)) # prints 'nan' and 
                   # 'RuntimeWarning: invalid value encountered in sqrt'

with errstate(invalid='raise'):
    print(np.sqrt(-1)) # prints nothing and raises FloatingPointError

See Section 2.2.2: Floating-point numbers for more details on this example and Section 15.3.3: Timing with a context manager for another example.

12.2 Finding errors: debugging

Errors in software code are sometimes referred to as bugs. Debugging is the process of finding and fixing bugs in code. This process can be performed at varying degrees of sophistication. The most efficient way is to use a tool called a debugger. Having unit tests in place is a good way to identify errors early; see Section 15.2.2: Using the unittest package. When it is not obvious where or what the problem is, a debugger is very useful.

12.2.1 Bugs

There are typically two kinds of bugs:

• An exception is raised and not caught.
• The code does not function properly.

The first case is usually easier to fix. The second can be more difficult as the problem can be a faulty idea or solution, a faulty implementation, or a combination of the two.

We are only concerned with the first case in what follows, but the same tools can be used to help find why the code does not do what it is supposed to.

12.2.2 The stack

When an exception is raised, you see the call stack. The call stack contains the trace of all the functions that called the code where the exception was raised.

A simple stack example is:

def f():
   g()
def g():
   h()
def h():
   1//0

f()

The stack, in this case, is f, g, and h. The output generated by running this piece of code looks like this:

Traceback (most recent call last):
  File "stack_example.py", line 11, in <module>
    f() 
  File "stack_example.py", line 3, in f
    g() 
  File "stack_example.py", line 6, in g
    h() File "stack_example.py", line 9, in h
    1//0 
ZeroDivisionError: integer division or modulo by zero

The error is printed. The sequence of functions leading up to the error is shown. The function f on line 11 was called, which in turn called g and then h. This caused ZeroDivisionError.

A stack trace reports on the active stack at a certain point in the execution of a program. A stack trace lets you track the sequence of functions called up to a given point. Often this is after an uncaught exception has been raised. This is sometimes called post-mortem analysis, and the stack tracepoint is then the place where the exception occurred. Another option is to invoke a stack trace manually to analyze a piece of code where you suspect there is an error, perhaps before the exception occurs.

In the following example, an exception is raised to provoke the generation of a stack trace:

def f(a):
   g(a)
def g(a):
   h(a)
def h(a):
   raise Exception(f'An exception just to provoke a strack trace and a value a={a}')

f(23)

This returns the following output:

Traceback (most recent call last):

  File ".../Python_experiments/manual_trace.py", line 17, in <module>
    f(23)

  File "../Python_experiments/manual_trace.py", line 11, in f
    g(a)

  File "../Python_experiments/manual_trace.py", line 13, in g
    h(a)

  File "/home/claus/Python_experiments/manual_trace.py", line 15, in h
    raise Exception(f'An exception just to provoke a strack trace and a value a={a}')

Exception: An exception just to provoke a strack trace and a value a=23

12.2.3 The Python debugger

Python comes with its own built-in debugger called pdb. Some development environments come with the debugger integrated. The following process still holds in most of these cases.

The easiest way to use the debugger is to enable stack tracing at the point in your code that you want to investigate. Here is a simple example of triggering the debugger based on the example mentioned in Section 7.3: Return values:

import pdb

def complex_to_polar(z):
    pdb.set_trace() 
    r = sqrt(z.real ** 2 + z.imag ** 2)
    phi = arctan2(z.imag, z.real)
    return (r,phi)
z = 3 + 5j 
r,phi = complex_to_polar(z)

print(r,phi)

The command pdb.set_trace() starts the debugger and enables the tracing of subsequent commands. The preceding code will show this:

> debugging_example.py(7)complex_to_polar()
-> r = sqrt(z.real ** 2 + z.imag ** 2) 
(Pdb)

The debugger prompt is indicated with (Pdb). The debugger stops the program execution and gives you a prompt that lets you inspect variables, modify variables, step through commands, and so on.

The current line is printed at each step, so you can follow where you are and what will happen next. Stepping through commands is done with the command n (next), like this:

> debugging_example.py(7)complex_to_polar() 
-> r = sqrt(z.real ** 2 + z.imag ** 2) 
(Pdb) n 
> debugging_example.py(8)complex_to_polar() 
-> phi = arctan2(z.imag, z.real) 
(Pdb) n 
> debugging_example.py(9)complex_to_polar() 
-> return (r,phi) 
(Pdb) 
...

The command n (next) will continue to the next line and print the line. If you need to see more than one line at a time, the  command l (list) shows the current line with the surrounding code:

> debugging_example.py(7)complex_to_polar() 
-> r = sqrt(z.real ** 2 + z.imag ** 2) 
(Pdb) l
  2
  3 import pdb
  4
  5 def complex_to_polar(z):
  6 pdb.set_trace()
  7 -> r = sqrt(z.real ** 2 + z.imag ** 2)
  8 phi = arctan2(z.imag, z.real)
  9 return (r,phi)
 10
 11 z = 3 + 5j
 12 r,phi = complex_to_polar(z) 
(Pdb)

The inspection of variables can be done by printing their values to the console using the command p (print) followed by the variable name. An example of printing variables is: 

> debugging_example.py(7)complex_to_polar() 
-> r = sqrt(z.real ** 2 + z.imag ** 2) 
(Pdb) p z 
(3+5j) 
(Pdb) n 
> debugging_example.py(8)complex_to_polar() 
-> phi = arctan2(z.imag, z.real) 
(Pdb) p r 
5.8309518948453007 
(Pdb) c 
(5.8309518948453007, 1.0303768265243125)

The command p (print) will print the variable; the command c (continue) continues execution.

Changing a variable mid-execution is useful. Simply assign the new value at the debugger prompt and step or continue the execution:

> debugging_example.py(7)complex_to_polar() 
-> r = sqrt(z.real ** 2 + z.imag ** 2) 
(Pdb) z = 2j 
(Pdb) z 
2j 
(Pdb) c 
(2.0, 1.5707963267948966)

Here the variable z is assigned a new value to be used throughout the remaining code. Note that the final printout has changed.

12.2.4 Overview – debug commands

In Table 12.2, the most common debug commands are shown. For a full listing and description of commands, see the documentation for more information [24]. Note that any Python command also works, for example, assigning values to variables.

If you want to inspect a variable with a name that coincides with any of the debugger's short commands, for example, h, you must use !h to display the variable.

12.2.5 Debugging in IPython

IPython comes with a version of the debugger called ipdb. At the time of writing this book, the differences to pdb are very minor but this may change.

There is a command in IPython, %pdb, that automatically turns on the debugger in case of an exception. This is very useful when experimenting with new ideas or code. An example of how to automatically turn on the debugger in IPython is:

In [1]: %pdb # this is a so - called IPython magic command 
Automatic pdb calling has been turned ON

In [2]: a = 10

In [3]: b = 0

In [4]: c = a/b
___________________________________________________________________
ZeroDivisionError                  Traceback (most recent call last) 
<ipython-input-4-72278c42f391> in <module>() 
—-> 1 c = a/b

ZeroDivisionError: integer division or modulo by zero 
> <ipython-input-4-72278c42f391>(1)<module>()
      -1 c = a/b
ipdb>

The IPython magic command %pdb at the IPython prompt automatically enables the debugger when exceptions are raised. Here the debugger prompt shows ipdb instead to indicate that the debugger is running.

12.3 Summary

The key concepts in this chapter were exceptions and errors. We showed how an exception is raised to be caught later in another program unit. You can define your own exceptions and equip them with messages and current values of given variables.

The code may return unexpected results without throwing an exception. The technique to localize the source of the erroneous result is called debugging. We introduced debugging methods and hopefully encouraged you to train them so that you have them readily available when needed. The need for serious debugging comes sooner than you might expect.