Another very effective way of debugging Python is to use the Python debugger: pdb. Instead of using it directly though, you should definitely check out the pdbpp library. pdbpp augments the standard pdb interface by providing some convenient tools, my favorite of which is the sticky mode, which allows you to see a whole function while you step through its instructions.

There are several different ways to use this debugger (whichever version, it's not important), but the most common one consists of simply setting a breakpoint and running the code. When Python reaches the breakpoint, execution is suspended and you get console access to that point so that you can inspect all the names, and so on. You can also alter data on the fly to change the flow of the program.

As a toy example, let's pretend we have a parser that is raising KeyError because a key is missing in a dictionary. The dictionary is from a JSON payload that we cannot control, and we just want, for the time being, to cheat and pass that control, since we're interested in what comes afterward. Let's see how we could intercept this moment, inspect the data, fix it, and get to the bottom of it, with pdbpp:

# pdebugger.py
# d comes from a JSON payload we don't control
d = {'first': 'v1', 'second': 'v2', 'fourth': 'v4'}
# keys also comes from a JSON payload we don't control
keys = ('first', 'second', 'third', 'fourth')

def do_something_with_value(value):
    print(value)

for key in keys:
    do_something_with_value(d[key])

print('Validation done.')

As you can see, this code will break when key gets the 'third' value, which is missing in the dictionary. Remember, we're pretending that both d and keys come dynamically from a JSON payload we don't control, so we need to inspect them in order to fix d and pass the for loop. If we run the code as it is, we get the following:

$ python pdebugger.py
v1
v2
Traceback (most recent call last):
  File "pdebugger.py", line 10, in <module>
    do_something_with_value(d[key])
KeyError: 'third'

So we see that that key is missing from the dictionary, but since every time we run this code we may get a different dictionary or keys tuple, this information doesn't really help us. Let's inject a call to pdb just before the for loop. You have two options:

import pdb
pdb.set_trace()

This is the most common way of doing it. You import pdb and call its set_trace method. Many developers have macros in their editor to add this line with a keyboard shortcut. As of Python 3.7 though, we can simplify things even further, to this:

breakpoint()

The new breakpoint built-in function calls sys.breakpointhook() under the hood, which is programmed by default to call pdb.set_trace(). However, you can reprogram sys.breakpointhook() to call whatever you want, and therefore breakpoint will point to that too, which is very convenient.

The code for this example is in the pdebugger_pdb.py module. If we now run this code, things get interesting (note that your output may vary a little and that all the comments in this output were added by me):

$ python pdebugger_pdb.py
(Pdb++) l
 16
 17 -> for key in keys:  # breakpoint comes in
 18 do_something_with_value(d[key])
 19

(Pdb++) keys  # inspecting the keys tuple
('first', 'second', 'third', 'fourth')
(Pdb++) d.keys()  # inspecting keys of `d`
dict_keys(['first', 'second', 'fourth'])
(Pdb++) d['third'] = 'placeholder'  # add tmp placeholder
(Pdb++) c  # continue
v1
v2
placeholder
v4
Validation done.

First, note that when you reach a breakpoint, you're served a console that tells you where you are (the Python module) and which line is the next one to be executed. You can, at this point, perform a bunch of exploratory actions, such as inspecting the code before and after the next line, printing a stack trace, and interacting with the objects. Please consult the official Python documentation (https://docs.python.org/3.7/library/pdb.html) on pdb to learn more about this. In our case, we first inspect the keys tuple. After that, we inspect the keys of d. We see that 'third' is missing, so we put it in ourselves (could this be dangerous—think about it). Finally, now that all the keys are in, we type c, which means (c)ontinue.

pdb also gives you the ability to proceed with your code one line at a time using (n)ext, to (s)tep into a function for deeper analysis, or to handle breaks with (b)reak. For a complete list of commands, please refer to the documentation or type (h)elp in the console.

You can see, from the output of the preceding run, that we could finally get to the end of the validation.

pdb (or pdbpp) is an invaluable tool that I use every day. So, go and have fun, set a breakpoint somewhere, and try to inspect it, follow the official documentation and try the commands in your code to see their effect and learn them well.