PART 1
Basic Scripting
Techniques

CHAPTER 1
Shell Script Debugging

Even though this book isn't a "how to script" manual, some concepts that are fundamental to writing successful scripts should be discussed. Debugging is one of them. Debugging code is a significant part of writing code. No matter how disciplined you are or how skilled you become at coding, you will have bugs in your code, in the form of either syntax or logic errors. The syntactical problems tend to be simpler to resolve since many times they show up when the code throws an error when it is run. The logical bugs, on the other hand, may be more difficult to track down since the code may run without error, but the resulting output does not match the design of the program. The more complex your code becomes as your skill increases, the more difficult these types of problems will be to detect.

Since writing bug-free code is nearly impossible, you need a few techniques up your sleeve that will help you finish, diagnose, repair, and clean up your code. This chapter presents a few ways to debug code that I have used consistently and that help me extract details from the inner workings of my scripts. These techniques validate that the code is living up to my expectations and demonstrate where the code needs more work to perform the intended task.

Shell Trace Options

The first technique—using the set command—is the simplest to implement and can give you great amounts of detail about how the logic is progressing and the values of variables internal to your script. Using the set command is really just using shell options to display verbose output when the script is running. One of the functions of the set command is to turn on and off the various options that are available in the shell. In this case, the option being set is -x, or xtrace. This is where the running script will, in addition to any normal output, display the expanded commands and variables of a given line of code before the code is run. With this increased output, you can easily view what is happening in the running script and possibly determine where your problem lies.

When you put the instruction set -x into your script, each of the commands that execute after that set instruction will be displayed, together with any arguments that were supplied to the command, including variables and their values. Each line of output will be preceded by a plus-sign (+) prompt to designate it as part of the trace output. Traced commands from the running shell that are being executed in a subshell are denoted by a double plus sign (++).

To demonstrate what the use of set -x can do for you, consider this script:

Note that the set -x line is currently commented out. When this script is entered in the file example and run, the behavior is as expected.

or

This is the output when the set -x line is uncommented:

or

The output is a verbose trace of the script's execution. Note that the lines without the plus sign are the output of the script that would be displayed if the script were run without tracing enabled. As you can see, this type of trace is highly useful in determining the value that variables contain during the execution of a script, as well as the route that the code took based on the conditions satisfied.

A shell option that is a slight variation of this output can also be used for troubleshooting. The -v option to the shell enables verbose mode and outputs the script code (as it is being executed) to the standard error file handle (often abbreviated as stderr). More specifically, in the case of a shell script, each line of code that is encountered during execution is output to stderr along with any other output from the script. (Chapter 9 contains more discussion of file handles.) The following is the output from the same script when the set -v line is implemented:

or

The verbose (-v) option to the shell is more useful if you simply want to see the running code of the script that you're working with (as opposed to the expanded values of variables) to make sure the code is working as designed with the xtrace (-x) option. Both options can be employed together by using set -xv, and you'll see both types of output at the same time, although it may be difficult to wade through.

Both the verbose and xtrace options are valuable in their own way for troubleshooting both logical and syntactical problems. As with all options to the shell, they can be turned on and off. The syntax for disabling an option is the opposite of that for turning on an option. Instead of using a minus (-) sign as you did before to enable an option such as in -x, you would use a plus sign, as in +x to disable the option. This will disable the option from that point on. This is very useful if you want to debug only a small portion of the script. You would enable the option just prior to the problem area of code, and disable it just after the problem area so you aren't inundated with irrelevant output.

Simple Output Statements

The next debugging technique—the use of echo or print commands in the code—is also very simple, but it is used frequently to gather specific variable values from a running script rather than displaying potentially large amounts of data using the set -x option. Typically these commands are used for simple output of a script to some type of display or file. In this case, however, they will be used as a checkpoint in the code to validate variable assignments.

These additional output instructions are used regularly in at least a couple of ways. The first way is to output the value of a specific variable at a specific time. Sometimes variables get changed when you aren't expecting them to be, and adding a simple output line will show this. The main advantage of this type of output compared to set -x is that you have the ability to format your output for ease of reading. While set -x has a valid use and is valuable in tracing through the running of a script, it can be cumbersome to isolate the exact piece of data that you're looking for. With an echo or print statement, you can display a single line of output with multiple variables that include some headings for easy reading. The following line is an example of the code you might use:

The output doesn't need to be polished since it is simply for your validation and troubleshooting, but you will want it to be meaningful so you can see the exact data you're looking for at its exact spot in the code.

The second way is to output a debugging line to verify that the logic is correct for known input data. If you are running a script that should have known results but does not, it may contain a logical error where what you've designed and what you've coded don't quite match. Such errors can be difficult to find. Adding some echo statements in key positions can reveal the flow of control through the script as it executes, and so validate whether you are performing the correct logical steps.

I've modified the script slightly to add echo statements at two key positions, but only one of the statements in each echo-statement pair will be executed because of the if statement. This way you not only see the output of the statement itself, but you know which condition of the if statement the code executed. In the following very simple example code, you can see that there is an echo statement as part of the original code. When there are many conditions and comparisons without output, these types of statements are very valuable in determining if your logic is correct.

Tip I tend not to format these debugging echo statements with the traditional indentation because they are usually temporary additions while I'm troubleshooting. Indenting them with the normal code makes them more difficult to find when I want them removed.

Controlling Output with Debug Levels

The problem with using echo statements as I described previously is that you have to comment or remove them when you don't want their output displayed. This is fine if your program is working to perfection and will not need further modification. However, if you're constantly making changes to a script that is actually being used, the need to add back or uncomment echo statements each time you debug can be tiresome. This next debugging technique improves on the basic echo statement by adding a debugging level that can be turned on or off. After you've prepped your script once, enabling or disabling debugging output is as simple as changing a single variable.

The technique is to set a debug variable near the beginning of the script. This variable will then be tested during script execution and the debug statements will be either displayed or suppressed based on the variable's value.

The following is our original example, modified once again for this technique:

This idea can be expanded to include many debug statements in the code, providing output of varying levels of detail during execution. By varying the value to which $debug is compared in the test (e.g., $debug -gt 2), you can, in principle, have an unlimited number of levels of debug output, with 1 being the most simple and the highest-numbered level of your choosing being the most complex. You can, of course, create any debug-level logic you wish. In the example here, I am checking if the debug variable is greater than some specified value. If it is, the debug output is displayed. With this model, if you have various debug output levels and your debug variable is assigned a value higher than the highest debug level, all levels below that one will be displayed. Here are a few lines of code to illustrate the point:

If these three lines were executed in a script, only the output from the first two would be displayed. If you were to change the logic of the test from "greater than" (-gt) to "equal to" (-eq), only the output of the last debug statement would be displayed.

My mind works best when things are simple. For simple scripts I usually set the debug value to either on or off. Multilevel debugging is more valuable for larger scripts, since the code can become quite complex and difficult to track. Using multiple debug levels in a complex script allows you to follow the code's logic as it executes, selecting the level of detail desired.

A further improvement to this technique is to design the script to accept a debug switch when the script is called. You can then use the switch to specify whatever value of debug level you desire for the information you're looking for, without having to modify the code every time you would like to view debugging output. See Chapter 5 for more information on how to process command-line switches passed to a script.

Simplifying Error Checking with a Function

The last debugging approach I'll discuss is an error-checking technique. Instead of simply checking the values of variables and debug statements, this method is more proactive. You evaluate the final condition of an executed command and output a notification if the command was unsuccessful.

The code is a very simple function that I include in a standard function library I use. (You can find information on function libraries in Chapter 2.) This function uses the $? shell internal variable. The shell sets this variable automatically to the value of the previous command's return code. This function uses that value and alerts you of the command's success or failure. A command's return code is a numeric value that defines the exit status of the most recently executed command. Traditionally, a successful completion of a command will yield a value of 0 for the $? shell variable. Error checking is an important part of all types of coding. Not only do you need to get the commands, logic, and functionality of the program correct along the desired path of execution, you should also check for problem conditions along the way. Anticipating potential problems will make your code more robust and resilient.

The function that is included here is called alert since it notifies you of any issues. A function is something like a mini-program within the main code, and it can be called like any other regular command. A good use for a function is to reduce duplication of code if you're going to perform a given task many times throughout the script. The alert function, like all others, needs to be included in the code (that is, defined) prior to it being called by the script. Once the function has been defined, it should be called following any critical commands. By critical, I mean those that are most important to the success of the script. For instance, if you have a script that does some file manipulation (such as finding files that match certain criteria and moving them around or modifying them), there will be plenty of lines of code, but the key commands might be find, mv, sed, and a few others. These are the commands that are performing real action, and I would consider them critical.

When you identify a line of code that you want to check, you should call the alert function directly following the execution of that command because that is when you can first retrieve the value of the $? shell variable and thus determine the effect of the executed command.

The alert function's code is simple. The function is called with $? as its first argument, and a string describing what is being reported as its second argument. If the value of $? is 0, the function echoes that the operation succeeded; otherwise it echoes that it didn't.

The following is an example of a command followed by a call to the alert function. The command simply mails the contents of a log file specified in the environment variable LOG to a predefined recipient specified in the variable TO.

Depending on the success or failure of the cat and mail commands, the output of the call to alert would look like this:

or like this:

with the LOG and TO variables expanded to their values.

The following code is a more advanced form of the previous alert function. It is simpler to call and has a couple of additional features. First, it has been combined with a global DEBUG variable so that it will only report issues if that variable is set. It has also been combined with a global STEP_THROUGH variable. When that variable is set to 1, the code pauses for input on any error it encounters. If the STEP_THROUGH variable is set to 2, the function pauses every time it has been called.

We first set the RET_CODE variable to the last command's return code and then determine if the DEBUG variable is either undefined or set to 0. The -z test determines if a variable has a zero length. If either of these conditions are true, the function will return to the main code from which it was called.

The next step is to determine if the return code of the command was nonzero, which implies a failure of some kind. If it is zero, the code echoes out a warning that states what the command was attempting to do and the return code that it received. The $* shell internal variable holds all the positional parameters that were passed to the function itself. If it was called with something like alert creating the archive of last months records and there was a problem, the output would look like this: Warn: creating the archive of last months records failed with a return code of 1. In a real case, the return-code value will vary.

The last line in this code segment determines if the DEBUG variable is greater than 9. If this is the case, the script will exit with the most recent failure's return code.

This last bit of code is where the function allows you to pause, either at only nonzero return codes or any time the alert function was called. You could improve the function by sending output to an optional log file for later review.

Manual Stepping

My final comments on debugging code stem from an interaction I had recently with a friend who was trying to debug an issue with her script. The code attempted to move around some files on the local disk as well as on a Network File System (NFS)-mounted file system. It was receiving a puzzling "permission denied" error. Nothing was obviously wrong with the code, and the permissions on the directories seemed correct. It wasn't until we started performing the steps in the script manually that we found the problem. A file move was attempting to overwrite a preexisting file in the destination directory with read-only permissions and obviously (hindsight, you know) this was what triggered the "permission denied" errors. When we initially looked at the code and the directories involved, we were focusing on the directory permissions and the user that needed to write to the directory. We failed to notice the permissions on the files in the directory.

I'm not suggesting that all problems are this easy to find. Debugging code can take hours, days, or even longer when the code is complex, but a few lessons can be learned from this simple example.

First, before you start writing a program, attempt to perform the code's steps manually where appropriate. This won't always be feasible, but when it is you may be able to weed out some trouble spots before they are mixed in with all the script's other tasks.

Second, try out the code with sample input and attempt to follow it through by performing the loops and conditionals as they are written. It is not an easy task, but attempt to look at the code as objectively as possible without making assumptions, and ask the too-obvious-to-ask questions about what is happening.

Last, seek out another set of eyes. Using a third party is an excellent way of finding problems, especially when you've been working on the same issue for a long time. Sometimes a peer with a fresh viewpoint is able to solve the problem right away.