Once you get beyond some trivial scripts, you'll soon find that a script that just runs command after command after command doesn't work for most real-world scripts. Instead, you need the capability to perform some sort of logic within the program, test conditions, and take alternative measures if certain tests fail. You may need to perform some operation on each file in a directory or back up only selected files. Shell scripting languages offer a variety of ways to test values and then execute different commands based on the results of the tests. All of these things fall under the concept of controlling how your scripts run.
This chapter covers:
Advanced variable referencing, useful for conditionals and iteration
Iterating with the
forandforeachstatementsUsing
if-then-elsestatements to test values and execute different commands based on the resultsTesting for a variety of values using the
switch-casestatementsLooping while a condition is true
Chapter 2 introduced the concepts of variables, named data holders. Each variable can hold a value, which you can later reference by placing a dollar sign in front of the variable name. When it comes to looping and conditional tests, however, you often need more options for referencing variables. This section shows you some of those options.
The Bourne shell, along with bash and ksh, provides another means to reference the value of a variable. The normal means is with a dollar sign:
$variable_nameThis example references the value of the variable variable_name. If you have not set a value into variable_name, you will get no value. This is not an error.
You can also use curly braces to reference a variable's value:
${variable_name}This method uses the curly braces to clearly delineate the start and end of the variable name. With just the dollar sign, you have a clear separation for the start of the variable name but not the end.
This alternate method proves very useful if you want to place a variable with other text immediately adjacent or, better still, when you need to append something to the value of the variable. For example, if you have a variable with the value abc and want to output abcdef, the natural way to do this would be to place the variable name next to the text def. Unfortunately, the following would output nothing except for a new line:
myvar=abc echo $myvardef # nothing
Note how the shell sees nothing to separate the name of the variable, myvar, from the text def. Thus, the shell interprets $myvardef as referencing a variable named myvardef. If you have not set this variable, which is likely, then myvardef will hold nothing.
But if you use the ${variable_name} alternate format, you can then make a clean separation between the variable name and the remaining text:
${myvar}defThese variable constructs are useful when you start to test variable values and use variables to control the flow of how your scripts run. One of the most common means to control the flow in a script is through looping.
Looping is the process of repeating the same script elements a given number of times. Looping works best when applied to one of the following tasks:
Performing the same operation on a number of files
Performing an operation for a fixed number of times, such as trying to reconnect to the network three times (and then giving up if the connection cannot be re-established)
Performing an operation on a given number of items
For all of these iterative tasks, you can use the shell for loop. The basic syntax of the for loop is:
forvariableinlist_of_itemsdocommand1command2...last_commanddone
You need to provide the name of a variable. The shell will set the variable to the current value on each iteration through the loop. The commands between the do and the done statements get executed on each iteration.
The list_of_items differs depending on what you want to use for iteration, such as a list of files.
Quite a few shell scripts need to loop over a list of files. Backup scripts, for example, might check each file in a directory to see if the file is newer than the last backup. If the file has been modified since the last backup, then you may want a script to back up the file.
The basic syntax for looping over a number of files is to use the for loop with a list of items that resolves to the files. For example:
forfilenamein * docommand1command2...last_commanddone
In this example, the * resolves to the list of files in the current directory. The for loop then iterates over each file, setting the variable filename to the name of the current file.
Iterating over all the files in a directory is a very common operation in shell scripts.
As another example, you can use the for loop to help create a backup script. Backup scripts save a copy of files to some form of external storage. In most larger environments, administrators run incremental backups. With incremental backups, administrators first perform a full backup of all files and then selectively back up only those files that have changed since the last full backup. Typically, administrators start the process again every so often, such as performing a full backup every month or every week. Every day in between, however, administrators perform incremental backups.
Chapter 13 covers more on administrative tasks such as backups.
In addition to looping over a set of files, you can write scripts that loop for a fixed number of iterations.
The previous example scripts have allowed the shell to fill in the list_of_items from a listing of files. You can also indicate specific items to iterate over. For example, to teach an expensive computer to count, you can set up a for loop like the following:
for i in 1 2 3 4 5 6 7 8 9 10 do done
In this example, the for loop will set the variable i to each of the values 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, performing whatever commands are in the do-done block.
You can place the for loop and the do statement on the same line, but you must separate the two elements with a semicolon as if they were separate commands (which in a sense, they are). For example:
# Counts by looping for a fixed number of times
# Note do on same line requires semicolon.
for i in 1 2 3 4 5 6 7 8 9 10; do
echo -n "...$i"
done
echo # Output newlineThe resulting output will be identical to the previous example.
As you can guess from the way the example scripts specify the values to iterate over, there is nothing stopping you from making a for loop count backward. For example:
# Counts backwards
for i in 10 9 8 7 6 5 4 3 2 1
do
echo -n "...$i"
done
echo # Output new line
echo "Blast off!"Enter this script and save it under the name counter2. When you run this script, you should see the following output:
$ sh counter2
...10...9...8...7...6...5...4...3...2...1
Blast off!This is the same loop as before, only the values are listed in reverse order. There is also a final echo statement making a space-flight reference. You can place values in any order.
The previous example shows a very efficient countdown to launching a spacecraft because there is no pause between iterations. By the time you see the 10, you should also see the 1. The craft is ready to launch. Aside from the remarkable efficiency, you may want to put some kind of wait into a script of this type. You can do that with the sleep command.
The sleep command takes one command-line argument, the length of time to sleep in seconds. (See the online manuals on the sleep command for using other units of sleep time.) For example:
sleep 5
This command causes the shell to wait for five seconds before going on to the next command.
Note that the sleep amount will not be exact, as your computer will be performing other work (running other programs) at the same time. It should be at least five seconds, however, with an amount very close to five seconds.
The bash shell supports a for loop syntax that appears a lot like that of the C programming language. The basic syntax is:
max=upper_limitfor ((i=1; i <= max ; i++)) docommands... done
This is like the C programming language, not the C shell.
In this example, the variable max holds the upper limit, such as 10. The for loop will iterate using the variable i, where i starts at 1 and continues until it reaches a value greater than 10. The i++ syntax refers to incrementing the variable i.
Some things to note in the syntax include:
You need two sets of parentheses on each side, (( and )).
You reference the loop iteration variable,
iin this case, without the$.This syntax is supported by the bash shell, although you may find it works with the Bourne shell on systems where bash also provides the Bourne shell, such as Linux and Mac OS X.
If you are familiar with C programming, then this syntax will appear like a C for loop, with some modifications for use in a shell. Otherwise, you probably want to avoid this syntax.
The C shell doesn't support the for loop but does support something similar, a foreach loop. The syntax for foreach is:
foreachvariable(list_of_items)command1command2...last_commandend
The loop executes once for each value in the list of items. On each iteration, the shell sets the variable to the current value from the list of items.
Some differences from the Bourne shell for syntax include:
Use
foreachinstead offor.There is no
in.There is no
doto start the block of commands.Instead of
done, the C shell usesend.The list of items must be enclosed within parentheses.
The Bourne shell if statement checks whether a condition is true. If so, the shell executes the block of code associated with the if statement. If the condition is not true, the shell jumps beyond the end of the if statement block and continues on. Only if the condition is true will the shell execute the block.
The basic syntax of the sh if statement follows:
if (condition_command) thencommand1command2...last_commandfi
If the condition_command resolves to a true value (see the section What Is Truth? for more on truth and falsehood), then the shell executes the commands in the block: command1, command2, and so on to the last_command in the block. Note how if statements end with a fi (if spelled backward). The fi ends the block of commands.
The condition_command includes numerical and textual comparisons, but it can also be any command that returns a status of zero when it succeeds and some other value when it fails.
The if statement is a part of just about every programming language. If you are not familiar with an if statement, however, think of the following general-purpose tasks:
If a file has been modified since the last backup, then back up the file.
If the Oracle database is not running, then send an email message to an administrator.
If a new version of a software package, such as the Mozilla Firefox Web browser, has been released, then update the software package.
If you were assigned any of these tasks—that is, writing a shell script to perform these functions—you would use an if statement. If you are tasked with assignments like these, however, you may need to reword the assignments. For example, the following tasks mirror those from the preceding list but don't use that handy word if:
Only modify those files that have been modified since the last backup.
Whenever the Oracle database is not running, send an email message to an administrator.
Update all applications for which a new package is available.
These task statements don't include if, but if you think about it for a while, the if statement fits right in. All of these statements involve a condition. If the condition is true, then the script is supposed to take some action. This type of task fits well with the if statement.
Another good use for an if statement (or a case statement, covered below) is a script that tries to figure out the proper commands to use on different operating systems. For example, if a script is running on Windows, the file copy command will be copy. If the script is running on Unix, then the copy command will be cp. Handling operating system issues is another good use for an if statement.
Whenever you have a task like these, think of the if statement. The What Is Truth? section later in this chapter includes examples to show how to create if statements in your scripts.
In addition to the normal if statement, you can extend the if statement with an else block. The basic idea is that if the condition is true, then execute the if block. If the condition is false, then execute the else block. The basic syntax is:
if (condition_command) thencommand1command2...last_commandelsecommand1command2...last_commandfi
Again, the entire statement ends with a fi, if spelled backward. If the condition_command is true, then the shell executes the first block of commands. If the condition_command is not true, then the shell executes the second block of commands.
There is also an elif that combines the else with a nested if statement. See the related section later in the chapter for more on elif.
By convention, Unix commands return a command result of 0 (zero) on success. On errors, most commands return a negative number, although some return a positive number such as 1 (one). Because the Bourne shell if statement is tied to running a program and testing for success, it only makes sense that 0 be considered true and 1, or any nonzero number, false. Commands return these numbers when they exit.
Try the following scripts to get an idea of how the if statement works, and especially the focus on running commands.
Try out these two scripts until you are comfortable with the way shells handle if statements. Just about every programming language has an if statement. Shells, however, by calling on a program to handle the condition, can appear odd to anyone who has a programming background. That's because the if statement usually compares variables in other programming languages instead of running commands.
Note that you can run the test command to compare variable values. See the section Testing with the test Command for more on this topic.
In addition to the return0 and return1 scripts shown here, Unix and Linux systems include two special programs, true and false, that return a true status and a false status, respectively. These are actual programs on disk (usually in /bin). Some shells include true and false as built-in commands. Bash, for example, does this.
Remember that the if statement wants to run a command. The if statement then examines the results of the command to decide whether the results were true. You can then use if with different commands to see the results.
For example, you can call the make command, which is used to control the compiling of software applications, especially C and C++ programs. The make command will return a success exit code (0) if everything went fine and make successfully built whatever it was supposed to build. The make command will return a failure exit code on any errors.
Note that you may need to install the make command if the command is not available on your system. See your system documentation for more on installing software packages. The following section assumes you have installed make.
Most Unix and Linux commands output their results. The ls command, for example, outputs the listing of the files as requested by you, the user. Use a greater-than sign, >, to redirect the standard output of a command to a given file. The basic syntax is:
command>output_file
You can try this with the ls command. For example:
$ ls > listing.txtThis command sends the output of the ls command into the file listing.txt. If the file doesn't exist, the shell creates the file. If the file already exists, the shell truncates the file. When the command completes, you should be able to view the contents of the file listing.txt, which should hold the same data as if the ls command sent its output to the screen.
To get rid of a command's output, redirect the output to /dev/null. You can try this at the command line:
$ ls > /dev/null $
The system should respond with another prompt with no output from ls. The greater-than sign, >, redirects the output of the command (ls in this case). Here, the > redirects the output of the ls command to the file /dev/null, often called the null device or the bit bucket. That's because /dev/null is a special file that the operating system considers a device. In this case, /dev/null consumes all input sent to it, kind of like a black-hole star.
The /dev directory is a special directory that holds what are called device files or device drivers, special code that interfaces with devices such as hard disks, DVD drives, USB keychain flash drives, and so on.
Chapter 8 covers more on redirecting the standard input, output, and error files. For now, just treat this command as sending the output of the ls command to the null device, which consumes all the bytes sent to it.
Note that you can also capture the output of a command into a variable. See the section Capturing the Output of Processes in Chapter 9 for more on this topic.
To get a taste of redirecting, try the following example, which combines redirection with the ls command inside an if-then conditional statement.
You can also try using the which command to test for a program's existence. The which command displays the full path to a command, if the command exists. If the command does not exist, the which command generates an error. For example:
$which ash/bin/ash $which zsh/usr/bin/which: no zsh in (/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin: /home2/ericfj/bin:/usr/java/j2sdk1.4.1_03/bin:/opt/jext/bin)
The which command in Mac OS X and the Cygwin utilities for Windows does not support returning a true or untrue value. The which command always returns a true value. Thus, the tests in the find_shells script generate falsely positive results.
Look at the repetition in the find_shells script. Such repetition cries out for using functions, the topic of Chapter 10. You can also use other techniques to reduce the duplication.
The Bourne shell syntax for the if statement allows an else block that gets executed if the test is not true. You can nest if statements, allowing for multiple conditions. As an alternative, you can use the elif construct, short for else if. The basic syntax is:
if (condition_command) thencommand1command2...last_commandelif (condition_command2) thencommand1command2...last_commandelsecommand1command2...last_commandfi
With the normal if statement, and in this case, if the condition_command returns zero (true), then the shell executes the commands underneath. In this case, if the condition_command returns a nonzero status, or false, then the shell will jump to the next condition to check, the condition_command2. If the condition_command2 returns zero, then the commands shell executes the commands underneath the elif statement. Otherwise, the shell executes the commands underneath the else statement.
Theoretically, the elif keyword is not needed, as you can nest if statements. But in most cases, if makes your scripts easier to understand.
As with for loops, you can nest if statements, although using the elif construct eliminates many of the nested if statements that would be required without it.
If you nest too many if statements, your scripts will become very hard to read. After two or three nested if statements, you should stop.
The following example shows how you can nest if statements.
Because the Bourne shell if statement was designed to work with commands, and because many scripts need to compare values, the Bourne shell includes the test command. The test command forms a sort of all-encompassing means to test files, values, and most everything else. Depending on the command-line options you pass, test can compare values, check for file permissions, and even look for whether files exist. The test command is normally implemented by the shell as a built-in command. But test also exists as a program on disk, typically in /usr/bin.
To control what kind of test is performed, you must figure out the needed command-line options for the test command to build up an expression to test. The test command uses the command-line options to determine what comparisons to make. Then the test command exits and returns a 0 if the expression tested was true and a nonzero value (1) if the expression was false. If test encounters an error, it returns a number greater than 1.
To build up a test, you use the command-line options to specify how to compare values. For example:
test $x -eq $y
This example tests whether the value of x is equal to that of y, using the -eq command-line option. This is assumed to be a numeric equality test. The shell will try to make numbers out of strings if needed.
The following table shows the numeric test options. The variables x and y should have a numeric value, obviously.
Test | Usage |
|---|---|
$x -eq $y | Returns true if x equals y |
$x -ne $y | Returns true if x does not equal y |
$x -gt $y | Returns true if x is greater than y |
$x -ge $y | Returns true if x is greater than or equal to y |
Returns true if x is less than y | |
$x -le $y | Returns true if x is less than or equal to y |
Passing no test to test results in a false value.
The test command can also compare text strings. The test command compares text strings by comparing each character in each string. The characters must match for the comparison to be true.
For some reason, text has been called strings or text strings in just about every programming language. The usage comes from the idea of a text string as a string of characters (often of bytes), one after another. The analogy doesn't really hold up, but the terminology has.
You can compare based on the text string values or check whether a string is empty or not. The following table shows the text string test options.
Test | Usage |
|---|---|
"dollar;s1" = "dollar;s2" | Returns true if s1 equals s2 |
"dollar;s1" != "dollar;s2" | Returns true if s1 does not equal s2 |
$s1 | Returns true if s1 is not null |
$s1 -z | Returns true if the length of s1 (the number of characters in s1) is zero |
$s1 -n | Returns true if the length of s1 (the number of characters in s1) is not zero |
The following Try It Out shows the tests from this table in use in a shell script.
In addition to the numeric and string tests, you can use the test command to test files. In fact, testing files is the primary use of the test command in most scripts.
The following table lists the file options for the test commands.
Test | Usage |
|---|---|
-d filename | Returns true if the file name exists and is a directory |
-e filename | Returns true if the file name exists |
-f filename | Returns true if the file name exists and is a regular file |
Returns true if the file name exists and you have read permissions | |
-s filename | Returns true if the file name exists and is not empty (has a size greater than zero) |
-w filename | Returns true if the file name exists and you have write permissions |
-x filename | Returns true if the file name exists and you have execute permissions |
These tests are described in Chapter 5.
Each of the test options introduced so far works alone. You can combine tests using the binary and negation tests. The following table shows these add-in tests.
Test | Usage |
|---|---|
! | Negates the test. |
-a | Returns true if two tests are both true and false otherwise. This is an AND operation. |
With the negation test, !, you can negate any of the other tests. So, for example, if you use the -eq option to compare two numeric values for equality, you can use the ! to negate the test, in other words to check for inequality:
if (test ! $x -eq $y ) then
echo "x != y. Expected."
else
echo "false: x = y."
fiIn this example, the $x -eq $y tests if the value of x equals the value of y. The test will return true if the values are equal and false if they are not equal. Placing an exclamation mark into the test converts the test to return true if the values are not equal and false if they are equal. (Try this test in the test_binary script following in this section.)
The options -a and -o form the binary test options. But even though these are listed as binary comparisons—that is, checks at the bit level—in most cases you can treat -a as a logical AND. Treat -o as a logical OR. For example:
x=3
y=10
if (test $x -eq $x -a $y -eq $y) then
echo "x = x and y = y. Expected."
else
echo "false: x = x and y = y."
fiIn this case, x equals x and y equals y. In addition:
if (test $x -eq $x -a $y -ne $y) then
echo "x = x and y = y."
else
echo "false: x = x and y != y. Expected."
fiThis example tests whether x equals x and y does not equal y. This is obviously false.
The -a option takes precedence over the -o option if both appear in the same call to the test command.
You can try out the AND, OR, and negation tests with the following example. These tests tend to get complex because each of these new tests requires an additional test (!) or tests (-a and -o) to run.
In addition to the test command, you can write your tests using [, a shorthand for test. It may seem weird, but [ is actually a command. You can check this with the which command:
$ which [
/usr/bin/[Note that ] is not a command. The ] is interpreted as a command-line argument to the [ command.
Even though it is a command, [, like test, is often built into the shell. Shells such as bash, for example, come with a built-in test and [ commands. You can see this by using the type command in bash:
$type [[ is a shell builtin $type testtest is a shell builtin
Note that type is also a built-in shell command:
$ type type
type is a shell builtinIf you use type with the name of a command, you will see output like the following:
$ type rm
rm is /bin/rmThis indicates that rm, for removing, or deleting, files, is a command. Shells such as bash, csh, and tcsh support aliases, where you can enter in an alias name and use the alias in place of a command. The type command on bash can also tell you about whether a command is aliased or not. For example:
$ type ls
ls is aliased to `ls --color=tty'The alias command lists all the aliases. When you run this command with bash, you will see output like the following:
$ alias
alias l.='ls -d .* --color=tty'
alias ll='ls -l --color=tty'
alias ls='ls --color=tty'
alias vi='vim'The output on your system may differ, depending on what aliases are set up in your shell.
With tcsh or csh, the output will appear slightly different. For example:
$ alias
h history
l. ls -d .* --color=tty
ll ls -l --color=tty
ls (ls -CF)
qtopia /opt/Qtopia/bin/qtopiadesktop
rm (rm -i)
vi vim
xcd cd !*; echo -n "^[]2;$cwd^G"As before, the output on your system may differ, depending on what aliases are set up in your shell.
Tying this together, you can write your tests using test or [, as you prefer. The syntax is:
if [test_options] thencommandselsecommandsfi
Use square brackets, [ and ], around the test options. Note that this example places the then construct on the next line. This is needed because this example skips the parenthesis. You can keep the then on the same line if you include parentheses, but this syntax looks odd:
if ( [test_options] ) thencommandselsecommandsfi
The test_options should be the command-line options and arguments normally passed to the test command. For example:
x=5
y=10
if [ $x -eq $y ]
then
echo "X = Y."
else
echo "X != Y. Expected."
fiThe [ syntax is supposed to make the tests easier to understand, but using [ can be confusing because of the way [ was implemented as a command. Even so, this syntax is used in most scripts in place of the test command.
Nested if statements work well, but as soon as you are confronted with a number of possible actions to take, nested if statements start to confuse. You can simplify complex tests with the case statement. Whenever you have a question or test with a number of choices and a script that requires specific actions for each choice, you can use the case statement. The case statement is similar to a set of if-elif constructs.
The syntax of the case statement follows:
casewordinvalue1)command1command2
...
last_command
;;
value2)
command1
command2
...
last_command
;;
esacWith this syntax, you compare a word, usually the value of a variable, against a number of values. If there is a match, the shell executes the commands from the given value up to the two semicolons (;;) that end an individual case.
The entire statement ends with esac, case spelled backward.
There is a big problem with the choose1 script, however. You can see this problem if you enter something unexpected. For example:
$sh choose1Which would you rather have, ksh, a platypus, or MS Word for Windows for Windows for Macintosh?fred$
If you enter an unexpected value, such as fred, the script does nothing. It cannot handle unexpected input. Of course, this means the choose1 script is not very robust and will likely fail if you release this software to the wide world.
To deal with this situation, the case statement also includes a catch-all value of *, which you can use to capture values that would normally fall through a case statement, matching nothing. The syntax follows:
casewordinvalue1)command1command2...last_command;;value2)command1command2...
last_command;; *)command1command2...last_command;; esac
The C shell doesn't support the case statement like the Bourne shell. The C shell, however, does provide a switch statement, very similar to the Bourne shell case statement.
The C shell syntax is:
switch ( word )
case value1:
commands
breaksw
case value1:
commands
breaksw
default
commands
endswInstead of case, you see a switch statement. (This is one area where the C shell is quite similar to the C programming language.) Each value is specified by a case construct. Each case block ends with a breaksw statement, short for break switch. (This mimics the C language break statement.)
A default block specifies the action to take if none of the cases matches.
The entire switch statement ends with an endsw statement, short for end switch.
Like the for loop, the while loop repeats its block of commands a number of times. Unlike the for loop, however, the while loop iterates until its while condition is no longer true. The basic syntax is:
while [ test_condition ]
do
commands...
doneThe while loop is sort of like a combination of an if statement and a for loop. Use the while loop when you don't know in advance how many iterations you need.
Note that the C shell sports a different syntax for a while loop:
while ( test_condition )
commands...
endThe until loop is very similar to the while loop. With while, the test must be true to execute the block of commands in the loop. With until, the test must be false. Otherwise, the two loops are the same.
Think of until as "loop until a condition is true" and while as "loop while this condition remains true." In other words, until just reverses the test. The following example shows this test reversal by modifying the previous example to use the until statement.
Yow. That is a lot of syntax for one chapter. This chapter includes the basics on how to control which commands in your script get executed. These basics include:
You can use the syntax
${variable}to access the value of a variable. This syntax is the same as$variable, but the curly braces clearly delineate the start and end of the variable name. This means the variable name cannot get merged into adjacent text.The
forloop allows your script to iterate a given number of times. A special variant of theforloop iterates over files using wildcard syntax, such as*and*.txt.The
ifstatement executes a command and tests whether the results of the command are true or false.A special command named
testprovides a zillion command-line options to test different conditions.You can use
[ ]in place of thetestcommand, placing thetestcommand-line options and arguments between square brackets.The
whileloop iterates while a condition is true. When the condition resolves to false, thewhileloop stops.The
untilloop reverses thewhileloop test. Theuntilloop iterates while the condition is not true.
The next chapter expands the use of shell scripts to the overarching computer environment, discussing operating system issues, how shells start up, and how to turn your shell scripts into executable commands.
Run the
choose1example script and enter the imaginary Microsoft product name. Be sure to have your license key ready.Extend the
mylsormyls2shell scripts to list the files in a different directory from the current directory, such as /usr/local. The script should still output the name of all the files in the directory.Enhance the script you wrote for Exercise 2 to place the name of the directory in a variable and access the variable in the
forloop.Enhance the script you wrote for Exercise 3 to ask the user for the name of the directory to list.
Enhance the script you wrote for Exercise 4. This new script should ask the user the directory to list and then should output
/after directory names and*after executable files. Do not output*if the file is a directory (and has execute permissions on the directory).