Chapter 7

Writing Simple Shell Scripts

You'd never get any work done if you typed every command that needs to be run on your Linux system when it starts. Likewise, you could work more efficiently if you grouped together sets of commands that you run all the time. Shell scripts can handle these tasks.

A shell script is a group of commands, functions, variables, or just about anything else you can use from a shell. These items are typed into a plaintext file. That file can then be run as a command. Most Linux systems use system initialization shell scripts during system startup to run commands needed to get services going. You can create your own shell scripts to automate the tasks you need to do regularly.

This chapter provides a rudimentary overview of the inner workings of shell scripts and how they can be used. You learn how shell scripts are responsible for the messages that scroll by on the system console during booting and how simple scripts can be harnessed to a scheduling facility (such as cron or at) to simplify administrative tasks.

Understanding Shell Scripts

Have you ever had a task that you needed to do over and over that took a lot of typing on the command line? Do you ever think to yourself, “Wow, I wish there were just one command I could type to do all of this”? Maybe a shell script is what you're after.

Shell scripts are the equivalent of batch files in MS-DOS, and can contain long lists of commands, complex flow control, arithmetic evaluations, user-defined variables, user-defined functions, and sophisticated condition testing. Shell scripts are capable of handling everything from simple one-line commands to something as complex as starting up your Linux system.

In fact, as you will read in this chapter, Linux systems do just that. They use shell scripts to check and mount all your file systems, set up your consoles, configure your network, launch all your system services, and eventually provide you with your login screen. While dozens of different shells are available in Linux, the default shell is called bash, the Bourne Again SHell.

Executing and debugging shell scripts

One of the primary advantages of shell scripts is that they can be opened in any text editor to see what they do. A big disadvantage is that large or complex shell scripts often execute more slowly than compiled programs. There are two basic ways to execute a shell script:

• The filename is used as an argument to the shell (as in bash myscript). In this method, the file does not need to be executable; it just contains a list of shell commands. The shell specified on the command line is used to interpret the commands in the script file. This is most common for quick, simple tasks.
• The shell script may also have the name of the interpreter placed in the first line of the script preceded by #! (as in #!/bin/bash) and have the execute bit of the file containing the script set (using chmod +x filename). You can then run your script just like any other program in your path simply by typing the name of the script on the command line.

When scripts are executed in either manner, options for the program may be specified on the command line. Anything following the name of the script is referred to as a command-line argument.

As with writing any software, there is no substitute for clear and thoughtful design and lots of comments. The pound sign (#) prefaces comments and can take up an entire line or exist on the same line after script code. It is best to implement more complex shell scripts in stages, making sure the logic is sound at each step before continuing. Here are a few good, concise tips to make sure things are working as expected during testing:

• In some cases, you can place an echo statement at the beginning of lines within the body of a loop and surround the command with quotes. That way, rather than executing the code, you can see what will be executed without making any permanent changes.
• To achieve the same goal, you can place dummy echo statements throughout the code. If these lines get printed, you know the correct logic branch is being taken.
• You can use set -x near the beginning of the script to display each command that is executed or launch your scripts using

$ bash –x myscript

• Because useful scripts have a tendency to grow over time, keeping your code readable as you go along is extremely important. Do what you can to keep the logic of your code clean and easy to follow.

Understanding shell variables

Often within a shell script, you want to reuse certain items of information. During the course of processing the shell script, the name or number representing this information may change. To store information used by a shell script in such a way that it can be easily reused, you can set variables. Variable names within shell scripts are case-sensitive and can be defined in the following manner:

NAME=value

The first part of a variable is the variable name, and the second part is the value set for that name. Be sure that the NAME and value touch the equal sign, without any spaces. Variables can be assigned from constants, such as text, numbers, and underscores. This is useful for initializing values or saving lots of typing for long constants. The following examples show variables set to a string of characters (CITY) and a numeric value (PI):

CITY="Springfield"
PI=3.14159265

Variables can contain the output of a command or command sequence. You can accomplish this by preceding the command with a dollar sign and open parenthesis, and following it with a closing parenthesis. For example, MYDATE=$(date) assigns the output from the date command to the MYDATE variable. Enclosing the command in backticks (') can have the same effect. In this case, the date command is run when the variable is set and not each time the variable is read.

echo '$HOME *** 'date''    $HOME *** 'date'
echo "$HOME 'date'"        /home/chris *** Tue Mar 20 16:56:52 EDT 2012

Using variables is a great way to get information that can change from computer to computer or from day to day. The following example sets the output of the uname -n command to the MACHINE variable. Then I use parentheses to set NUM_FILES to the number of files in the current directory by piping (|) the output of the ls command to the word count command (wc -l).

MACHINE='uname –n'
NUM_FILES=$(/bin/ls | wc –l)

Variables can also contain the value of other variables. This is useful when you have to preserve a value that will change so you can use it later in the script. Here, BALANCE is set to the value of the CurBalance variable:

BALANCE="$CurBalance"

Special shell positional parameters

There are special variables that the shell assigns for you. One set of commonly used variables is called positional parameters or command line arguments and is referenced as $0, $1, $2, $3…$n. $0 is special and is assigned the name used to invoke your script; the others are assigned the values of the parameters passed on the command line, in the order they appeared. For instance, let's say you had a shell script named myscript that contained the following:

#!/bin/bash
# Script to echo out command-line arguments
echo "The first argument is $1, the second is $2."
echo "The command itself is called $0."

The following shows what would happen if you ran that command with foo and bar as arguments:

$ myscript foo bar
The first argument is foo, the second is bar.
The command itself is called /home/chris/bin/myscript.

As you can see, the positional parameter $0 is the full path or relative path to myscript, $1 is foo, and $2 is bar.

Another variable, $#, tells you how many parameters your script was given. In the example, $# would be 2. The $@ variable holds all the arguments entered at the command line. Another particularly useful special shell variable is $?, which receives the exit status of the last command executed. Typically, a value of zero means the command exited successfully, and anything other than zero indicates an error of some kind. For a complete list of special shell variables, refer to the bash man page.

Reading in parameters

Using the read command, you can prompt the user for information, then store and use that information later in your script. Here's an example of a script that uses the read command:

#!/bin/bash
read -p "Type in an adjective, noun and verb (past tense): " adj1 noun1 verb1
echo "He sighed and $verb1 to the elixir. Then he ate the $adj1 $noun1."

In this script, after prompting for an adjective, noun, and verb, the user is expected to enter words that are then assigned to the adj1, noun1, and verb1 variables. Those three variables are then included in a silly sentence, which is displayed on the screen. If the script were called sillyscript, here's an example of how it might run:

$ sillyscript
Type in an adjective, noun and verb (past tense): hairy football danced
He sighed and danced to the elixir. Then he ate the hairy football.

Parameter expansion in bash

As mentioned earlier, if you want the value of a variable, you precede it with a $ (for example, $CITY). This is really just shorthand for the notation ${CITY}; curly braces are used when the value of the parameter needs to be placed next to other text without a space. Bash has special rules that allow you to expand the value of a variable in different ways. Going into all the rules is probably overkill for a quick introduction to shell scripts, but the follow list presents some common constructs that you're likely to see in bash scripts you find on your Linux system.

• ${var:-value}—If variable is unset or empty, expand this to value.
• ${var#pattern}—Chop the shortest match for pattern from the front of var's value.
• ${var##pattern}—Chop the longest match for pattern from the front of var's value.
• ${var%pattern}—Chop the shortest match for pattern from the end of var's value.
• ${var%%pattern}—Chop the longest match for pattern from the end of var's value.

Try typing the following commands from a shell to test how parameter expansion works:

$ THIS="Example"
$ THIS=${THIS:-"Not Set"}
$ THAT=${THAT:-"Not Set"}
$ echo $THIS
Example
$ echo $THAT
Not Set

In the examples here, the THIS variable is initially set to the word Example. In the next two lines, the THIS and THAT variables are set to their current values or to Not Set, if they are not currently set. Notice that because I just set THIS to the string Example, when I echo the value of THIS it appears as Example. However, because THAT was not set, it appears as Not Set.

In the following example, MYFILENAME is set to /home/digby/myfile.txt. Next, the FILE variable is set to myfile.txt and DIR is set to /home/digby. In the NAME variable, the filename is cut down to simply myfile; then, in the EXTENSION variable, the file extension is set to txt. (To try these out, you can type them at a shell prompt as in the previous example, and then echo the value of each variable to see how it is set.) Type the code on the left. The material on the right side describes the action.

Performing arithmetic in shell scripts

Bash uses untyped variables, meaning it normally treats variables as strings or text, but can change them on-the-fly if you want it to. Unless you tell it otherwise with declare, your variables are just a bunch of letters to bash. But when you start trying to do arithmetic with them, bash converts them to integers if it can. This makes it possible to do some fairly complex arithmetic in bash.

Integer arithmetic can be performed using the built-in let command or through the external expr or bc commands. After setting the variable BIGNUM value to 1024, the three commands that follow would all store the value 64 in the RESULT variable. The bc command is a calculator application that is available in most Linux distributions. The last command gets a random number between 0 and 10 and echoes the results back to you.

BIGNUM=1024
let RESULT=$BIGNUM/16
RESULT='expr $BIGNUM / 16'
RESULT='echo "$BIGNUM / 16" | bc'
let foo=$RANDOM%10; echo $foo

Another way to incrementally grow a variable is to use $(()) notation with ++I added to increment the value of I. Try typing the following:

$ I=0
$ echo The value of I after increment is $((++I))
$ echo The value of I before and after increment is $((I++)) and $I

Repeat either of those commands to continue to increment the value of $I.

To see a complete list of the kinds of arithmetic you can perform using the let command, type help let at the bash prompt.

Using programming constructs in shell scripts

One of the features that makes shell scripts so powerful is that their implementation of looping and conditional execution constructs is similar to those found in more complex scripting and programming languages. You can use several different types of loops, depending on your needs.

The “if…then” statements

The most commonly used programming construct is conditional execution, or the if statement. It is used to perform actions only under certain conditions. There are several variations of if statements for testing various types of conditions.

The first if...then example tests if VARIABLE is set to the number 1. If it is, then the echo command is used to say that it is set to 1. The fi statement then indicates that the if statement is complete and processing can continue.

VARIABLE=1
if [ $VARIABLE -eq 1 ] ; then
echo "The variable is 1"
fi

Instead of using –eq, you can use the equal sign (=), as shown in the following example. The = works best for comparing string values, while -eq is often better for comparing numbers. Using the else statement, different words can be echoed if the criterion of the if statement isn't met ($STRING = "Friday"). Keep in mind that it's good practice to put strings in double quotes.

STRING="Friday"
if [ $STRING = "Friday" ] ; then
echo "WhooHoo.  Friday."
else
echo "Will Friday ever get here?"
fi

You can also reverse tests with an exclamation mark (!). In the following example, if STRING is not Monday, then "At least it's not Monday" is echoed.

STRING="FRIDAY"
if [ "$STRING" != "Monday" ] ; then
   echo "At least it's not Monday"
fi

In the following example, elif (which stands for “else if”) is used to test for an additional condition (for example, whether filename is a file or a directory).

filename="$HOME"
if [ -f "$filename" ] ; then
   echo "$filename is a regular file"
elif [ -d "$filename" ] ; then
   echo "$filename is a directory"
else
   echo "I have no idea what $filename is"
fi

As you can see from the preceding examples, the condition you are testing is placed between square brackets [ ]. When a test expression is evaluated, it returns either a value of 0, meaning that it is true, or a 1, meaning that it is false. Notice that the echo lines are indented. The indentation is optional and done only to make the script more readable.

Table 7.1 lists the conditions that are testable and is quite a handy reference. (If you're in a hurry, you can type help test on the command line to get the same information.)

Table 7.1 Operators for Test Expressions

There is also a special shorthand method of performing tests that can be useful for simple one-command actions. In the following example, the two pipes (||) indicate that if the directory being tested for doesn't exist (-d dirname), then make the directory (mkdir $dirname).

# [ test ] || action
# Perform simple single command if test is false
dirname="/tmp/testdir"
[ -d "$dirname" ] || mkdir "$dirname"

Instead of pipes, you can use two ampersands to test if something is true. In the following example, a command is being tested to see if it includes at least three command-line arguments.

# [ test ] && {action}
# Perform simple single action if test is true
[ $# -ge 3 ] && echo "There are at least 3 command line arguments."

You can combine the && and || operators to make a quick, one-line if-then-else statement. The following example tests that the directory represented by $dirname already exists. If it does, a message says the directory already exists. If it doesn't, the statement creates the directory:

# dirname=mydirectory
# [ -e $dirname ] && echo $dirname already exists || mkdir $dirname

The case command

Another frequently used construct is the case command. Similar to a switch statement in programming languages, this can take the place of several nested if statements. The following is the general form of the case statement:

case "VAR" in
   Result1)
      { body };;
   Result2)
      { body };;
   *)
      { body } ;;
esac

Among other things, you can use the case command to help with your backups. The following case statement tests for the first three letters of the current day (case 'date +%a' in). Then, depending on the day, a particular backup directory (BACKUP) and tape drive (TAPE) are set.

# Our VAR doesn't have to be a variable,
# it can be the output of a command as well
# Perform action based on day of week
case 'date +%a' in
   "Mon")
         BACKUP=/home/myproject/data0
         TAPE=/dev/rft0
# Note the use of the double semi-colon to end each option
         ;;
# Note the use of the "|" to mean "or"
   "Tue" | "Thu")
         BACKUP=/home/myproject/data1
         TAPE=/dev/rft1
         ;;
   "Wed" | "Fri")
         BACKUP=/home/myproject/data2
         TAPE=/dev/rft2
         ;;
# Don't do backups on the weekend.
   *)
         
BACKUP="none"
         TAPE=/dev/null
         ;;
esac

The asterisk (*) is used as a catchall, similar to the default keyword in the C programming language. In this example, if none of the other entries are matched on the way down the loop, the asterisk is matched, and the value of BACKUP becomes none. Note the use of esac, or case spelled backwards, to end the case statement.

The “for…do” loop

Loops are used to perform actions over and over again until a condition is met or until all data has been processed. One of the most commonly used loops is the for...do loop. It iterates through a list of values, executing the body of the loop for each element in the list. The syntax and a few examples are presented here:

for VAR in LIST
do
    { body }
done

The for loop assigns the values in LIST to VAR one at a time. Then for each value, the body in braces between do and done is executed. VAR can be any variable name, and LIST can be composed of pretty much any list of values or anything that generates a list.

for NUMBER in 0 1 2 3 4 5 6 7 8 9
do
   echo The number is $NUMBER
done
for FILE in '/bin/ls'
do
   echo $FILE
done

You can also write it this way, which is somewhat cleaner:

for NAME in John Paul Ringo George ; do
   echo $NAME is my favorite Beatle
done

Each element in the LIST is separated from the next by whitespace. This can cause trouble if you're not careful because some commands, such as ls -l, output multiple fields per line, each separated by whitespace. The string done ends the for statement.

If you're a die-hard C programmer, bash allows you to use C syntax to control your loops:

LIMIT=10
# Double parentheses, and no $ on LIMIT even though it's a variable!
for ((a=1; a <= LIMIT ; a++)) ; do
  echo  "$a"
done

The “while…do” and “until…do” loops

Two other possible looping constructs are the while...do loop and the until...do loop. The structure of each is presented here:

while condition      until condition
do                   do
   { body }            { body }
done                 done

The while statement executes while the condition is true. The until statement executes until the condition is true—in other words, while the condition is false.

Here is an example of a while loop that will output the number 0123456789:

N=0
while [ $N –lt 10 ] ; do
   echo –n $N
   let N=$N+1
done

Another way to output the number 0123456789 is to use an until loop as follows:

N=0
until [ $N –eq 10 ] ; do
   echo –n $N
   let N=$N+1
done

Trying some useful text manipulation programs

Bash is great and has lots of built-in commands, but it usually needs some help to do anything really useful. Some of the most common useful programs you'll see used are grep, cut, tr, awk, and sed. As with all the best UNIX tools, most of these programs are designed to work with standard input and standard output, so you can easily use them with pipes and shell scripts.

The general regular expression parser

The name general regular expression parser (grep) sounds intimidating, but grep is just a way to find patterns in files or text. Think of it as a useful search tool. Gaining expertise with regular expressions is quite a challenge, but once you master it, you can accomplish many useful things with just the simplest forms.

For example, you can display a list of all regular user accounts by using grep to search for all lines that contain the text /home in the /etc/passwd file as follows:

$ grep /home /etc/passwd

Or you could find all environment variables that begin with HO using the following command:

$ env | grep ˆHO

To find a list of options to use with the grep command, type man grep.

Remove sections of lines of text (cut)

The cut command can extract fields from a line of text or from files. It is very useful for parsing system configuration files into easy-to-digest chunks. You can specify the field separator you want to use and the fields you want, or you can break up a line based on bytes.

The following example lists all home directories of users on your system. This grep command line pipes a list of regular users from the /etc/passwd file and then displays the sixth field (-f6) as delimited by a colon (-d':'). The hyphen at the end tells cut to read from standard input (from the pipe).

$ grep /home /etc/passwd | cut  -d':' -f6 -

Translate or delete characters (tr)

The tr command is a character-based translator that can be used to replace one character or set of characters with another or to remove a character from a line of text.

The following example translates all uppercase letters to lowercase letters and displays the words mixed upper and lower case as a result:

$ FOO="Mixed UPpEr aNd LoWeR cAsE"
$ echo $FOO | tr [A-Z] [a-z]
mixed upper and lower case

In the next example, the tr command is used on a list of filenames to rename any files in that list so that any tabs or spaces (as indicated by the [:blank:] option) contained in a filename are translated into underscores. Try running the following code in a test directory:

for file in * ; do
   f='echo $file | tr [:blank:] [_]'
   [ "$file" = "$f" ] || mv -i -- "$file" "$f"
done

The stream editor (sed)

The sed command is a simple scriptable editor, and thus can perform only simple edits, such as removing lines that have text matching a certain pattern, replacing one pattern of characters with another, and so on. To get a better idea of how sed scripts work, there's no substitute for the online documentation, but here are some examples of common uses.

You can use the sed command to essentially do what I did earlier with the grep example: search the /etc/passwd file for the word home. Here the sed command searches the entire /etc/passwd file, searches for the word home, and prints any line containing the word home.

$ sed –n '/home/p' /etc/passwd

In this example, sed searches the file somefile.txt and replaces every instance of the string Mac with Linux. Notice that the letter g is needed at the end of the substitution command to cause every occurrence of Mac on each line to be changed to Linux. (Otherwise, only the first instance of Mac on each line is changed.) The output is then sent to the fixed_file.txt file. The output from sed goes to stdout, so this command redirects the output to a file for safekeeping.

$ sed 's/Mac/Linux/g' somefile.txt > fixed_file.txt

You can get the same result using a pipe:

$ cat somefile.txt | sed 's/Mac/Linux/g' > fixed_file.txt

By searching for a pattern and replacing it with a null pattern, you delete the original pattern. This example searches the contents of the somefile.txt file and replaces extra blank spaces at the end of each line (s/ *$) with nothing (//). Results go to the fixed_file.txt file.

$ cat somefile.txt | sed 's/ *$//' > fixed_file.txt

Using simple shell scripts

Sometimes the simplest of scripts can be the most useful. If you type the same sequence of commands repetitively, it makes sense to store those commands (once!) in a file. The following sections offer a couple of simple, but useful, shell scripts.

Telephone list

This idea has been handed down from generation to generation of old UNIX hacks. It's really quite simple, but it employs several of the concepts just introduced.

#!/bin/bash
# (@)/ph
# A very simple telephone list
# Type "ph new name number" to add to the list, or
# just type "ph name" to get a phone number
PHONELIST=∼/.phonelist.txt
# If no command line parameters ($#), there
# is a problem, so ask what they're talking about.
if [ $# -lt 1 ] ; then
  echo "Whose phone number did you want? "
   exit 1
fi
# Did you want to add a new phone number?
if [ $1 = "new" ] ; then
  shift
  echo $* >> $PHONELIST
  echo $* added to database
  exit 0
fi
# Nope. But does the file have anything in it yet?
# This might be our first time using it, after all.
if [ ! -s $PHONELIST ] ; then
  echo "No names in the phone list yet! "
  exit 1
else
  grep -i -q "$*" $PHONELIST     # Quietly search the file
  if [ $? -ne 0 ] ; then         # Did we find anything?
    echo "Sorry, that name was not found in the phone list"
    exit 1
  else
    grep -i "$*" $PHONELIST
  fi
fi
exit 0

So, if you created the telephone list file as ph in your current directory, you could type the following from the shell to try out your ph script:

$ chmod 755 ph
$ ./ph new "Mary Jones" 608-555-1212
Mary Jones 608-555-1212 added to database
$ ./ph Mary
Mary Jones 608-555-1212

The chmod command makes the ph script executable. The ./ph command runs the ph command from the current directory with the new option. This adds Mary Jones as the name and 608-555-1212 as the phone number to the database ($HOME/.phone.txt). The next ph command searches the database for the name Mary and displays the phone entry for Mary. If the script works, add it to a directory in your path (such as $HOME/bin).

Backup script

Because nothing works forever and mistakes happen, backups are just a fact of life when dealing with computer data. This simple script backs up all the data in the home directories of all the users on your Fedora or RHEL system.

#!/bin/bash
# (@)/my_backup
# A very simple backup script
#
# Change the TAPE device to match your system.
# Check /var/log/messages to determine your tape device.
# You may also need to add scsi-tape support to your kernel.
TAPE=/dev/rft0
# Rewind the tape device $TAPE
mt $TAPE rew
# Get a list of home directories
HOMES='grep /home /etc/passwd | cut –f6 –d': ''
# Back up the data in those directories
tar cvf $TAPE $HOMES
# Rewind and eject the tape.
mt $TAPE rewoffl

Summary

Writing shell scripts gives you the opportunity to automate many of your most common system administration tasks. This chapter covered common commands and functions you can use in scripting with the bash shell. It also provided some concrete examples of scripts for doing backups and other procedures.

In the next chapter, you transition from learning about user features into examining system administration topics. Chapter 8 covers how to become the root user, as well as how to use administrative commands, monitor log files, and work with configuration files.

Exercises

Use these exercises to test your knowledge of writing simple shell scripts. These tasks assume you are running a Fedora or Red Hat Enterprise Linux system (although some tasks will work on other Linux systems as well). If you are stuck, solutions to the tasks are shown in Appendix B (although in Linux, there are often multiple ways to complete a task).

 Today is Sat Dec 10 15:45:04 EST 2011.
 You are in /home/joe and your host is abc.example.com.

 There are X parameters that include Y.
 The first is A, the second is B, the third is C.

 The street I grew up on was $mystreet and the town was $mytown