Chapter 14
Automating Tasks and Shell Scripting
This chapter covers ways to automate tasks on your system by using task schedulers. This chapter also provides an introduction to the basics of creating shell scripts, or executable text files written to conform to shell syntax. Shell scripting and automation are useful on your local machine, but they are vital in DevOps and Site Reliability Engineering where the goal is to automate anything that can be automated to save human brain power for tasks that require imaginative thinking.
Shell scripts run like any other command under Linux and can contain complex logic or a simple series of Linux command-line instructions. You can also run other shell scripts from within a shell program. The features and functions for several Linux shells are discussed in this chapter after a short introduction to working from the shell command line. You find out how to write and execute a simple shell program using bash, one of the most popular Linux shells and the default shell in Ubuntu and most other distributions.
What Is a Shell?
We have used the shell since Chapter 10, “Command-Line Beginner’s Class,” but have only glossed over the idea, calling it a command line and terminal. Simply put, a shell is a program that users employ to type commands. In our case, it is an interpreter that supplies both a command language and a program scripting language that can control what the operating system does.
Several shells are available for users. The most widely known and widely distributed shells early in the UNIX era were the Bourne Shell, sh, and the C Shell, csh. The C Shell used syntax from the C programming language whereas the Bourne Shell developed a unique set of features that today have become standard to all UNIX/Linux shells.
Here is a list of the most commonly found shells these days:
▸ Almquist Shell (ash): This was written as a BSD-licensed replacement for the Bourne Shell, with just slightly fewer features (mostly eliminating those not frequently used).
▸ Bourne-Again Shell (bash): This was written by the GNU Project as a GPL-licensed version of the Bourne Shell with some enhancements.
▸ Debian Almquist Shell (dash): This is written for use in resource-constrained environments and is considered a modern replacement for the Almquist Shell.
▸ Korn Shell (ksh or pdcsh): This was written by David Korn while he worked at Bell Labs and is based on the Bourne Shell source code. It has been released under the Eclipse Public License; the pdcsh version is the one you are most likely to encounter as it is released with a public domain license.
▸ TENEX C Shell (tcsh): This is based on and compatible with the C Shell, but released with a BSD license.
▸ Z Shell (zsh): This is a modern Bourne-Again Shell replacement with some improvements and is the default shell in macOS.
The automation and tasks described in this chapter should work on most shells.
By default, Ubuntu uses the Bourne-Again Shell (bash). It is also aliased to /bin/sh, so whenever you run a shell script using sh notation, you are using bash unless you have chosen to use a different shell.
To temporarily change your shell, just for your current session, enter the name of the shell you want to use. If it is not yet installed, most of them are available in the Ubuntu software repositories via apt.
To permanently change your shell, you use an interactive program, chsh, which stands for “change shell.”
Scheduling Tasks
There are three ways to schedule commands in Ubuntu, all of which work in different ways. The first is the at command, which specifies a command to run at a specific time and date relative to today. The second is the batch command, which is actually a script that redirects you to the at command with some extra options set so your command runs when the system is quiet. The last option is the cron daemon, which is the Linux way of executing tasks at a given time.
Using at and batch to Schedule Tasks for Later
If you want to run a time-intensive task, but you do not want to do it while you are logged in, you can tell Ubuntu to run it later with the at command, which you must install. The package name is the same as the tool: at. To use at, you need to tell it the time at which you want the task to run and then press Enter. You then see a new prompt that starts with at>, and everything you type there until you press Ctrl+D will be the commands that at will run.
When the designated time arrives, at performs each action individually and in order, which means later commands can rely on the results of earlier commands. In the following example, run at just after 8:00 p.m., at is used to download and extract the latest Linux kernel at a time when the network should be quiet:
matthew@seymour:~$ at now + 7 hours at> wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.bz2 at> tar xvfjp linux-3.0.tar.bz2 at> <EOT> job 2 at 2011-07-08 20:01
Specifying now + 7 hours as the time does what you would expect: at was run at 8:00 p.m., so the command will run just after 3:00 a.m.
If you have a more complex job, you can use the –f parameter to have at read its commands from a file, like this:
echo wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.00.tar.bz2; tar xvfjp linux-3.0.tar.bz2 > myjob.job at –f myjob.job tomorrow
As you can see, at is flexible about the time format it takes; you can specify it in three ways:
▸ Using the now parameter, you can specify a number of minutes, hours, days, or weeks relative to the current time. For example, now + 4 weeks runs the command four weeks from today.
▸ You can also specify several special times, including tomorrow, midnight, noon, or teatime (4:00 p.m.). If you do not specify a time with tomorrow, your job is set for precisely 24 hours from the current time.
▸ You can specify an exact date and time by using HH:MM MM/DD/YY format (for example, 16:40 22/12/20 for 4:40 p.m. on December 22, 2020).
When your job is submitted, at reports the job number, date, and time that the job will be executed; the queue identifier; and the job owner (you). It also captures all your environment variables and stores them along with the job so that when your job runs, it can restore the variables, preserving your execution environment.
The job number and job queue identifier are both important. When you schedule a job using at, it is placed into queue a by default, which means it runs at your specified time and takes up a normal amount of resources.
An alternative command, batch, is really just a shell script that calls at with a few extra options. These options (-q b -m now, in case you are interested) set at to run on queue b (-q b), mail the user on completion (-m), and run immediately (now). The queue part is what is important: Jobs scheduled on queue b will only be executed when system load falls below 0.8—that is, when the system is not running at full load. Furthermore, those jobs run with a lower niceness, meaning a queue jobs usually have a niceness of 2, whereas queue b jobs have a niceness of 4.
Because batch always specifies now as its time, you need not specify your own time; the task will simply run as soon as the system is quiet. Having a default niceness of 4 means that batched commands get fewer system resources than a queue job gets (at’s default) and fewer system resources than with most other programs. You can optionally specify other queues by using at. Queue c runs at niceness 6, queue d runs at niceness 8, and so on. However, it is important to note that the system load is only checked before the command is run. If the load is lower than 0.8, your batch job runs. If the system load subsequently rises beyond 0.8, your batch job continues to run, albeit in the background, thanks to its niceness value.
When you submit a job for execution, you are also returned a job number. If you forget this or just want to see a list of other jobs you have scheduled to run later, use the atq command with no parameters. If you run this as a normal user, it prints only your jobs; running it as a super user prints everyone’s jobs. The output is in the same format as when you submit a job, so you get the ID number, execution time, queue ID, and owner of each job.
If you want to delete a job, use the atrm command followed by the ID number of the job you want to delete. This next example shows atq and atrm being used to list jobs and delete one:
matthew@seymour:~$ atq 14 2012-01-20 23:33 a matthew 16 2012-02-03 22:34 a matthew 17 2012-01-25 22:34 a matthew 15 2012-01-22 04:34 a matthew 18 2012-01-22 01:35 b matthew matthew@seymour:~$ atrm 16 matthew@seymour:~$ atq 14 2012-01-20 23:33 a matthew 17 2012-01-25 22:34 a matthew 15 2012-01-22 04:34 a matthew 18 2012-01-22 01:35 b matthew
In this example, job 16 is deleted using atrm, and so it does not show up in the second call to atq.
The default configuration for at and batch is to allow everyone to use it, which is not always the desired behavior. Access is controlled through two files: /etc/at.allow and /etc/at.deny. By default, at.deny exists but is empty, which allows everyone to use at and batch. You can enter usernames into at.deny, one per line, to stop those users from scheduling jobs.
Alternatively, you can use the at.allow file; this does not exist by default. If you have a blank at.allow file, no one except root is allowed to schedule jobs. As with at.deny, you can add usernames to at.allow, one per line, and those users are able to schedule jobs. You should use either at.deny or at.allow: When someone tries to run at or batch, Ubuntu checks for her username in at.allow. If it is in there, or if at.allow does not exist, Ubuntu checks for her username in at.deny. If her username is in at.deny or if at.deny does not exist, the user is not allowed to schedule jobs.
Using cron to Run Jobs Repeatedly
The at and batch commands work well if you just want to execute a single task at a later date, but they are less useful if you want to run a task frequently. Instead, the cron daemon exists for running tasks repeatedly based on system (and user) requests. The cron daemon has a similar permissions system to at: Users listed in the cron.deny file are not allowed to use cron, and users listed in the cron.allow file are. An empty cron.deny file—the default—means everyone can set jobs. An empty cron.allow file means that no one (except root) can set jobs.
There are two types of jobs: system jobs and user jobs. Only root can edit system jobs, whereas any user whose name appears in cron.allow or does not appear in cron.deny can run user jobs. System jobs are controlled through the /etc/crontab file, which by default looks like this:
SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 17 * * * * root cd / && run-parts -report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts –report [ccc]/etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts –report [ccc]/etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts –report [ccc]/etc/cron.monthly )
The first two lines specify which shell should be used to execute the job (it defaults to the shell of the user who owns the crontab file, usually /bin/bash) and the search path for executables that will be used. It’s important that you avoid using environment variables in this path statement because they might not be set when the job runs.
The next line starts with a pound sign (#) and so is treated as a comment and ignored. The next four lines are the important parts: They are the jobs themselves.
Each job is specified in seven fields that define the time to run, owner, and command. The first five commands specify the execution time in quite a quirky order: minute (0–59), hour (0–23), day of the month (1–31), month of the year (1–12), and day of the week (0–7). For day of the week, both 0 and 7 are Sunday, which means that 1 is Monday, 3 is Wednesday, and so on. If you want to specify “all values” (that is, every minute, every hour, every day, and so on), use an asterisk, *.
The next field specifies the username of the owner of the job. When a job is executed, it uses the username specified here. The last field is the command to execute.
So, the first job runs at minute 17, every hour of every day of every month, and executes the command run-parts /etc/cron.hourly. The run-parts command is a simple script that runs all programs inside a given directory (in this case, /etc/cron.hourly). So, in this case, the job executes at 00:17 (17 minutes past midnight), 01:17, 02:17, 03:17, and so on and uses all the programs listed in the cron.hourly directory.
The next job runs at minute 25 and hour 6 of every day of every month, running run-parts /etc/cron.daily. Because of the hour limitation, this script runs only once per day, at 6:25 a.m. Note that it uses minute 25 rather than minute 17 so that daily jobs do not clash with hourly jobs. You should be able to guess what the next two jobs do simply by looking at the commands they run.
Each of those four directories (cron.hourly, cron.daily, cron.weekly, and cron.monthly) contains a collection of shell scripts that are run by run-parts. For example, in cron.daily you have scripts like logrotate, which handles backing up of log files, and makewhatis, which updates the whatis database. You can add other system tasks to these directories if you want to, but be careful to ensure that your scripts are correct.
Caution
The cron daemon reads all the system crontab files and all user crontab files once a minute (on the minute; that is, at 6:00:00, 6:01:00, and so on) to check for changes. However, any new jobs it finds will not be executed until at least 1 minute has passed.
For example, if it is 6:01:49 (that is, 49 seconds past 1 minute past 6:00 a.m.) and you set a cron job to run at 6:02, it does not execute. At 6:02, the cron daemon rereads its configuration files and sees the new job, but it is not able to execute it. If you set the job to run at 6:02 a.m. every day, it is executed the following morning and every subsequent morning.
This same situation exists when deleting jobs. If it is 6:01:49 and you have a job scheduled to run at 6:02, deleting it makes no difference: cron runs it before it rereads the crontab files for changes. However, after it has reread the crontab file and noticed that the job is no longer there, it will not be executed in subsequent days.
There are alternative ways of specifying dates. For example, you can use sets of dates and times by using hyphens or commas; for example, hours 9–15 would execute at 9, 10, 11, 12, 13, 14, and 15 (from 9:00 a.m. to 3:00 p.m.), whereas 9, 11, 13, 15 would skip the even hours. Note that it is important that you not put spaces into these sets because the cron daemon interprets a space as the next field. You can define a step value with a slash (/) to show time division; for example, */4 for hours means “every 4 hours all day,” and 0-12/3 means “every 3 hours from midnight to noon.” You can also specify day and month names rather than numbers, using three-character abbreviations: Sun, Mon, Tue, Fri, Sat for days or Jan, Feb, Mar, Oct, Nov, Dec for months.
As well as system jobs, there are user jobs for those users who have the correct permissions. User jobs are stored in the /var/spool/cron directory, with each user having his own file named after his username (for instance, /var/spool/cron/philip or /var/spool/cron/root). These files contain the jobs the user wants to run and take roughly the same format as the /etc/crontab file, with the exception that the owner of the job should not be specified because it is always the same as the filename.
To edit your own crontab file, type crontab -e. This brings up a text editor, where you can enter your entries. (By default the editor is vim, also known by its older name vi, but you can set the EDITOR environment variable to change that.) The format of this file is a little different from the format for the main crontab file because in this case there is no need to specify the owner of the job, as it is always you.
So, this time each line is made up of six fields: minute (0–59), hour (0–23), day of the month (1–31), month of the year (1–12), day of the week (0–7), and the command to run. If you are using vim and are new to it, press i to enter insert mode to edit your text; then press Esc to exit insert mode. To save and quit, type a colon followed by wq and press Enter.
When programming, we tend to use a sandbox subdirectory in our home directory where we keep all sorts of temporary files that we were just playing around with. We can use a personal job to empty that directory every morning at 6:00 a.m. to get a fresh start each morning. Here is how that would look in our crontab file:
0 6 * * * rm –rf /home/matthew/sandbox/*
If you are not allowed to schedule jobs, you will be stopped from editing your crontab file.
After your jobs are placed, you can use the command crontab -l to list your jobs. This just prints the contents of your crontab file, so its output is the same as the line you just entered.
If you want to remove just one job, the easiest thing to do is type crontab -e to edit your crontab file in vim; then, after having moved the cursor to the job you want to delete, type dd to delete that line. If you want to delete all your jobs, you can use crontab -r to delete your crontab file.
Read the man page for more about cron.
Using rtcwake to Wake Your Computer from Sleep Automatically
Some of us keep our computers running 24/7. Perhaps you don’t want to do so, and you need to have your system up and running at a certain time every day, but you can’t guarantee that you will be able to be present to turn it on. It is possible to use rtcwake to place the computer in sleep or suspend mode instead of turning it off and then wake up the computer later. To do this, you must have sudo permissions. Here is an example:
matthew@seymour:~$ sudo rtcwake -m mem -s -3600
This command tells the computer to suspend to RAM, or sleep, which means to save the current state of the computer in memory and shut down everything else and then to wake the computer after 3600 seconds, which is one hour.
Here is the basic syntax of the command:
sudo rtcwake -m [type of suspend] -s [number of seconds]
There are five types of suspend available to use with -m:
▸ disk—(hibernate) The current state of the computer is written to disk, and the computer is powered off.
▸ mem—(sleep) The current state of the computer is written to RAM, and the computer is put into a low-power state, using just enough power to keep the memory preserved.
▸ no—The computer is not suspended immediately. Only the wakeup time is set. This allows you to continue working; you have to remember to put the computer to sleep manually.
▸ off—The computer is turned off completely. Wake will not work with this setting for everyone and is not officially supported, but it does work with some computers. It is included here for those who like to live dangerously.
▸ standby—The computer is put into standby mode, which saves some power over running normally but not nearly as much as the other options. This is the default setting and will be used if you omit -m.
Setting the wake time can be done more than one way:
▸ As shown earlier, you can use -s, which specifies the number of seconds before waking.
▸ You can also use -t, which allows you to set a specific time to wake but formatted in the number of seconds since the beginning of UNIX time (00:00:00 UTC on 1/1/1970). The date command can help you find this number, which is a commonly used method of performing time-related tasks in the UNIX/Linux world. You can do so like this: sudo rtcwake -m no -t $(date +%s -d 'tomorrow 06:30').
See the man files for rtcwake and date for help and more options.
Here are a few tips to help you get started:
▸ The letters RTC stand for “real-time clock,” which refers to the hardware clock that is set in your BIOS and is kept running by the battery on your motherboard. If your computer needs a new battery, as evidenced by the time needing to be reset every time you turn the computer back on, or if you have other clock-related problems, this command will not work for you.
▸ If you have problems using sleep, hibernate, or suspend on your system, this command will not work for you.
▸ You probably want to avoid using this command on a notebook computer. Overheating and/or dead batteries are a real possibility if a system wakes itself while the computer is in a laptop bag.
▸ If you want to run a specific command when the computer wakes up, you can do this the same way you chain other commands to run in a series: Put && after rtcwake and before the command you want to run when rtcwake has completed, as discussed in Chapter 12, “Command-Line Master Class, Part 2.”
Basic Shell Control
Ubuntu includes a rich assortment of capable, flexible, and powerful shells. Each shell is different but has numerous built-in commands and configurable command-line prompts and might include features such as command-line history, the ability to recall and use a previous command line, and command-line editing. As an example, the bash shell is so powerful that it is possible to write a minimal web server entirely in bash’s language using 114 lines of script. (See the link for the bash home page at the end of this chapter.)
The Shell Command Line
Having a basic understanding of the capabilities of the shell command line can help you write better shell scripts. If, after you have finished reading this short introduction, you want to learn more about the command line, check out Chapter 11, “Command-Line Master Class, Part 1.” You can use the shell command line to perform a number of different tasks, including the following:
▸ Searching files or directories with programs using pattern matching or expressions. These commands include the GNU gawk (linked as awk) and the grep family of commands, including egrep and fgrep.
▸ Getting data from and sending data to a file or command, known as input and output redirection.
▸ Feeding or filtering a program’s output to another command (called using pipes).
A shell can also have built-in job-control commands to launch the command line as a background process, suspend a running program, selectively retrieve or kill running or suspended programs, and perform other types of process control.
You can run multiple commands on a single command line by using a semicolon to separate commands:
matthew@seymour:~$ w ; free ; df
18:14:13 up 4:35, 2 users, load average: 0.97, 0.99, 1.04
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
matthew tty7 :0 13:39 4:35m 24:34 0.32s gnome-session
matthew pts/0 :0.0 17:24 0.00s 1.19s 4.98s gnome-terminal
total used free shared buffers cached
Mem: 4055692 1801104 2254588 0 134096 757532
-/+ buffers/cache: 909476 3146216
Swap: 8787512 0 8787512
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 14421344 6509276 7179508 48% /
none 2020136 336 2019800 1% /dev
none 2027844 3004 2024840 1% /dev/shm
none 2027844 224 2027620 1% /var/run
none 2027844 0 2027844 0% /var/lock
none 2027844 0 2027844 0% /lib/init/rw
/dev/sda6 284593052 144336704 125799860 54% /home
This example displays the output of the w, free, and df commands. You can extend long shell command lines inside shell scripts or at the command line by using the backslash character (), as follows:
matthew@seymour:~$ echo "this is a long > command line and" ; echo "shows that multiple commands > may be strung out." this is a long command line and shows that multiple commands may be strung out.
The first three lines of this example are a single command line. In that single line are two instances of the echo command. Note that when you use the backslash as a line-continuation character, it must be the last character on the command line (or in your shell script, as you see later in this chapter, in the “Writing and Executing a Shell Script” section).
Using the basic features of the shell command line is easy, but mastering the use of all features can be difficult. Entire books have been devoted to using shells, writing shell scripts, and using pattern-matching expressions. The following sections provide an overview of some features of the shell command line related to writing scripts.
Understanding grep
If you plan to develop shell scripts to expand the capabilities of pattern-matching commands such as grep, you will benefit from learning more about using expressions. One of the definitive guides to using the pattern-matching capabilities of UNIX and Linux commands is Mastering Regular Expressions by Jeffrey E. F. Friedl.
Shell Pattern-Matching Support
The shell command line enables you to use strings of specially constructed character patterns for wildcard matches. This is a different, simpler capability than that supported by GNU utilities such as grep, which can use more complex patterns, known as expressions, to search through files or directories or to filter data input to or out of commands.
The shell’s pattern strings can be simple or complex, but even using a small subset of the available characters in simple wildcards can yield constructive results at the command line. Common characters used for shell pattern matching include the following:
▸ *—Matches any character. For example, to find all files in the current directory ending in .txt, you could use this:
matthew@seymour:~$ ls *.txt
▸ ?—Matches a single character. For example, to find all files in the current directory ending in the extension .d?c (where ? could be 0–9, a–z, or A–Z), you could use the following:
matthew@seymour:~$ ls *.d?c
▸ [xxx] or [x-x]—Matches a range of characters. For example, to list all files in a directory with names containing numbers, you could use this:
matthew@seymour:~$ ls *[0-9]*
▸ x—Matches or escapes a character such as ? or a tab character. For example, to create a file with a name containing a question mark, you could use the following:
matthew~$ touch foo?
Note that the shell might not interpret some characters or regular expressions in the same manner as a Linux command, and mixing wildcards and regular expressions in shell scripts can lead to problems unless you’re careful. For example, finding patterns in text is best left to regular expressions used with commands such as grep; simple wildcards should be used for filtering or matching filenames on the command line. And although both Linux command expressions and shell scripts can recognize the backslash as an escape character in patterns, the dollar sign ($) has two wildly different meanings (single-character pattern matching in expressions and variable assignment in scripts).
Caution
Make sure you read your command carefully when using wildcards; an all-too-common error is to type something like rm -rf * .txt with a space between the * and .txt. By the time you wonder why the command is taking so long, Bash will already have deleted most of your files. The problem is that it will treat the * and the .txt separately. * will match everything, so Bash will delete all your files.
Redirecting Input and Output
You can create, overwrite, and append data to files at the command line, using a process called input and output redirection. The shell recognizes several special characters for this process, such as >, <, or >>.
In this example, the output of the ls command is redirected to create a file named textfiles.listing:
matthew@seymour:~$ ls *.txt >textfiles.listing
Use output redirection with care because it is possible to overwrite existing files. For example, specifying a different directory but using the same output filename overwrites the existing textfiles.listing:
matthew@seymour:~$ ls /usr/share/doc/mutt-1.4/*.txt >textfiles.listing
Fortunately, most shells are smart enough to recognize when you might do something foolish. Here, the bash shell warns that the command is attempting to redirect output to a directory:
matthew@seymour:~$ mkdir foo matthew@seymour:~$ ls >foo bash: foo: Is a directory
Output can be appended to a file without overwriting existing content by using the append operator, >>. In this example, the directory listing is appended to the end of textfiles.listing instead of overwriting its contents:
matthew@seymour:~$ ls /usr/share/doc/mutt-1.4/*.txt >>textfiles.listing
You can use input redirection to feed data into a command by using the < like this:
matthew@seymour:~$ cat < textfiles.listing
You can use the shell here operator, <<, to specify the end of input on the shell command line:
matthew@seymour:~$ cat >simple_script <<DONE > echo ""this is a simple script"" > DONE matthew@seymour:~$ cat simple_script echo ""this is a simple script""
In this example, the shell feeds the cat command you are typing (input) until the pattern DONE is recognized. The output file simple_script is then saved and its contents verified. You can use this same technique in scripts to create content based on the output of various commands and define an end-of-input or delimiter.
Piping Data
Many Linux commands can be used in concert in a single, connected command line to transform data from one form to another. Stringing together Linux commands in this fashion is known as using or creating pipes. Pipes are created on the command line with the bar operator (|). For example, you can use a pipe to perform a complex task from a single command line like this:
matthew@seymour:~$ find /d2 -name '*.txt' -print | xargs cat | tr ' ' ' ' | sort | uniq >output.txt
This example takes the output of the find command to feed the cat command (via xargs) the names of all text files in the /d2 directory. The content of all matching files is then fed through the tr command to change each space in the data stream into a carriage return. The stream of words is then sorted, and identical adjacent lines are removed using the uniq command. The output, a raw list of words, is then saved in the file named output.txt.
Background Processing
The shell allows you to start a command and then launch it into the background as a process by using an ampersand (&) at the end of a command line. This technique is often used at the command line of an X terminal window to start a client and return to the command line. For example, to launch another terminal window using the xterm client, you can use the following:
matthew@seymour:~$ xterm & [3] 1437
The numbers echoed back show a number (3 in this example), which is a job number, or reference number for a shell process, and a process ID number, or PID (1437 in this example). You can kill the xterm window session by using the shell’s built-in kill command, along with the job number, like this:
matthew@seymour:~$ kill %3
Or you can kill the process by using the kill command, along with the PID, as follows:
matthew@seymour:~$ kill 1437
You can use background processing in shell scripts to start commands that take a long time, such as backups:
matthew@seymour:~$ tar -czf /backup/home.tgz /home &
Writing and Executing a Shell Script
Why should you write and use shell scripts? Shell scripts can save you time and typing, especially if you routinely use the same command lines multiple times every day. Although you could also use the history function (by pressing the up or down arrow keys while using bash or using the history command), a shell script can add flexibility with command-line argument substitution and built-in help.
Although a shell script doesn’t execute faster than a program written in a computer language such as C, a shell program can be smaller in size than a compiled program. The shell program does not require any additional library support other than the shell or, if used, existing commands installed on your system. The process of creating and testing shell scripts is also generally simpler and faster than the development process for equivalent C language commands.
Note
Hundreds of commands included with Ubuntu are actually shell scripts, and many other good shell script examples are available over the Internet; a quick search yields numerous links to online tutorials and scripting guides from fellow Linux users and developers. For example, the startx command, used to start an X Window session from the text console, is a shell script used every day by most users. To learn more about shell scripting with bash, see the “Advanced Bash-Scripting Guide,” listed in the “References” section at the end of this chapter. You’ll also find Teach Yourself Shell Programming in 24 Hours a helpful guide to learning more about using the shell to build your own commands.
When you are learning to write and execute your first shell scripts, start with scripts for simple but useful tasks. Begin with short examples and then expand the scripts as you build on your experience and knowledge. Make liberal use of comments (lines preceded with a pound sign, #) to document each section of your script. Include an author statement and an overview of the script as additional help, along with a creation date or version number. Write shell scripts using a text editor such as vi because it does not automatically wrap lines of text. Line wrapping can break script syntax and cause problems. If you use the nano editor, include its -w flag to disable line wrapping.
In this section, you learn how to write a simple shell script to set up a number of aliases (command synonyms) whenever you log on. Instead of typing all the aliases every time you log on, you can put them in a file by using a text editor, such as vi, and then execute the file. Normally these changes are saved in system-wide shell configuration files under the /etc directory to make the changes active for all users or in your .bashrc, .cshrc (if you use tcsh), or .bash_profile files in your home directory.
Here is what is contained in myenv, a sample shell script created for this purpose (for bash):
#!/bin/sh alias ll='ls –L' alias ldir='ls –aF' alias copy='cp'
This simple script creates command aliases, or convenient shorthand forms of commands, for the ls and cp commands. The ll alias provides a long directory listing: The ldir alias is the ls command but prints indicators (for directories or executable files) in listings. The copy alias is the same as the cp command. You can experiment and add your own options or create aliases of other commands with options you frequently use.
You can execute myenv in a variety of ways under Linux. As shown in this example, you can make myenv executable by using the chmod command and then execute it as you would any other native Linux command:
matthew@seymour:~$ chmod +x myenv
This line turns on the executable permission of myenv, which can be checked with the ls command and its -l option like this:
matthew@seymour:~$ ls -l myenv -rwxr-xr-x 1 matthew matthew 0 2010-07-08 18:19 myenv
Running the New Shell Program
You can run your new shell program in several ways. Each method produces the same results, which is a testament to the flexibility of using the shell with Linux. One way to run your shell program is to execute the file myenv from the command line as if it were a Linux command:
matthew@seymour:~$ ./myenv
A second way to execute myenv under a particular shell, such as pdksh, is as follows:
matthew@seymour:~$ pdksh myenv
This invokes a new pdksh shell and passes the filename myenv as a parameter to execute the file. A third way requires you to create a directory named bin in your home directory and to then copy the new shell program into this directory. You can then run the program without the need to specify a specific location or to use a shell. You do so like this:
matthew@seymour:~$ mkdir bin matthew@seymour:~$ mv myenv bin matthew@seymour:~$ myenv
This works because Ubuntu is set up by default to include the executable path $HOME/bin in your shell’s environment. You can view this environment variable, named PATH, by piping the output of the env command through fgrep, like so:
matthew@seymour:~$ env | fgrep PATH /usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin: /usr/X11R6/bin:/sbin:/home/matthew/bin
As you can see, the user (matthew in this example) can use the new bin directory to hold executable files. Another way to bring up an environment variable is to use the echo command along with the variable name (in this case, $PATH):
matthew@seymour:~$ echo $PATH /usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/bball/bin
Caution
Never put . in your $PATH to execute files or a command in the current directory; doing so presents a serious security risk, especially for the root operator, and even more so if . is first in your $PATH search order. Trojan scripts placed by crackers in directories such as /tmp can be used for malicious purposes and will be executed immediately if the current working directory is part of your $PATH.
Storing Shell Scripts for System-wide Access
After you execute the command myenv, you should be able to use ldir from the command line to get a list of files under the current directory and ll to get a list of files with attributes displayed. However, the best way to use the new commands in myenv is to put them into your shell’s login or profile file. For Ubuntu users, and nearly all Linux users, the default shell is bash, so you can make these commands available for everyone on your system by putting them in the /etc/bashrc file. System-wide aliases for tcsh are contained in files with the extension .csh under the /etc/profile.d directory. The pdksh shell can use these command aliases, as well.
Note
To use a shell other than bash after logging in, use the chsh command from the command line or the system-config-users client during an X session. You’ll be asked for your password (or the root password, if using system-config-users) and the location and name of the new shell. The new shell will become your default shell, but only if its name is in the list of acceptable system shells in /etc/shells.
Interpreting Shell Scripts Through Specific Shells
The majority of shell scripts use a shebang line (#!) at the beginning to control the type of shell used to run the script; this bang line calls for an sh-incantation of bash:
#!/bin/sh
A shebang line (it is short for “sharp” and “bang,” two names for # and !) tells the Linux kernel that a specific command (a shell, or in the case of other scripts, perhaps awk or Perl) is to be used to interpret the contents of the file. Using a shebang line is common practice for all shell scripting. For example, if you write a shell script using bash but want the script to execute as if run by the Bourne shell, sh, the first line of your script contains #!/bin/sh, which is a link to the dash shell. Running dash as sh causes dash to act as a Debian Almquist shell. This is the reason for the symbolic link sh, which points to dash.
The Shebang Line
The shebang line is a magic number, as defined in /usr/share/misc/magic—a text database of magic numbers for the Linux file command. Magic numbers are used by many different Linux commands to quickly identify a type of file, and the database format is documented in the section five manual page named magic (read by using man 5 magic). For example, magic numbers can be used by the Linux file command to display the identity of a script (no matter what filename is used) as a shell script using a specific shell or other interpreter, such as awk or Perl.
You might also find different or new environmental variables available to your scripts by using different shells. For example, if you launch csh from the bash command line, you find several new variables or variables with slightly different definitions, such as the following:
matthew@seymour:~$ env ... VENDOR=intel MACHTYPE=i386 HOSTTYPE=i386-linux HOST=thinkpad.home.org
On the other hand, bash might provide these variables or variables of the same name with a slightly different definition, such as the following:
matthew@seymour:~$ env ... HOSTTYPE=i386 HOSTNAME=thinkpad.home.org
Although the behavior of a shebang line is not defined by POSIX, variations of its use can prove helpful when you are writing shell scripts. For example, as described in the wish man page, you can use a shell to help execute programs called within a shell script without needing to hardcode pathnames of programs. The wish command is a windowing Tool Control Language (tcl) interpreter that can be used to write graphical clients. Avoiding the use of specific pathnames to programs increases shell script portability because not every UNIX or Linux system has programs in the same location.
For example, if you want to use the wish command, your first inclination might be to write this:
#!/usr/local/bin/wish
Although this works on many other operating systems, the script fails under Linux because wish is located in the /usr/bin directory. However, if you write the command line this way, you can use the wish command (as a binary or a shell script itself):
#!/bin/sh exec wish "$@"
Using Variables in Shell Scripts
When writing shell scripts for Linux, you work with three types of variables:
▸ Environment variables—You can use these variables, which are part of the system environment, in your shell program. You can define new variables, and you can also modify some of them, such as PATH, within a shell program.
▸ Built-in variables—Variables such as options used on the command (interpreted by the shell as a positional argument) are provided by Linux. Unlike environment variables, you cannot modify built-in variables.
▸ User variables—These variables are defined within a script when you write a shell script. You can use and modify them at will within the shell script, but they are not available to be used outside the script.
A major difference between shell programming and other programming languages is that in shell programming, variables are not typed—that is, you do not have to specify whether a variable is a number or a string, and so on.
Assigning a Value to a Variable
Suppose that you want to use a variable called lcount to count the number of iterations in a loop within a shell program. You can declare and initialize this variable as follows:
Command |
Environment |
|---|---|
|
|
|
|
Note
Under pdksh and bash, you must ensure that the equals sign (=) does not have spaces before and after it.
To store a string in a variable, you can use the following:
Command |
Environment |
|---|---|
|
|
|
|
Use the preceding variable form if the string doesn’t have embedded spaces. If a string has embedded spaces, you can do the assignment as follows:
Command |
Environment |
|---|---|
|
|
|
|
Accessing Variable Values
You can access the value of a variable by prefixing the variable name with a dollar sign ($). That is, if the variable name is var, you can access the variable by using $var.
If you want to assign the value of var to the variable lcount, you can do so as follows:
Command |
Environment |
|---|---|
|
|
|
|
Positional Parameters
Passing options from the command line or from another shell script to your shell program is possible.
These options are supplied to the shell program by Linux as positional parameters, which have special names provided by the system. The first parameter is stored in a variable called 1 (number 1) and can be accessed by using $1 within the program. The second parameter is stored in a variable called 2 and can be accessed by using $2 within the program, and so on. One or more of the higher-numbered positional parameters can be omitted while you’re invoking a shell program.
Understanding how to use these positional parameters and how to access and use variables retrieved from the command line is necessary when developing more advanced shell programs.
A Simple Example of a Positional Parameter
Consider this example: If a shell program mypgm expects two parameters—such as a first name and a last name—you can invoke the shell program with only one parameter, the first name. However, you cannot invoke it with only the second parameter, the last name.
Here is a shell program called mypgm1, which takes only one parameter (a name) and displays it on the screen:
#!/bin/sh #Name display program if [ $# -eq 0 ] then echo "Name not provided" else echo "Your name is "$1 fi
If you execute mypgm1, as follows:
matthew@seymour:~$ bash mypgm1
you get the following output:
Name not provided
However, if you execute mypgm1 as follows:
matthew@seymour:~$ bash mypgm1 Sandra
you get this output:
Your name is Sandra
The shell program mypgm1 also illustrates another aspect of shell programming: the built-in variables provided to the shell by the Linux kernel. In mypgm1, the built-in variable $# provides the number of positional parameters passed to the shell program. You learn more about working with built-in variables in the next major section of this chapter.
Using Positional Parameters to Access and Retrieve Variables from the Command Line
Using positional parameters in scripts can be helpful if you need to use command lines with piped commands requiring complex arguments. Shell programs containing positional parameters can be even more convenient if the commands are infrequently used. For example, if you use your Ubuntu system with an attached voice modem as an answering machine, you can write a script to issue a command that retrieves and plays the voice messages. The following lines convert a saved sound file (in .rmd or voice-phone format) and pipe the result to your system’s audio device:
#!/bin/sh # play voice message in /var/spool/voice/incoming rmdtopvf /var/spool/voice/incoming/$1 | pvfspeed -s 8000 | pvftobasic >/dev/audio
You can then easily play back a voice message by using this script (in this case named pmm):
matthew@seymour:~$ pmm name_of_message
Shell scripts that contain positional parameters are often used for automating routine and mundane jobs, such as system log report generation, file system checks, user resource accounting, printer use accounting, and other system, network, or security administration tasks.
Using a Simple Script to Automate Tasks
You could use a simple script, for example, to examine your system log for certain keywords. If the script is run via your system’s scheduling table, /etc/crontab, it can help automate security monitoring. By combining the output capabilities of existing Linux commands with the language facilities of the shell, you can quickly build a useful script to perform a task that normally requires a number of command lines. For example, you can create a short script, named greplog, like this:
#!/bin/sh
# name: greplog
# use: mail grep of designated log using keyword
# version: v.01 08aug02
#
# author: bb
#
# usage: greplog [keyword] [logpathname]
#
# bugs: does not check for correct number of arguments
# build report name using keyword search and date
log_report=/tmp/$1.logreport.`date '+%m%d%y'`
# build report header with system type, hostname, date and time
echo "=============================================================="
>$log_report
echo " S Y S T E M O N I T O R L O G" >>$log_report
echo uname -a >>$log_report
echo "Log report for" `hostname -f` "on" `date '+%c'` >>$log_report
echo "=============================================================="
>>$log_report ; echo "" >>$log_report
# record log search start
echo "Search for->" $1 "starting" `date '+%r'` >>$log_report
echo "" >>$log_report
# get and save grep results of keyword ($1) from logfile ($2)
grep -i $1 $2 >>$log_report
# build report footer with time
echo "" >>$log_report
echo "End of" $log_report at `date '+%r'` >>$log_report
# mail report to root
mail -s "Log Analysis for $1" root <$log_report
# clean up and remove report
rm $log_report
exit 0
In this example, the script creates the variable $log_report, which will be the filename of the temporary report. The keyword ($1) and first argument on the command line are used as part of the filename, along with the current date (with perhaps a better approach being to use $$ instead of the date, which will append the script’s PID as a file extension). Next, the report header containing some formatted text, the output of the uname command, and the hostname and date are added to the report. The start of the search is then recorded, and any matches of the keyword in the log are added to the report. A footer containing the name of the report and the time is then added. The report is mailed to root with the search term as the subject of the message, and the temporary file is deleted.
You can test the script by running it manually and feeding it a keyword and a pathname to the system log, /var/log/syslog, like this:
matthew@seymour:~$ sudo greplog FAILED /var/log/syslog
Note that your system should be running the syslogd daemon. If any login failures have occurred on your system, the root operator might get an email message that looks like this:
Date: Sun, 23 Oct 2016 16:23:24 -0400 From: root <[email protected]> To: [email protected] Subject: FAILED ============================================================== S Y S T E M M O N I T O R L O G Linux system 4.4.0-22-generic #1 Sun Oct 9 20:21:24 EDT 2016 +GNU/Linux Log report for righthere.home.org on Sun 23 Oct 2016 04:23:24 PM EDT ============================================================== Search for-> FAILED starting 04:23:24 PM Oct 23 16:23:04 righthere login[1769]: FAILED LOGIN 3 FROM (null) FOR bball, +Authentication failure End of /tmp/FAILED.logreport.102303 at 04:23:24 PM
To further automate the process, you can include command lines using the script in another script to generate a series of searches and reports.
Built-in Variables
Built-in variables are special variables that Linux provides to the shell that you can use to make decisions within a shell program. You cannot modify the values of these variables within the shell program.
The following are some of the built-in variables:
▸ $#—The number of positional parameters passed to the shell program
▸ $?—The completion code of the last command or shell program executed within the shell program (returned value)
▸ $0—The name of the shell program
▸ $*—A single string of all arguments passed at the time of invocation of the shell program
To show these built-in variables in use, here is a sample program called mypgm2:
#!/bin/sh #my test program echo "Number of parameters is $#" echo "Program name is $0" echo "Parameters as a single string is $*"
If you execute mypgm2 from the command line in pdksh and bash as follows:
matthew@seymour:~$ bash mypgm2 Alan Turing
you get the following result:
Number of parameters is 2 Program name is mypgm2 Parameters as a single string is Alan Turing
Special Characters
Some characters have special meaning to Linux shells; these characters represent commands, denote specific use for surrounding text, or provide search parameters. Special characters provide a sort of shorthand by incorporating rather complex meanings into simple characters. Table 14.1 shows some special characters.
Table 14.1 Special Shell Characters
Character |
Explanation |
|---|---|
|
Indicates the beginning of a shell variable name |
|
Pipes standard output to the next command |
|
Starts a comment |
|
Executes a process in the background |
|
Matches one character |
|
Matches one or more characters |
|
Redirects output |
|
Redirects input |
|
Indicates command substitution (the backquote or backtick—the key above the Tab key on most keyboards) |
|
Redirects output (to append to a file) |
|
Waits until the following end-of-input string ( |
|
Specifies a range of characters |
|
Specifies all characters a through z |
|
Specifies characters a or z |
Space |
Acts as a delimiter between two words |
Special characters are very useful when you’re creating shell scripts, but if you inadvertently use a special character as part of a variable name or string, your program behaves incorrectly. As you learn in later parts of this chapter, you can use one of the special characters in a string if you precede it with an escape character (, or backslash) to indicate that it isn’t being used as a special character and shouldn’t be treated as such by the program.
A few special characters deserve special note: double quotes ("), single quotes ('), backslash (), and backtick (`), all of which are discussed in the following sections.
Using Double Quotes to Resolve Variables in Strings with Embedded Spaces
If a string contains embedded spaces, you can enclose the string in double quotes (") so that the shell interprets the whole string as one entity instead of as more than one.
For example, if you assigned the value abc def (abc followed by one space, followed by def) to a variable called x in a shell program as follows, you would get an error because the shell would try to execute def as a separate command:
Command |
Environment |
|---|---|
|
|
|
|
The shell executes the string as a single command if you surround the string in double quotes, as follows:
Command |
Environment |
|---|---|
|
|
|
|
The double quotes resolve all variables within the string. Here is an example for pdksh and bash:
var="test string" newvar="Value of var is $var" echo $newvar
Here is the same example for tcsh:
set var="test string" set newvar="Value of var is $var" echo $newvar
If you execute a shell program containing the preceding three lines, you get the following result:
Value of var is test string
Using Single Quotes to Maintain Unexpanded Variables
You can surround a string with single quotes (') to stop the shell from expanding variables and interpreting special characters. When used for the latter purpose, the single quote is an escape character, similar to the backslash, which you learn about in the next section. Here, you learn how to use the single quote to avoid expanding a variable in a shell script. An unexpanded variable maintains its original form in the output.
In the following examples, the double quotes from the preceding examples have been changed to single quotes:
For pdksh and bash:
var='test string' newvar='Value of var is $var' echo $newvar
For tcsh:
set var = 'test string' set newvar = 'Value of var is $var' echo $newvar
If you execute a shell program containing these three lines, you get the following result:
Value of var is $var
As you can see, the variable var maintains its original format in the results rather than being expanded.
Using the Backslash as an Escape Character
As you learned earlier, the backslash () serves as an escape character that stops the shell from interpreting the succeeding character as a special character. Say that you want to assign the value $test to a variable called var. If you use the following command, the shell reads the special character $ and interprets $test as the value of the variable test. No value has been assigned to test; a null value is stored in var as follows:
Command |
Environment |
|---|---|
|
|
|
|
Unfortunately, this assignment might work for bash and pdksh, but it returns an “undefined variable” error if you use it with tcsh. Use the following commands to correctly store $test in var:
Command |
Environment |
|---|---|
|
|
|
|
The backslash before the dollar sign ($) signals the shell to interpret the $ as any other ordinary character and not to associate any special meaning to it. You could also use single quotes (') around the $test variable to get the same result.
Using the Backtick to Replace a String with Output
You can use the backtick (`) character to signal the shell to replace a string with its output when executed. This is called command substitution. You can use this special character in shell programs when you want the result of the execution of a command to be stored in a variable. For example, if you want to count the number of lines in a file called test.txt in the current directory and store the result in a variable called var, you can use the following commands:
Command |
Environment |
|---|---|
|
|
|
|
Comparison of Expressions in pdksh and bash
Comparing values or evaluating the differences between similar bits of data—such as file information, character strings, or numbers—is a task known as comparison of expressions. Comparison of expressions is an integral part of using logic in shell programs to accomplish tasks. The way the logical comparison of two operators (numeric or string) is done varies slightly in different shells. In pdksh and bash, a command called test can be used to achieve comparisons of expressions. In tcsh, you can write an expression to accomplish the same thing.
This section covers comparison operations using the pdksh or bash shells. Later in the chapter, you learn how to compare expressions in the tcsh shell.
The pdksh and bash shell syntax provide a command named test to compare strings, numbers, and files. The syntax of the test command is as follows:
test expression or [ expression ]
Both forms of the test commands are processed the same way by pdksh and bash. The test commands support the following types of comparisons:
▸ String comparison
▸ Numeric comparison
▸ File operators
▸ Logical operators
String Comparison
You can use the following operators to compare two string expressions:
▸ =—Compares whether two strings are equal
▸ !=—Compares whether two strings are not equal
▸ -n—Evaluates whether the string length is greater than zero
▸ -z—Evaluates whether the string length is equal to zero
Next are some examples using these operators when comparing two strings, string1 and string2, in a shell program called compare1:
#!/bin/sh string1="abc" string2="abd" if [ $string1 = $string2 ]; then echo "string1 equal to string2" else echo "string1 not equal to string2" fi if [ $string2 != string1 ]; then echo "string2 not equal to string1" else echo "string2 equal to string2" fi if [ $string1 ]; then echo "string1 is not empty" else echo "string1 is empty" fi if [ -n $string2 ]; then echo "string2 has a length greater than zero" else echo "string2 has length equal to zero" fi if [ -z $string1 ]; then echo "string1 has a length equal to zero" else echo "string1 has a length greater than zero" fi
If you execute compare1, you get the following result:
string1 not equal to string2 string2 not equal to string1 string1 is not empty string2 has a length greater than zero string1 has a length greater than zero
If two strings are not equal in size, the system pads out the shorter string with trailing spaces for comparison. That is, if the value of string1 is "abc" and that of string2 is "ab", string2 is padded with a trailing space for comparison purposes; it has the value "ab " (with a space after the letters).
Numeric Comparison
The following operators can be used to compare two numbers:
▸ -eq—Compares whether two numbers are equal
▸ -ge—Compares whether one number is greater than or equal to the other number
▸ -le—Compares whether one number is less than or equal to the other number
▸ -ne—Compares whether two numbers are not equal
▸ -gt—Compares whether one number is greater than the other number
▸ -lt—Compares whether one number is less than the other number
The following shell program compares three numbers, number1, number2, and number3:
#!/bin/sh number1=5 number2=10 number3=5 if [ $number1 -eq $number3 ]; then echo "number1 is equal to number3" else echo "number1 is not equal to number3" fi if [ $number1 -ne $number2 ]; then echo "number1 is not equal to number2" else echo "number1 is equal to number2" fi if [ $number1 -gt $number2 ]; then echo "number1 is greater than number2" else echo "number1 is not greater than number2" fi if [ $number1 -ge $number3 ]; then echo "number1 is greater than or equal to number3" else echo "number1 is not greater than or equal to number3" fi if [ $number1 -lt $number2 ]; then echo "number1 is less than number2" else echo "number1 is not less than number2" fi if [ $number1 -le $number3 ]; then echo "number1 is less than or equal to number3" else echo ""number1 is not less than or equal to number3" fi
When you execute the shell program, you get the following results:
number1 is equal to number3 number1 is not equal to number2 number1 is not greater than number2 number1 is greater than or equal to number3 number1 is less than number2 number1 is less than or equal to number3
File Operators
You can use the following operators as file comparison operators:
▸ -d—Determines whether a file is a directory
▸ -f—Determines whether a file is a regular file
▸ -r—Determines whether read permission is set for a file
▸ -s—Determines whether a file exists and has a length greater than zero
▸ -w—Determines whether write permission is set for a file
▸ -x—Determines whether execute permission is set for a file
Assume that a shell program called compare3 is in a directory with a file called file1 and a subdirectory dir1 under the current directory. Assume that file1 has a permission of r-x (read and execute permission) and dir1 has a permission of rwx (read, write, and execute permission). The code for the shell program would look like this:
#!/bin/sh if [ -d $dir1 ]; then echo ""dir1 is a directory" else echo ""dir1 is not a directory" fi if [ -f $dir1 ]; then echo ""dir1 is a regular file" else echo ""dir1 is not a regular file" fi if [ -r $file1 ]; then echo ""file1 has read permission" else echo ""file1 does not have read permission" fi if [ -w $file1 ]; then echo ""file1 has write permission" else echo ""file1 does not have write permission" fi if [ -x $dir1 ]; then echo ""dir1 has execute permission" else echo ""dir1 does not have execute permission" fi
If you execute the shell program, you get the following results:
dir1 is a directory file1 is a regular file file1 has read permission file1 does not have write permission dir1 has execute permission
Logical Operators
You use logical operators to compare expressions using Boolean logic—that is, comparing values using characters representing NOT, AND, and OR:
▸ !—Negates a logical expression
▸ -a—Logically ANDs two logical expressions
▸ -o—Logically ORs two logical expressions
This example named logic uses the file and directory mentioned in the previous compare3 example:
#!/bin/sh if [ -x file1 -a -x dir1 ]; then echo file1 and dir1 are executable else echo at least one of file1 or dir1 are not executable fi if [ -w file1 -o -w dir1 ]; then echo file1 or dir1 are writable else echo neither file1 or dir1 are executable fi if [ ! -w file1 ]; then echo file1 is not writable else echo file1 is writable fi
If you execute logic, it yields the following result:
file1 and dir1 are executable file1 or dir1 are writable file1 is not writable
Comparing Expressions with tcsh
As stated earlier, the method for comparing expressions in tcsh is different from the method used under pdksh and bash. The comparison of expression demonstrated in this section uses the syntax necessary for the tcsh shell environment.
String Comparison
You can use the following operators to compare two string expressions:
▸ ==—Compares whether two strings are equal
▸ !=—Compares whether two strings are not equal
The following examples compare two strings, string1 and string2, in the shell program compare1:
#!/bin/tcsh set string1 = "abc" set string2 = "abd" if (string1 == string2) then echo "string1 equal to string2" else echo "string1 not equal to string2" endif if (string2 != string1) then echo "string2 not equal to string1" else echo "string2 equal to string1" endif
If you execute compare1, you get the following results:
string1 not equal to string2 string2 not equal to string1
Number Comparison
You can use the following operators to compare two numbers:
▸ >=—Determines whether one number is greater than or equal to the other number
▸ <=—Determines whether one number is less than or equal to the other number
▸ >—Determines whether one number is greater than the other number
▸ <—Determines whether one number is less than the other number
The next examples compare three numbers, number1, number2, and number3, in a shell program called compare2:
#!/bin/tcsh set number1=5 set number2=10 set number3=5 if (number1 > number2) then echo "number1 is greater than number2" else echo "number1 is not greater than number2" endif if (number1 >= number3) then echo "number1 is greater than or equal to number3" else echo "number1 is not greater than or equal to number3" endif if (number1 < number2) then echo "number1 is less than number2" else echo "number1 is not less than number2" endif if (number1 <= number3) then echo "number1 is less than or equal to number3" else echo "number1 is not less than or equal to number3" endif
When executing the shell program compare2, you get the following results:
number1 is not greater than number2 number1 is greater than or equal to number3 number1 is less than number2 number1 is less than or equal to number3
File Operators
You can use the following operators as file comparison operators:
▸ -d—Determines whether a file is a directory
▸ -e—Determines whether a file exists
▸ -f—Determines whether a file is a regular file
▸ -o—Determines whether a user is the owner of a file
▸ -r—Determines whether read permission is set for a file
▸ -w—Determines whether write permission is set for a file
▸ -x—Determines whether execute permission is set for a file
▸ -z—Determines whether the file size is zero
The following examples are based on a shell program called compare3, which is in a directory with a file called file1 and a subdirectory dir1 under the current directory. Assume that file1 has a permission of r-x (read and execute permission), and dir1 has a permission of rwx (read, write, and execute permission).
The following is the code for the compare3 shell program:
#!/bin/tcsh if (-d dir1) then echo "dir1 is a directory" else echo "dir1 is not a directory" endif if (-f dir1) then echo "file1 is a regular file" else echo "file1 is not a regular file" endif if (-r file1) then echo "file1 has read permission" else echo "file1 does not have read permission" endif if (-w file1) then echo "file1 has write permission" else echo "file1 does not have write permission" endif if (-x dir1) then echo "dir1 has execute permission" else echo "dir1 does not have execute permission" endif if (-z file1) then echo "file1 has zero length" else echo "file1 has greater than zero length" endif
If you execute the file compare3, you get the following results:
dir1 is a directory file1 is a regular file file1 has read permission file1 does not have write permission dir1 has execute permission file1 has greater than zero length
Logical Operators
You use logical operators with conditional statements. You use the following operators to negate a logical expression or to perform logical ANDs and ORs:
▸ !—Negates a logical expression
▸ &&—Logically ANDs two logical expressions
▸ ||—Logically ORs two logical expressions
This example named logic uses the file and directory mentioned in the previous compare3 example:
#!/bin/tcsh if ( -x file1 && -x dir1 ) then echo file1 and dir1 are executable else echo at least one of file1 or dir1 are not executable endif if ( -w file1 || -w dir1 ) then echo file1 or dir1 are writable else echo neither file1 or dir1 are executable endif if ( ! -w file1 ) then echo file1 is not writable else echo file1 is writable endif
If you execute logic, it yields the following result:
file1 and dir1 are executable file1 or dir1 are writable file1 is not writable
The for Statement
You use the for statement to execute a set of commands once each time a specified condition is true. The for statement has a number of formats. The first format used by pdksh and bash is as follows:
for curvar in list
do
statements
done
You should use this format if you want to execute statements once for each value in list. For each iteration, the current value of the list is assigned to vcurvar. list can be a variable containing a number of items or a list of values separated by spaces. The second format is as follows:
for curvar
do
statements
done
In this format, the statements are executed once for each of the positional parameters passed to the shell program. For each iteration, the current value of the positional parameter is assigned to the variable curvar. You can also write this format as follows:
for curvar in $
do
statements
done
Remember that $@ gives you a list of positional parameters passed to the shell program, quoted in a manner consistent with the way the user originally invoked the command.
Under tcsh, the for statement is called foreach, and the format is as follows:
foreach curvar (list)
statements
end
In this format, statements are executed once for each value in list, and, for each iteration, the current value of list is assigned to curvar.
Suppose that you want to create a backup version of each file in a directory to a subdirectory called backup. You can do the following in pdksh and bash:
#!/bin/sh
for filename in *
do
cp $filename backup/$filename
if [ $? -ne 0 ]; then
echo "copy for $filename failed"
fi
done
In this example, a backup copy of each file is created. If the copy fails, a message is generated.
The same example in tcsh is as follows:
#!/bin/tcsh
foreach filename (`/bin/ls`)
cp $filename backup/$filename
if ($? != 0) then
echo "copy for $filename failed"
endif
end
The while Statement
You can use the while statement to execute a series of commands while a specified condition is true. The loop terminates as soon as the specified condition evaluates to false. It is possible that the loop will not execute at all if the specified condition initially evaluates to false. You should be careful with the while command because the loop never terminates if the specified condition never evaluates to false.
Endless Loops Have a Place in Shell Programs
Endless loops can sometimes be useful. For example, you can easily construct a simple command that constantly monitors the 802.11 link quality of a network interface by using a few lines of script:
#!/bin/sh while : do /sbin/iwconfig wlan0 | grep Link | tr ' ' ' ' Done
The script outputs the search, and then the tr command formats the output. The result is a simple animation of a constantly updated single line of information:
Link Quality:92/92 Signal level:-11 dBm Noise level:-102 dBm
You can also use this technique to create a graphical monitoring client for X that outputs traffic information and activity about a network interface:
#!/bin/sh
xterm -geometry 75x2 -e
bash -c
"while :; do
/sbin/ifconfig eth0 |
grep 'TX bytes' |
tr '
' '
' ;
done"
This simple example uses a bash command-line script (enabled by -c) to execute a command line repeatedly. The command line pipes the output of the ifconfig command through grep, which searches the output of ifconfig and then pipes a line containing the string “TX bytes” to the tr command. The tr command then removes the carriage return at the end of the line to display the information inside an /xterm X11 terminal window, automatically sized by the -geometry option:
RX bytes:4117594780 (3926.8 Mb) TX bytes:452230967 (431.2 Mb)
Endless loops can be so useful that Linux includes a command that repeatedly executes a given command line. For example, you can get a quick report about a system’s hardware health by using the sensors command. Instead of using a shell script to loop the output endlessly, you can use the watch command to repeat the information and provide simple animation:
matthew@seymour:~$ watch “sensors -f | cut -c 1-20”
In pdksh and bash, use the following format for the while flow control construct:
while expression
do
statements
done
In tcsh, use the following format:
while (expression)
statements
End
If you want to add the first five even numbers, you can use the following shell program in pdksh and bash:
#!/bin/bash loopcount=0 result=0 while [ $loopcount -lt 5 ] do loopcount=`expr $loopcount + 1` increment=`expr $loopcount * 2` result=`expr $result + $increment` doneecho "result is $result"
In tcsh, you can write this program as follows:
#!/bin/tcsh set loopcount = 0 set result = 0 while ($loopcount < 5) set loopcount = `expr $loopcount + 1` set increment = `expr $loopcount * 2` set result = `expr $result + $increment` end echo "result is $result"
The until Statement
You can use the until statement to execute a series of commands until a specified condition is true.
The loop terminates as soon as the specified condition evaluates to true.
In pdksh and bash, the following format is used:
until expression
do
statements
done
As you can see, the format of the until statement is similar to that of the while statement, but the logic is different: In a while loop, you execute until an expression is false, whereas in an until loop, you loop until the expression is true. An important part of this difference is that while is executed zero or more times (so it is potentially not executed at all), but until is repeated one or more times, meaning it is executed at least once.
If you want to add the first five even numbers, you can use the following shell program in pdksh and bash:
#!/bin/bash loopcount=0 result=0 until [ $loopcount -ge 5 ] do loopcount=`expr $loopcount + 1` increment=`expr $loopcount * 2` result=`expr $result + $increment` done echo "result is $result"
The example here is identical to the example for the while statement except that the condition being tested is just the opposite of the condition specified in the while statement.
The tcsh shell does not support the until statement.
The repeat Statement (tcsh)
You use the repeat statement to execute only one command a fixed number of times.
If you want to print a hyphen (-) 80 times with one hyphen per line on the screen, you can use the following command:
repeat 80 echo '-'
The select Statement (pdksh)
You use the select statement to generate a menu list if you are writing a shell program that expects input from the user online. The format of the select statement is as follows:
select item in itemlist
do
Statements
Done
itemlist is optional. If it isn’t provided, the system iterates through the item entries one at a time. If itemlist is provided, however, the system iterates for each entry in itemlist, and the current value of itemlist is assigned to item for each iteration, which then can be used as part of the statements being executed.
If you want to write a menu that gives the user a choice of picking a Continue or a Finish, you can write the following shell program:
#!/bin/ksh
select item in Continue Finish
do
if [ $item = "Finish" ]; then
break
fi
done
When the select command is executed, the system displays a menu with numeric choices—in this case, 1 for Continue and 2 for Finish. If the user chooses 1, the variable item contains the value Continue; if the user chooses 2, the variable item contains the value Finish. When the user chooses 2, the if statement is executed, and the loop terminates.
The shift Statement
You use the shift statement to process the positional parameters, one at a time, from left to right. Recall that the positional parameters are identified as $1, $2, $3, and so on. The effect of the shift command is that each positional parameter is moved one position to the left, and the current $1 parameter is lost.
The shift statement is useful when you are writing shell programs in which a user can pass various options. Depending on the specified option, the parameters that follow can mean different things or might not be there at all.
The format of the shift command is as follows:
shift number
The parameter number is the number of places to be shifted and is optional. If it is not specified, the default is 1; that is, the parameters are shifted one position to the left. If it is specified, the parameters are shifted number positions to the left.
The if Statement
The if statement evaluates a logical expression to make a decision. An if condition has the following format in pdksh and bash:
if [ expression ]; then
Statements
elif [ expression ]; then
Statements
else
Statements
fi
if conditions can be nested. That is, an if condition can contain another if condition within it. It isn’t necessary for an if condition to have an elif or else part. The else part is executed if none of the expressions that are specified in the if statement are true and are not evaluated if preceding elif statements are true. The word fi is used to indicate the end of the if statements, which is very useful if you have nested if conditions. In such a case, you should be able to match fi to if to ensure that all if statements are properly coded.
In the following example for bash or pdksh, a variable var can have either of two values: Yes or No. Any other value is invalid. This can be coded as follows:
if [ $var = "Yes" ]; then echo "Value is Yes" elif [ $var = "No" ]; then echo "Value is No" else echo "Invalid value" fi
In tcsh, the if statement has two forms. The first form, similar to the one for pdksh and bash, is as follows:
if (expression) then
Statements
else if (expression) then
Statements
else
Statements
endif
Using the example of the variable var having only two values, Yes and No, here is how it is coded with tcsh:
if ($var == "Yes") then echo "Value is Yes" else if ($var == "No" ) then echo "Value is No" else echo "Invalid value" endif
The second form of the if condition for tcsh is as follows:
if (expression) command
In this format, only a single command can be executed if the expression evaluates to true.
The expr Statement
You use expr to evaluate an expression. It can evaluate numbers and mathematic expressions. It can also be used to evaluate strings and perform comparisons. Some examples follow.
To add or subtract, multiply or divide, use one of the following:
expr 12 + 2 expr 26-24 expr 6 * 3 expr 11 / 3
There are two things to note in this set of operations.
▸ When you multiply, you must use an escape character before the multiplication operator for the command to parse correctly, which is why you see (the escape character) before the * (the multiplication character).
▸ When you divide, only the whole number result will be returned, so, for example, expr 11 / 2 returns 3. To get the remainder, use the following, which returns 2:
expr 11 / 2
A number of string-related operations are available. Each involves adding a second word to the command before the string to be evaluated.
To find the length of a string, use the following:
expr length string
If the string includes spaces, you must again use the escape character, like this:
expr length linux is cool 13
You can do many more things with expr. Read the man page for more information.
The case Statement
You use the case statement to execute statements depending on a discrete value or a range of values matching the specified variable. In most cases, you can use a case statement instead of an if statement if you have a large number of conditions.
The format of a case statement for pdksh and bash is as follows:
case str in
str1 | str2)
Statements;;
str3|str4)
Statements;;
*)
Statements;;
esac
You can specify a number of discrete values—such as str1, str2, and so on—for each condition, or you can specify a value with a wildcard. The last condition should be an asterisk (*) and is executed if none of the other conditions are met. For each of the specified conditions, all the associated statements until the double semicolon (;;) are executed.
You can write a script that echoes the name of the month if you provide the month number as a parameter. If you provide a number that isn’t between 1 and 12, you get an error message. The script is as follows:
#!/bin/sh case $1 in 01 | 1) echo "Month is January";; 02 | 2) echo "Month is February";; 03 | 3) echo "Month is March";; 04 | 4) echo "Month is April";; 05 | 5) echo "Month is May";; 06 | 6) echo "Month is June";; 07 | 7) echo "Month is July";; 08 | 8) echo "Month is August";; 09 | 9) echo "Month is September";; 10) echo "Month is October";; 11) echo "Month is November";; 12) echo "Month is December";; *) echo "Invalid parameter";; esac
You need to end the statements under each condition with a double semicolon (;;). If you do not, the statements under the next condition are also executed.
The format for a case statement for tcsh is as follows:
switch (str)
case str1|str2:
Statements
breaksw
case str3|str4:
Statements
breaksw
default:
Statements
breaksw
endsw
You can specify a number of discrete values—such as str1, str2, and so on—for each condition, or you can specify a value with a wildcard. The last condition should be the default and is executed if none of the other conditions are met. For each of the specified conditions, all the associated statements until breaksw are executed.
You can write the example that echoes the month when a number is given, shown earlier for pdksh and bash, in tcsh as follows:
#!/bin/tcsh set month = 5 switch ( $month ) case 1: echo "Month is January" ; breaksw case 2: echo "Month is February" ; breaksw case 3: echo "Month is March" ; breaksw case 4: echo "Month is April" ; breaksw case 5: echo "Month is May" ; breaksw case 6: echo "Month is June" ; breaksw case 7: echo "Month is July" ; breaksw case 8: echo "Month is August" ; breaksw case 9: echo "Month is September" ; breaksw case 10: echo "Month is October" ; breaksw case 11: echo "Month is November" ; breaksw case 12: echo "Month is December" ; breaksw default: echo "Oops! Month is Octember!" ; breaksw endsw
You need to end the statements under each condition with breaksw. If you do not, the statements under the next condition are also executed.
The break and exit Statements
You should be aware of two other statements: the break statement and the exit statement.
You can use the break statement to terminate an iteration loop, such as a for, until, or repeat command.
You can use exit statement to exit a shell program. You can optionally use a number after exit. If the current shell program has been called by another shell program, the calling program can check for the code (the $? or $status variable, depending on shell) and make a decision accordingly.
Using Functions in Shell Scripts
As with other programming languages, in Ubuntu shell programs also support functions. A function is a piece of a shell program that performs a particular process; you can reuse the same function multiple times within the shell program. Functions help eliminate the need for duplicating code as you write shell programs.
The following is the format of a function in pdksh and bash:
func(){
Statements
}
You can call a function as follows:
func param1 param2 param3
The parameters param1, param2, and so on are optional. You can also pass the parameters as a single string—for example, $@. A function can parse the parameters as if they were positional parameters passed to a shell program from the command line as command-line arguments but instead use values passed inside the script. For example, the following script uses a function named Displaymonth() that displays the name of the month or an error message if you pass a month number out of the range 1 to 12. This example works with pdksh and bash:
#!/bin/sh
Displaymonth() {
case $1 in
01 | 1) echo "Month is January";;
02 | 2) echo "Month is February";;
03 | 3) echo "Month is March";;
04 | 4) echo "Month is April";;
05 | 5) echo "Month is May";;
06 | 6) echo "Month is June";;
07 | 7) echo "Month is July";;
08 | 8) echo "Month is August";;
09 | 9) echo "Month is September";;
10) echo "Month is October";;
11) echo "Month is November";;
12) echo "Month is December";;
*) echo "Invalid parameter";;
esac
}
Displaymonth 8
The preceding program displays the following output:
Month is August
References
▸ www.gnu.org/software/bash/—The bash home page at the GNU Software Project
▸ www.tldp.org/LDP/abs/html/—Mendel Cooper’s “Advanced Bash-Scripting Guide”
▸ www.freeos.com/guides/lsst/—Linux shell scripting tutorial
▸ https://kornshell.com—The KornShell website
▸ https://web.cs.mun.ca/~michael/pdksh/—The pdksh home page
▸ www.tcsh.org—Find out more about tcsh here
▸ www.zsh.org/—Examine zsh in more detail here