The << label redirector essentially forces the input to a command to be the shell program’s text, which is read until there is a line that contains only label. The input in between is called a here-document. Here-documents aren’t very interesting when used from the command prompt. In fact, it’s the same as the normal use of standard input except for the label. We could have used a here-document in the previous example of >>, like this (EOF, for “end of file,” is an often-used label):

$ cat >> .mailrc << EOF
> alias fred [email protected]
> EOF
$

Here-documents are meant to be used from within shell scripts; they let you specify “batch” input to programs. A common use of here-documents is with simple text editors like ed(1). Task 7-1 uses a here-document in this way.

We can use ed to delete the body lines, leaving just the header. To do this, we need to know something about the syntax of mail messages, specifically, that there is always a blank line between the header lines and the message text. The ed command /^$/,$d does the trick: it means, “Delete from the first blank line^[81] through the last line of the file.” We also need the ed commands w (write the changed file) and q (quit). Here is the code that solves the task:

ed $1 << EOF
/^$/,$d
w
q
EOF

Normally, the shell does parameter (variable) substitution, command substitution, and arithmetic substitution on text in a here-document, meaning that you can use shell variables and commands to customize the text. This evaluation is disabled if any part of the delimiter is quoted, as done in the previous example. (This prevents the shell from treating $d as a variable substitution.)

Often though, you do want the shell to perform its evaluations: perhaps the most common use of here-documents is for providing templates for form generators or program text for program generators. Task 7-2 is a simple task for system administrators that shows how this works.

You can get a list of all users on the system in various ways; perhaps the easiest is to use cut to extract the first field of /etc/passwd, the file that contains all user account information. Fields in this file are separated by colons (:).^[82]

Given such a list of users, the following code does the trick:

pgmname=$1
for user in $(cut -f1 -d: /etc/passwd); do
    mail $user << EOF
Dear $user,

A new version of $pgmname has been installed in $(whence pgmname).

Regards,
Your friendly neighborhood sysadmin.
EOF
done

The shell substitutes the appropriate values for the name of the program and its directory.

The redirector << has two variations. First, you can prevent the shell from doing parameter, command and arithmetic substitution by surrounding the label in single or double quotes. (Actually, it’s enough to quote just one character in the label.) We saw this in the solution to Task 7-1.

The second variation is <<-, which deletes leading TABs (but not spaces) from the here-document and the label line. This allows you to indent the here-document’s text, making the shell script more readable:

pgmname=$1
for user in $(cut -f1 -d: /etc/passwd); do
    mail $user <<- EOF
        Dear $user,

        A new version of $pgmname has been installed in $(whence pgmname).

        Regards,

        Your friendly neighborhood sysadmin.
        EOF
done

Of course, you need to choose your label so that it doesn’t appear as an actual input line.

A common idiom in shell programming is to use print to generate some text to be further processed by one or more commands:

# start with a mild interrogation
print -r "$name, $rank, $serial_num" | interrogate -i mild

This could be rewritten to use a here-document, which is slightly more efficient, although not necessarily any easier to read:

# start with a mild interrogation
interrogate -i mild << EOF
$name, $rank, $serial_num
EOF

Starting with ksh93n,^[83] the Korn shell provides a new form of here-document, using three less-than signs:

               program <<< WORD

In this form, the text of WORD (followed by a trailing newline) becomes the input to the program. For example:

# start with a mild interrogation
interrogate -i mild <<< "$name, $rank, $serial_num"

This notation first originated in the Unix version of the rc shell, where it is called a “here string.” It was later picked up by the Z shell, zsh (see Appendix A), from which the Korn shell borrowed it. This notation is simple, easy to use, efficient, and visually distinguishable from regular here-documents.

The next few redirectors in Table 7-1 depend on the notion of a file descriptor. This is a low-level Unix I/O concept that is vital to understand when programming in C or C++. It appears at the shell level when you want to do anything that doesn’t involve standard input, standard output and standard error. You can get by with a few basic facts about them; for the whole story, look at the open(2), creat(2), read(2), write(2), dup(2), dup2(2), fcntl(2), and close(2) entries in the Unix manual. (As the manual entries are aimed at the C programmer, their relationship to the shell concepts won’t necessarily be obvious.)

File descriptors are integers starting at 0 that index an array of file information within a process. When a process starts, it has three file descriptors open. These correspond to the three standards: standard input (file descriptor 0), standard output (1), and standard error (2). If a process opens Unix files for input or output, they are assigned to the next available file descriptors, starting with 3.

By far the most common use of file descriptors with the Korn shell is in saving standard error in a file. For example, if you want to save the error messages from a long job in a file so that they don’t scroll off the screen, append 2> file to your command. If you also want to save standard output, append > file1 2> file2.

This leads to Task 7-3.

We’ll call this function start. The code is very terse:

function start {
    "$@" > logfile 2>&1 &
}

This line executes whatever command and parameters follow start. (The command cannot contain pipes or output redirectors.) It first sends the command’s standard output to logfile.

Then, the redirector 2>&1 says, “Send standard error (file descriptor 2) to the same place as standard output (file descriptor 1).” 2>&1 is actually a combination of two redirectors in Table 7-1: n > file and >& n. Since standard output is redirected to logfile, standard error will go there too. The final & puts the job in the background so that you get your shell prompt back.

As a small variation on this theme, we can send both standard output and standard error into a pipe instead of a file: command 2>&1 | ... does this. (Why this works is described shortly.) Here is a function that sends both standard output and standard error to the logfile (as above) and to the terminal:

function start {
    "$@" 2>&1 | tee logfile &
}

The command tee(1) takes its standard input and copies it to standard output and the file given as argument.

These functions have one shortcoming: you must remain logged in until the job completes. Although you can always type jobs (see Chapter 1) to check on progress, you can’t leave your office for the day unless you want to risk a breach of security or waste electricity. We’ll see how to solve this problem in Chapter 8.

The other file-descriptor-oriented redirectors (e.g., <& n) are usually used for reading input from (or writing output to) more than one file at the same time. We’ll see an example later in this chapter. Otherwise, they’re mainly meant for systems programmers, as are <&- (force standard input to close) and >&- (force standard output to close), <& n - (move file descriptor n to standard input) and >& n - (move file descriptor n to standard output).

Finally, we should just note that 0< is the same as <, and 1> is the same as >. (In fact, 0 is the default for any operator that begins with <, and 1 is the default for any operator that begins with >.)

$ mycommand -h fred -w wilma 2>&1 | more

program > file1 2>&1          Standard output and standard error to file1
program 2>&1 > file1          Standard error to terminal and standard output to file1

Normally, when you provide a pathname after an I/O redirector such as < or >, the shell tries to open an actual file that has the given filename. However, there are two kinds of pathnames where the shell instead treats the pathnames specially.

The first kind of pathname is /dev/fd/ N, where N is the file descriptor number of an already open file. For example:

# assume file descriptor 6 is already open on a file
print 'something meaningful' > /dev/fd/6   # same as 1>&6

This works even on systems that don’t have a /dev/fd directory. This kind of pathname may also be used with the various file attribute test operators of the [[...]] command.

The second kind of pathname allows access to Internet services via either the TCP or UDP protocol. The pathnames are:

To use these files for two-way I/O, open a new file descriptor using the exec command (which is described in Chapter 9), using the “read and write” operator, <>. Then use read -u and print -u to read from and write to the new file descriptor. (The read command and the -u option to read and print are described later in this chapter.)

The following example, courtesy of David Korn, shows how to do this. It implements the whois(1) program, which provides information about the registration of Internet domain names:

host=rs.internic.net
port=43
exec 3<> /dev/tcp/$host/$port
print -u3 -f "%s
" "$@"
cat <&3

Using the exec built-in command (see Chapter 9), this program uses the “read-and-write” operator, <>, to open a two-way connection to the host rs.internic.net on TCP port 43, which provides the whois service. (The script could have used port=whois as well.) It then uses the print command to send the argument strings to the whois server. Finally, it reads the returned result using cat. Here is a sample run:

$ whois.ksh kornshell.com

Whois Server Version 1.3

Domain names in the .com, .net, and .org domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

   Domain Name: KORNSHELL.COM
   Registrar: NETWORK SOLUTIONS, INC.
   Whois Server: whois.networksolutions.com
   Referral URL: http://www.networksolutions.com
   Name Server: NS4.PAIR.COM
   Name Server: NS0.NS0.COM
   Updated Date: 02-dec-2001


>>> Last update of whois database: Sun, 10 Feb 2002 05:19:14 EST <<<

The Registry database contains ONLY .COM, .NET, .ORG, .EDU domains and
Registrars.

Network programming is beyond the scope of this book. But for most things, you will probably want to use TCP connections instead of UDP connections if you do write any networking programs in ksh.

As we’ve seen countless times in this book, print simply prints its arguments to standard output. You should use it instead of the echo command, whose functionality differs from system to system.^[86] (The Korn shell’s built-in version of echo emulates whatever the system’s standard version of echo does.) Now we’ll explore the print command in greater detail.

$ P
                     ATH=/bin:/usr/bin:/etc:/usr/ucb:/usr/local/bin:/home/billr/bin

If you need to produce formatted reports, the shell’s print command can be combined with formatting attributes for variables to produce output data that lines up reasonably. But you can only do so much with these facilities.

The C language’s printf(3) library routine provides powerful formatting facilities for total control of output. It is so useful that many other Unix-derived programming languages, such as awk and perl, support similar or identical facilities. Primarily because the behavior of echo on different Unix systems could not be reconciled, and recognizing printf’s utility, the POSIX shell standard mandates a printf shell-level command that provides the same functionality as the printf(3) library routine. This section describes how the printf command works and examines additional capabilities unique to the Korn shell’s version of printf.

The printf command can output a simple string just like the print command.

printf "Hello, world
"

The main difference that you will notice at the outset is that, unlike print, printf does not automatically supply a newline. You must specify it explicitly as .

The full syntax of the printf command has two parts:

printf format-string [arguments ...]

The first part is a string that describes the format specifications; this is best supplied as a string constant in quotes. The second part is an argument list, such as a list of strings or variable values, that correspond to the format specifications. (If there are more arguments than format specifications, ksh cycles through the format specifications in the format string, reusing them in order, until done.) A format specification is preceded by a percent sign (%), and the specifier is one of the characters described shortly. Two of the main format specifiers are %s for strings and %d for decimal integers.

The format string combines text to be output literally with specifications describing how to format subsequent arguments on the printf command line. For example:

$ printf "Hello, %s
" World
Hello, World

Because the printf command is built-in, you are not limited to absolute numbers:

$ printf "The answer is %d.
" 12+10+20
The answer is 42.

The allowed specifiers are shown in Table 7-4.

The printf command can be used to specify the width and alignment of output fields. A format expression can take three optional modifiers following % and preceding the format specifier:

%flags width.precision format-specifier

The width of the output field is a numeric value. When you specify a field width, the contents of the field are right-justified by default. You must specify a flag of “-” to get left-justification. (The rest of the flags are discussed shortly.) Thus, "%-20s" outputs a left-justified string in a field 20 characters wide. If the string is less than 20 characters, the field is padded with whitespace to fill. In the following examples, a | is output to indicate the actual width of the field. The first example right-justifies the text:

printf "|%10s|
" hello

It produces:

|     hello|

The next example left-justifies the text:

printf "|%-10s|
" hello

It produces:

|hello     |

The precision modifier, used for decimal or floating-point values, controls the number of digits that appear in the result. For string values, it controls the maximum number of characters from the string that will be printed.

You can specify both the width and precision dynamically, via values in the printf argument list. You do this by specifying asterisks, instead of literal values.

$ myvar=42.123456
$ printf "|%*.*G|
" 5 6 $myvar
|42.1235|

In this example, the width is 5, the precision is 6, and the value to print comes from the value of myvar.

The precision is optional. Its exact meaning varies by control letter, as shown in Table 7-5:

Finally, one or more flags may precede the field width and the precision. We’ve already seen the “-” flag for left-justification. The rest of the flags are shown in Table 7-6.

If printf cannot perform a format conversion, it returns a non-zero exit status.

Similar to print, the built-in printf command interprets escape sequences within the format string. However, printf accepts a larger range of escape sequences; they are the same as for the $'...' string. These sequences are listed later in Table 7-9.

$ printf '42 is %.3.5d in base 5
' 42
42 is 132 in base 5

The other side of the shell’s string I/O facilities is the read command, which allows you to read values into shell variables. The basic syntax is:

read var1 var2 ...

There are a few options, which we cover in Section 7.2.3.5, later in this chapter. This statement takes a line from the standard input and breaks it down into words delimited by any of the characters in the value of the variable IFS (see Chapter 4; these are usually a space, a TAB, and newline). The words are assigned to variables var1, var2, etc. For example:

$ read fred bob
                  dave pete
$ print "$fred"
dave

$ print "$bob"
pete

If there are more words than variables, excess words are assigned to the last variable. If you omit the variables altogether, the entire line of input is assigned to the variable REPLY.

You may have identified this as the missing ingredient in the shell programming capabilities we’ve seen so far. It resembles input statements in conventional languages, like its namesake in Pascal. So why did we wait this long to introduce it?

Actually, read is sort of an escape hatch from traditional shell programming philosophy, which dictates that the most important unit of data to process is a text file, and that Unix utilities such as cut, grep, sort, etc., should be used as building blocks for writing programs.

read, on the other hand, implies line-by-line processing. You could use it to write a shell script that does what a pipeline of utilities would normally do, but such a script would inevitably look like:

while (read a line) do
    process the line
    print the processed line
end

This type of script is usually much slower than a pipeline; furthermore, it has the same form as a program someone might write in C (or some similar language) that does the same thing much, much faster. In other words, if you are going to write it in this line-by-line way, there is no point in writing a shell script. (The authors have gone for years without writing a script with read in it.)

console s531
tty01   gl35a
tty03   gl35a
tty04   gl35a
tty07   t2000
tty08   s531

TERM=vt99       # assume this as a default
line=$(tty)
while read dev termtype; do
    if [[ $dev == $line ]]; then
        TERM=$termtype
        export TERM
        print "TERM set to $TERM."
        break
    fi
done

function findterm {
    TERM=vt99       # assume this as a default
    line=$(tty)
    while read dev termtype; do
        if [[ $dev == $line ]]; then
            TERM=$termtype
            export TERM
            print "TERM set to $TERM."
            break
        fi
    done
}

findterm < /etc/terms

TERM=vt99       # assume this as a default
line=$(tty)
while read dev termtype; do
    if [[ $dev == $line ]]; then
        TERM=$termtype
        export TERM
        print "TERM set to $TERM."
        break
    fi
done < /etc/terms

while read dev termtype < /etc/terms
do
    ...
done

{
    TERM=vt99       # assume this as a default
    line=$(tty)
    while read dev termtype; do
        if [[ $dev == $line ]]; then
            TERM=$termtype
            export TERM
            print "TERM set to $TERM."
            break
        fi
    done
} < /etc/terms

{ TERM=vt99; line=$(tty); while ... ; } < /etc/terms

#
# System Console is a Shande 531s
console s531
#
# Prof. Subramaniam's line has a Givalt GL35a
tty01   gl35a
...

if [[ $dev != #* && $dev == $line ]]; then
    ...

grep "^[^#]" /etc/terms | {
    TERM=vt99
    ...
}

ls "$@" | {
    let width=0
    while read fname; do
        if (( ${#fname} > $width )); then
            let width=${#fname}
        fi
    done
    let "width += 2"
    let numcols="int(${COLUMNS:-80} / $width)"
}

set -A filenames $(ls "$@")
typeset -L$width fname
let count=0

while (( $count < ${#filenames[*]} )); do
    fname=${filenames[$count]}
    print "$fname  c"
    let count++
    if [[ $((count % numcols)) == 0 ]]; then
         print          # output a newline
    fi
done

if (( count % numcols != 0 )); then
    print
fi

print -n 'terminal? '
read TERM
print "TERM is $TERM"

terminal? vt99
TERM is vt99

print -n 'terminal? ' >&2
read TERM
print TERM is $TERM

read TERM?'terminal? '
print "TERM is $TERM"

set -A termnames gl35a t2000 s531 vt99
print 'Select your terminal type:'
while true;  do
    {
        print '1) gl35a'
        print '2) t2000'
        print '3) s531'
        print '4) vt99'
    } >&2
    read REPLY?'terminal? '

    if (( REPLY >= 1 && REPLY <= 4 )); then
        TERM=${termnames[REPLY-1]}
        print "TERM is $TERM"
        export TERM
        break
    fi
done

A line with a
 escape sequence

$ read -r fredline < fred
$ print "$fredline"
A line with a
 escape sequence
$

$ read fredline < fred
$ print "$fredline"
A line with an escape sequence
$

while read -s cmd; do
    # process the command
done

print -n "OK, Mr. $prisoner, enter your name, rank and serial number: "
# wait two hours, no more
if read -t $((60 * 60 * 2)) name rank serial
then
    # process information
    ...
else
    # prisoner is being silent
    print 'The silent treatment, eh? Just you wait.'
    call_evil_colonel -p $prisoner
    ...
fi

typeset -L30 f1 f2
while read -u3 f1 && read -u4 f2; do
    print "$f1$f2"
done 3<$1 4<$2

DAVE
Height: 177.8 cm.
Weight: 79.5 kg.
Hair: brown
Eyes: brown

SHIRLEY
Height: 167.6 cm.
Weight: 65.5 kg.
Hair: blonde
Eyes: blue

DAVE                          SHIRLEY
Height: 177.8 cm.             Height: 167.6 cm.
Weight: 79.5 kg.              Weight: 65.5 kg.
Hair: brown                   Hair: blonde
Eyes: brown                   Eyes: blue

Brace expansion is a feature borrowed from the Berkeley csh command interpreter and also available in the popular bash shell. Brace expansion is a way of saving typing when you have strings that are prefixes or suffixes of each other. For example, suppose you have the following files:

$ ls
cpp-args.c  cpp-lex.c  cpp-out.c  cpp-parse.c

You could type vi cpp-{args,lex,parse}.c if you wished to edit three out of the four C files, and the shell would expand this into vi cpp-args.c cpp-lex.c cpp-parse.c. Furthermore, brace substitutions may be nested. For example:

$ print cpp-{args,l{e,o}x,parse}.c
cpp-args.c cpp-lex.c cpp-lox.c cpp-parse.c

This is a handy feature. We haven’t covered it up until now because it’s possible that your version of ksh may not have it. It is an optional feature that is enabled when ksh is compiled. However, it is enabled by default when ksh93 is compiled from source code.

Process substitution allows you to open multiple process streams and feed them into a single program for processing. For example:

awk '...' <(generate_data) <(generate_more_data)

(Note that the parentheses are part of the syntax; you type them literally.) Here, generate_data and generate_more_data represent arbitrary commands, including pipelines, that produce streams of data. The awk program processes each stream in turn, not realizing that the data is coming from multiple sources. This is shown graphically in Figure 7-1.a.

Figure 7-1. Process substitution for both input and output data streams

Process substitution may also be used for output, particularly when combined with the tee(1) program, which sends its input to multiple output files and to standard output. For example:

               generate_data | tee >(sort | uniq > sorted_data) 
                    >(mail -s 'raw data' joe) > raw_data

This command uses tee to (1) send the data to a pipeline that sorts and saves the data, (2) send the data to the mail program to user joe, and (3) redirect the original data into a file. This is represented graphically in Figure 7-1.b. Process substitution, combined with tee, allows you to create nonlinear data graphs, freeing you from the straight “one input, one output” paradigm of traditional Unix pipes.

Process substitution is only available on Unix systems that support the /dev/fd/N special files for named access to already open file descriptors. (This is different from the use of /dev/fd/N described earlier in this chapter, where the shell itself interprets the pathname. Here, because external commands must be able to open files in /dev/fd, the feature must be directly supported by the operating system.) Most modern Unix systems, including GNU/Linux, support this feature. Like brace substitution, it must be enabled at compile time, and may not be available in your version of ksh. As with brace expansion, it is enabled by default when ksh93 is compiled from source code.

We’ve touched upon command-line processing (see Figure 7-2) throughout this book; now is a good time to make the whole thing explicit.^[90] Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, and then does the following for each command:

Figure 7-2. Steps in command-line processing

That’s a lot of steps — and it’s not even the whole story! But before we go on, an example should make this process clearer. Assume that the following command has been run:

alias ll="ls -l"

Further assume that a file exists called .hist537 in user fred’s home directory, which is /home/fred, and that there is a double-dollar-sign variable $$ whose value is 2537 (we’ll see what this special variable is in the next chapter).

Now let’s see how the shell processes the following command:

ll $(whence cc) ~fred/.*$(($$%1000))

Here is what happens to this line:

Although this list of steps is fairly straightforward, it is not the whole story. There are still two ways to subvert the process: by quoting, and by using the advanced command eval.

You can think of quoting as a way of getting the shell to skip some of the 13 steps above. In particular:

Table 7-8 contains some simple examples that show how these work; they assume the statement dave=bob was run and user fred’s home directory is /home/fred.

If you are wondering whether to use single or double quotes in a particular shell programming situation, it is safest to use single quotes unless you specifically need parameter, command, or arithmetic substitution.

Using double quotes on variable values is increasingly important when dealing with the results of wildcard expansion. Today, it is not unusual to have files and directories available on Unix systems that actually physically exist on Microsoft Windows and Apple Macintosh systems. On those systems, spaces and other unusual characters, such as apostrophes and back-quotes, are common in filenames. Thus, to pass the full pathname into your application, be sure you quote things properly.

Task 7-5 is a more advanced example of command-line processing that should give you deeper insight into the overall process.

Recall from Chapter 4 that we found a simple way to set up the prompt string PS1 so that it always contains the current directory: PS1='($PWD)-> '.

One problem with this setup is that the resulting prompt strings can get very long. One way to shorten them is to substitute tilde notation for users’ home directories. This cannot be done with a simple string expression analogous to the above. The solution is somewhat complicated and takes advantage of the command-line processing rules.

The basic idea is to create a “wrapper” around the cd command, as we did in Chapter 5, that installs the current directory with tilde notation as the prompt string. We will see how to make this wrapper function shortly. The code we need to insert tilde notation is complicated in its own right; we develop it first.

We start with a function that, given a pathname as argument, prints its equivalent in tilde notation if possible. In order to write this function, we assume that we already have an associative array named tilde_ids, in which the subscripts are home directories and the values are user names. Thus, print ${tilde_ids[/home/arnold]} would print the value arnold. Here’s the function, named tildize:

function tildize {
    # subdir of our home directory
    if [[ $1 == $HOME* ]]; then
        print "~${1#$HOME}"
        return 0
    fi

    # loop over homedirs trying to match current dir
    typeset homedir
    for homedir in ${!tilde_ids[*]}; do
        if [[ $1 == ${homedir}?(/*) ]]; then
            print "~${tilde_ids[$homedir]}${1#$homedir}"
            return 0
        fi
    done
    print "$1"
    return 1
}

The first if clause checks if the given pathname is under the user’s home directory. If so, it substitutes tilde (~) for the home directory in the pathname and returns.

If not, we loop over all the subscripts in tilde_ids, comparing each one to our current directory. The test matches home directories by themselves or with some other directory appended (the ?(/*) part.) If a user’s home directory is found, ~ user is substituted for the full home directory in the given pathname, the result is printed, and the function exits.

Finally, if the for loop exhausts all users without finding a home directory that is a prefix of the given pathname, tildize simply echoes back its input.

Now, how do we create the tilde_ids array? We use the function init_tilde_db. It should be called once, from the .profile file when we log in. The tilde_ids array must be explicitly declared as an associative array using typeset -A:

# tilde_ids[] is global associative array
# mapping directories to user names
typeset -A tilde_ids

function init_tilde_db {
    typeset user homedir    # local vars
    awk -F: '{ print $1, $6 }' /etc/passwd |
        while read user homedir; do
            if [[ $homedir != / ]]; then
                tilde_ids[$homedir]=$user
            fi
        done
}

We use the awk utility to extract the first and sixth fields of the file /etc/passwd, which contain user IDs and home directories, respectively.^[93] In this case, awk acts like cut. The -F: is analogous to -d:, which we saw in Chapter 4, except that awk prints the values on each line separated by spaces, not colons (:).

awk’s output is fed into a while loop that checks the pathname given as argument to see if it contains some user’s home directory. (The conditional expression eliminates “users” like daemon and root, whose home directories are root and therefore are contained in every full pathname.)

Now that we have the tildize function, you might think we could use it in a command substitution expression like this:

PS1='$(tildize $PWD)> '

In fact, you’d be right.^[94] But there’s a hidden cost here. The function is run every time that the shell prints the prompt. Even if all you do is hit ENTER, the shell runs the tildize function. If there are lots of users on your system, the shell loops through all of the home directories, each time. To avoid this, we write a cd function that only updates the prompt when we actually change directories. The following code should go into your .profile or environment file, along with the definition of tilde_ids and tildize:

init_tilde_db  # set up array once, upon login

function cd {
    command cd "$@"  # run real cd
    typeset es=$?    # save exit status in a local var
    PS1="$(tildize $PWD)> "
    return $es
}

cd $PWD        # set prompt

As we saw in Chapter 5, writing a function with the same name as a built-in command looks pretty strange at first glance. But, following the POSIX standard, the Korn shell distinguishes between “special” built-in commands and regular built-in commands. When the shell looks for commands to execute, it finds functions before it finds regular built-in commands. cd is a regular built-in command, so this works. Within the function, we use the the cleverly named command command to actually get at the real cd command.^[95] The statement command cd "$@" passes the function’s arguments on to the real cd in order to change the directory. (As a side note, the shell defines an alias command='command ', which allows you to use command with aliases.)

When you log in, this code sets PS1 to the initial current directory (presumably your home directory). Then, whenever you enter a cd command, the function runs to change the directory and reset the prompt.

Of course, the function tildize can be any code that formats the directory string. See the exercises at the end of this chapter for a couple of suggestions.

print $"hello, world"  A well-known greeting among computer scientists

$ print $'A string with 'single quotes' and "double quotes" in it'     
A string with 'single quotes' and "double quotes" in it

We have seen that quoting lets you skip steps in command-line processing. Then there’s the eval command, which lets you go through the process again. Performing command-line processing twice may seem strange, but it’s actually very powerful: it lets you write scripts that create command strings on the fly and then pass them to the shell for execution. This means that you can give scripts “intelligence” to modify their own behavior as they are running.

The eval statement tells the shell to take eval’s arguments and run them through the command-line processing steps all over again. To help you understand the implications of eval, we’ll start with a trivial example and work our way up to a situation in which we’re constructing and running commands on the fly.

eval ls passes the string ls to the shell to execute; the shell prints a list of files in the current directory. Very simple; there is nothing about the string ls that needs to be sent through the command-processing steps twice. But consider this:

listpage="ls | more"
$listpage

Instead of producing a paginated file listing, the shell treats | and more as arguments to ls, and ls complains that no files of those names exist. Why? Because the pipe character “appears” in step 5 when the shell evaluates the variable, after it has actually looked for pipe characters (in step 2). The variable’s expansion isn’t even parsed until step 10. As a result, the shell treats | and more as arguments to ls, so that ls tries to find files called | and more in the current directory!

Now consider eval $listpage instead of just $listpage. When the shell gets to the last step, it runs the command eval with arguments ls, |, and more. This causes the shell to go back to Step 1 with a line that consists of these arguments. It finds | in Step 2 and splits the line into two commands, ls and more. Each command is processed in the normal (and in both cases trivial) way. The result is a paginated list of the files in your current directory.

Now you may start to see how powerful eval can be. It is an advanced feature that requires considerable programming cleverness to be used most effectively. It even has a bit of the flavor of artificial intelligence, in that it enables you to write programs that can “write” and execute other programs.^[96] You probably won’t use eval for everyday shell programming, but it’s worth taking the time to understand what it can do.

As a more interesting example, we’ll revisit Task 4-1, the very first task in the book. In it, we constructed a simple pipeline that sorts a file and prints out the first N lines, where N defaults to 10. The resulting pipeline was:

sort -nr $1 | head -${2:-10}

The first argument specifies the file to sort; $2 is the number of lines to print.

Now suppose we change the task just a bit so that the default is to print the entire file instead of 10 lines. This means that we don’t want to use head at all in the default case. We could do this in the following way:

if [[ -n $2 ]]; then
    sort -nr $1 | head -$2
else
    sort -nr $1
fi

In other words, we decide which pipeline to run according to whether or not $2 is null. But here is a more compact solution:

eval sort -nr $1 ${2:+"| head -$2"}

The last expression in this line evaluates to the string | head -$2 if $2 exists (is not null); if $2 is null, then the expression is null too. We backslash-escape dollar signs ($) before variable names to prevent unpredictable results if the variables’ values contain special characters like > or |. The backslash effectively puts off the variables’ evaluation until the eval command itself runs. So the entire line is either:

eval sort -nr $1 | head -$2

if $2 is given or:

eval sort -nr $1

if $2 is null. Once again, we can’t just run this command without eval because the pipe is “uncovered” after the shell tries to break the line up into commands. eval causes the shell to run the correct pipeline when $2 is given.

Next, we’ll revisit Task 7-3 from earlier in this chapter, the start function that lets you start a command in the background and save its standard output and standard error in a logfile. Recall that the one-line solution to this task had the restriction that the command could not contain output redirectors or pipes. Although the former doesn’t make sense when you think about it, you certainly would want the ability to start a pipeline in this way.

eval is the obvious way to solve this problem:

function start {
    eval "$@" > logfile 2>&1 &
}

The only restriction that this imposes on the user is that pipes and other such special characters must be quoted (surrounded by quotes or preceded by backslashes).

Task 7-6 is a way to apply eval in conjunction with various other interesting shell programming concepts.

make is known primarily as a programmer’s tool, but it seems as though someone finds a new use for it every day. Without going into too much extraneous detail, make keeps track of multiple files in a particular project, some of which depend on others (e.g., a document depends on its word processor input file(s)). It makes sure that when you change a file, all of the other files that depend on it are processed.

For example, assume you’re writing a book in DocBook XML. You have files for the book’s chapters called ch01.xml, ch02.xml, and so on. The generated PostScript output for these files are ch01.ps, ch02.ps, etc. The tool to convert DocBook XML into PostScript is called (for some strange reason) gmat. You run commands like gmat ch N .xml to do the processing. (gmat knows to create ch01.ps from ch01.xml; you don’t need to use shell redirection.) While you’re working on the book, you tend to make changes to several files at a time.

In this situation, you can use make to keep track of which files need to be reprocessed, so that all you need to do is type make, and it figures out what needs to be done. You don’t need to remember to reprocess the files that have changed.

How does make do this? Simple: it compares the modification times of the input and output files (called sources and targets in make terminology), and if the input file is newer, make reprocesses it.

You tell make which files to check by building a file called makefile that has constructs like this:

                  target : source1 source2 ...
                  commands to make target

This essentially says, “For target to be up to date, it must be newer than all of the sources. If it’s not, run the commands to bring it up to date.” The commands are on one or more lines that must start with TABs: e.g., to make ch07.ps:

ch07.ps : ch07.xml
        gmat ch07.xml

Now suppose that we write a shell function called makecmd that reads and executes a single construct of this form. Assume that the makefile is read from standard input. The function would look like the following code.

function makecmd {
    read target colon sources
    for src in $sources; do
        if [[ $src -nt $target ]]; then
            while read cmd && [[ $cmd == 	* ]]; do
                print "$cmd"
                eval $cmd
            done
            break
        fi
    done
}

This function reads the line with the target and sources; the variable colon is just a placeholder for the :. Then it checks each source to see if it’s newer than the target, using the -nt file attribute test operator that we saw in Chapter 5. If the source is newer, it reads, prints, and executes the commands until it finds a line that doesn’t start with a TAB or it reaches end-of-file. (The real make does more than this; see the exercises at the end of this chapter.) After running the commands, it breaks out of the for loop, so that it doesn’t run the commands more than once. (It isn’t necessary to strip the initial TAB from the command. The shell discards the leading whitespace automatically.)

ccom < filename.c | as | optimize > filename.o

cat filename.c | ccom | as | optimize > filename.o

cat $srcname | ccom | as | optimize > $objname

eval cat $srcname | ccom | as | optimize > $objname

eval cat $srcname " | ccom" " | as" " | optimize" > $objname

compile=" | ccom"
assemble=" | as"
optimize=" | optimize"

eval cat $srcname $compile $assemble $optimize > $objname

optimize=""
if -O given then
    optimize=" | optimize"
fi

eval cat $srcname | ccom | as > $objname

assemble="| as"
if $srcname ends in .s then
    compile=""
fi

eval cat $srcname | as > $objname

# initialize option-related variables
do_link=true
debug=""
link_libs=""
clib="-lc"
exefile=""

# initialize pipeline components
compile=" | ccom"
assemble=" | as"
optimize=""

# process command-line options
while getopts "cgl:[lib]o:[outfile]O files ..." opt; do
    case $opt in
        c )    do_link=false ;;
        g )    debug="-g" ;;
        l )    link_libs+=" -l $OPTARG" ;;
        o )    exefile="-o $OPTARG" ;;
        O )    optimize=" | optimize" ;;
    esac
done
shift $(($OPTIND - 1))

# process the input files
for filename in "$@"; do
    case $filename in
        *.c )
            objname=${filename%.c}.o ;;
        *.s )
            objname=${filename%.s}.o
            compile="" ;;
        *.o )
            objname=$filename  # just link it directly with the rest
            compile=""
            assemble="" ;;
        *   )
        print "error: $filename is not a source or object file."
        exit 1 ;;
    esac

    # run a pipeline for each input file
    eval cat $filename $compile $assemble $optimize > $objname
    objfiles+=" $objname"
    compile=" | ccom"
    assemble=" | as"
done

if [[ $do_link == true ]]; then
    ld $exefile $objfiles $link_libs $clib
fi