CHAPTER 7
Shell Language Portability
So far, this book has mostly discussed the portable subset of shell languages, with an occasional warning that a useful technique may not always be portable. This chapter discusses shell portability in much greater detail, starting with more discussion on what portability is and how bugs and features differ. The next sections discuss some of the most common additional features and extensions, with brief notes on where you might find them. This includes substitution rules, redirections, and even additional syntax structures found in some shells. There is also a discussion of which features may be omitted for a stripped-down shell.
Following the discussion of extensions is a list of common shell variants, including ways to identify them and ways to invoke them to get behavior closer to the POSIX standard. Following this is a discussion of ways in which a script can configure itself for more complete shell functionality, whether by defining additional features as shell functions or by searching for a more powerful shell to execute itself with.
More on Portability
A portable program is one that runs correctly on any standard system. But what about nonstandard systems? What about buggy systems?
There is no perfect answer. When writing portable code, give some thought to what you are likely to run into. The autoconf maintainers recommend avoiding any features introduced more recently than V7 UNIX (1977). Most modern systems have a shell at least a little newer, although some famously continue to ship with a "traditional" Bourne shell. This advice may sound very drastic, but it is almost certainly the right choice for autoconf scripts because the entire purpose of autoconf is to be able to figure out what is wrong on your system and work around it. In that case, having a script fail on an old system would be exceptionally frustrating. If you are writing the installer for a set of 3D video drivers for X11, by contrast, you can probably make few more assumptions about your target platform.
Err on the side of caution. The assumption that Linux systems would always use bash as the primary shell probably seemed reasonable once, but Debian and Ubuntu desktop systems have switched to dash. Furthermore, many Linux programs have been run in emulation on BSD systems, where their installers get run using the BSD /bin/sh, usually an ash derivative.
Think about your use cases. A script that is going to be run by end users probably needs to simply work out of the box on their systems. A script that is used by developers and is expected to be installed and ported to a new system can be a little more flexible; it's enough that it is easy to make it run. For something that is used internally, it may be fine to need a few minutes to migrate a script to a new box. Even then, attention to portability makes your life easier; a few minutes is a lot easier to manage than a few days.
In theory, you are usually best off avoiding extensions. However, extensions may not always avoid you; you may have to port existing code that relies on them, or you may find that an extension changes the behavior of a program that was written without it. Likewise, while it would be nice to simply avoid prestandard or broken shells, sometimes you have no choice.
Standardization
Standardization offers a useful way to think about the shell portability question, as it does for most other portability questions. If you are not quite sure what to do, targeting the POSIX shell is a good first step. This gives you a good baseline that nearly any system can meet with only a little work. This may not be enough for some programs; for instance, it is not enough for autoconf and may be a poor choice for something like an installer. Dependencies beyond the POSIX spec are almost always a poor choice. While ksh, bash, and pdksh are quite common, they are not universal. If you are finding that the additional features in these shells are particularly crucial to a script, it may be a warning sign that you have gotten into an area where another programming language may be a better choice. While the shell can certainly be used as a general-purpose scripting language, it is probably not as good of a general-purpose scripting language as Perl, Python, or Ruby. One of the shell's core strengths is its universality; if you start relying on specific features of bash version 3 or ksh93, you have lost that universality.
Most systems provide at least one shell that is reasonably close to a POSIX shell, but it is not always /bin/sh. In some cases, the best POSIX-compliant shell on a system may be an optional component, so some users won't install it. Depending on your target audience, it may be perfectly adequate to declare such a shell to be a system requirement, and tell people where to get it for their particular target system.
Standardization of a programming language describes two sets of requirements. One is implementation requirements—a shell must meet these requirements to be considered compliant with the standard. The other is requirements of programs in the language—a program must meet these requirements in order to run reliably on implementations. It is often helpful to view a standard as a contract; a vendor claiming POSIX compliance for a shell is promising to run your code correctly if you write your code correctly, allowing you to refer to the standard to determine whether something is a bug or not.
The POSIX standard distinguishes between behavior required for conformance, behavior permitted in a conforming shell, and optional extensions. Support for $() command substitution is required; a shell that lacks this is not conformant. When running commands in a pipeline, a shell may run any or all of them in subshells; the standard allows for one of the commands to be run in the parent shell (as the Korn shell does) but does not require it. A program that relies on a particular decision about which commands are run in subshells is not portable among POSIX shells, but any answer a shell gives to that question is compliant with the POSIX standard.
POSIX explicitly blesses some extensions, warning that they may exist and have special behavior; for instance, POSIX reserves the [[ token as having possible special meaning to the shell, even though it does not specify anything at all about the syntax or semantics of such a token. This allows ksh and bash to be considered compliant (at least on this issue) even though they provide an extra feature (see the section "Built-In Tests" later in this chapter).
Brace expansion (also described in this chapter in the section "Portability Issues: Welcome to the Club") actually violates the POSIXstandard; it causes at least some well-formed shell programs to behave contrary to the standard, but the standard doesn't make any allowances for this. Many extensions are arguably standard violations, and shells that provide them may allow you to disable them. However, in portable code, you have to be aware of these extensions and avoid tripping on them.
Bugs
Previously, I've mostly ignored the topic of bugs. Bugs are not the same thing as lack of a standard feature; they are special cases where a particular feature misbehaves. Bugs are usually more narrowly defined. A shell that lacks $() command substitution simply lacks it all the time; any test program will confirm its absence. A bug often manifests only under particular circumstances, making it much harder to figure out what went wrong. For instance, one early shell (long since patched) omitted the last trailing slash of any command-line argument containing two trailing slashes. The shell doesn't lack the ability to pass slashes in arguments, or even pairs of slashes, and it doesn't truncate characters otherwise; it's just a special case. Finding this out and identifying the problem could be a real pain.
Unfortunately for shell programmers, obscure bugs are plentiful. The worst are mostly in systems that have mostly left commercial use, but there are plenty left floating around. The good news is that you will rarely encounter them, but there are plenty of special cases. Most bugs are specific to a particular version of a shell; a bug might exist in zsh 4.x, but not in zsh 3.x. While features are usually added but not removed in newer shells, bugs can come and go. The documentation for autoconf has a particularly large list of shell bugs, including a few you can probably safely ignore:
www.gnu.org/software/autoconf/manual/html_node/Portable-Shell.html
Portability Issues: Welcome to the Club
These portability issues are not unique to shell programming; C programmers have been living with a similar problem for a very long time. The ANSI/ISO C89 standard, released in 1989 (and again in 1990, which is a long story), offered substantial improvements to the language; and code written in "C89" offers substantial improvements for developers, compared with code written for previous language versions. However, many vendors continued to ship compilers that did not implement this language at all for a long time. The net result is a complicated tangle of portability rules, habits people have developed, urban legends, and more. A huge number of the tests generally performed by configure scripts have been essentially guaranteed to produce particular answers on many modern systems; indeed, most of the systems where they wouldn't work as expected never got Y2K upgrades.
When the ANSI/ISO C99 standard came out, I decided that I had probably had about enough of worrying about portability to pre-ANSI compilers. While it is true that they still exist, and some vendors still ship them, there is not much point in trying to deal with them; instead, if I even find myself on such a system, I'll get gcc and move on. This is practical for C code because I already know I'm going to have to compile it on a new target system. It is not as practical for shell code because it imposes an additional step of shell development that might not otherwise apply.
Common Extensions and Omissions
This section introduces features common enough that you should be aware of them, even if you don't plan to use them. They matter anyway because they may change the behavior of programs that were otherwise valid. Furthermore, you may find them useful enough to justify imposing some requirements on shells your code will run on. Some, like the additional POSIX parameter expansion features, are found in nearly all modern shells. Others are found only in a few shells, such as ksh, zsh, or bash.
Other Kinds of Expansion and Substitution
The parameter substitution and globbing rules shown so far are a minimal subset widely available even on fairly old shells. However, there are a number of additional options you might run into. This section introduces brace expansion, additional forms of parameter substitution common to POSIX shells, arithmetic substitution, and some additional globbing features.
Brace Expansion
Brace expansion is a variety of expansion introduced by csh, and later adopted by ksh, bash, and zsh. While brace expansion is primarily used with file names, it is not a form of file globbing. In brace expansion, lists of alternatives in braces are expanded into multiple words. The brace expression {a,b} expands into two words, a and b. When a brace expression occurs in part of a word, the whole word is duplicated for each of the resulting words: a{b,c} expands to ab ac.
At first look, brace expansion looks a lot like a form of globbing, but there are several significant differences between brace expansion and globbing. The first is that brace expansion generates all specified names, whether or not any of them exist as files:
$ ls a{b,c}
ls: cannot access ab: No such file or directory
ls: cannot access ac: No such file or directory
Pathname globbing produces names sorted in order; by contrast, brace expansion always generates names in the order they were given:
$ echo X{h,e,l,l,o}
Xh Xe Xl Xl Xo
Brace expansions can be nested, as well. Nesting works from the inside out; inner groups are expanded to create more words, then those words are used for the next level of expansion. The following two examples are equivalent:
$ echo a{b{c,d},e{f,g}}
abc abd aef aeg
$ echo a{bc,bd,ef,eg}
abc abd aef aeg
Brace expansion occurs after parameter substitution, but before globbing. In zsh, if brace expansion results in glob patterns, the shell displays an error message instead of executing a command if any of the glob patterns do not match any files.
Brace expansion does not occur within double quotes. In shells that provide brace expansion, it occurs prior to other forms of expansion; thus you will not see brace expansion on the results of parameter substitution or globbing. Brace expansion only occurs if you explicitly include braces in your code, making it easy to avoid being affected by it unintentionally.
Brace expansion is available in ksh93, pdksh, bash, and zsh. In pdksh, bash, and zsh, brace expansion can be disabled. In zsh and pdksh, brace expansion is disabled by default in POSIX mode. In bash, it must be disabled separately by set +B. In practice, it is very rare for brace expansion to break a script that otherwise works, but it is possible. Brace expansion is often used to generate a list of file names, leading many users to assume that it is part of globbing. However, brace expansion can be used for many other purposes in a script. While this feature is not portable enough to rely on, it is powerful and expressive, and it is good to be aware of it.
Additional Parameter Expansion Features
A number of additional parameter expansion forms are provided by POSIX shells (see Table 7-1). These are additional variants similar in syntax to the ${param:-word} forms discussed in Chapter 4.
Table 7-1. A Few More Parameter Expansions
| Pattern | Description |
${#parameter} |
Length of parameter. (0 if null or unset.) |
${parameter#pattern} |
Substitute parameter, removing the shortest match of pattern from the beginning. |
${parameter##pattern} |
Substitute parameter, removing the longest match of pattern from the beginning. |
${parameter%pattern} |
Substitute parameter, removing the shortest match of pattern from the end. |
${parameter%%pattern} |
Substitute parameter, removing the longest match of pattern from the end. |
The ${#parameter} construct expands to the length of $parameter. It is sometimes used in expressions like test ${#parameter} -gt 0, which is another way of spelling test -n "${parameter}". The advantage of ${#parameter} is that it can be used without nearly as much worry about quoting; it expands into a plain integer value (sometimes zero), which never requires quoting.
The four pattern-based substitution rules all use a half-anchored pattern. Normally, a shell pattern must match all of the text it is being compared with; it is anchored at both ends of the text. In these rules, however, the pattern can match part of the text. The shell substitutes, not the matching text, but whatever text did not match. These are easiest to illustrate by example:
$ file=/home/seebs/shell/example.txt
$ echo "${file%/*}"
/home/seebs/shell
$ echo "${file%%/*}"
$ echo "${file#*/}"
home/seebs/shell/example.txt
$ echo "${file##*/}"
example.txt
The shortest-match forms are an exception to the general rule that pattern matching always matches the longest string it can. Instead, they look for the shortest possible substring that matches the given pattern. So, in the first example, the shortest substring (at the end of the whole file name) matching the pattern /* is /example.txt. The shell can't match any shorter pattern because that is the first slash (counting from the end); it doesn't match a longer pattern because it doesn't have to.
The longest match forms behave more like normal pattern matching but are only anchored on one side. Because the string starts with a /, followed by other characters, the whole string is matched by /* when looking for the longest match.
The rules that match at the beginning behave much the same way. The shortest match of */ just removes the leading /; remember that a * matches anything at all, including an empty string (this is why ${file#*} just substitutes $file). The longest match is the full path, leaving the file name.
The reason these rules remove the match, rather than leaving it, is that shell patterns are sometimes a little weak on complicated expressions. It is often useful to be able to obtain the name of a file with a file name suffix removed, but it is very hard to write a shell pattern to express "everything but a file name suffix." For instance, in a build script, you might have a list of source files and want to identify the corresponding object files. You can replace the suffix easily:
$ file=hello.c
$ echo "${file%.c}".o
hello.o
As with other parameter substitution rules, the word on the right-hand side is itself subject to parameter substitution. If you want to know what a pattern matched, you can use two of these rules to find out:
$ file=/home/seebs/shell/example.txt
$ echo "${file%/*}"
/home/seebs/shell
$ echo "${file#${file%/*}}"
/example.txt
In this example, the pattern used is the substitution of ${file%/*}, or /home/seebs/shell. Removing this from the beginning of $file shows /example.txt, which is the exact pattern matched by the /* in the first substitution. This does not always work if the inner substitution yields pattern characters:
$ file=[0-9]/hello
$ echo "${file%/*}"
[0-9]
$ echo "${file#${file%/*}}"
[0-9]/hello
The shell interprets [0-9] as a pattern, not as a literal string, when substituting it on the right-hand side. However, you can prevent this using quoting:
$ file=[0-9]/hello
$ echo "${file%/*}"
[0-9]
$ echo "${file#"${file%/*}"}"
/hello
This latter behavior is documented in the POSIX standard, and pdksh, ksh93, and bash all do it; however, at least some ash derivatives ignore the quotes (for more information on quoting inside parameter substitution, see the detailed discussion in Chapter 6).
If you want the effect of this sort of substitution in a pre-POSIX shell, you can usually duplicate it using sed or expr. Remember that these utilities use regular expressions, not shell patterns. To strip the directory name off a file, you would use the regular expression .*/, not just */:
file=`expr "$file" : '.*/(.*)$'`
This feature is a significant performance advantage in general, although less so in shells that implement expr as a builtin. (If you are wondering why you shouldn't just use basename and dirname, the answer is that they are not as universally available as expr; see the discussion of utility portability in Chapter 8.)
Arithmetic Substitution
The POSIX spec provides for arithmetic substitution, using $((expression)). The shell evaluates expression and substitutes its result. This is substantially similar to $(expr expression), but there are three key differences. The first is that the syntax is different; arithmetic substitution does not need special shell syntax characters in expression quoted, but expr does. Furthermore, expr requires each operator to be a separate argument; arithmetic substitution can parse expressions where operators have not been separated out by spaces. The following examples are equivalent:
$ echo $(expr ( 3 + 1 ) * 5 )
20
$ echo $(( (3+1)*5 ))
20
The parentheses and asterisk need to be quoted for expr but do not in an arithmetic substitution; similarly, arithmetic substitution doesn't need spaces around the operators. If (3+1)*5 were passed to expr as an argument, it would be interpreted as a string, and no arithmetic would be performed.
The second major difference is that the available operators vary widely. The length and : (regular expression match) operators of expr (see Chapter 2 for a discussion of regular expressions and expr; the length operator just yields the length of its argument) are not available; instead, all operations work on numeric values. However, arithmetic substitution provides many additional operators, such as bitwise operators. In expr, & and | are logical operators (similar to the && and || operators in the shell). In arithmetic substitution, they are bitwise operators:
$ echo $(( 1 | 2 ))
3
$ echo $(expr 1 | 2)
1
In expr, the first operand of | is evaluated, and if it is neither zero nor an empty string, its value is the result of the expression; otherwise, the second operand is evaluated and is the result. So, 1 | 2 evaluates 1, finds out that it is not zero or an empty string, and results in that value. In arithmetic substitution, 1 | 2 is the bitwise union of the two numbers, so the result is 3.
Finally, the third difference is that the arithmetic substitution itself can (in most shells) assign values to shell variables. While idiomatically it is often clearer to write x=$((x+1)), you can also write $((x=x+1)). This feature is not available in some implementations of ash. Note that the final result is still substituted, so $((x=1)) expands to 1; if it is on a line by itself, the shell tries to find a program named 1 to run.
Arithmetic expressions have a number of shorthands. A variable name is implicitly expanded when encountered without a leading $. Thus $((x)) is the same as $(($x)). Shell arithmetic may be limited to a signed 32-bit value. However, many shells provide additional functionality; some shells support floating point operations in arithmetic substitution, or provide C-style increment (++) and decrement (--) operators.
In portable scripts (assuming you're willing to rely on arithmetic substitution at all), you can count on basic C-style integer math, including bitwise operations. Don't rely on assignment to variables, the increment or decrement operators, or floating-point math.
Even if you don't plan to use arithmetic substitution, you have to be aware of it if you are using $()-style command substitution and subshells. In shells that support arithmetic substitution, $(( and )) are tokens. To be on the safe side (and to avoid possible bugs), separate parentheses when nesting them. This comes back to the maximal munch rule described in Chapter 4; when $(( is a token, then $((foo)|bar) is a syntax error because there is no corresponding )) token.
Globbing Extensions
The ksh, zsh, and bash shells offer additional globbing options. These are not particularly portable to other shells. There are two major sets to consider. One is the pattern grouping operators, introduced originally in ksh. These are available in ksh and pdksh; they are available in bash if the extglob shell option has been set and in zsh if the KSH_GLOB option has been set. These operators all work on grouped sets of patterns, called pattern lists. A pattern list consists of one or more patterns; if there are multiple patterns, they are separated by pipes (|) as they would be in a case statement. A special character followed by a pattern list in parentheses becomes a special glob operator. There are five such operators, as shown in Table 7-2.
Table 7-2. Extra KSH GLOB Operators
| Pattern | Description |
@(pattern-list) |
Exactly one of the patterns in pattern-list. |
?(pattern-list) |
Zero or one of the patterns in pattern-list. |
*(pattern-list) |
Zero or more of the patterns in pattern-list. |
+(pattern-list) |
One or more of the patterns in pattern-list. |
!(pattern-list) |
None of the patterns in pattern-list. |
There are a number of additional variants possible in ksh93, but this subset is available in bash, pdksh, and zsh as well.
The @, ?, *, and + variants are reasonably intuitive if you have worked with regular expressions. The @ pattern operator functions a bit like a character class, only matching larger patterns. The pattern *.@(tar.gz|tar.bz2|tgz|tbz2) matches any file name ending in one of the four suggested suffixes, which are common names for compressed tar archives. Note that while the @ operator itself matches only one of the provided patterns, this pattern quite happily matches x.tgz.tbz2.tar.gz; the @ operator matches the trailing suffix, the period matches the period before that suffix, and the whole rest of the pattern matches the initial *. The @ operator is similar to a regular expression using {1,1}; it is used only to introduce the pattern list. The ?, *, and + operators perform the same function as their equivalents in an extended regular expression (although they come before the pattern list, rather than following it).
The ! operator can be a lot more confusing to use. The pattern *!(foo)* can still match a file name containing the word foo because the foo can match one of the asterisks. In fact, even the pattern !(foo) can match a file name containing the word foo, as long as the file name contains something else as well. To match any file name without the word foo in it, use !(*foo*). Getting used to the way in which the generally greedy behavior of pattern expressions mingles with a negation operator can take time. Similarly, the expression !(foo)?(bar) can match a file named foobar; the initial !(foo) matches the string foobar, and the ?(bar) matches zero repetitions of the pattern bar.
Another globbing feature is recursive expansion, available primarily in zsh. The Z shell recognizes a special pattern rule of (word/)# as matching zero or more subdirectories matching word. As a particular shorthand, **/ is equivalent to (*/)# and can match both anything in the current directory and anything in any subdirectory of the current directory (or their subdirectories; it's recursive). With the -G option, ksh93 also recognizes **/, and also recognizes ** as a shorthand for every file or directory in the current directory or its subdirectories. This feature is moderately dangerous; it can easily do surprising or unwanted things, and it is not especially portable. If you want to find files in subdirectories, look into the find command.
Alias Substitution
Aliases allow a command name to be replaced with some other command name or code. Unlike other kinds of substitution, aliases can result in shell control structures. (However, it is not remotely portable to try to alias shell reserved words.) An alias typically takes the following form:
alias text=replacement
In general, aliases should be valid command names. Aliases are never substituted when quoted. This can offer a workaround for concerns about aliases interfering with the behavior of a script, as the name "ls" (including the quotes) is not subject to alias substitution. However, it is subject to confusing anyone trying to read your code.
The behavior of aliases varies noticeably between shells, and not all shells provide this feature. I do not recommend relying on this except in cases where you are quite sure what shell will execute a given piece of code. Some shells allow arbitrarily complicated alias expressions, whereas others can alias only simple command names.
The real problem with aliases in scripts, however, is not the portability problem; it is the maintainability problem. Just as C code that relies heavily on preprocessor behavior can be extremely difficult to understand or debug, shell code that uses aliases often becomes unmaintainable quickly. The primary use of aliases in historical code has been developing shorthands or abbreviations for common tasks. Use shell functions instead.
Syntax Extensions
A few shells offer additional syntactic features that do not fit well in other categories. This section reviews three of particular and common interest: additional forms of redirection, support for arrays, and the [[expr]] syntax for built-in test functionality, similar to the test program.
Redirections
There are a handful of additional redirection syntax features available in some shells. Both ksh and bash offer a rich selection of additional redirections. There are a few features unique to ksh93 and others found also in pdksh.
The >|file redirection operator opens and truncates file. This is the normal behavior for > redirection. However, in ksh and bash, the shell option noclobber prevents redirecting to existing files; this redirection overrides that option.
The bash-only <<<word operator operates a little like a here document, but instead of sending following lines of the shell script to a command as standard input, it expands word and uses that as the command's standard input. So, cat <<<$foo is similar to echo $foo. This is useful for commands that need only a small amount of input directed to them.
Another bash extension is the >&N- redirection operator (and the corresponding <&N-). These operators move one file descriptor to another, closing the original. The shell command exec 3>&2-is equivalent to running first exec 3>&2, then exec 2>&-.
Both bash and ksh support the <> operator, which opens a file for both reading and writing. If no descriptor number is provided, <> opens a file as standard input.
The final three operators are found only in ksh. The first is the coprocess operator (|&), which opens a special process in the background. This is only sort of a redirection; it is in some ways more like a command separator, and it occupies the same basic function as a pipe. When a program is run with |& as a redirection, the program is run in the background, with input and output attached to a special file descriptor maintained by the shell. You can write data to the coprocess by running commands >&p and read by running commands <&p. The coprocess continues until it exits, but you can close its input stream (which will typically cause most filter programs to exit) by redirecting its input stream to another file descriptor, and then closing it, as in the following code:
exec 3>&p
exec 3>&-
The first line duplicates the coprocess's input to file descriptor 3, and the second closes it. (Remember that the shell's output to the coprocess is the coprocess's input.)
The coprocess feature is moderately difficult to duplicate in any other shell; it may not be practical to rework a program that depends on it, so avoid depending on this feature in portable scripts.
Finally, ksh93 provides two additional redirection forms that are available only on systems that provide the /dev/fd directory containing special files that represent the standard file descriptors (so /dev/fd/2 is standard error). On these systems, you can specify input to, or output from, a command as though it were a file argument using the syntax <(list) to refer to the output of list, and >(list) to refer to its input. The shell runs the command in the background, with its output connected to a particular file descriptor; then the shell provides the name of that file descriptor's special file as an argument to a command. This is only useful with programs that expect their arguments to be file names:
$ echo <(ls)
/dev/fd/4
All of these redirection options are a bit specialized, and I do not recommend relying on any of them in portable scripts. Still, forewarned is forearmed. I have often wished that the coprocess feature had made it into the POSIX shell; it is one of the most persuasive arguments for writing a script that requires ksh.
Arrays
One of the most significant weaknesses of the shell as a general programming language is the lack of arrays. Arrays are available in ksh, zsh, and bash. All shells that support arrays support integer array subscripts. Additionally, zsh and ksh93 support associative arrays, which use text keys rather than integer values. Array subscripts (for regular arrays) are treated as arithmetic expressions, according to the rules described previously for arithmetic substitution. The number of elements in an array may be limited; pdksh restricts array subscripts to the range 0–1023.
In general, arrays are created either using the set command or by direct assignment. The expression x=(a b c) creates x as an array variable holding three values: ${x[0]} is a, ${x[1]} is b, and ${x[2]} is c. (In zsh, the indexes start at 1 unless the KSH_ARRAYS option has been set.)
In ksh93 and zsh, associative arrays are declared using the typeset -A command:
typeset -A foo
foo[a]=hello,
foo[b]=world
echo ${foo[a]} ${foo[b]}hello, world
The associative array feature is probably familiar to programmers who have worked in just about any modern language; it first became commonly known to UNIX users through the awk utility.
Arrays are not portable to most shells. In practice, portable shell scripts must use the positional parameters as an array or engage in elaborate constructions using eval to create variable names dynamically (see the examples in Chapter 5). If you only need one array, it is fairly practical to use the positional parameters as that array (although, if you need more than nine items, you will have to get clever in traditional shells). If you need more than one, you can store long strings using delimiter characters (typically colons or spaces), then use the set command to extract them into an array. If your delimiter character is spaces, this is easy; the following code extracts the members of an array into the positional parameters:
set -- $array
Similarly, you can store the positional parameters into a variable using $*:
array=$*
If you need to use a different delimiter, you have to set (and restore) $IFS:
save_IFS=$IFS
IFS=:
set -- $array
IFS=$save_IFS
When substituted, $* delimits the positional parameters with the first character of $IFS, providing symmetry. Remember that there is no field splitting in assignment; you do not need to quote $* to assign from it.
Another array-handling option is to switch to m4sh, which gives you some limited "compile-time" array functionality; you can use m4 arrays to develop scripts that act somewhat as though the shell had arrays. Finally, depending on the data you need to work with, you may be able to use a temporary file and a tool like awk or sed to extract and modify values. This is pretty high overhead, though; I prefer to just use eval.
Built-In Tests
The ksh, bash, and zsh shells support a more flexible conditional test expression using double brackets. This is somewhat different syntactically from the [ synonym for the test command, much as arithmetic substitution differs from expr. The syntax is [[expression]], and the rules for evaluating expression are somewhat simpler than the rules for the test utility. As with arithmetic substitution, expression is not subject to some of the normal shell features, such as field splitting. The exact set of tests supported varies somewhat from one shell to another, but in general this form handles most of the same expressions as the test program. As an additional feature, the shell recognizes && and || operators in these expressions (although some shells do not recognize -a and -o in these expressions).
Of particular note is that all three shells recognize file names of the form /dev/fd/N as referring to file descriptor N while processing these expressions. Thus even if the special /dev/fd files do not exist on a particular system, [[ -t /dev/fd/0 ]] succeeds if standard input is considered to be a terminal device.
Unlike the regular test program (whether it is implemented as a builtin or not), the [[expression]] form does not recognize operators that were quoted, and operators are never optional. This eliminates the two common problems that require special treatment of values in conditional expressions; there is never any ambiguity over what is, or is not, an operator, so tricks like prefixing values with X are unneeded.
The [[expression]] syntax is not described by POSIX, although POSIX does reserve the [[ and ]] tokens as potentially having special meaning to the shell.
The select Loop
One of the really interesting features introduced in ksh is the select loop, which allows the user to pick an item from a list in an unambiguous manner:
echo "Where would you like to go for your vacation?"
select answer in Oz Detroit
do
echo $answer
break
done
The output of this script in ksh looks like this:
$ ksh vacation
Where would you like to go for your vacation?
1) Oz
2) Detroit
#? 1
Oz
$
This can be used to select items from lists. As with a for loop, parameter substitution or command substitution on the list is subject to field splitting, so you can build a list and then let the user pick a word from it. This feature is available in ksh, bash, and zsh; it is not present in ash, however, and is not in the POSIX standard.
The select control structure loops until the loop is explicitly terminated. Because select is implemented as a control structure, a script using it is always a syntax error in another shell. However, you can come surprisingly close; a detailed discussion of this is included at the end of this chapter in the section "Emulating Features."
Common Omissions
It is not always obvious what ought to count as a standard feature that has been omitted and what ought to count as an extension that has not been provided. The pattern-matching parameter substitution forms previously listed are defined by POSIX but are a little more esoteric than more basic features, such as the use of ! to negate the exit status of a command.
Stripped-down shells usually start by omitting interactive features (such as command history, expansion of parameters in prompts, and so on). Some shells omitted shell functions in the distant past, but no one's seen a shell without shell functions in years.
Another common way to strip a shell down is to omit built-in commands. As long as the commands also exist as separate programs, this may hurt performance slightly but has no other impact. However, some shells omit builtins that cannot be run as external commands, such as getopts. (In fact, every modern POSIX-like system seems to have this, with the obvious caveat that the default shell on Solaris is still pre-POSIX.)
In general, the biggest impact of a stripped-down system will be in utility programs, rather than in the core shell language itself. For instance, many embedded systems lack sed, awk, or the printf utility. Utility portability issues are discussed at greater length in Chapter 8.
Common Shells and Their Features
This section introduces some of the most common shells you are likely to encounter, giving a brief overview for each of where it fits in the shell family tree, what sorts of features it has or lacks, and how to invoke it for maximal POSIX compliance if you need that. These shells are introduced in alphabetical order by name; for example, dash is under the section "Debian Almquist Shell." Shells specific to a given system are prefixed with the system's name, as in the "Solaris /usr/xpg4/bin/sh" section.
You can sometimes guess which shell you are in by checking the value of $SHELL, but this is useless in determining which shell has been used as /bin/sh.
Almquist Shell
The Almquist shell (ash) ) was developed by Kenneth Almquist as a compatible replacement for the Bourne shell shipped with SVR4 UNIX, plus POSIX features. Modern variants are POSIX-compliant by default. You can also find ash on many other systems; a variant of it is included in busybox, and it is also used in Cygwin and Minix. This is also the ancestor of the Debian ash (called dash), described in the "Debian Almquist Shell" section later in this chapter. The big strength of ash is that it is small, reasonably efficient, and fast. Some versions of ash are a little light on features like command-line editing, variable expansion in prompts, and other interactive features, but it is fine for scripting.
How to Identify
There is no simple way to figure out that you are running under ash. There is no standard pre-defined magic variable provided by the shell. Because ash is often used as /bin/sh, you can't check the shell's name, either. Luckily, there are relatively few version-specific quirks. The closest way I have found to identify ash is to check for everything else. If a shell is not a variety of ksh, bash, or zsh and does not seem to be a pre-POSIX shell, it may very well be ash.
Version Information
There is no formal numbering of ash versions. The initial release was in 1989, and since then ash has been in continuous development on the various BSD systems. Particular versions have been extracted from NetBSD (most often) and imported into Linux or other systems, but there are not usually version strings to identify them.
Major Compatibility Notes
There are two major bugs in early versions of ash that could affect portability, both involving command substitution. Probably the most significant is that in older versions of ash, command substitution of a single built-in command does not spawn a subshell, so the built-in command can modify the parent shell's environment. The other is that command substitution inside variable expansion did not work in one of the early versions migrated to Linux systems; ${FOO=`echo hello`} did not work as expected.
Getting POSIX Behavior
Conveniently simple, ash is by design a fairly closely matched POSIX shell. Very early versions were missing a few features, but the versions being distributed today are unlikely to hold many surprises.
Bourne-Again Shell
The GNU Bourne-again shell is probably one of the largest and most feature-filled variants. It has been in development since 1987. Unlike most of the other shells described as Bourne shell derivatives, bash incorporates a couple of features from csh. There are a lot of similarities between the extensions in bash and the extensions in ksh.
How to Identify
Check the environment variable $BASH_VERSION. This variable is set even when the shell is running in POSIX mode and contains the version number of the current shell.
Version Information
Early versions of bash (1.x) had a number of surprising behaviors that are mostly gone now. The 2.x and later versions use a new syntax for the output of set; older versions of bash, and other shells, may not be able to read this output. Finally, the 3.x versions introduced the support needed for the bash debugger; this is not available for older versions.
Major Compatibility Notes
Early versions of bash provided !-style history expansion, as used in csh. This only affects interactive use but is a major surprise in that it is one of the only cases where something inside single quotes can be expanded. In modern versions, this feature must be explicitly enabled. Also, bash introduces source as a synonym for the . command. In most cases, the compatibility problem is not that bash cannot run scripts written for other shells, but that other shells cannot run scripts written for bash. The bash shell provides a broad variety of builtins, often with extensions and added features, and the same caveats apply to these. The general caveats of modern POSIX-like shells, such as arithmetic substitution, apply to bash as well.
Getting POSIX Behavior
To force POSIX behavior, invoke bash with the --posix option or run set -o posix in the shell. The environment variable $POSIXLY_CORRECT also forces this behavior when set; setting it during the operation of a script takes effect immediately. Finally, if the bash program has the name sh, it goes into POSIX mode once it has read its startup files. You must also separately disable brace expansion (set +B) if you want better conformance; the feature is left on because it is very rare for a script that does not intend to use it to get affected accidentally.
Debian Almquist Shell
The Debian branch of the Almquist shell is an import of ash to use as a standard system shell. It was adopted because it is smaller and faster than bash and also with an eye to reducing the tendency for Linux scripts to be unportable to other UNIX-like systems. It was ported to Linux in 1997 by Herbert Xu and renamed to dash in 2002. It first showed up as /bin/sh on desktop Ubuntu around version 6.10, and is expected to be /bin/sh in Debian Lenny (frozen, but not shipped, as of this writing).
How to Identify
As with other ash variants, there is no obvious way to tell that you are running in dash. For instance, on a modern desktop Ubuntu system, /bin/sh is a symlink to /etc/alternatives/sh, which is a symlink to the selected shell, usually /bin/dash by default.
Version Information
On a desktop system, the package management system will usually have a version number available:
$ dpkg -l | grep dash
ii dash 0.5.4-8ubuntu1 POSIX-compliant shell
The exact way to extract this information varies from one system to another. However, the version number here does not necessarily correlate to a particular version of the ash shell. In general, a system providing dash provides a version modern enough to ignore the historical early quirks and lets you just write for the POSIX shell spec.
Major Compatibility Notes
There are no major surprises with dash, but be aware that many scripts on Linux systems may behave surprisingly in a non-bash shell. As a result, you may find that a system administrator has changed the default shell back to bash, so you have to watch out for bash extensions even if you think the shell should be dash.
Getting POSIX Behavior
As with ash, POSIX behavior is the default.
Korn Shell
The Korn shell was developed at Bell Labs by David Korn. It has been in use internally in various forms since 1982. An early version from 1986 has been distributed some, but widespread external use started with the 1988 releases. A variant of ksh was around in SVR4, and many System V–derived commercial UNIX systems have provided it. The current versions are available under an open source license, but earlier versions were not.
How to Identify
There is no simple way to identify whether you are in ksh, let alone what version. In ksh93, the special shell parameter ${.sh.version} contains the shell's version string; in ksh93t (June 2008) and later, this can also be accessed as $KSH_VERSION. Some systems provide a utility called what for identifying the versions of commands:
$ what $(which ksh93)
/usr/pkg/bin/ksh93
[. . .]
$Id: Version M 1993-12-28 q $
At a prompt, if you set the shell for its emacs-style command-line editing mode (set -o emacs), typing Ctrl-V displays the version information.
If you are willing to do some extra work, you can detect ksh by testing for the select control structure and then excluding other shells that offer simpler tests. The following script determines whether a shell has the select primitive and runs the last command of a pipeline in the parent shell:
eval "echo 1 | select no_select in false; do break; done" > /dev/null 2>&1
if $no_select; then
echo "no select"
else
echo "select"
fi
This script incorrectly indicates that there is no select control structure in pdksh or bash (because they run the select in a subshell). However, you can check for them using $KSH_VERSION and $BASH_VERSION, respectively. This script detects the select structure in zsh, ksh88, and ksh93; if this test determines that select is available, you can check $ZSH_VERSION to determine whether or not you are in zsh, and if you are not, you must be in some variety of ksh.
You cannot simply check ${.sh.version} because the invalid (for any other shell) parameter name causes a script to abort. Even eval is not enough to protect you from this in some shells, but a subshell can come to the rescue:
if ( test -n "${.sh.version}" ) 2>/dev/null ; then
echo "ksh93"
else
echo "not ksh93"
fi
You have to use a subshell here; ksh88, ash, and traditional sh all abort when trying to expand that variable.
Version Information
There are three major revisions of ksh: ksh86, ksh88, and ksh93. You are unlikely to encounter ksh86 in the wild. The ksh88 version is still used in some commercial UNIX systems as an implementation of the POSIX shell. There are a number of new features in ksh93, such as associative arrays and floating-point arithmetic, as well as a variable namespace feature using parameter names containing periods. Brace expansion is found in ksh93, but not in ksh88.
Major Compatibility Notes
The only compatibility problems you are likely to encounter with ksh are with scripts that happen to match some of the ksh syntax extensions. Brace and tilde expansion are both performed by ksh93; ksh88 performs tilde expansion, but not brace expansion.
Unlike most other shells (including pdksh), ksh runs the last command of a pipeline that runs in the parent shell; in ksh, echo hello | read greeting sets $greeting in the shell. This rarely breaks programs that were not expecting it, but it can be a source of portability problems if you rely on it.
Getting POSIX Behavior
There is no switch to make ksh behave more like a POSIX shell than it usually does. However, its features are mostly extensions, and all of the modern POSIX features are available by default. A POSIX script will, with rare exceptions, execute without difficulty in ksh.
Public Domain Korn Shell
The public domain Korn shell is a clone of ksh. It was written because ksh was not open source, and many systems lacked a POSIX shell. Since then, ksh has become open source, POSIX shells have become much more common, and bash has become much better for scripting. However, pdksh is still found on a number of systems. There are a few features in pdksh not found in ksh88 or ksh93, and pdksh has acquired some of the new ksh93 features.
How to Identify
The special shell parameter $KSH_VERSION contains the version information of the shell. Most versions of ksh do not, but ksh93t (June 2008) adds $KSH_VERSION as a synonym for ${.sh.version}.
Every installation of pdksh I've seen over the last fifteen years has been version 5.2.14. The mainline ksh shell became more widely available, and pdksh hasn't been substantially upgraded since 2001. While it has some quirks, pdksh is stable.
Major Compatibility Notes
Unlike ksh, pdksh runs the last command of a pipeline in a subshell. There are other subtle differences between ksh and pdksh, described in the pdksh documentation, but most scripts written for ksh88 will run in pdksh.
Getting POSIX Behavior
Like bash, pdksh supports a POSIX mode in which it deviates from ksh behavior in favor of the POSIX specification; this can be controlled through set -o posix or the $POSIXLY_CORRECT environment variable.
Solaris /usr/xpg4/bin/sh
The /usr/xpg4/bin/sh program, when it has been installed, is a ksh88 shell modified a little to be a bit more like a POSIX shell. The name comes from the X/Open Portability Guide, Issue 4 (X/Open Company, 1992), which is one of the precursors to modern UNIX standards.
How to Identify
As with other ksh88 shells, there is no way to identify this shell from within a script (but see the previous "Korn Shell" section for some workarounds).
Version Information
This is a late, fairly well bug-fixed ksh88. It does not come in multiple versions.
Major Compatibility Notes
The only compatibility problems you are likely to encounter with this ksh variant are with scripts that happen to match some of the ksh syntax extensions. Being based on ksh88, this shell does not have brace expansion. Unlike most other shells (including pdksh), ksh runs the last command of a pipeline that runs in the parent shell; in ksh, echo hello | read greeting sets $greeting in the shell. This rarely breaks programs that were not expecting it, but it can be a source of portability problems if you rely on it.
Getting POSIX Behavior
This shell is already configured to offer POSIX shell functionality. It has no configuration choices to change this.
SVR2 Bourne Shell
The SVR2 Bourne shell, or derivatives of it, are the oldest shells I know of that are still in use today. Specifically, the Tru64 UNIX shell is an SVR2 derivative. (The documentation claims it is an SVR3.2 shell, but it has characteristic behaviors of older shells.)
How to Identify
The SVR2 Bourne shell is the only shell I am aware of in which the historical behavior survives of expanding "$@" to an empty quoted string when there are no parameters. It also lacks the getopts builtin. The following code identifies a shell with the old "$@" behavior:
( set dummy; shift; set dummy "$@"; shift; echo $# )
In an SVR3 or later shell, this should consistently print 0; in the SVR2 shell, it prints 1.
Version Information
There were a couple of variants of this, but most are now gone. The 8th Edition UNIX shell was a derivative, but it added modern "$@".
Major Compatibility Notes
In theory, the SVR2 Bourne shell wipes out the positional parameters when calling a shell function. However, the only known living version of this shell includes the SVR3 fix for this bug, and the positional parameters are restored after calling a shell function. This is the only shell lacking getopts or modern "$@" semantics.
This shell lacks the ! modifier to invert the status of a command, and it recognizes ^ as a synonym for | as a command separator. See also "Major Compatibility Notes" under the "SVR4 Bourne Shell" section; this shell has all of the quirks of the later shell.
Getting POSIX Behavior
You can't, but you can look for another shell on the system. A Korn shell is available on most variants. If you need to target this system, you may want to use an execution preamble to switch to that.
SVR4 Bourne Shell
The SVR4 Bourne shell program is extraordinarily stable, offering essentially a stable feature set since 1988, with occasional bug fixes or small updates in some systems. It is not a POSIX shell. For most modern users, this is the only non-POSIX shell you will find in regular use. This is the shell used by Solaris and Irix systems as /bin/sh.
How to Identify
While the SVR4 shell has no direct identifying cues as to its version, you can detect that you are probably running in it by running eval "! false" and checking $?. Most other shells will consider this to succeed, yielding 0; the SVR4 shell reports failure because there is no command named !.
Version Information
The SVR4 shell has only minor bug fixes and enhancements between the original SVR4 releases and the current version. The shell's version is determined by the system version; use uname to find that.
Major Compatibility Notes
This shell is included in Solaris and Irix, even today, and that is the reason to worry about the portability of POSIX-specified features. While there are other systems with pre-POSIX shells installed, these are by far the most common. Many systems seem to have migrated to the POSIX shell sometime in the last ten years or so, but these vendors have stayed with the old one for compatibility with older scripts, some of which might have their semantics changed by an update.
The SVR4 shell lacks the ! modifier used to reverse the return status of a command. It cannot access positional variables past $9; ${10} is an invalid construct in it. It supports backtick command substitution, but not $() command substitution. In the SVR4 shell, ^ is equivalent to a pipe on the command line; it must be quoted to be used in normal words or arguments to other commands.
The SVR4 shell provides getopts, unset, and modern "$@" behavior. (In fact, these were all introduced in SVR3 or so.)
A particular surprise is that, while set -- args sets the positional parameters, set -- does not clear them.
Getting POSIX Behavior
You can't; if you need POSIX behavior, you have to use another shell. Luckily, Solaris ships with several shells. Some of them are optional, but zsh appears to be installed by default on every remotely recent system and can be used as a POSIX shell. See the following section, "Execution Preambles," for information about getting into a more modern shell.
Traditional Shell
The V7 shell (the shell of 7th Edition UNIX) is generally regarded as the starting point of the modern Bourne shell. It can be identified as much by what it lacks as by what it provides. In practice, every shell has since evolved, but it is worth considering this shell simply for contrast. Table 7-3 gives a brief overview of major features that were not found in the V7 shell and when they were added.
Table 7-3. Shell Features and Their Arrival
| Feature | First Available | Notes |
| Functions | SVR2 | Shell functions did not support local positional parameters at first. |
unset |
SVR2 | Not always found on small or specialized shells. |
| Function arguments | SVR3 | Positional parameters can be used safely after a function call. |
getopts |
SVR3 | Replaces the getopt utility. |
| 8-bit support | SVR3 | Previous shells used 8th bit for quoting information. |
| Symbolic signal names | SVR4 | Previous shells allowed only numeric signal numbers. |
Z Shell
The Z shell is an interesting offshoot or variant; it has been around for a long time, but by default is noticeably incompatible with the Bourne shell derivatives. However, it is also extremely configurable. Just as bash can emulate a POSIX shell, zsh can do a pretty good job of emulating ksh88 or a POSIX shell. This is important for portable code because zsh may be the closest thing to a POSIX shell available on some systems. The Z shell has been in development since 1990.
How to Identify
The special shell parameter $ZSH_VERSION indicates the version of zsh being run.
Version Information
You will rarely see versions prior to the 3.x version series in the wild. 4.x is more common now, and 4.2 is considered stable as of this writing.
Major Compatibility Notes
The most surprising change for users is that variable expansions are not subject to field splitting in zsh. The Z shell documentation describes this as a bug in other shells. (They are not alone in this view; Plan 9's rc shell went the same way.) You can override this behavior by setting the shell compatibility option or explicitly with setopt shwordsplit. There is an important exception: "$@" works as expected.
However, when emulating plain sh, zsh performs too much word splitting on the common idiom ${1+"$@"}. You can work around this using zsh's fairly powerful aliasing feature:
alias -g '${1+"$@"}'='"$@"'
It may be simpler to use "$@" without qualification; it works even on nearly all traditional shells still in use today.
Getting POSIX Behavior
In modern zsh, you can issue the command emulate sh or emulate ksh to set the shell into an emulation mode, primarily useful for running scripts. If zsh is invoked with the name sh or ksh, it automatically uses the corresponding emulation mode. (There is also a csh emulation mode, but it is of no use for POSIX shell scripting.)
Execution Preambles
Portable shell scripts face the common problem that sometimes a crucial feature is not available in a given shell. In some cases, the feature is important enough to justify going to some lengths to obtain a more standard shell environment. Sometimes, the goal is just to have predictable behavior. The configure scripts generated by autoconf use a great deal of startup code to ensure predictable behavior across a broad range of platforms. The following sample illustrates code to do this for a few shells:
# Be Bourne compatible
if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then
emulate sh
NULLCMD=:
#Zsh 3.x and 4.x performs word splitting on ${1+"$@"}, which
#is contrary to our usage. Disable this feature.
alias -g '${1+"$@"}'='"$@"'
elif test -n "${BASH_VERSION+set}" && (set -o posix) >/dev/null 2>&1; then
set -o posix
fi
DUALCASE=1; export DUALCASE #for MKS sh
This preamble causes three common shells (zsh, bash, and the MKS Toolkit sh used on some Windows systems) to behave more like a standard POSIX shell than they otherwise might.
There are three primary things you can do with an execution preamble. The first is simply to set shell options or variables that you use later in a script to simplify your code. The second is to feed the script into a shell that has a particular feature you need. Finally, the third option is to actually modify the script before executing it (whether through the same shell or a different one). This section discusses the general principles of developing and using execution preambles. For more information, look into the m4sh utility, which is used to build more portable shell scripts. As an m4sh script, the preceding sample preamble code (and a great deal more) would be written as follows:
AS_SHELL_SANITIZE
$as_unset CDPATH
This provides a fairly predictable and standardized environment, with a number of utility features and functions defined in a fairly portable way. While m4sh scripts are somewhat different from conventional shell scripts, they are extremely good at running in a variety of outright hostile environments. If you need bulletproofing, this may be the way to go.
Setting Options and Variables
In many cases, merely tweaking a couple of shell options will get you behavior that is standard enough to be useful. The "Common Shells and Their Features" section covers some of these. Another technique is to use variables to hold command option flags, command names, or other values that vary from one system to another. If a command is available only on some machines, and possibly optional on others, you can use a variable to hold the command's name. A historic example of this is the use of a RANLIB variable in makefiles. On some systems, the ranlib utility had to be run on archives; on other systems, it was not only unnecessary but unavailable. The solution is to store the name of the utility to run after creating an archive in a variable. You can do this in shell scripts, too:
save_IFS=$IFS
IFS=:
ranlib=:
for dir in $PATH; do
if test -x "$dir"/ranlib; then
ranlib="$dir"/ranlib
break
fi
done
IFS=$save_IFS
$ranlib archive
If there is a ranlib utility in the user's path, it is identified by the loop and stored in the $ranlib variable. The quotes around $dir are there because someone's path could contain directories containing spaces. If there is no ranlib utility available, the script continues anyway, running the : command (which ignores its arguments). Using variables to hold command names can simplify a lot of shell development.
This technique only works for commands, not for shell syntax. Some shells provide better semantics for shell functions when they are declared as function name() { . . . } rather than just as name() { . . . }. However, you cannot set a variable to expand to function and use it in this context because function is a keyword to those shells, and the result of substitution is not a keyword.
You can also use similar techniques to hold particular command-line arguments or other values that affect the behavior of a program. Imagine that you want to display a line of text without a trailing new line; there is no consistently portable way to do this, unfortunately (the flaws with echo are discussed in more detail in Chapter 8, which discusses utility portability). However, there are two very common ways to deal with this problem, and a script can test whether either of them is available:
case `echo -n "c"` in
-n*c) func_echo_n() { echo "$@"; } ;;
*c) func_echo_n() { echo -n "$@"; } ;;
*) func_echo_n() { echo "$@c"; } ;;
esac
func_echo_n Testing...
echo "Ok."Testing...Ok.
This script defines a function called func_echo_n that will echo without a trailing new line if either of the common mechanisms works. (If neither does, the script just displays its output with a new line.) System V systems often supported a c suffix to do this, while BSD systems tended to recognize the -n flag. If neither works, the output of the trial command begins with -n and ends with a c. If the output ends with a c but did not begin with -n, then the -n flag is accepted and presumably works. If the output does not end with a c, then the c worked. This does not guarantee success, but it does prevent printing extraneous output; in the worst case, there will be new lines but no stray -n or c strings floating around. (Outside of embedded systems, though, you should probably just use printf.)
Picking a Better Shell
Sometimes access to a particular feature is sufficiently crucial to make it necessary to run a script in a shell that provides it. Some of the POSIX features are extremely useful in shell programming, and it is quite possible to be surprised when you find yourself compelled to add support for a target you were sure was never going to come up.
One workaround is to find a shell providing the needed features and ensure that your script is always run in that shell. For a script full of bash-isms, the following preamble ensures execution in bash or warns the user as to what has gone wrong:
if test -z "$BASH_VERSION"; then
save_IFS=$IFS
IFS=:
for dir in /bin /usr/bin /usr/local/bin /usr/pkg/bin /sw/bin $PATH; do
bash="$dir/bash"
if test -x "$bash" && test -n `"$bash" -c 'echo $BASH_VERSION'`
2>/dev/null; then
IFS=$save_IFS
exec "$bash" "$0" "$@"
fi
done
echo >&2 "Help! I must be executed in bash."
exit 1
fi
echo $BASH_VERSION
This preamble searches $PATH for a bash shell, and it exits if it cannot find one. A few points are of interest. One is the use of a common set of likely directories to search before $PATH, in case the user has an ill-considered search path. The test running bash to ensure it is something that produces output when asked to expand $BASH_VERSION is in single quotes because, if it used double quotes, it would require an unusually large number of backslashes. The expansion has to be done in the target shell, so the command to pass to it should be "echo $BASH_VERSION". However, this quoted string occurs inside backticks, and the shell's initial scan of the command substitution also consumes backslashes (which it doesn't inside single quotes). So, to pass $ to the bash called by the subshell, you would have to write \$:
if test -x "$bash" && test -n `"$bash" -c "echo \$BASH_VERSION"` 2>/dev/null;then
This is a great example of a case where selecting the right quoting mechanism makes your life easier.
It is possible to base this kind of testing on a feature test, as well. For instance, if you are fairly confident that the only target system you have with a pre-POSIX shell is Solaris, the following preamble gets you a fairly good POSIX shell:
if eval "! false" > /dev/null; then
true
else
exec /usr/bin/zsh "$0" "$@"
fi
if test -n "$ZSH_VERSION"; then
emulate sh
NULLCMD=:
fi
If the shell executing this does not know about the ! command prefix, the eval operation fails, and the else branch is taken, executing the script with zsh (which supports that syntax).
The second test causes zsh to run in its standard shell mode, which is usually a good choice for a script (and has no effect in other shells). There is a lot more you can do for an execution preamble, but a simple preamble like this may be enough to get your script running quickly on the targets you need it on. The combination of switching to a different shell, and then configuring that shell to behave the way you want it to, is quite powerful. If you are thinking about more than one possible system, of course, the preamble gets longer. You would want to search for multiple shells, not just zsh, and search a reasonable path. Because every step of this is a new opportunity to make mistakes, you should probably not write an execution preamble much longer than the previous example; if you need more, this is where tools like m4sh become really useful. As with most tools, using an existing tool is generally better than writing your own. In particular, since much of the benefit of shell programming is ease of development, if you start getting bogged down in details someone else has already slogged through, you are probably not getting a good return on your time.
Self-Modifying Code
If the feature you need is simple enough, it may be possible to emulate it in the current shell. The standard configure scripts generated by autoconf use this technique to emulate the special shell variable $LINENO in shells that don't provide it. Doing this correctly is fairly hard, and doing it portably requires a great deal of attention to additional special cases; if you write a sed script, and one of the systems you need to run on has a buggy sed, you haven't gained anything.
Don't be too hasty to use this; it has very limited applicability for most cases, and in general, you are better off with a generic execution preamble. Still, it is an option worth considering. The following fragment of the configure script shows how $LINENO can be replaced. (This is a small fraction of the code involved, dealing only with the actual substitution.)
sed '=' <$as_myself |
sed '
N
s,$,-,
: loop
s,^(['$as_cr_digits']*)(.*)[$]LINENO([^'$as_cr_alnum'_]),1213,
t loop
s,-$,,
s,^['$as_cr_digits']*
,,
' >$as_me.lineno &&
chmod +x $as_me.lineno ||
{ echo "$as_me: error: cannot create $as_me.lineno; rerun with a POSIX shell
" >&2
{ (exit 1); exit 1; }; }
. ./$as_me.lineno
exit
This script, like much of autoconf, shows attention to a number of portability details commonly overlooked. The first line runs the script (the file name is stored in $as_myself) through sed, using the = command to print the line number before each line. (The default behavior of sed is to print every line after executing all commands, so the lines are printed after their line numbers.) The next sed script (explained in detail in Chapter 11) replaces each instance of $LINENO with the current line number; the output of this is stored in $as_me.lineno.
This script fragment highlights something that can be visually confusing in long blocks of quoted code for another language (in this case, sed). In this line, it looks at first as though variable names are being quoted for some reason:
s,^(['$as_cr_digits']*)(.*)[$]LINENO([^'$as_cr_alnum'_]),1213,
In fact, the variable names are outside the quotes, and everything else is in them. The single quote immediately preceding $as_cr_digits is the end of a quoted string starting on the second line (the line containing only sed '). The variables $as_cr_digits and $as_cr_alnum hold strings of standard ASCII digits and letters. This preserves behavior even on systems with unusual character sets or with defective character range handling. These variables in question are known to contain no spaces, so they don't cause the argument to sed to get broken into multiple words. If this were ambiguous, they might have been placed in double quotes:
s,^(['"$as_cr_digits"']*)(.*)[$]LINENO([^'"$as_cr_alnum"'_]),1213,
Another interesting choice is illustrated here; the . command is used to read and execute the created script. If the original script had used exec, the shell would have executed the created script, using its name for $0, and error messages would be from configure.lineno rather than configure. Furthermore, the positional parameters would need to be passed in again; this way, the script environment is preserved. A bare exit command exits with the return code of the previous command, and the return code of the . command is the return code of the executed code (assuming it was successful in finding and reading that code at all).
Emulating Features
In many cases, it is impossible to replace a shell feature. However, in a few cases, it may be possible to come surprisingly close. The following shell function can be used to replace select in most cases, replacing the select keyword with while func_select:
func_select () {
func_select_args=0
case $1 in
[!_a-zA-Z]* | *[!_a-zA-Z0-9]* )
echo >&2 "func_select: '$1' is not a valid variable name."
return 1
;;
esac
func_select_var=$1
shift
case $1 in
in) shift;;
*) echo >&2 "func_select: usage: func_select var in ..."; return 1;;
esac
case $# in
0) echo >&2 "func_select: usage: func_select var in ..."; return 1;;
esac
for func_select_arg
do
func_select_args='expr $func_select_args + 1`
eval func_select_a_$func_select_args=$func_select_arg
done
REPLY=""
while :
do
if test -z "$REPLY"; then
func_select_i=1
while test $func_select_i -le $func_select_args
do
eval echo ""$func_select_i) $func_select_a_$func_select_i""
func_select_i='expr $func_select_i + 1`
done
fi
echo >&2 "${PS3:-#? }"
if read REPLY;then
if test -n "${REPLY}"; then
case $REPLY in
0* | *[!0-9]* )
eval $func_select_var=
;;
*)
if test "$REPLY" -ge 1 && test "$REPLY" -le $func_select_args; then
eval $func_select_var=$func_select_a_$REPLY
else
eval $func_select_var=
fi
;;
esac
return 0
fi
else
eval $func_select_var=
return 1
fi
done
}
Of course, if you've been following along, this function hardly requires comment. This is a large enough block of code, however, that it may be worth a bit of exploration. While none of the features used here are new, the combinations merit some discussion.
The first section of the function simply sets up variables. All variables are prefixed with func_select_, except for $REPLY (which is part of the normal behavior this function is supposed to emulate). The function validates its arguments minimally, insisting that the result variable have a valid identifier name and that at least one additional argument was provided. After this validation, the function builds an emulated array (see the in-depth discussion in Chapter 5) storing each of the choices in a numbered variable.
The main loop begins by setting $REPLY to an empty string. On each pass of the loop, if $REPLY is empty, the list is printed; this ensures that the list is printed the first time through. After that, the script prints a prompt and attempts to read a new value. If it fails, the output variable is emptied and the function returns. If a value is read, the function always returns; the only way to repeat the loop is if $REPLY is empty.
The test for a valid $REPLY value accepts only strings of digits, starting with a non-zero digit; this is accomplished by rejecting any pattern containing nondigits or starting with a zero. It would also be possible to strip leading zeroes. (In fact, one of the bugs of this implementation is that it does not strip leading and trailing spaces, which the real select does.) If a valid digit string is found, and it is between 1 and $func_select_args inclusive, the output variable is given the corresponding stored value.
Even this function has a few design choices reflecting a desire for portability. If you could safely assume you did not need to run in pre-POSIX shells, the $func_select_a_N variables would not be needed; you could use the positional parameters. When targeting a specific system, there might well be a better way to print the prompt; for instance, the printf command might be usable. (This example didn't use it because it was developed for use on an embedded system.) So one weakness of this is that the prompt is echoed with a trailing new line, which changes the output of the program slightly.
For purposes of getting a script that uses select running quickly on a shell other than ksh or bash, however, this is probably good enough. It works on shells as old as the traditional Bourne shell used in Solaris, and it also runs in modern shells. Be wary of that last part; it is important to check a portability feature like this against new shells, not just the older shells that originally needed it.
In fact, this emulation can be even closer in some shells; in ash, for instance, following the function declaration with an alias can give you essentially complete compatibility:
alias select='while func_select'
While this may look like a significant improvement, I do not recommend it. Aliasing behavior is a bit quirky and fairly unportable. Although aliases are now standard in POSIX shells, they are not universally available, and they are a rich source of unexpected errors. They are a wonderful interactive feature, but you should avoid them in scripting even when using a shell that supports them.
What's Next?
Chapter 8 takes a look at the major portability issues you are likely to encounter with common utility programs. While these programs are technically not part of the shell, they are essential to most shell programs. Chapter 8 also gives you information about what the common versions are, how to find good versions of a utility on a system, and what common features are not as portable as you might think.