Problem

You want your shell commands to read data from a file.

Solution

Use input redirection, indicated by the < character, to read data from a file:

 wc < my.file

Discussion

Just as the > sends output to a file, so the < takes input from a file. The choice and shape of the characters were meant to give a visual clue as to what was going on with redirection. Can you see it? (Think “arrowhead.”)

Many shell commands will take one or more filenames as arguments, but when no filename is given will read from standard input. Those commands can be invoked as either command filename or command < filename with the same result. That’s the case here with wc, but also with cat and others.

It may look like a simple feature, and be familiar if you’ve used the DOS command line before, but it is a significant feature of shell scripting (which the DOS command line borrowed) and was radical in both its power and its simplicity when first introduced.

See Also

• Recipe 2.6, “Saving Output to Other Files”

Problem

You need input to your script, but don’t want a separate file.

Solution

Use a here-document with the << characters, redirecting the text from the command line rather than from a file. When put into a shell script, the script file then contains the data along with the script.

Here’s an example of a shell script in a file we call ext:

$ cat ext
#
# here is a "here" document
#
grep $1 <<EOF
mike x.123
joe  x.234
sue  x.555
pete x.818
sara x.822
bill x.919
EOF
$

It can be used as a shell script for simple phone number lookups:

$ ext bill
bill x.919
$

or:

$ ext 555
sue x.555
$

Discussion

The grep command looks for occurrences of the first argument in the files that are named, or if no files are named it looks to standard input.

A typical use of grep is something like this:

grep somestring file.txt

or:

grep myvar *.c

In our ext script we’ve parameterized the grep by making the string that we’re searching for be the parameter of our shell script ($1). Whereas we often think of grep as searching for a fixed string through various different files, here we are going to vary what we search for, but search through the same data every time.

We could have put our phone numbers in a file, say phonenumbers.txt, and then used that filename on the line that invokes the grep command:

grep $1 phonenumbers.txt

However, that requires two separate files (our script and our datafile) and raises the question of where to put them and how to keep them together.

So, rather than supplying one or more filenames to search through, we set up a here-document and tell the shell to redirect standard input to come from that (temporary) document.

The << syntax says that we want to create such a temporary input source, and the EOF is just an arbitrary string (you can choose what you like) to act as the terminator of the temporary input. It is not part of the input, but acts as the marker to show where it ends. The regular shell script (if any) resumes after the marker.

We also might add -i to the grep command to make our search case-insensitive. Thus, using grep -i $1 <<EOF would allow us to search for “Bill” as well as “bill”.

See Also

• man grep

• Recipe 3.3, “Preventing Weird Behavior in a Here-Document”

• Recipe 3.4, “Indenting Here-Documents”

Problem

Your here-document is behaving weirdly. You wanted to maintain a simple list of donors using the method described previously for phone numbers, so you created a file called donors that looked like this:

$ cat donors
#
# simple lookup of our generous donors
#
grep $1 <<EOF
# name amt
pete $100
joe  $200
sam  $ 25
bill $ 9
EOF
$

But when you tried running it you got weird output:

$ ./donors bill
pete bill00
bill $  9
$ ./donors pete
pete pete00
$

Solution

Turn off the shell scripting features inside the here-document by escaping any or all of the characters in the ending marker:

grep $1 <<'EOF'
pete $100
joe  $200
sam  $ 25
bill $ 9
EOF

Discussion

It’s a very subtle difference, but the <<EOF can be replaced with <<EOF, or <<'EOF', or even <<EOF—they all work. It’s not the most elegant syntax, but it’s enough to tell bash that you want to treat the “here” data differently.

Normally (i.e., unless you use this escaping syntax), says the bash manpage, “…all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion.”

So what’s happening in our original donors script is that the amounts are being interpreted as shell variables. For example, $100 is being seen as the shell variable $1 followed by two zeros. That’s what gives us pete00 when we search for “pete” and bill00 when we search for “bill”.

When we escape some or all of the characters of the EOF, bash knows not to do the expansions, and the behavior is the expected behavior:

$ ./donors pete
pete $100

Of course, you may want the shell expansion on your data—it can be useful in the correct circumstances—but that isn’t what we want here. We’ve found it to be a useful practice to always escape the marker, as in <<'EOF' or <<EOF, to avoid unexpected results, unless you know that you really want the expansion to be done on your data.

See Also

• Recipe 3.2, “Keeping Your Data with Your Script”

• Recipe 3.4, “Indenting Here-Documents”

Problem

The here-document is great, but it’s messing up your shell script’s formatting. You want to be able to indent for readability.

Solution

Use <<-, and then you can use tab characters (only!) at the beginning of lines to indent this portion of your shell script:

$ cat myscript.sh
...
     grep $1 <<-'EOF'
        lots of data
        can go here
        it's indented with tabs
        to match the script's indenting
        but the leading tabs are
        discarded when read
        EOF
    ls
...
$

Discussion

The hyphen (-) just after the << is enough to tell bash to ignore the leading tab characters. This is for tab characters only and not arbitrary whitespace. Note that this is especially important with the EOF or any other marker designation. If you have spaces there, it will not recognize the EOF as your ending marker, and the “here” data will continue through to the end of the file (swallowing the rest of your script). Therefore, you may want to always left-justify the EOF (or other marker) just to be safe, and let the formatting go on this one line.

See Also

• Recipe 3.2, “Keeping Your Data with Your Script”

• Recipe 3.3, “Preventing Weird Behavior in a Here-Document”

Problem

You need to get input from the user.

Solution

Use the read statement:

read

or:

read -p "answer me this " ANSWER

or:

read -t 3 -p "answer quickly: " ANSWER

or:

read PRE MID POST

Discussion

In its simplest form, a read statement with no arguments will read user input and place it into the shell variable REPLY.

If you want bash to print a prompt string before reading the input, use the -p option. The next word following the -p will be the prompt, but quoting allows you to supply multiple words for a prompt. Remember to end the prompt with punctuation and/or a space, as the cursor will wait for input right at the end of the prompt string.

The -t option sets a timeout. The read statement will return after the specified number of seconds regardless of whether the user has responded. Our example uses both the -t and -p options together, but you can use the -t option on its own. As of bash version 4 you can even specify fractional numbers of seconds, like .25 or 3.5 for the timeout value. The exit status ($?) will be greater than 128 if the read timed out.

If you supply multiple variable names in the read statement, then read parses the input into words, assigning them in order. If the user enters fewer words, the extra variables will be set to null. If the user enters more words than there are variables in the read statement, then all of the extra words will be part of the last variable in the list.

See Also

• help read for more options to the read builtin

• Recipe 3.8, “Prompting for a Password”

• Recipe 6.11, “Looping with a read”

• Recipe 13.6, “Parsing Text with a read Statement”

• Recipe 14.12, “Validating Input”

Problem

You need to get a simple yes or no input from the user, and you want to be as user-friendly as possible. In particular, you do not want to be case-sensitive, and you want to provide a useful default if the user presses the Enter key.

Solution

If the actions to take are simple, use the self-contained function in Example 3-1.

# cookbook filename: func_choose

# Let the user make a choice about something and execute code based on
# the answer
# Called like: choose <default (y or n)> <prompt> <yes action> <no action>
# e.g. choose "y" 
#       "Do you want to play a game?" 
#       /usr/games/GlobalThermonuclearWar 
#       'printf "%b" "See you later Professor Falkin.
"' >&2
# Returns: nothing
function choose {

    local default="$1"
    local prompt="$2"
    local choice_yes="$3"
    local choice_no="$4"
    local answer

    read -p "$prompt" answer
    [ -z "$answer" ] && answer="$default"

    case "$answer" in
        [yY1] ) eval "$choice_yes"
            # error check
            ;;
        [nN0] ) eval "$choice_no"
            # error check
            ;;
        *     ) printf "%b" "Unexpected answer '$answer'!" >&2 ;;
    esac
} # end of function choose

If the actions are complicated, use the function in Example 3-2 and handle the results in your main code.

# cookbook filename: func_choice.1

# Let the user make a choice about something and return a standardized
# answer. How the default is handled and what happens next is up to
# the if/then after the choice in main.
# Called like: choice <prompt>
# e.g. choice "Do you want to play a game?"
# Returns: global variable CHOICE
function choice {

    CHOICE=''
    local prompt="$*"
    local answer

    read -p "$prompt" answer
    case "$answer" in
        [yY1] ) CHOICE='y';;
        [nN0] ) CHOICE='n';;
        *     ) CHOICE="$answer";;
    esac
} # end of function choice

The code in Example 3-3 calls the choice function to prompt for and verify a package date. Assuming $THISPACKAGE is set, the function displays the date and asks for verification. If the user types y, Y, or presses Enter, then that date is accepted. If the user enters a new date, the function loops and verifies it (for a different treatment of this problem, see Recipe 11.7).

# cookbook filename: func_choice.2
CHOICE=''
until [ "$CHOICE" = "y" ]; do
    printf "%b" "This package's date is $THISPACKAGE
" >&2
    choice "Is that correct? [Y/,<New date>]: "
    if [ -z "$CHOICE" ]; then
        CHOICE='y'
    elif [ "$CHOICE" != "y" ]; then
        printf "%b" "Overriding $THISPACKAGE with $CHOICE
"
        THISPACKAGE=$CHOICE
    fi
done

# Build the package here

Next we’ll show different ways to handle some yes or no questions. Carefully read the prompts and look at the defaults. In both cases the user can simply hit the Enter key, and the script will then take the default the programmer intended:

# If the user types anything except a case-insensitive 'n', they will
# see the error log
choice "Do you want to look at the error logfile? [Y/n]: "
if [ "$CHOICE" != "n" ]; then
    less error.log
fi

# If the user types anything except a case-insensitive 'y', they will
# not see the message log
choice "Do you want to look at the message logfile? [y/N]: "
if [ "$CHOICE" = "y" ]; then
    less message.log
fi

Finally, the function in Example 3-4 asks for input that might not exist.

# cookbook filename: func_choice.3

choice "Enter your favorite color, if you have one: "
if [ -n "$CHOICE" ]; then
    printf "%b" "You chose: $CHOICE
"
else
    printf "%b" "You do not have a favorite color.
"
fi

Discussion

Asking the user to make a decision is often necessary in scripting. For getting arbitrary input, see Recipe 3.5. For choosing an option from a list, see Recipe 3.7.

If the possible choices and the code to handle them are fairly straightforward, the first self-contained function is easier to use, but it’s not always flexible enough. The second function is flexible at the expense of having to do more in the main code.

Note that we’ve sent the user prompts to STDERR so that the main script output on STDOUT may be redirected without the prompts cluttering it up.

See Also

• Recipe 3.5, “Getting User Input”

• Recipe 3.7, “Selecting from a List of Options”

• Recipe 11.7, “Figuring Out Date and Time Arithmetic”

Problem

You need to provide the user with a list of options to choose from and you don’t want to make them type any more than necessary.

Solution

Use bash’s builtin select construct to generate a menu, then have the user choose by typing the number of the selection (see Example 3-5).

# cookbook filename: select_dir

directorylist="Finished $(for i in /*;do [ -d "$i" ] && echo $i; done)"

PS3='Directory to process? ' # Set a useful select prompt
until [ "$directory" == "Finished" ]; do

    printf "%b" "a

Select a directory to process:
" >&2
    select directory in $directorylist; do

        # User types a number which is stored in $REPLY, but select
        # returns the value of the entry
        if [ "$directory" == "Finished" ]; then
            echo "Finished processing directories."
            break
        elif [ -n "$directory" ]; then
            echo "You chose number $REPLY, processing $directory..."
            # Do something here
            break
        else
            echo "Invalid selection!"
        fi # end of handle user's selection

    done # end of select a directory
done # end of until dir == finished

Discussion

The select statement makes it trivial to present a numbered list to the user on STDERR, from which they may make a choice. Don’t forget to provide an “exit” or “finished” choice, though Ctrl-D will end the select and empty input will print the menu again.

The number the user typed is returned in $REPLY, and the value of that entry is returned in the variable you specified in the select construct.

See Also

• help select

• help read

• Recipe 3.6, “Getting Yes or No Input”

Problem

You need to prompt the user for a password, but you don’t want it echoed on the screen.

Solution

Use the read command to read the user’s input, but with a special option to turn off echoing:

read -s -p "password: " PASSWD
printf "%b" "
"

Discussion

The -s option tells the read command not to echo the characters typed (s is for silent) and the -p option says that the next argument is the prompt to be displayed prior to reading input.

The line of input that is read from the user is put into the variable named $PASSWD.

We follow the read with a printf to print out a newline. The printf is necessary because read -s turns off the echoing of characters. With echoing disabled, when the user presses the Enter key no newline is echoed and any subsequent output would appear on the same line as the prompt. Printing the newline gets us to the next line, as you would expect. It may even be handy for you to write the code all on one line to avoid intervening logic (putting it on one line also prevents mistakes should you cut and paste this line elsewhere):

read -s -p "password: " PASSWD ; printf "%b" "
"

Be aware that if you read a password into an environment variable it is in memory in plain text, and thus may be accessed via a core dump or /proc/core (if your OS provides /proc/). It is also in the process environment, which may be accessible by other processes. You may be better off using certificates with SSH, if possible. In any case, it is wise to assume that root and possibly other users on the machine may gain access to the password, so you should handle the situation accordingly.

See Also

• help read

• Recipe 10.6, “Trapping Interrupts”

• Recipe 14.14, “Leaking Passwords into the Process List”

• Recipe 14.20, “Using Passwords in Scripts”

• Recipe 14.21, “Using SSH Without a Password”

• Recipe 19.9, “Making Your Terminal Sane Again”