The previous chapter showed the differences between controlling expect with spawn_id versus using the -i flag. The send command can be controlled in the same way. For example, the two lines are equivalent—both send the string foo to the process corresponding to the spawn id in the proc variable.

set spawn_id $proc; send "foo"
send −i $proc "foo"

While the first line is longer, setting the spawn id is simpler if a single process is the focus of interaction for a group of commands. For example, if a login is performed, the implicit method (using spawn_id) looks like this:

set spawn_id $proc
expect "login:"
send "$name
"
expect "Password:"
send "$password
"
expect "$prompt"

Using explicit −i parameters requires more characters and is more difficult to read.

expect −i $proc "login:"
send −i $proc "$name
"
expect −i $proc "Password:"
send −i $proc "$password
"
expect −i $proc "$prompt"

Setting spawn_id makes it possible to easily localize which process is being interacted with. If the process has to be changed to a different one, only the set command has to change.

Procedures are excellent ways of localizing variables, thereby reducing complexity. For example, the following procedure takes a spawn id as an argument and performs a login interaction.

proc login {id} {
    set spawn_id $id

    expect "login:"
    send "$name
"
    expect "Password:"
    send "$password
"
    expect "$prompt"
}

If only a single process is being controlled, it is convenient to name the formal parameter spawn_id. Then, no explicit set is needed. It occurs implicitly at the time the procedure is called. Here is the same procedure definition rewritten using this technique:

proc login {spawn_id} {
    expect "login:"
    send "$name
"
    expect "Password:"
    send "$password
"
    expect "$prompt"
}

Unlike the expect command, the send command has no built-in support for sending to multiple processes simultaneously. The support is not necessary, since it is possible to achieve the same effect by writing a group of send commands or by writing one in a loop or procedure. For example, a string can be sent to a list of processes with the following command:

foreach spawn_id $procs {
    send $string

}

Notice that spawn_id is implicitly set by the foreach command. This is analogous to a formal parameter except that spawn_id retains the last value after the foreach finishes. This could be avoided by using a different variable and then passing it explicitly with −i, or alternatively by placing the entire command in a procedure.

In comparison, waiting for output from multiple processes cannot be simulated easily, hence it is built in. A number of other options are built into the send command because they are inherently difficult to achieve. These will be covered in the remainder of this chapter.

Many programs echo their input. For example, if you send the date command to the shell, you will see the string date followed by a date. More precisely, you will see everything that you would ordinarily see at a terminal. This includes formatting, too.

send "date
"
expect -re $prompt

The command above ends with expect_out(buffer) set to "date Sun Jun 13 18:54:11 EDT 1993 %2" (your date or prompt may be different). The %2 at the end is a typical prompt. More importantly, the string date has been echoed. Also, each line ends with a , including the one you sent with a .

The echoing of date has nothing to do with the send command. To put this another way, there is no way to send the string and have send not echo it because send is not echoing it in the first place. The spawned process is.

In many cases, the spawned process actually delegates the task of echoing to the terminal driver, but the result is the same—you see your input to the process as output from the process.

Often, echoed input can be handled by accounting for it in patterns. Chapter 7 (p. 173) demonstrated a simple example of this. In general, when dealing with the shell, you can just search for the prompt, and you need not worry about anything that comes before it. One unfortunate possibility is that your command looks like a prompt. In this case, you have to adjust your pattern or take some other action. Another possibility is to change your prompt to something that will never resemble your input. This is often much easier than selecting a new pattern.

Yet another possibility is to turn off the echoing entirely. Few programs let you do this, but fortunately all shells do, and shells can be used to start any other programs. This provides a general solution. For example, you can spawn a shell and then send the command "stty -echo“, after which your commands will no longer be echoed. "stty echo" reenables echoing.

After sending stty to a spawned shell, other commands started by the shell are similarly affected. Imagine starting a cat process from within the shell and then sending characters to it. Normally these characters would be echoed (no matter what cat’s destination). By preceeding it with "stty -echo“, no characters will be echoed.

Disabling the echo can greatly simplify scripts. For example, cat never prompts. But because of the default echoing, its output accumulates waiting to be read with the expect command. If you do not use expect, the operating system will eventually block cat from further execution because there is no more space for its output. The solution is either to do an expect every so often to flush output of cat, or to turn off echoing entirely. By turning off the echoing, you avoid having to use expect while sending to cat. I will show an example of this in Chapter 20 (p. 461).

When dealing with programs that run in cooked mode, you must observe certain precautions. In particular, the terminal driver has a big impact on how characters are understood by the program. The terminal driver makes all the translations that it normally does when you are typing by hand. For instance, the return character is translated to a , and the line kill character (typically ^U) removes all the characters typed so far in the current line.

Some characters generate signals. For instance, ^C and ^Z are usually tied to signals that interrupt and suspend a process. For example, in cooked mode a process never reads a ^Z. Rather, the terminal driver turns it into a signal which stops the process. When sending a ^Z by hand, control returns to a shell automatically. However, if the process was created directly by spawn, there is no shell to which to return, so once the process stops, it will not say "suspended" or anything else. The process is really stopped and will remain so until it receives a continue signal. Unless you are explicitly testing how a process reacts to a signal, there is no point in sending it characters like these. (Expect does not need to suspend processes anyway. To do job control, Expect scripts merely change spawn_id or use an −i flag.)

Another problem that arises in cooked mode is limitations in the device driver itself. For instance, you must avoid sending “too many” characters in a row without a return. Most UNIX systems do not allow you to send more than 256 characters while in cooked mode. Some allow as many as 1000 characters, but there is always a limit and it is always surprisingly low. These low limits may sound hard to believe, but systems can get away with them because people never type commands longer than this. Indeed, people rarely type commands longer than 80 characters.

These limits occur even on window systems. If you paste a large hunk of text into a shell window, the shell may lock up, beep, or perform some other distasteful behavior.

Don’t confuse these limits with the length of a command that can be fed to the shell. That limit is typically 10K on older systems and a megabyte or more on modern systems. Stemming from the kernel, that limit holds for arguments to any program invocation (including the spawn and exec commands within Expect). The cooked mode limits, however, are due to the terminal drivers and occur only in interactive use.

Generally, these limits are easy to avoid. Write scripts so that they run programs just like a real human does. Send a command, read a response, send a command, read a response. Don’t send multiple commands at once—unless that is what a real user would do.

I used the following script to test the number of characters that could be entered on a system that sent a ^G when the limit was exceeded.

spawn /bin/sh
expect "\$ "           ;# match literal $ in shell prompt
for {set i 0} 1 {incr i} {
    send "/"
    expect "07"  break  
           "/"
}
puts "terminal driver accepted $i characters"

In raw mode, this problem does not exist. Any number of characters can be entered without pressing return.

It is often useful to send large quantities of information. A common scenario in which this arises is when sending mail via a commercial mail system such as GEnie or CompuServe. To reduce costs, you create and edit mail messages offline. Once the messages are prepared, you can then log in, start up the mailer, and feed the messages to it. This allows you to use your own editor as well as lower your costs for connect time.

The fragment of the script that sends the message from your system to the remote system can be very simple. For example, if the message is already stored in the variable message, one send command is sufficient.

send $message

The size of things that can be sent this way is limited only by the amount of memory that can be dynamically allocated on your system. On modern computers, this limit is well above a megabyte, and a 32-bit computer should have no problem with 50Mb or even 100Mb.

Before sending very large files, it is wise to ensure that the remote side is not echoing (as described earlier). If the echoing cannot be disabled, the process output must be consumed, such as by an expect command. Otherwise, your operating system has to store all these characters in memory. Most systems limit unread characters to 10K or even less. If more characters arrive, the operating system temporarily stops the spawned process until the Expect process can read them into its own buffers. Expect’s own buffers are limited only by the amount of virtual memory on the computer.

This behavior of the operating system works in reverse as well. Expect can easily write output faster than a spawned process can read it. Most programs simply do not read data as quickly as another program can write it. Again, the operating system has to buffer the input, and if too much arrives, the Expect process itself will be temporarily stopped until the reading processing has consumed sufficient output so as to make room for more. The temporary halt is transparent to Expect itself. It may just seem like the send command has taken a long time to execute.

Some operating systems do not buffer interactive input well. Upon receiving more characters than they have space for, they might ignore new characters, respond with error messages, or other anomalous behavior. Many serial devices (e.g., modems, terminal drivers) behave similarly.

One solution is to break the outgoing string into pieces, sending a piece at a time separated by a delay. Here is a procedure that prints out 10 characters at a time, pausing one second between each group.

proc ten_chars_per_sec {s} {
    while {[string length $s] > 10} {
        send [string range $s 0 9]
        set s [string range $s 10 end]
        sleep 1
    }
    send $s
}

ten_chars_per_sec loops over a string, sending the first ten characters, recreating the string without the first ten characters, and delaying for one second before looping. When there are less than ten characters remaining, the loop exits, and the remaining characters are sent out by the final command. If s happens to be empty (because the original length was perfectly divisible by 10), the command "send $s" has no effect at all.

The delay is achieved by executing a sleep command. Although the sleep command is efficient, the string manipulation might not be. If the string was large enough, constantly shifting the string could be fairly expensive. While the ten_chars_per_sec procedure can be rewritten more efficiently, Expect addresses the problem directly by providing explicit support for this common operation. The send command has a -s flag which causes the operation to occur slowly.

The slowness is controlled by the variable send_slow. The variable is a list of two numbers. The first is the number of characters to send at a time. The second number is a time to pause after each group of characters is sent. Here is an example:

set send_slow {2 .001}
send -s "Now is the time"

The fragment above sends the string "Now is the time" two characters at a time. Each pair of characters is separated by .001 seconds (i.e., one millisecond). This might be appropriate for a system that has a hardware serial interface that can accept two characters at a time before overflowing—a very common situation.

Arbitrarily large groups of characters and pauses can be used as well. This is useful when dealing with logical interfaces. They may allocate huge buffers, but without flow control, even the largest buffers can overflow. For example, to send 1000 characters with one second between each group of characters, you would say:

set send_slow {1000 1}
send -s $text

The first number, the number of characters, is always integral. The second number, the timeout, may be either integral or real (scientific notation is acceptable).

The following script may be used as a filter. It copies the standard input to the standard output at the rate of one hundred characters per second—a very slow rate.

#!/usr/local/bin/expect —
set send_slow {1 .01}      ;# send 1 character every .01 seconds
while {1} {
    expect_user eof exit -re ".+"
    send_user -s $expect_out(buffer)
}

With the -h flag, the send command sends characters in a way that resembles a human typing. Unlike "send -s“, "send -h" enables variability and randomness. The time periods between every character can be different from one character to the next.

A varying sending speed is quite useful when trying to simulate the effect of interactive loads. For example, suppose you are testing a computer to see if it can handle a particular mix of CPU-intensive background processes and some interactive processes. Processing human interaction usually requires only a tiny fraction of the CPU. But by default, Expect scripts skip all the delays that real humans produce. Such scripts produce a very inaccurate simulation of interactive performance since the scheduler handles Expect processes as if they were just CPU-intensive processes.

The algorithm used to time character output is based upon a Weibull distribution, a common statistical tool for generating pseudo-random inter-arrival times. A few modifications have been made to suit its use in Expect. The output is controlled by the value of the send_human variable which takes a five-element list. The first two elements are average interarrival times of characters in seconds. The first is used by default. The second is used at word endings, to simulate the subtle pauses that occasionally occur at such transitions. The third parameter (the shape) is a measure of variability where .1 is quite variable, 1 is reasonably variable, and 10 is almost invariable. The extremes are 0 to infinity.^[43] The last two parameters are, respectively, a minimum and maximum interarrival time.

Intuitively, the elements are used from left to right. First, one of the two average arrival times is chosen. Then, it is shaped according to the variability and a pseudo-random factor. Finally, the result is clipped according to the minimum and maximum elements. The ultimate average can be quite different from the stated average if enough times are clipped by the minimum and maximum values.

As an example, the following command produces characters like a fast and consistent typist:

set send_human {.1 .3 1 .05 2}
send -h "I'm hungry.  Let's do lunch."

Characters are output at an average of one every .1 seconds, except for word endings where they average one every .3 seconds. (Word ending are transitions from an alphanumeric character to anything but an alphanumeric character.) The minimum time between characters is .05 seconds and the maximum time is 2 seconds. The shape parameter will be described further later.

The following might be more suitable after a hangover:

set send_human {.4 .4 .2 .5 100}
send -h "Goodd party lash night!"

Errors are not simulated. It is impossible for Expect to know what a “reasonable error” is or how to correct it—this depends on the particular spawned program. You can set up error correction situations yourself by embedding mistakes and corrections in a send argument. For example, if the spawned process understands that a backspace removes earlier characters, the previous example can be “fixed” as follows:

send -h "Goodd party lasht night!"

The shape parameter controls the variability. The term shape refers to the control that the parameter has over a bell-like curve in which the time is likely to appear. The various curves are not particularly intuitive. However, you can examine several runs I made printing out the characters 0123456789 with an average time between characters of .1 seconds, and a minimum time of .05 seconds and maximum of 2 seconds.

The following graphs show the times between characters with the shape ranging from .125 to 16. The left-hand axis is the time in seconds that passed before the particular character was sent. Each character is shown in the axis at the bottom of the graph.

16 is almost large enough that no variation is seen. The line is essentially straight. Each character was sent almost exactly .1 seconds after the previous one. The graph showing the shape of 8 has a slight bend in the middle, but you probably would not notice the differences if you were watching the characters print. A scheduler might not either.

Figure 12-1. 

A shape of 4 allows a little bit more variability. You would probably not notice the difference however. The extremes are still within .07 seconds of one another.

A shape of 2 allows enough variability that the minimum time bound is reached. All times below .05 seconds are truncated back up to .05 seconds. The maximum time is .06 seconds above the maximum time when the shape was 4.

Figure 12-2. 

A shape of 1 shows significant inconsistencies, yet the values are still within .16 seconds of one another. A shape of between 1 to 2 probably represents a good typist.

A shape of .5 is the lowest in this sequence that could still be considered consistent. Several values have hit the minimum time. Without this bound, there would be a spread of a half of a second.

Figure 12-3. 

A shape of .25 shows significant inconsistencies. The maximum bound of 2 seconds has been reached. Only three of the characters have times that are not truncated by the minimum or maximum bounds. This is probably a fair simulation of someone using the hunt-and-peck technique.

A shape of .125 is extremely variable. All but one of the inter-character times have been truncated. This probably does not describe the typing of any human. Nonetheless, it could conceivably be used to test some facet of the scheduler.

Figure 12-4. 

One final note—the timeouts described by send_slow and send_human have no interaction with the expect command and its timeouts. They are completely separate.

In Chapter 6 (p. 153), I described how to detect a null character. A null can be sent by calling send with the -null flag. By default, one null is sent to the currently spawned process.

send -null

An optional integer may be used to send several nulls. For example, the following command sends three nulls.

send -null 3

send uses the pattern "-null" while expect uses "null“. The absence of a hyphen is historical—all of the special patterns lack a hyphen.

A break is a special condition of a serial communications line. It does not correspond to a particular ASCII character sequence, so it requires special handling in order to produce it using send. A break condition is generated by calling send with the -break flag.

send -break

The spawned program to which you are sending must understand and expect the break. Most programs do not understand a break as input even if they can generate it on output. For example, the tip program translates the input character sequence ~# into an output break condition. tip itself does not accept the break condition from the user.

Historically, keyboards contained special keys to generate break conditions. However, such keyboards are rare today. Hence, most modern programs are like tip, accepting some combination of keystrokes to signify a break.

The only situation in which you are likely to generate a break directly by using "send -break" is when the spawned process is not an actual process but is instead a serial device. I have not yet described how to make such a connection, but I will in Chapter 13 (p. 286).

A break condition cannot be detected using the expect command. However, breaks can be turned into signals. This is similar to how a ^C in cooked mode is interpreted as an interrupt. I will discuss signals and how to detect them in Chapter 14 (p. 303). See your local stty documentation for more information on the break condition.

Because the send command looks for special arguments such as -s and -h, you cannot send strings like these without taking some special action. An argument of "--" (two hyphens) forces send to interpret the following argument literally.^[44] So if you want to send -h, you can say:

send -- "-h"

The "--" does not affect strings that do not otherwise have a special meaning to send. Thus you can preface any string with it. For example, the following two commands both do exactly the same thing—they send the string foo.

send -- "foo"
send "foo"

This behavior is particularly useful if you are sending the contents of a variable that can have any value.

send $unknown                            ;# DANGEROUS
send -- $unknown                            ;# SAFE

Without the "--,”, it is possible that the unknown variable might contain something that looks like one of send’s special flags. Using the "--" guarantees that the command is interpreted correctly.

You might also use the "--" with literals when you are generating Expect commands from another program rather than writing them by hand. For example, imagine that you are writing a C program which in turn spits out Expect commands. Your code might look like this:

printf("send "%s"",unknown);      /* DANGEROUS */
printf("send -- "%s"",unknown);   /* SAFE */

While the resulting Expect script supplies the send command with literal values, the same problem arises. The C program is not checking to make sure that the value it is supplying to the send command does not look like one of the send flags. The solution, again, is to use the "--“.

In Chapter 6 (p. 140), I described how to handle spawned programs that generate character graphics. It is occasionally useful to generate character graphics from the script itself. Expect provides no built-in support for doing this. It is not necessary because you can call external programs such as tput.^[45] For example, the cup operation moves the cursor. The following command moves it to row 23, column 4:

send_user -raw [exec tput cup 23 4]

Executing many calls to tput can be slow because it is a separate program. However, its different outputs can be saved in variables and reused, thereby avoiding excessive calls of exec. More sophisticated handling is possible with Curses extensions to Tcl. These are listed in the Tcl FAQ.

Both the send command and the puts command cause characters to be output. However, they have differences which should already be apparent. But I will compare and contrast them here just to make things clear. Note that all of the variations on send follow the style of the send command itself. For example, it is possible to call send_user with the -h flag or send_log with the -- flag.

The primary difference between send and puts is that send works with processes started by spawn, whereas puts works with files opened by Tcl’s open command. For example, to write an AT-style reset command (the characters "ATZ “) to a modem with a serial interface, puts could be used as follows:

set file [open "/dev/modem" "w"]
puts -nonewline $file "ATZ
"

Doing the same thing with send requires a spawned process instead of an open file.

spawn tip modem
send −i $spawn_id "ATZ
"

In this example, the "−i $spawn_id" is not needed. It is just here to contrast it with the $file argument in the puts. Without the −i specification, send writes to the tip process anyway because spawn automatically sets spawn_id. In contrast, with no file specification, puts writes to the standard output. The previous open has no effect on the default destination of puts.

puts "ATZ
"    ;# write to standard output
send "ATZ
"    ;# write to currently spawned process

In UNIX, processes can be viewed as files (to a degree). The puts command, thus, can write to processes as well—but only those opened by open. The open command starts processes (and process pipelines if a "|" prefaces the command). For example, the following fragment starts the word-count program, wc, and then sends it a list of words to count. The result of that is sent to lpr, which prints the word count out on a hardcopy printer.

set file [open "|wc | lpr" "w"]
puts $wordlist

By using the flag "w+" with open, it is possible to use both gets and puts.

set file [open "|tip modem" "w+"]
puts "ATZ
"

Unfortunately, some (but not all) interactive programs do rather peculiar things with their standard input. For example, tip expects to read its input from a terminal interface and it tries to put this interface into raw mode. In this case, there is no terminal interface so tip will fail.

Some programs are not so demanding, but many are. Expect is the best way of dealing with these programs. In general then, puts is used for communicating with files and send is used for communicating with processes.

send and puts do not work with each other’s identifiers. You cannot take a file created by open and use it with send. Nor can you take a spawn id and give it to puts.

puts $spawn_id "ATZ
"     ;# WRONG
send −i $file "ATZ
"      ;# WRONG

This does not mean, however, that puts and send cannot be usefully used together. Because they both deal with strings, they can be called with the same input. For example, upon receiving input (either from expect or gets), the output can be sent to both files and processes.

expect -re ".+" {
    send −i $id1 $expect_out(buffer)
    puts $file1 $expect_out(buffer)
}

Waiting for input from both processes and files is a little trickier and will be described further in Chapter 13 (p. 285).

This chapter has already covered the major features of the send command. send has several other features which were mentioned in earlier chapters. In particular, -raw is useful in raw mode (see Chapter 8 (p. 193)). The send command is also controlled indirectly by log_user (see Chapter 7 (p. 171)).

In contrast, the puts command is unaffected by log_user. Similarly, puts has no analog to -raw. The puts command provides no automatic translation mechanisms to deal with cooked or raw mode.

One final difference between puts and send is that puts terminates lines with a newline (unless you use the -nonewline flag). This is convenient when it comes to writing text files but not when controlling interactive programs that rarely want newlines. Most interactive programs either read characters one at a time or look for commands to end with a . The send command has no preference one way or the other and instead makes this explicit in every command. If you prefer, embed your send commands in a procedure that always tacks on the line terminator of your choice.