fsck, the UNIX file system check program, can be run from a shell script only with the -y or -n options. The manual defines the -y option as follows:

The -n option is safer, but almost uselessly so. This kind of interface is inexcusably bad, yet many programs have the same style. ftp, a file transfer program, has an option that disables interactive prompting so that it can be run from a script. But it provides no way to take alternative action should an error occur.

Expect is a tool for controlling interactive programs. It solves the fsck problem, providing all the interactive functionality non-interactively. Expect is not specifically designed for fsck and can handle errors from ftp as well.

The problems with fsck and ftp illustrate a major limitation in the user interface offered by shells such as sh, csh, and others (which I will generically refer to as “the shell” from now on). The shell does not provide a way of reading output from and writing input to a program. This means the shell can run fsck, but only by missing out on some of its useful features. Some programs cannot be run at all from a shell script. For example, passwd cannot be run without a user interactively supplying the input. Similar programs that cannot be automated in a shell script are telnet, crypt, su, rlogin, and gdb. A large number of application programs are written with the same fault of demanding user input.

Expect was designed specifically to interact with interactive programs. An Expect programmer can write a script describing the dialogue. Then the Expect program can run the “interactive” program non-interactively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.

Figure 1-1. 

Expect programs can be written in C or C++, but are almost always written using Tcl.^[4] Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you will not have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, UNIX programs can be called (exec). I will provide a quick introduction to the language in Chapter 2 (p. 23).

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. The expect command describes a list of patterns to watch for. Each pattern is followed by an action. If the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings "welcome“, "failed“, or "busy“, and then it evaluates one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other pattern matches in a certain amount of time.

expect {
    "welcome"            break
    "failed"            abort
    timeout            abort
    "busy"        {
        puts "busy"
        continue
    }
}

It is surprising how little scripting is necessary to produce something useful. Below is a script that dials a phone. It is used to reverse the charges so that long-distance phone calls are charged to the computer. It is invoked with the phone number as its argument.

spawn tip modem
expect "connected"
send "ATD$argv
"
# modem takes a while to connect
set timeout 60
expect "CONNECT"

The first line runs the tip program so that the output of a modem can be read by expect and its input written by send. Once tip says it is connected, the modem is told to dial using the command ATD followed by the phone number. The phone number is retrieved from argv, which is a variable predefined to contain the original argument with which the script was called.

The fourth line is just a comment noting that the variable being set in the next line controls how long expect will wait before giving up. At this point, the script waits for the call to complete. No matter what happens, expect terminates. If the call succeeds, the system detects that a user is connected and prompts with "login:“.

Actual scripts do more error checking, of course. For example, the script could retry if the call fails. But the point here is that it does not take much code to produce useful scripts. This six-line script replaced a 60Kb executable (written in C) that did the same thing!

In Chapter 16 (p. 347), I will talk more about the dialback concept and show a different way to do it.

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line. Instead, it interactively prompts for it—twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.)

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that—automate passwd so that it can be called from a shell script.

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password
"
expect "password:"
send "$password
"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern "password:“. expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The indicates a carriage-return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a non-interactive passwd, but the script has to do it because passwd assumes it is interacting with a human who does not type consistently.

The final command "expect eof" causes the script to wait for the end-of-file in the output of passwd. Similar to timeout, eof is another keyword pattern. This final expect effectively waits for passwd to complete execution before returning control to the script.

Take a step back for a moment. Consider that this problem could be solved in a different way. You could edit the source to passwd (should you be so lucky as to have it) and modify it so that given an optional flag, it reads its arguments from the command line just the way that the Expect script does. If you lack the source and have to write passwd from scratch, of course, then you will have to worry about how to encrypt passwords, lock and write the password database, etc. In fact, even if you only modify the existing code, you may find it surprisingly complicated code to look at. The passwd program does some very tricky things. If you do get it to work, pray that nothing changes when your system is upgraded. If the vendor adds NIS, Kerberos, shadow passwords, a different encryption function, or some other new feature, you will have to revisit the code.

(gdb) print &var
$1 = (int *) 0xe008

send "print &var
"
expect "0x*
" {
    send_user "$expect_out(0,string)
"
}

Some programs behave differently when run interactively and non-interactively, and they do so intentionally. For example, most programs prompt only when running interactively. Non-interactively, prompting is not needed.

A more serious problem occurs when dealing with programs that change the way they buffer output depending on whether they are running interactively or not. Programs using the standard I/O library provided by UNIX automatically buffer their output when running non-interactively. This causes problems when you need to see the output immediately. Expect can make the programs think they are running interactively, thereby resolving the buffering problem.

As another example, shells force the user to press control characters (^Z, ^C) and keywords (fg, bg) to switch jobs. These cannot be used from shell scripts. Similarly, the shell running non-interactively does not deal with history and other features designed solely for interactive use. This presents a similar problem as with passwd earlier. Namely, it is impossible to construct shell scripts which test certain shell behavior. The result is that these aspects of the shell will inevitably not be rigorously tested. Using Expect, it is possible to drive the shell using its interactive job control features. A spawned shell thinks it is running interactively and handles job control as usual.

Expect’s interact command turns control of a process over to you so that you can type directly to the process instead of through send commands.

Consider fsck, the UNIX program I mentioned earlier, which checks file system consistency. fsck provides almost no way of answering questions in advance. About all you can say is “answer everything yes” or “answer everything no”.

The following fragment shows how a script can automatically answer some questions differently than others. The script begins by spawning fsck and then in a loop answering yes to one type of question and no to another. The \ prevents the next character from being interpreted as a wildcard. In this example, the asterisk is a wild card but the question mark is not and matches a literal question mark.

while 1 {
    expect {
        eof                     {break}
        "UNREF FILE*CLEAR\?"   {send "y
"}
        "BAD INODE*FIX\?"      {send "n
"}
        "\? "                  {interact +}
    }
}

The last question mark is a catch-all. If the script sees a question it does not understand, it executes the interact command, which passes control back to you. Your keystrokes go directly to fsck. When done, you can exit or return control to the script, here triggered by pressing the plus key. If you return control to the script, automated processing continues where it left off.

Without Expect, fsck can be run non-interactively only with very reduced functionality. It is barely programmable and yet it is the most critical of system administration tools. Many other tools have similarly deficient user interfaces. In fact, the large number of these is precisely what inspired the original development of Expect.

The interact command can be used to partially automate any program. Another popular use is for writing scripts that telnet through a number of hosts or front-ends, automatically handling protocols as encountered. When such a script finally reaches a point that you would like to take over, you can do so. For example, you could browse through remote library catalogs this way. Using Expect, scripts can make a number of different library systems seem like they are all connected rather than different and disconnected.

The interact command also provides arbitrarily complex macros. If you find yourself repeatedly typing out long names, you can create a short character sequence to type instead. A simple example is the following interact command, which sends the (meaningless but long and hard to type) string "set def qwk/term=vt100 yhoriz=200" when (the short and easy to type) "y2" is entered. This ability to abbreviate is useful when dealing with interactive programs that require you to enter the same gobbledegook over and over again.

interact {
    "y2" {send "set def qwk/term=vt100 yhoriz=200"}
    "~~d" {send [exec date]}
}

This fragment also sends the current date if you press "~~d“. Arbitrary actions can be invoked, including other expect and interact commands. This example uses the exec command to run the UNIX date command.

Macros can be used in the other direction, too. If the program you are dealing with prints things out that you do not want to see or want to see but in a different way, you can change the appearance entirely.

The interact command is just a shorthand for the simplest and most common types of filtering that can be done in interactions. It is possible to build arbitrarily sophisticated mechanisms using the tools in Expect.

For example, commands can be filtered out of interfaces to prevent users from entering commands that you would prefer they not enter. Programs can also be wrapped even more tightly. The adb program, for instance, can crash a UNIX system with a slip of the finger or, more likely, inexperience. You can prevent this from happening by securely wrapping the adb program in a script. Not only does this increase the safety of your system, but your system administrators no longer all have to be masters of its intricacies.

The UNIX dump program is another program with an unlikable interface. For example, dump often guesses incorrectly about the length of a tape and will prompt for a new tape even if one is not needed. An Expect script can be used to respond to dump so that it can continue automatically. A script can answer just this question or any question from dump.

Expect can, in general, be applied to interfaces that you simply do not like for whatever reason. For example, you might like to automate sequences of your favorite game, perhaps because you have long since mastered some of it and would like to practice the end game but without laboriously replaying the beginning each time.

It is possible to customize exactly how much of the underlying programs show through. You can even make interactions entirely invisible so that users do not have to be irritated by underlying programs that they have no interest in anyway. Their attitude is “Use any tool. Just get the job done.” And they are right.

Expect can be combined with Tk, a toolkit for the X Window System. The combination of Expect and Tk is called Expectk.^[5] Using Expectk, it is possible to take an existing interactive application and give it a Motif-like X interface without changing any of the underlying program. No recompiling is necessary, and because the underlying programs are not changed, there is no need to retest them again. All your efforts can be focused on the graphical user interface. Making it look pretty is all you have to do.

Figure 1-2. 

While Expectk will allow you to build X applications, it is limited in the amount it can automate existing X applications. Currently, Expect can automate xterm and other applications that specifically provide automation support, but Expect cannot automate any arbitrary X application.

Expect is also limited in its knowledge of character graphics such as is done by Curses. Nonetheless, with a little scripting, testing and automation of character graphics can be accomplished, and in Chapter 19 (p. 453), I will describe one way of doing this. Ultimately, I believe that Expect will do best with this capability built in rather than provided via scripts. However, the requirements and interfaces are not obvious, and further experimentation and design is required. ExpecTerm is an example implementation of a built-in character-graphic capability based on an earlier version of Expect. ExpecTerm is available from the Tcl archive (see page 19).

Expect’s language is general-purpose. It has control structures and data structures so that you can do just about anything. This can make up for the lack of programmability in other programs. For example, most debuggers do not provide very sophisticated methods of programmed control. Typically, you cannot declare variables and are restricted to simple loops.

The core of Expect’s language facilities are provided by Tcl, a language used by many other tools. “Tcl” stands for Tool Command Language. It comes as a library intended to be embedded in applications, providing a generic language facility to any application. Tcl solves a long-standing problem in designing application languages.

Actually, most tools do not have very sophisticated internal control languages. The shell represents the extreme—it has a very rich language and can even call upon other programs. Start-up scripts such as .cshrc and .profile can be very complex. Few programs have such flexibility. And that is just as well. Such programs each have their own language and it is daunting to master each new one.

In the middle of the spectrum are programs such as ftp that have a limited ad hoc language—such as permitted in the file .netrc—not very flexible but better than nothing. A lot of programs are designed like ftp—their languages are limited, not extensible, and different from one tool to the next.

At the far end are programs such as telnet which have no control language whatsoever. The bulk of all interactive programs are like ftp and telnet—severely lacking in programmability.

This lack of programmability is understandable. Consider writing an application that controls the printer. It is not worth writing a big language that requires a scanner, a parser, symbol table, etc. The application alone might only be 10Kb. However, it is impossible to predict what uses you might put your program to in the future. Suppose you get another printer. Then your program needs to understand this. Suppose you give your program to someone else and they have two printers but they are of two different types. Suppose they want to reserve one printer for printouts of a special kind for a day—printing checks on payday, for example. You can see that the possibilities are endless. So rather than customizing the application each time and extending the language as required (if you have not boxed yourself into a corner already), it makes sense to use a general-purpose language from the beginning.

Programs that already use Tcl are all set. They have a nice language from the outset. But many tools we are blessed with were written before Tcl existed. And most of them will not be rewritten. Development of most of these programs has stopped long ago (although for $$$ you can buy new versions). Fortunately, Expect allows you to layer Tcl on top of any interactive program without making any changes whatsoever to the original program. In effect, Expect makes nonprogrammable programs programmable.

Just as you personally can interact with multiple programs at the same time, so can Expect. Analogous to the way you can say fg and bg to switch between processes in a shell, Expect can also switch its attention. Of course, Expect does it a lot more quickly than you can. The result is that Expect can act as “glue” for programs that were never designed to operate with other programs. An amusing example is the original chess program distributed with V7 and BSD UNIX written by Ken Thompson. It was designed to interact with a user—it prompts for moves and echoes user input in algebraic notation. Unfortunately, it does not accept its own output as input. Even if it did, there is no way to pipe both inputs and outputs between two processes simultaneously from the shell.

With a few lines of Expect, it is possible to make one chess process play another, including both the communication and the massaging of the output so that it is acceptable as input. I will show this in Chapter 10 (p. 229), along with some more serious uses for these techniques.

Expect is useful for automating processes in the background such as through cron. By using Expect with cron, you can not only automate the interactions but also automate starting the process in the first place. For example, you might want to copy over files each night between two networked but otherwise unrelated systems. And you can be arbitrarily selective about choosing files. You might want to copy over only executables that are less than 1 Mb and were created on a Saturday unless the current load exceeds 2.5 and the previous transfer took less than 5 seconds, or some other such complicated set of rules. There are no ftp clients that have such complex front-ends. But you can write Expect scripts to make whatever decisions seem appropriate while driving ftp.

It is even possible to have a cron process interactively contact you while it is in the background. For example, an ftp script may need a password to continue what it is doing. It can search the network for you and then ask you for it. Or you can have it look for a set of users. After getting the password, Expect will go back and use it to complete the task. By having Expect query for passwords, you do not need to embed them in scripts.

Doing backups from cron is another common reason to use Expect. If the backup program needs another tape, an Expect script can tell it to go on (for example, if your tapes are physically longer than the backup program thinks), or it can, again, contact you for assistance.

Most of the examples in this chapter use programs that are common to all UNIX systems. But Expect is not restricted to these programs. You can apply Expect to other programs. Even (gasp!) programs that you have written.

In the previous section, I described how you might automate ftp. But if you have a different program on your system to do file transfer, that’s fine. Expect can use it. Expect does not have a built-in mechanism for file transfer, remote login, rebooting your colleague’s workstation, or a million other useful things. Instead, Expect uses whatever local utilities you already have. This means that you do not have to learn a new file transfer protocol or a new host-to-host security system. If you use ".rhosts“, then that is what Expect will use when it does remote host operations. If you use Kerberos, then Expect will use that. And so on.

I have mentioned a number of UNIX programs that can be controlled by Expect. Expect can run any UNIX program, not just interactive ones. So you can invoke programs like cat or ls if you need them too.

Expect can do just about anything that any shell script can do. But that is not the point of Expect. Indeed, Expect scripts are meant to be small. By wrapping badly behaving programs with Expect scripts, you can use them from other scripting languages, such as the shell. The resulting scripts behave just like any UNIX program. Users have no way of knowing that the programs are just scripts.

Expect makes use of a number of features that are present in all UNIX systems. The family of standards known as “POSIX” describes most but not all of these features. So while Expect can run on any UNIX system, it may have trouble providing all of its features on non-UNIX systems that nonetheless claim strict POSIX compliance. While Expect works just fine on some non-UNIX POSIX systems and can work in a limited way on all POSIX systems, I prefer to be conservative in my claims, so I use the phrase “UNIX systems” when referring to the systems on which Expect runs.

Fortunately, Expect can be used to control other operating systems indirectly. Since Expect is capable of making network connections (through telnet, rlogin, tip, etc.), it can remotely contact other computers even while running on a UNIX computer. In this way, it is very common to use Expect scripts to control non-UNIX computers such as Lisp machines, PostScript printers, modems, pagers, etc.

Testing and setting modems and other network devices (routers, bridges, etc.) is a particularly popular use of Expect. It is possible to write scripts that regularly test sets of modems to make sure that they are functional and that previous users have not left them in a bad state. Such scripts can even simulate a real user—placing a phone call, connecting to a remote host, and even logging in. An Expect script can remember the port numbers and other trivia that users do not bother to note until they have scrolled off the screen. So if a line turns out to be faulty, the script can record this and other information to a log.

Expect is a real program used in real applications. In my own environment, my colleagues depend on many Expect scripts to do important and critical tasks. We also use Expect to whip up demos and to automate ad hoc programs.

I have already mentioned lots of other uses for Expect. Many more cannot be described without also going into detail about proprietary or one-of-a-kind programs that people are stuck with using every day. But I will mention some companies and examples of how each uses Expect to give you a warm fuzzy feeling that Expect is widely accepted.

3Com does software quality assurance with Expect. Silicon Graphics uses it to do network measurements such as echo response time using telnet. The World Bank uses it to automate file transfers and updates. IBM uses it as part of a tape backup production environment. HP uses it to automate queries to multiple heterogenous commercial online databases. Sun uses it to sweep across their in-house network testing computer security. Martin Marietta uses it to control and extract usage statistics from network routers. Tektronix uses it to test software simulations of test instruments. The National Cancer Institute uses it to administer accounts across multiple platforms. Cisco uses it for network control and testing. Xerox uses it for researcher collaboration. Motorola uses it to control and backup multiple commercial databases. Data General uses it for software quality engineering. The Baha’i World Centre uses it to automate and coordinate data collection and storage from different telephone exchange locations. Amdahl uses it to automatically retrieve stock prices. CenterLine Software uses it for software quality assurance. Encore Computer uses it to simulate 500 users logging into one system at the same time. AT&T uses it to copy files between internal hosts through a firewall to and from external Internet hosts. Sandia National Laboratories uses it to control unreliable processes that need to be watched constantly. Schlumberger uses it to manage a network environment including routers, repeaters, bridges, and terminal servers all from different manufacturers. ComputerVision built an automated testbed using it. This is just a fraction of some of the users and uses of Expect. The whole list is truly astonishing, and it continues to grow rapidly.

In addition to this list of internal applications, companies such as IBM, AT&T, and Data General have incorporated Expect into applications that they sell. One of the best known commercial products utilizing Expect is DejaGnu, written by Rob Savoye for Cygnus Support. DejaGnu is a software testing utility designed to simplify the running of large collections of regression tests. Using Tcl and Expect, DejaGnu provides a framework in which tests can be written, run, and analyzed quickly and easily. Due to the power of Expect, DejaGnu is capable of testing both interactive and non-interactive applications, including embedded applications and applications on other hosts to support cross-platform and remote target board development. Cygnus has created freely available DejaGnu test suites for the GNU C and C++ compiler, debugger, assembler, and binary utilities. These test suites can be used to test any similar program or port of such a program, whether it is freely available or proprietary. According to the Free Software Report, Vol. 3, No. 1, “Cygnus supports over 70 platform configurations of the GNU compilers fully tested by DejaGnu. DejaGnu executes 8000 test cases in 16,000,000 documented tests for a typical release.”

It is not necessary to license Expect. Most of Expect is in the public domain, but two parts of it have copyrights. The Tcl core and the regular expression engine inside Expect are copyrighted but otherwise freely available, allowing the software to be used for any purpose and without fee, and with few restrictions on the code other than maintaining the copyright internally. The full copyright notices follow:

The following notice is required on the NIST-authored portions of Expect:

So you can use Expect in other freely-redistributable or commercial packages. You can even use pieces of the code in other software products. Just don’t blame the authors for any problems encountered with the software.

Expect may already be installed on your system, typically in /usr/local/bin. If you cannot find Expect, ask your system administrator. If you do not have a system administrator, you can obtain Expect by following the instructions below. Expect requires no special permissions to install, nor does it have to be installed in any particular place. You can even try it out in your own directory.

Expect includes a number of examples, several of which are useful as tools in their own right. Indeed, quite a few have man pages of their own and can be installed along with Expect. If the examples are not installed, you can find them in the example directory of the Expect distribution. Ask your local system administrator where the distribution is.

The examples provided with Expect are subject to change, but below is a list of just a few of the examples. The README file in the example directory contains a complete list as well as full explanations about each of the examples:

These and additional examples are available with the Expect distribution. The README file in the distribution also describes the location of the Expect archive which holds even more scripts. You can also contribute your own scripts to the archive. Particularly large or sophisticated applications (such as those which combine Expect with other extensions) can be found separately in the Tcl archive (see page 19).

There is a high probability that you already have Expect on your system. Expect is shipped by many vendors with their operating system utilities. Expect can also be found on many software distributions including GCT from Testing Foundations, USENET Software from UUNET, the Sun User Group CD-ROM, the Prime Time Freeware CD-ROM, the Lemis CD-ROM, and others. Even if you do not have one of these, some other user on your system may have already retrieved Expect from an Internet repository.

The entire Expect distribution can be obtained from many sites around the Internet. Your best bet to finding a nearby site is to use an Archie server. Archie maintains a database of thousands of computers across the Internet and what programs they are willing to supply. If you do not know how to use Archie, you can obtain instructions by sending email to [email protected] with the word help as the contents of the message.^[6]

As this book is being written, Archie reports at least one site in the following countries where Expect can be obtained:

Currently, the site ftp.cme.nist.gov always contains the latest version of Expect (or a pointer to it). To retrieve it from there, you can use anonymous ftp or request automatic email delivery.

To get Expect from ftp.cme.nist.gov via ftp, retrieve the file pub/expect/README. This will tell you what other files you need to retrieve and what to do after that. Typically, you need a copy of Expect and a copy of Tcl. You may also want to get a copy of Tk. These can be found in the same directory as Expect.

If you are not directly on the Internet but can send mail, you can request email delivery of the files. Send a message to "[email protected]“. The message body should be:

send pub/expect/README

The site ftp.cs.berkeley.edu always contains the latest releases of Tcl and Tk. This site permits anonymous ftp. If you would like to request email from that site, any of the other sites listed above, or any other anonymous ftp sites, you can use an "ftp by mail” agent, such as the one provided by Digital Equipment’s Western Research Laboratories. Complete instructions for using ftp-mail may be retrieved by emailing to [email protected]. The subject should be "ftpmail" and the message body should be "help“.

This book contains a great deal of information about Expect and Tcl. Yet there are other resources that you may find useful. I will occasionally refer to some of these resources. The others are just for extra reading on your own.

Cygnus Support
1937 Landings Drive
Mountain View, CA  94043
+1 (415) 903-1400
[email protected]

Computerized Processes Unlimited
4200 S. I-10 Service Rd., Suite 205
Metairie, LA 70006
+1 (504) 889-2784
[email protected]