Tk provides commands to build user interfaces for the X Window System. With Tk, you can build graphic user interfaces (GUIs) entirely using Tcl and its extensions. In this chapter, I will cover how to use Expect with Tk. Some of the examples are particularly noteworthy. These include a GUI for setting passwords, a terminal emulator, and a mechanism to watch for patterns in character graphic applications.
Tk is one of the most popular Tcl extensions—and deservedly so. Tk provides a layer of abstraction that is quite high-level and yet still provides a lot of flexibility. It is possible to do things much more quickly in Tk than in other toolkits, and even better—it is likely that you will do things with Tk that you never would have even tried without it.[66]
Unlike the Tcl chapter, I will give only a brief overview of Tk here—it is not the primary focus of this book and is only mentioned a few times in other chapters. Nonetheless, a little knowledge of Tk is necessary to understand the gist of this chapter. I will make up for the lack of a complete introduction by giving a little more information when I use some of the Tk commands for the first time. However, I will skip parts of Tk that are not immediately relevant. For more information, refer to the Tk reference material. For more information on X, refer to The X Window System Series from O’Reilly and Associates.
Experienced Tk programmers may skip the introductory material in the next section and go right to the Expectk section beginning on page 428.
Tk provides commands to build user interfaces for the X Window System. For example, the button command creates an object that can be “pushed” analogously to a physical pushbutton. The scrollbar command creates an object that can be manipulated to change the view in other objects. Tk GUIs are built from these and other commands. The actions that Tk GUIs control use the same Tcl commands that you already know.
Commands to create objects are simple, and all of the commands follow a similar style. For example, the following command creates a button labelled "Get File“.
button .getbutton -text "Get File"
.getbutton is the name of the newly created button. Any further references to the button in the script are made using this string. Objects such as buttons and scrollbars are called widgets.
Commands may be associated with most widgets by using the -command flag. For example, the following command creates a button that when pushed, invokes the getfile command.
button .getbutton -text "Get File" -command "getfile"
The -command flag can name any Tcl command. This is one of the keys to the power of Tk. The screen layout is defined with simple commands, all in Tk and Tcl. And their behavior is defined with simple commands, also, all in Tk and Tcl. Of course, when used with Expect, any Expect commands may also be given.
Besides push buttons, there are also radio buttons (pushing one in forces others out) and check buttons (they stay in until you push them again). Besides buttons, Tk comes with a variety of widgets such as scrollbars, menus, drawing canvases, etc. Some of these can be used to contain other widgets. For example, a file browser might be made up of a listbox widget and a scrollbar widget, sitting within a frame widget. The frame widget groups other widgets together much like a directory groups files together in the file system. In fact, the analogy is paralleled in the naming convention. The "." in widget names is analogous to a "/" in filenames. The widget .files could be a frame containing the list widget .files.list and the scrollbar widget .files.sb. This naming convention is not required, but it is normally followed if it makes the code simpler to read. Following the analogy, the widget "." represents a top-level window that is not enclosed by anything else.
Each widget name automatically becomes the name of a command to manipulate the widget. For example, the button created in the previous section could be flashed this way:
.getbutton flash
The size of the listbox widget stored in the variable $lb could be returned this way:
$lb size
Before widgets can appear on the screen, their relationship to other widgets must be described. For example, two buttons may be displayed inside a frame using Tk’s pack command as follows:
pack .but1 .but2 -in .files
The -in flag makes the button widgets appear inside the .files widget. If the hierarchic naming convention is used (described in the previous section), the -in flag and the container widget can be omitted.
pack .files.but1 .files.but2
The pack command decides on the layout and size of the widgets, although additional flags may be used for guidance. The pack command is so named because it typically packs widgets together as tightly as possible so as to make the best possible use of the space in the window. Other commands are available to describe widget relationships in different ways, but in this chapter, the examples all use pack.
Tk’s bind command describes what actions should be taken when events occur. Events include user input (keystrokes, button clicks) as well as window changes (resizes, exposures).
The following bind command declares that the exit command should be executed when the user enters a Control-C in the widget .files.
bind .files <Control-c> exit
The next command defines a binding for the first mouse button.
bind .files <Button1> {puts "You pressed button 1"}
It is also possible to associate events on a class of widgets. The following binding causes all buttons to invoke highlight_button when the cursor enters the window.
bind Button <Any-Enter> {highlight_button %W}
When the event occurs, strings in the action beginning with % are replaced with more event information. For example, %W is replaced with the actual name of the window.
Tk scripts are similar to X programs in that they are usually event driven. Event-driven programs spend most of their time waiting for events to arrive. As events arrive, they are processed by executing any actions that have been declared with bindings. This simple idea of waiting for events and processing them is known as the event loop.
There is no explicit command in scripts where the event loop occurs. The event loop is simply entered after all the commands in the scripts have been read and processed.
You can include the Expect and Tk extensions together when you build your Tcl-based application. Alternatively if you have Tk installed on your system, the Expect Makefile automatically builds a program called Expectk.
Expectk is a mixture of Expect and Tk. Most of the commands work as before. However, some new commands exist and some old commands have different names.
The most common use of Expectk is to take existing command-line-oriented programs and wrap them with X GUIs. As with Expect, no changes have to be made to the original programs.
Wrapping existing programs avoids several common problems. For example, changing an underlying program requires that you test the result, including features that were working before you touched the program. And it is much more work to test GUIs than to test command-line interfaces. Another problem is version control. If you modify a program, you will have two versions of the original program to maintain—a command-line version and a GUI version. And of course, all of these problems presume that you have the source in the first place, which is often not the case.
Expectk allows you to focus on the GUI, since the original application already exists. This reduces the amount of testing that has to be done and avoids version problems—you will be using the same version of the application whether you use its command-line interface or its GUI interface.
All of the benefits of Tk itself carry through to Expectk. Scripts are much shorter than their compiled equivalents in C or C++. Scripts require no lengthy compilations and can be modified quickly and easily. And the widget library offered by Tk is high-level and compares favorably to more traditional widget libraries. In addition, a GUI builder exists for Tk. Written by Sven Delmas, XF allows you to build a GUI by pointing and clicking, reducing the amount of work required to create the graphical elements and layout of the GUI. The Tcl FAQ describes how to obtain XF. For more information on the FAQ, see Chapter 1 (p. 20).
Expectk can also be useful when creating brand new applications. Many applications do not require an intimate connection to a GUI. By separating the GUI from the back end, you can concentrate on each part separately. Write simple command-line interfaces to the application and you will not have to be debugging X code just to test out your application. Similarly, it is much easier to test a GUI without having to worry about the application code at the same time.
Expectk scripts are executed using the Expectk program. Expectk is very similar to Expect. For example, you can invoke scripts in an analogous way, such as with the command line:
% expectk scriptExpectk scripts may also be executed as:
% script
if the first line of the script starts with:
#!/usr/local/bin/expectk
The — and -f flags work with Expectk just like they do with Expect. However, many of the other flags are different because there are so many additional flags required by Tk and X. For example, as with most X applications, the -display flag allows you to specify an X server. Expectk uses a skeleton framework provided by Tk which is subject to change with little control from Expect. Thus, your best bet to finding out the current list of flags is a quick look at the source.
Both Tk and Expect have a command named send.[67] Expectk detects this collision and lets Tk “win” the fight for command names. So if you type "send“, you get Tk’s send command. Alternate names are provided for any Expect commands that could collide. In particular, Expect commands that do not already begin with "exp" can be invoked by prefixing them with "exp_“. For example, Expect’s send command can be invoked as exp_send.
% exp_send "foo
"If you accidentally call Tk’s send when you want Expect’s send, you will see the following error:
% send "foo
"
wrong # args: should be "send interpName arg ?arg ..."
The alias exp_send works in both Expect and Expectk, so you can use exp_send all the time if you find it simpler to remember or read. You should also stick with exp_send if you are writing code that is to be portable to both Expectk and Expect.
tkpasswd is an Expectk script that creates a GUI for changing passwords conveniently. You might wonder how there could be any value in a GUI for such a trivial program; however, people who change passwords frequently (such as system administrators) will find many benefits. For instance, the GUI provides the same interface whether you are changing local passwords (/etc/passwd) or remote passwords (NIS). The GUI can show accounts in different ways, such as sorted by name or uid. The GUI also highlights accounts that have no passwords—a potential security problem. Lastly, the GUI can reject passwords that are inappropriate. Naturally, all of this is done without modifying the passwd program itself.
Even with these and other features, the script is only 300 lines (of which 60 are empty or comment lines). About 100 lines are related to laying out the graphics. Only about 10 of the lines are directly related to driving the passwd program, but it is worthwhile to examine other parts of the program to see how, for example, the press of a button is translated into a behavior change in the passwd interaction. The script comes with the Expect distribution as an example.
When run, the script displays the image shown on page 431. At the top are several radio buttons which control the script. In the middle is a user browser. Below this are several more buttons and an entry widget in which passwords can be entered.
The script begins with the usual incantation and commentary (trimmed for publication):
#!/usr/local/bin/expectk -- # tkpasswd - Change passwords using Expectk
The first two buttons choose between a local password database (/etc/passwd) or an NIS database. When either button is activated, the variable passwd_cmd is assigned the list of "passwd" and "cat /etc/passwd" or the list "yppasswd" and "ypcat passwd“. The first element in each list is the appropriate UNIX command to set a password, and the second element is the matching UNIX command to read the password database.
Each time a button is pressed, the get_users command is also called to reload the correct password database.
I will briefly describe some of the graphical aspects but just for the first set of buttons. This will help to give you more of a feel for Tk if you have never used it.
Both buttons are embedded in a frame with a raised border so that it is easy for the user to see that they are related. "-bd 1" means the border is one pixel wide. The "-anchor w" aligns the
buttons on the west side of the frame. The pack command places the buttons into the frame and then the frame itself is placed on the screen. The "-fill x" makes the buttons and the frame expand horizontally to fill the display. (The actual width is determined later.)
frame .type -relief raised -bd 1
radiobutton .passwd -text passwd -variable passwd_cmd
-value {passwd {cat /etc/passwd}}
-anchor w -command get_users -relief flat
radiobutton .yppasswd -text yppasswd -variable passwd_cmd
-value {yppasswd {ypcat passwd}}
-anchor w -command get_users -relief flat
pack .passwd .yppasswd -in .type -fill x
pack .type -fill xIn another frame, three more buttons control how users are sorted in the display. As before, a value is assigned to a control variable (sort_cmd) which conveniently is just the right UNIX command to sort a file. Providing sorted and unsorted displays is important because the NIS database is provided in a randomized order (which usually cries for sorting) while the local database is provided in the original order (and may or may not need sorting).
frame .sort -relief raised -bd 1 radiobutton .unsorted -text unsorted -variable sort_cmd -value " " -anchor w -relief flat -command get_users radiobutton .name -text name -variable sort_cmd -value "| sort" -anchor w -relief flat -command get_users radiobutton .uid -text uid -variable sort_cmd -value "| sort -t: -n +2" -anchor w -relief flat -command get_users pack .unsorted .name .uid -in .sort -fill x pack .sort -fill x
In the center of the display is a frame containing a user browser (user list and scroll bar). The users are displayed in a text widget. The currently selected user is displayed in green. Users with no passwords are highlighted in red (to suggest danger). On a monochrome monitor, black on white and white on black is used with an additional border to distinguish between the usual white on black entries.
The default number of users shown is 10 but the window is defined so that the user can increase or decrease it, in which case the user list expands or contracts appropriately. The remainder of the display is fixed.
The width of the users is set at 14—enough for an eight character user name, a blank, and a five character user id. Everything else in the display is fixed to this width and the user is not allowed to change it.
frame .users -relief raised -bd 1
text .names -yscrollcommand ".scroll set" -width 14
-height 1 -font "*-bold-o-normal-*-120-*-m-*"
-setgrid 1
.names tag configure nopassword -relief raised
.names tag configure selection -relief raised
if {[tk colormodel .]=="color"} {
.names tag configure nopassword -background red
.names tag configure selection -background green
} else {
.names tag configure nopassword -background black
-foreground white
.names tag configure selection -background white
-foreground black
}
scrollbar .scroll -command ".names yview" -relief raised
pack .scroll -in .users -side left -fill y
pack .names -in .users -side left -fill y
pack .users -expand 1 -fill y
wm minsize . 14 1
wm maxsize . 14 999
wm geometry . 14x10A field within a frame labelled "Password" is provided in which the user can enter new passwords. The focus is moved to the entry field allowing the user to enter passwords no matter where the cursor is in the display. Special bindings are added (later) which allow scrolling via the keyboard as well.
frame .password_frame -relief raised -bd 1 entry .password -textvar password -relief sunken -width 1 focus .password bind .password <Return> password_set label .prompt -text "Password:" -bd 0 pack .prompt .password -in .password_frame -fill x -padx 2 -pady 2 pack .password_frame -fill x
Several more buttons are created and placed at the bottom of the display. Rather than putting them at the top, they are placed at the bottom because it is likely they will be pressed at most once. In contrast, the buttons at the top are likely to be pressed many times.
The first button controls whether passwords are checked against a dictionary. It sets the variable dict_check appropriately, and the dictionary is loaded if it has not been already.
set dict_loaded 0 checkbutton .dict -text "check dictionary" -variable dict_check -command { if !$dict_loaded load_dict } pack .dict -fill x -padx 2 -pady 2
A quit button causes the program to exit if pressed.
button .quit -text quit -command exit button .help_button -text help -command help pack .quit .help_button -side left -expand 1 -fill x -padx 2 -pady 2
A help button pops up a help window describing how to use the program. The actual text is omitted here.
proc help {} {
catch {destroy .help}
toplevel .help
message .help.text -text <...help text here...>
button .help.ok -text "ok" -command {destroy .help}
pack .help.text
pack .help.ok -fill x -padx 2 -pady 2
}It is interesting to note that all the preceding code is just to set up the display and takes about a third of the program.
The get_users procedure reloads the password database. It is called when any of the top buttons are activated.
After clearing the current list, the procedure executes the appropriate UNIX commands to read and sort the password database. The particular commands are defined by the radio buttons. They select which database to read and how to sort it.
The remainder of the procedure adds the users to the list of names, appropriately tagging any that have null passwords. User ids are displayed as well. User names that have no other information with them are pointers back to the NIS database. They are displayed without user ids but nothing else is done. The script does not have to worry about them because the passwd program itself rejects attempts to set them.
proc get_users {} { global sort_cmd passwd_cmd global selection_line global nopasswords ;# line #s of users with no passwds global last_line ;# last line of text box .names delete 1.0 end set file [open "|[lindex $passwd_cmd 1] $sort_cmd"] set last_line 1 set nopasswords {} while {[gets $file buf] != −1} { set buf [split $buf :] if [llength $buf]>2 { # normal password entry .names insert end "[format "%-8s %5d" [ lindex $buf 0] [lindex $buf 2]] " if 0==[string compare [lindex $buf 1] ""] { .names tag add nopassword {end - 1 line linestart} {end - 1 line lineend} lappend nopasswords $last_line } } else { # +name style entry .names insert end "$buf " } incr last_line } incr last_line −1 close $file set selection_line 0 }
At various places in the script, feedback is generated to tell the user what is going on. For simplicity, feedback is displayed in the same field in which the password is entered. This is convenient because the user probably does not want the password left on the screen for long anyway. (Making the password entirely invisible could be done by making some minor changes to the bindings.) The feedback is selected (highlighted) so that it disappears as soon as the user begins to enter a new password.
proc feedback {msg} {
global password
set password $msg
.password select from 0
.password select to end
update
}The dictionary takes considerable time to load into memory (about 10 seconds for 25,000 words on a Sun Sparc 2) so it is not loaded unless the user specifically activates the "check dictionary" button. The first time it is pressed, this procedure is executed. For each word, it creates an element in an array called dict. No value is necessary. Later on, passwords will be looked up in the dictionary just by testing if they exist as an element in the dict array—a very fast operation.
Calling the UNIX grep command would spread the load out, but it would also expose the password to anyone running ps. Instead, I tried my best to speed this procedure up (without resorting to C). Using split on the entire file reduced the run-time by about one third from that taken by the more obvious gets in a while loop. While this can backfire if a file is larger than available memory, it works well with reasonably sized dictionaries.
Since Tcl rescans commands each time they are executed, it is possible to improve performance simply by using shorter commands. (However, the benefit only becomes apparent when there are no other bottlenecks left.) I achieved another 10% increase in speed by temporarily renaming the set command to s. Interestingly, renaming dict to d had almost no impact so I left it unchanged. Substituting a single bare character for the “” made no difference at all.
proc load_dict {} {
global dict dict_loaded
feedback "loading dictionary..."
if 0==[catch {open /usr/dict/words} file] {
rename set s
foreach w [split [read $file] "
"] {s dict($w) ""}
close $file
rename s set
set dict_loaded 1
feedback "dictionary loaded"
} else
feedback "dictionary missing"
.dict deselect
}
}The weak_password procedure is a hook in which you can put any security measures you like. As written, all it does is reject a word if it appears in the dictionary. The mechanism to look up a word was described earlier.
# put whatever security checks you like in here
proc weak_password {password} {
global dict dict_check
if $dict_check {
feedback "checking password"
if [info exists dict($password)] {
feedback "sorry - in dictionary"
return 1
}
}
return 0
}After entering a password, the password_set procedure is invoked to set the password. The interactive command is extracted from the radio buttons and it is spawned. If the prompt is for an "old password“, the script queries the user for it and then passes it on. The new password is sent as many times as requested without telling the user. (All passwords have to be entered twice. Short passwords have to be entered four times.) Any unrecognized response is passed back to the user.
proc password_set {} {
global password passwd_cmd selection_line
if {$selection_line==0} {
feedback "select a user first"
return
}
set user [lindex [.names get selection.first selection.last] 0]
if [weak_password $password] return
feedback "setting password . . ."
set cmd [lindex $passwd_cmd 0]
spawn -noecho $cmd $user
log_user 0
set last_msg "error in $cmd"
while 1 {
expect {
-nocase "old password:" {
exp_send "[get_old_password]
"
} "assword:" {
exp_send "$password
"
} -re "(.*)
" {
set last_msg $expect_out(1,string)
} eof break
}
}
set status [wait]
if [lindex $status 3]==0 {
feedback "set successfully"
} else {
feedback $last_msg
}
}The script is intended to be run by a superuser. Traditionally, the superuser is not prompted for old passwords so no entry field is permanently dedicated in the display for this purpose. However, in case the user is prompted, a window is popped up to collect the old password. This also handles the case when a non-superuser tries to change their own password. Trying to change any other password will be trapped by the passwd program itself, so the script does not have to worry about it.
The procedure temporarily moves the focus to the popup so the user does not have to move the mouse. After pressing return, the popup goes away and the focus is restored.
proc get_old_password {} {
global old
toplevel .old
label .old.label -text "Old password:"
catch {unset old}
entry .old.entry -textvar old -relief sunken -width 1
pack .old.label
pack .old.entry -fill x -padx 2 -pady 2
bind .old.entry <Return> {destroy .old}
set oldfocus [focus]
focus .old.entry
tkwait visibility .old
grab .old
tkwait window .old
focus $oldfocus
return $old
}Once enough procedures are defined, the script can initialize the user list and radio buttons. Initially, the local password database is selected and displayed without sorting.
.unsorted select .passwd invoke
The remaining effort in the script is in handling user input. The global variable selection_line identifies the user whose password is about to be changed. The make_selection procedure scrolls the user list if the selected user is not displayed. Lastly, the selected user is highlighted.
proc make_selection {} {
global selection_line last_line
.names tag remove selection 0.0 end
# don't let selection go off top of screen
if {$selection_line < 1} {
set selection_line $last_line
} elseif {$selection_line > $last_line} {
set selection_line 1
}
.names yview -pickplace [expr $selection_line-1]
.names tag add selection $selection_line.0
[expr 1+$selection_line].0
}The select_next_nopassword procedure searches through the list of users that do not have passwords. Upon finding one, it is highlighted. The procedure is long because it can search in either direction and can start searching from the middle of the list and loop around if necessary.
proc select_next_nopassword {direction} {
global selection_line last_line
global nopasswords
if 0==[llength $nopasswords] {
feedback "no null passwords"
return
}
if $direction==1 {
# get last element of list
if $selection_line>=[lindex $nopasswords [
expr [llength $nopasswords]-1]] {
set selection_line 0
}
foreach i $nopasswords {
if $selection_line<$i break
}
} else {
if $selection_line<=[lindex $nopasswords 0] {
set selection_line $last_line
}
set j [expr [llength $nopasswords]-1]
for {} {$j>=0} {incr j −1} {
set i [lindex $nopasswords $j]
if $selection_line>$i break
}
}
set selection_line $i
make_selection
}The select procedure is called to determine which user has been clicked on with the mouse. Once it has, it updates selection_line and the display.
proc select {w coords} {
global selection_line
$w mark set insert "@$coords linestart"
$w mark set anchor insert
set first [$w index "anchor linestart"]
set last [$w index "insert lineend + 1c"]
scan $first %d selection_line
$w tag remove selection 0.0 end
$w tag add selection $first $last
}The bindings are straightforward. Mouse button one selects a user. ^C causes the application to exit. In the style of emacs, ^P and ^N move the user up and down by one. Meta-n and meta-p invoke select_next_nopassword to find the next or previous user without a password. These bindings are defined for the entry field in which the new password is entered. Because this field always has the focus, the user can select different users and enter passwords without touching the mouse.
bind Text <1> {select %W %x,%y} bind Entry <Control-c> {exit} bind .password <Control-n> {incr selection_line 1; make_selection} bind .password <Control-p> {incr selection_line −1; make_selection} bind .password <Meta-n> {select_next_nopassword 1} bind .password <Meta-p> {select_next_nopassword −1}
When waiting inside of expect (or interact) commands, the Tk event loop is still active. This means that the user can, for example, press buttons on the screen and Tk will respond to the buttons while the script is executing an expect command.
While any specific action is being executed from the Tk event loop, the original expect command cannot return. If the user presses a button that kicks off, say, another expect command, the original expect command cannot return until the new expect command returns. (This is true for any command in Tk, not just expect commands.)
If both expect commands read from the same spawn id, the later one will see all the buffered data already received. The new expect command can match data that the original expect command had read but not matched.
All of the descriptions so far are identical to the way Expect works without Tk but in the presence of signals. expect commands that are triggered by signal handlers suspend any currently active expect commands.
It is possible to have actions execute whenever input arrives and matches a pattern. This is accomplished using the expect_background command. Patterns declared this way are called background patterns. These patterns can match whenever the Tk event loop is active. This is similar to the way Tk’s bind command works.
Contrast expect_background with expect. Althought only a single expect command can be active at any time, any number of background patterns can be active simultaneously.
The expect_background command takes the same arguments as the expect command. Both commands also handle spawn ids in the same way. So by default, expect_background associates patterns with the current spawn id. Other spawn ids are associated by using the −i flag.
For example, the following command adds any input received from spawn id $shell to the end of the text widget ".text“.
expect_background −i $shell -re ".+" {
.text insert end $expect_out(0,string)
}The expect_background command returns immediately. However, the patterns are remembered by Expect. Whenever any input arrives, it is compared to the patterns. If they match, the corresponding action is executed. The patterns are remembered until another expect_background is entered for the same spawn id. For example, the following command effectively cancels the previous expect_background command:
expect_background −i $shell
Multiple spawn ids and patterns can be provided in a single expect_background command. Each time a particular spawn id appears, it replaces the previous background pattern associated with that spawn id. It is possible to declare multiple spawn ids together and change or delete some or all of them separately.
Multiple spawn ids are accepted using the "-i "$id1 $id2 $id3"" notation or via an indirect spawn id specification (see Chapter 11 (p. 264)). When indirect spawn id lists change, the background patterns are immediately disassociated from the old spawn ids and reassociated with the new spawn ids.
In Chapter 11 (p. 262), I described how the -info flag is used to return the association patterns from expect_before and expect_after. The -info flag works with expect_background as well.
When a background pattern matches, the associated action is evaluated. Evaluation of the action follows the same rules as for a regular expect command. However, inside the action, background patterns for the same spawn id are blocked from further matching. This prevents input that arrived later (i.e., in the middle of an action) from being processed while input associated with the pending action is still being processed.
Any command may be used in the action of a background pattern including another expect or expect_background. expect_background commands allow background patterns from a different spawn id to begin matching immediately—even before the current action finishes. expect commands are executed as usual (i.e., immediately), even if they are for the same spawn id as the one associated with the currently executing background action.
It is not possible to wait using both expect and expect_background for output from the same spawn id at precisely the same time. The behavior in such a situation is undefined.
The following script creates two text widgets that work like primitive terminals. One allows interaction with a telnet process and the other with a shell. The script has a bind command to pass user keystrokes to the processes and an expect_background command to handle the output of the two processes.
Notice that the expect_background command discards
characters since output lines ordinarily end with
but the text widget only expects
as its line terminator. No further intelligence is provided for more sophisticated emulation. For example, absolute cursor motion is not supported. Nonprintable characters appear on the screen as hex escapes.
# start a shell and text widget for its output
spawn $env(SHELL)
set shell $spawn_id
text .shell -relief sunken -bd 1
pack .shell
# start a telnet and a text widget for its output
spawn telnet
set telnet $spawn_id
text .telnet -relief sunken -bd 1
pack .telnet
expect_background {
-i $telnet -re "[^x0d]+" {
.telnet insert end $expect_out(0,string)
.telnet yview -pickplace insert
}
-i $shell -re "[^x0d]+" {
.shell insert end $expect_out(0,string)
.shell yview -pickplace insert
}
-i $any_spawn_id "x0d" {
# discard
}
}
bind Text <Any-Enter> {focus %W}
bind .telnet <Any-KeyPress> {exp_send −i $telnet "%A"}
bind .shell <Any-KeyPress> {exp_send −i $shell "%A"}The previous example was very simple-minded. The characters from the output of the spawned processes were copied to their own text widget. The only attempt at formatting was to handle line endings. Most programs expect more than this. For example, tabs are usually expanded to spaces, and backspaces cause the terminal cursor to move left instead of right.
More sophisticated programs require character addressing. By sending special terminal manipulation character sequences (I will just call them sequences from now on), programs can write to arbitrary character locations on the screen. The following terminal emulator supports this. You can use it to run programs such as emacs and vi.
As before, a text widget is used for display. Its name is stored in the variable term. For simplicity, the code only supports a single emulator, assumes a fixed size display of 24 rows of 80 columns, and runs a shell. The following code starts the process and creates the text widget.
# tkterm - term emulator using Expect and Tk text widget set rows 24 ;# number of rows in term set cols 80 ;# number of columns in term set term .t ;# name of text widget used by term log_user 0 # start a shell and text widget for its output set stty_init "-tabs" eval spawn $env(SHELL) stty rows $rows columns $cols < $spawn_out(slave,name) set term_spawn_id $spawn_id text $term -width $cols -height $rows
Once the terminal widget has been created, it can be displayed on the screen with a pack command. But this is not necessary. You may want to use the terminal widget merely as a convenient data structure in which case it need never be displayed. In contrast, the following line packs the widget on to the screen in the usual way.
pack $term
The task of understanding screen manipulation sequences is complicated. It is made more so by the lack of a standard for it. To make up for this, there are packages that support arbitrary terminal types through the use of a terminal description language. So the script has to declare how it would like to hear terminal manipulation requests. The two common packages that provide this are termcap and terminfo. Because termcap has a BSD heritage and terminfo has a SV heritage, it is not uncommon to find that you need both termcap and terminfo. On my own system as delivered from the vendor, half of the utilities use termcap and half use terminfo!
Surprisingly, it is much easier to design a terminal description from scratch than it is to mimic an existing terminal description. Part of the problem is that terminfo and termcap do not cover all the possibilities nor is their behavior entirely well defined. In addition, most terminals understand a large number of sequences—many more than most databases describe. But because the databases can be different for the same terminal from one computer to another, an emulator must emulate all of the sequences whether they are in the database or not. Even sitting down with a vendor’s manuals is not a solution because other vendors commonly extend other vendor’s definitions.
Fortunately, few sequences are actually required. For instance, most cursor motion can be simulated with direct addressing. This turns out to be more efficient than many relative cursor motion operations as I will explain later.
The following code establishes descriptions in both termcap and terminfo style using the terminal type of "tk“. The code succeeds even if termcap and terminfo are not supported on the system. This code actually has to be executed before the spawn shown earlier in order for the environment variables to be inherited by the process.
I will briefly describe the termcap definition. (The terminfo definition is very similar so I will skip those.) The definition is made up of several capabilities. Each capability describes one feature of the terminal. A capability is expressed in the form xx=value, where xx is a capability label and value is the actual string that the emulator receives. For instance the up capability moves the cursor up one line. Its value is the sequence: escape, "[“, "A“. These sequences are not interpreted at all by Tcl so they may look peculiar. The complicated-looking sequence (cm) performs absolute cursor motion. The row and column are substituted for each %d before it is transmitted. The remaining capabilities are nondestructive space (nd), clear screen (cl), down one line (do), begin standout mode (so) and end standout mode (se).
set env(LINES) $rows
set env(COLUMNS) $cols
set env(TERM) "tk"
set env(TERMCAP) {tk:
:cm=E[%d;%dH:
:up=E[A:
:nd=E[C:
:cl=E[HE[J:
:do=^J:
:so=E[7m:
:se=E[m:
}
set env(TERMINFO) /tmp
set ttsrc "/tmp/tk.src"
set file [open $tksrc w]
puts $file {tk,
cup=E[%p1%d;%p2%dH,
cuu1=E[A,
cuf1=E[C,
clear=E[HE[J,
ind=
,
cr=
,
smso=E[7m,
rmso=E[m,
}
close $file
catch {exec tic $tksrc}
exec rm $tksrcFor simplicity, the emulator only understands the generic standout mode rather than specific ones such as underlining and highlighting. The global variable term_standout describes whether characters are being written in standout mode. Text in standout mode is tagged with the tag standout, here defined by white characters on a black background.
set term_standout 0 ;# if in standout mode
$term tag configure standout
-background black
-foreground whiteThe text widget maintains the terminal display internally. It can be read or written in a few different ways. Access is possible by character, by line, or by the entire screen. Lines are newline delimited. It is convenient to initialize the entire screen (i.e., each line) with blanks. Later, this will allow characters to be inserted anywhere without worrying if the line is long enough already. In the following procedure, term_init, the "insert $i.0" operation adds a line of blanks to row i beginning at column 0.
proc term_init {} {
set blankline [format %*s $cols ""]
for {set i 1} {$i <= $rows} {incr i} {
$term insert $i.0 $blankline
}For historical reasons, the first row in a text widget is 1 while the first column is 0. The variables cur_row and cur_col describe where characters are next written. Here, they are initialized to the upper-left corner.
set cur_row 1
set cur_col 0The visible insertion cursor is maintained as a mark. It generally tracks the insertion point. Here, it is also set to the upper-left corner.
$term mark set insert $cur_row.$cur_col } term_init
The term_init procedure is called immediately to initialize the text widget.
A few more utility routines are useful. The term_clear procedure clears the screen by throwing away the contents of the text widget and reinitializing it.
proc term_clear {} {
global term
$term delete 1.0 end
term_init
}The term_down procedure moves the cursor down one line. If the cursor is already at the end of the screen, the text widget appears to scroll. This is accomplished by deleting the first line and then creating a new one at the end.
proc term_down {} {
global cur_row rows cols term
if {$cur_row < $rows} {
incr cur_row
} else {
# already at last line of term, so scroll screen up
$term delete 1.0 "1.end + 1 chars"
# recreate line at end
$term insert end [format %*s $cols ""]
}
}There is no correspondingly complex routine to scroll up because the termcap/terminfo libraries never request it. Instead, they simulate it with other capabilities. In fact, the termcap/terminfo libraries never request that the cursor scroll past the bottom line either. However, programs like cat and ls do, so the terminal emulator understands how to handle this case.
The term_insert procedure writes a string to the current location on the screen. It is broken into three parts. The first part writes from anywhere on a line up to the end. If the string is long enough and wraps over several lines, the next section writes the full lines that wrap. Finally, the last section handles the last characters that do not make a full line. Characters are tagged with the standout tag if the emulator is in standout mode.
Each one of these sections does its work by first deleting the existing characters and then inserting the new characters. This is a good example of where termcap/terminfo fail to have the ability to adequately describe a terminal. The text widget is essentially always in “insert” mode but termcap/terminfo have no way of describing this.
One capability of which the script does not take advantage, is that termcap/terminfo can be told not to write across line boundaries. On that basis, this procedure could be simplified by removing the second and third parts. Again, however, programs such as cat and ls expect to be able to write over line boundaries. The term_insert procedure does not worry about scrolling once the bottom of the screen is reached. term_down takes care of that already.
proc term_insert {s} {
global cols cur_col cur_row
global term term_standout
set chars_rem_to_write [string length $s]
set space_rem_on_line [expr $cols - $cur_col]
if {$term_standout} {
set tag_action "add"
} else {
set tag_action "remove"
}
##################
# write first line
##################
if {$chars_rem_to_write > $space_rem_on_line} {
set chars_to_write $space_rem_on_line
set newline 1
} else {
set chars_to_write $chars_rem_to_write
set newline 0
}
$term delete $cur_row.$cur_col
$cur_row.[expr $cur_col + $chars_to_write]
$term insert $cur_row.$cur_col [
string range $s 0 [expr $space_rem_on_line-1]
]
$term tag $tag_action standout $cur_row.$cur_col
$cur_row.[expr $cur_col + $chars_to_write]
# discard first line already written
incr chars_rem_to_write -$chars_to_write
set s [string range $s $chars_to_write end]
# update cur_col
incr cur_col $chars_to_write
# update cur_row
if $newline {
term_down
}
##################
# write full lines
##################
while {$chars_rem_to_write >= $cols} {
$term delete $cur_row.0 $cur_row.end
$term insert $cur_row.0 [string range $s 0 [expr $cols-1]]
$term tag $tag_action standout $cur_row.0 $cur_row.end
# discard line from buffer
set s [string range $s $cols end]
incr chars_rem_to_write -$cols
set cur_col 0
term_down
}
#################
# write last line
#################
if {$chars_rem_to_write} {
$term delete $cur_row.0 $cur_row.$chars_rem_to_write
$term insert $cur_row.0 $s
$term tag $tag_action standout $cur_row.0
$cur_row.$chars_rem_to_write
set cur_col $chars_rem_to_write
}
term_chars_changed
}At the very end of term_insert is a call to term_chars_changed. This is a user-defined procedure called whenever visible characters have changed. For example, if you want to find when the string foo appears on line 4, you could write:
proc term_chars_changed {} {
global $term
if {[string match *foo* [$term get 4.0 4.end]]} . . .
}Some other tests suitable for the body of term_chars_changed are:
# Test if "foo" exists at line 4 col 7
if {[string match foo* [$term get 4.7 4.end]]}
# Test if character at row 4 col 5 is in standout mode
if {-1 != [lsearch [$term tag names 4.5] standout]} ...
You can also retrieve information:
# Return contents of screen $term get 1.0 end # Return indices of first string on lines 4 to 6 that are # in standout mode $term tag nextrange standout 4.0 6.end
And here is possible code to modify the text on the screen:
# Replace all occurrences of "foo" with "bar" on screen
for {set i 1} {$i<=$rows} {incr i} {
regsub -all "foo" [$term get $i.0 $i.end] "bar" x
$term delete $i.0 $i.end
$term insert $i.0 $x
}
The last utility procedure is term_update_cursor. It is called to update the visible cursor.
proc term_update_cursor {} {
global cur_row cur_col term
$term mark set insert $cur_row.$cur_col
term_cursor_changed
}The term_update_cursor procedure also calls a user-defined procedure, term_cursor_changed. A possible definition might be to test if the cursor is at some specific location:
proc term_cursor_changed {} {
if {$cur_row == 1 && $cur_col == 0} ...
}
By default, both procedures do nothing:
proc term_cursor_changed {} {}
proc term_chars_changed {} {}
term_exit is another user-defined procedure. term_exit is called when the spawned process exits. Here is a definition that causes the script itself to exit when the process does.
proc term_exit {} {
exit
}The last user-defined procedure is term_bell. term_bell is executed when the terminal emulator needs its bell rung. The following definition sends an ASCII bell character to the standard output.
proc term_bell {} {
send_user "a"
}Now that all of the utility procedures are in place, the command to read the sequences is straightforward. For instance, a backspace character causes the current column to be decremented. A carriage-return sets the current column to 0. Compare this to the code on page 442.
Notice how simple the code is for absolute cursor motion. It is basically two assignment statements. Because it is so simple, there is no need to supply termcap/terminfo with information on relative cursor motion commands. They cannot be substantially faster.[68]
expect_background {
-i $term_spawn_id
-re "^[^x01-x1f]+" {
# Text
term_insert $expect_out(0,string)
term_update_cursor
} "^
" {
# (cr,) Go to beginning of line
set cur_col 0
term_update_cursor
} "^
" {
# (ind,do) Move cursor down one line
term_down
term_update_cursor
} "^" {
# Backspace nondestructively
incr cur_col −1
term_update_cursor
} "^a" {
term_bell
} eof {
term_exit
} "^x1b\[A" {
# (cuu1,up) Move cursor up one line
incr cur_row −1
term_update_cursor
} "^x1b\[C" {
# (cuf1,nd) Nondestructive space
incr cur_col
term_update_cursor
} -re "^x1b\[([0-9]*);([0-9]*)H" {
# (cup,cm) Move to row y col x
set cur_row [expr $expect_out(1,string)+1]
set cur_col $expect_out(2,string)
term_update_cursor
} "^x1b\[Hx1b\[J" {
# (clear,cl) Clear screen
term_clear
term_update_cursor
} "^x1b\[7m" {
# (smso,so) Begin standout mode
set term_standout 1
} "^x1b\[m" {
# (rmso,se) End standout mode
set term_standout 0
}
}Finally, some bindings are provided. The meta key is simulated by sending an escape. Most programs understand this convention, and it is convenient because it works over telnet links.
bind $term <Any-Enter> {
focus %W
}
bind $term <Meta-KeyPress> {
if {"%A" != ""} {
exp_send −i $term_spawn_id " 33%A"
}
}
bind $term <Any-KeyPress> {
if {"%A" != ""} {
exp_send −i $term_spawn_id -- "%A"
}
}Some bindings can be described using capabilities. For instance, the capability for function key 1 could be described in either of two ways:
:k1=EOP: termcap-style :kf1=EOP: terminfo-style
bind $term <F1> {exp_send −i $term_spawn_id " 33OP"}
This book describes a version of Expect that does not provide built-in support for understanding character graphics. Nonetheless, it is possible to use the terminal emulator in the previous section to partially or fully automate character-graphic applications.
For instance, each expect-like operation could be a loop that repeatedly performs various tests of interest on the text widget contents. In the following code, the entrance to the loop is protected by "tkwait var test_pats“. This blocks the loop from proceeding until the test_pats variable is changed. The variable is changed by the term_chars_changed procedure, invoked whenever the screen changes. Using this idea, the following code waits for a % prompt anywhere on the first line:
proc term_chars_changed {} {
uplevel #0 set test_pats 1
}
while 1 {
if {!$test_pats} {tkwait var test_pats}
set test_pats 0
if {[regexp "%" [$term get 1.0 1.end]]} break
}
Writing a substantial script this way would be clumsy. Furthermore, it prevents the use of control flow commands in the actions. One solution is to create a procedure that does all of the work handling the semaphore and hiding the while loop.
Based on a procedure (shown later) called term_expect, the rogue script in Chapter 6 (p. 139) can be rewritten with the following code. This code is similar to the earlier version except that instead of patterns, tests are composed of explicit statements. Any nonzero result causes term_expect to be satisfied whereupon it executes the associated action. For instance, the first test looks for % in either the first or second line on the screen. The meaning of the rest of the script should be obvious.
while 1 {
term_expect {regexp "%" [$term get 1.0 2.end]}
exp_send "rogue
"
term_expect
{regexp "Str: 18" [$term get 24.0 24.end]} {
break
} {regexp "Str: 16" [$term get 24.0 24.end]}
exp_send "Q"
term_expect {regexp "quit" [$term get 1.0 1.end]}
exp_send "y"
}In contrast to the original rogue script, there is no interact command at the end of this one. Because of the bindings, the script is always listening to the keyboard! To prevent this implicit interact, remove or override the KeyPress bindings that appear at the end of the terminal emulator.
Since the tests can be arbitrarily large lists of statements, they are grouped with braces. For example:
term_expect { set line [$term get 1.0 2.end] regexp "%" $line } { action } timeout { puts "timed out!" }
Timeouts follow a similar syntax as before. A test for an eof is not provided since a terminal emulator should not exit just because the applications making use of it do so. In this example, a shell prompt is used to detect when the rogue program has exited.
The term_expect procedure lacks some of the niceties of expect and should be viewed as a framework for designing a built-in command. Feel free to modify it. Your experiences will help in the ultimate design of a built-in command.
An implementation of term_expect is shown in this section. The code is quite complex and really beyond the level at which this book is aimed. Fortunately, it is not necessary to understand in order to use it. Nonetheless, I will briefly describe how it works. If you follow it all, you are doing very well indeed.
The code assumes that the terminal emulator is available because the text widget maintains the memory of what is on the screen. Although the terminal emulator is necessary, the text widget and, indeed, Tk itself can be obviated by maintaining an explicit representation such as a list of strings representing rows of the terminal. However, even with Tk and the terminal emulator, the timeout and the scope handling makes the code intricate. Without them, the code would be more similar to the fragment on page 452.
Timeouts are implemented using an after command which sets a strobe at the end of the timeout period. In order to avoid an old after command setting the strobe for a later term_expect command, a new strobe variable is generated each time.[69] A global variable provides a unique identifier for this purpose and is initialized separately.
set term_counter 0 ;# distinguish different timers
The procedure begins by deciding the amount of time to wait before timing out. This is rather involved because it looks in the local scope and the global scope as well as providing a default value. This imitates the behavior of the real expect command.
proc term_expect {args} {
set timeout [
uplevel {
if [info exists timeout] {
set timeout
} else {
uplevel #0 {
if {[info exists timeout]} {
set timeout
} else {
expr 10
}
}
}
}
]Two unique global variables are used as strobes—to indicate that an event (data or timeout) has occurred. The strobe variable holds the name of a global variable changed when the terminal changes or the code has timed out. Later, the code will wait for this variable to change. To distinguish between the two types of events, tstrobe is another strobe changed only upon timeout. (It is possible to use a single tri-valued strobe, but the coding is much trickier.)
global term_counter
incr term_counter
global [set strobe _data_[set term_counter]]
global [set tstrobe _timer_[set term_counter]]The term_chars_changed procedure is modified to fire the strobe. Note the use of double quotes around the body of term_chars_changed in order to allow substitution of the strobe command in this scope.
proc term_chars_changed {} "uplevel #0 set $strobe 1"The next lines set the strobes to make sure that the screen image can be tested immediately since the screen could initially be in the expected state. The after command arranges for the timer strobe to be set later.
set $strobe 1 ;# force an initial test
set $tstrobe 0 ;# no timeout yet
if {$timeout >= 0} {
set mstimeout [expr 1000*$timeout]
after $mstimeout "set $strobe 1; set $tstrobe 1"
set timeout_act {}
}If the user omits the final action, the number of arguments will be uneven. Later code is simplified by adding an empty action in this case.
set argc [llength $args]
if {$argc%2 == 1} {
lappend args {}
incr argc
}If the test is the bare string "timeout“, its action is saved for later. Both the string and the action are removed from the list of tests.
for {set i 0} {$i<$argc} {incr i 2} {
set act_index [expr $i+1]
if {![string compare timeout [lindex $args $i]]} {
set timeout_act [lindex $args $act_index]
set args [lreplace $args $i $act_index]
incr argc -2
break
}
}Now the procedure loops, waiting for the screen to be changed. A test first checks if the strobe has already occurred. If not, tkwait waits. This suspends the loop when no screen activity is occurring. Once the strobe occurs, the rest of the loop executes. If the timeout has occurred or any of the tests are true, the loop breaks so that the action can be evaluated.
while {![info exists act]} {
if {![set $strobe]} {
tkwait var $strobe
}
set $strobe 0
if {[set $tstrobe]} {
set act $timeout_act
} else {
for {set i 0} {$i<$argc} {incr i 2} {
if {[uplevel [lindex $args $i]]} {
set act [lindex $args [incr i]]
break
}
}
}
}To keep the environment clean, the global strobe variables are deleted. If a timeout could occur in the future, the unset is similarly scheduled; otherwise the variables are deleted immediately. The term_chars_changed procedure is reset so that it does not continue setting the data strobe.
proc term_chars_changed {} {}
if {$timeout >= 0} {
after $mstimeout unset $strobe $tstrobe
} else {
unset $strobe $tstrobe
}Finally, the action is evaluated. If a flow control command (such as break) was executed, it is returned in such a way that the caller sees it as well. (See the Tcl manual for more detail on this.)
set code [catch {uplevel $act} string] if {$code > 4} {return -code $code $string} if {$code == 4} {return -code continue} if {$code == 3} {return -code break} if {$code == 2} {return -code return} if {$code == 1} {return -code error -errorinfo $errorInfo -errorcode $errorCode $string} return $string }
Add scroll bars to the terminal emulator on page 442. Make it allow for resizeable text widgets.
The terminal emulator is based on an ANSI terminal. Change the emulator so that it emulates a particular terminal that is not ANSI-conforming. This could be useful if you have to interact with a program or service that is hardwired for a terminal type. Make the emulator understand any type of terminal.
Write a version of the UNIX
scriptcommand that automatically strips out any character graphics as the output is logged to a file.Modify the
term_expectprocedure on page 453 so that it does not require Tk. Use a list of character strings to emulate a Tk text widget. Then try it with an array. Which is faster? Is this what you had expected?Expand on the previous exercise, by emulating multiple terminals. Provide “hotkeys” so that you can switch between different terminal sessions in a single keystroke.
Write a script for browsing through the archives of the
comp.lang.tclnewsgroup. Display the subjects in a scrollable window, allowing them to be ordered by date, subject, or author. Upon selection, download the posting and display it.Modify the script from the previous exercise so that postings may be saved locally or cached so that the script does not have to
ftpthem again if they are selected.Modify the
tkpasswdscript so that it rejects passwords containing fewer than two digits and two alphabetic characters, one uppercase and one lowercase. Use exercise 4 on page 160.The
tkpasswdscript echoes passwords as you type them. There is little point intkpasswdobscuring the password. However, obscuring the password is useful in other scripts. A secure way to accomplish this is to use an entry widget that displays a string of asterisks. Most keypresses add an additional asterisk, whereas keys such as backspace remove an asterisk. Keypresses also add or subtract from a separate variable that is not displayed but contains the real password. Create the bindings necessary to implement this idea.
[66] Despite my enthusiasm for Tk, it is no longer the only game in town. New interpreted window systems are popping up everywhere. The good news is that many are incorporating Expect-like functionality within them. As an example, Neils Mayer’s WINTERP is similar to Tcl and Tk but is based on David Betz’s XLISP and OSF’s Motif. WINTERP includes the Expect library described in Chapter 21 (p. 485).
[67] I will not talk about the function of Tk’s send command until Chapter 18 (p. 405).
[68] The definition for nondestructive space might be seen as a concession to speed, but in fact it is required by some buggy versions of termcap which operate incorrectly if the capability not defined. The other relative motion capabilities are assumed by the terminal driver for non-character-graphic tools such as cat and ls.
[69] Tk 4 promises to provide support for cancelling after commands. This would remove the need for separate strobe variables.