A.1.1 Experimental Design

The experiment was designed to test two independent variables: two user interface styles (avatars-only versus avatars+list) and the number of visitors in the welcome area. Group size was set at 4, 8, or 16 to cover a reasonable spectrum of sizes. The number of exhibits in the welcome area was fixed at 10.

A number of complementary dependent variables were defined (Table A.2). One concern was conversation fluency: How quickly would participants be able to respond to questions directed at them by other visitors? This variable was defined as the time between the appearance of a question directed to a participant and his or her initiation of a typed reply.

Table A.2 Dependent variables and hypotheses for the avatar experiment.

A second concern was group memory: Would participants be able to keep track of who had been part of the ongoing conversation? We measured this variable through a memory test administered at the end of the study, in which participants indicated for a list of 20 user names whether each individual had participated in the ongoing conversation (half of the people had, half had not, but all were present at one time or another).

Two other dependent variables were defined to measure subjective outcomes. Three Likert scales were developed (Figure A.2). The first questioned participants’ feelings of awareness of others, the second asked whether they felt their interactions with others to be direct, and the third measured confusion about conversation partners. The final dependent variable was a simple yes-no judgment, indicating whether the user would feel comfortable initiating a conversation in the welcome area.

Figure A.2 Likert scales assessing three aspects of satisfaction with two user interfaces.

Hypotheses were developed for each combination of independent and dependent variables. Based on our formative evaluations, we expected that:

• Conversation fluency and memory will be improved for avatars+list. The list provides redundant information about who is there, and we expected this to aid users in keeping track of what was happening and in remembering what happened afterward.
• Subjective reactions will be more positive for the avatars alone. Although the list provides useful information, it does so by pulling attention away from the overall scene, thereby reducing levels of awareness and engagement and increasing confusion.
• Both of the above effects will vary with the size of the group. Specifically, the facilitating effects of the visitor list will be stronger for the larger group sizes, and the preference for the avatars will be stronger for the smaller group sizes.
• For the yes-no question, there will be a preference for the avatars+list, because the combination offers more options for conversation management.

Participants were randomly assigned to sample groups in a mixed design: The (visitor) group-size variable was manipulated as a repeated measure (i.e., all users were exposed to groups of all three sizes), while user interface style was between-groups (i.e., two independent groups of test users worked with the two versions of the software). A total of 24 participants were recruited, with 6 men and 6 women assigned to each group.

A. 1.2 Procedures

Participants first read and signed an informed consent form (similar to the one shown in Figure 7.6). They completed a user background survey that assessed their general computer background, their experience with virtual environments in particular, and their experience with science fairs (similar to that in Figure 7.7). Each participant then read printed instructions that described the tasks they were about to carry out: The instructions included a screen shot of the welcome area with 10 exhibits and 4 visitors. The different elements of the screen were described with reference to this screen.

The participants were told that they would be playing the role of Alicia, a community member visiting the fair for the first time. Other visitors would be arriving and leaving during her visit. Alicia’s main goal is to explore the 10 exhibits. However, the participants were also told that on occasion another visitor will direct a question to Alicia, and they should respond to these questions as quickly and accurately as possible.

The experimental trials then proceeded. Over a period of about 10 minutes, a total of 30 other visitors arrived at the fair; some moved on while others stayed and chatted. During this time, Alicia was asked to respond to three questions. The order and precise timing of the questions was determined quasi-randomly for each participant. One question appeared when 3 other people were present, one when 7 others were present, and one when 15 other visitors were in the welcome area.

The “other visitors” who arrived, chatted, and directed questions to Alicia were simulated by the computer; the questions asked were balanced for length and complexity. This control of the simulated visitors and their questions was an important feature of the experiment design. It eliminated any possible confound due to personality or conversation skill of real visitors. Response times were collected automatically; the clock was started when a question was displayed and stopped as soon as the test participant typed a character in the input area of the chat tool. Participants’ responses to the questions were recorded for later review.

Following the science fair activity, the participants first completed a user reaction questionnaire that included the three Likert scales, along with more general comments and suggestions for improvement. They then completed a memory test of conversation partners that indicated whether each of 20 names had participated in the conversation at the fair; no advance warning was given of this test.

A. 1.3 Results and Analysis

The effects of group size and user interface style on conversation fluency were assessed with a two-factor, repeated-measures ANOVA. The raw data appear in Table A.3, and the means for each test condition are graphed in Figure A.3. The ANOVA revealed a main effect of user interface style (F(1,66) = 10.68, p < 0.01), with the avatar+list interface producing faster times overall. There was also a main effect of group size, with response times increasing as the number of visitors increased (F(2,66) = 42.02, p < 0.001). However, as the graph conveys, the analysis also revealed the predicted interaction of user interface style with group size (F(2,66) = 11.12, p < 0.001). The advantage for the avatar+list interface was significant only for the largest group size.

Table A.3 Raw data for response times to answer question directed to test participants. Note that group size is a repeated measure, and user interface style is between-subjects.

Figure A.3 Average time to respond to questions in conditions with varying numbers of other visitors and different user interface styles.

The raw data for percentage correct on the memory test appear in Table A.4. As the difference in means suggests, the participants using the avatars+list user interface were more accurate in identifying the names of visitors who had participated in the ongoing conversation. A one-tailed t-test for independent samples confirms this conclusion: t (22) = 1.78, p < 0.05.

Table A.1 Percentage correct on conversation partner identification for participants using the two user interfaces. Scores are percentage correct for 20 names.

Participants’ responses to the rating scales appear in Table A.5, broken down by user interface style and the reaction assessed by the rating scale (see Figure A.2). Judgments have been converted to an ordinal scale (from 1 = Strongly Disagree to 5 = Strongly Agree). Note that for the third scale, the assertion was worded negatively—a lower rating reflects a more positive response. The mean differences in the answers to questions 1 and 2 are consistent with the predicted preference for avatars alone. However, the effect was reversed for the confusion scale, with participants using the avatars alone reporting a greater sense of confusion.

Table A.5 Judgments made on satisfaction rating scales; judgments have been converted to an ordinal scale (1 = Strongly Disagree to 5 = Strongly Agree).

Because the rating scale data are ordinal in nature, the data were analyzed with the nonparametric Wilcoxon U statistic. The rank of each rating across the two user interface conditions was calculated for each of the three scales. The statistic calculates an estimate of what the average rank for each group would be by chance alone, and tests to see if the average rank obtained is different from the estimate. The Wilcoxon U for the avatar-only condition was 20 for the awareness scale, 33 for the directness scale, and 18 for the confusion scale. All of these differences are significant at p < 0.05.

The final analysis examined participants’ response to whether they would choose to initiate a conversation in the welcome area. Answers to this question are tabulated in Table A.6. These categorical data were analyzed with a chi-square test, which compares the observed frequency in each cell with the frequency expected by chance. The expected frequency is estimated from the margin values (e.g., the expected frequency of the upper left cell is equal to 12/24*14, or 7). This analysis revealed that there was no significant relationship between user interface style and likelihood of initiating a conversation (X²(1) = 0.38, ns).

Table A.6 Frequency of responses regarding likelihood of initiating a conversation in the welcome area of the virtual science fair.

A. 1.4 Drawing Conclusions

The findings of the comparative study confirmed most of the experimental hypotheses. Participants using the avatars-only version of the user interface felt greater awareness and directness in their conversations. However, they also seemed to experience greater confusion, perhaps because the speech bubbles are distributed around the “room,” and it is difficult to keep track of the entire population of visitors and of what is being said. In the avatars+list condition, the list is a stable indicator of who is present and more attention can be focused on the content of the speech bubbles.

The performance data are also generally consistent with the hypotheses: Response times and vistor group memory were better in general for the users working with the avatars+list interface. Importantly, however, the response time advantage was present only for the largest visitor group size. In combination with the satisfaction results, this suggests that the issue is more complex than deciding whether or not to provide the secondary list view. In most cases, users will prefer avatars alone and perform just as well without the extra list support. Only when the visitor group size is large is a redundant information display needed. Display of a visitor list might thus be made contingent on the current visitor group size, or perhaps as an option under user control.

The findings regarding conversation initiation contribute little information. It seems likely that other factors are more important in determining users’ conversation behavior (e.g., who is visiting and users’ general personality). Despite the lack of a difference between test conditions, this result should be seen as good news, in that the choice of user interface will not bias people one way or the other in taking conversational initiative.