CHAPTER 23

TEACHING COMPUTER SCIENTISTS TO MAKE USE

Mary Beth Rosson, John M. Carroll, and Con Rodi

School of Information Sciences and Technology, Pennsylvania State University, USA

CHALLENGES IN TEACHING STUDENTS TO MAKE USE

Design is an ill-structured problem, and the art of design is a complex skill regardless of the domain (Simon 1972). Designing software systems with use in mind is no exception. Starting conditions and design options are never completely specified; any given design problem has many possible solutions that present a variety of competing trade-offs (Brooks 1995, Reitman 1965). To recognize and address these trade-offs, designers must recruit many different perspectives and values, meaning that an individual designer rarely holds all the expertise relevant to a design problem. For interactive software system design, the impact that a designed artifact will have on people—both at an individual level and at a cultural or societal level—are an essential and pervasive concern that few software designers are equipped to consider.

When human–computer interaction (HCI) first emerged as a scientific discipline, it borrowed heavily from existing theories of psychology (Carroll 1997). Attention focused on the mechanisms of perception and attention, interpretation of visual displays, and relatively low-level planning and actions (Card, Moran and Newell 1983, Carroll and Thomas 1982, Norman 1996). More recently, HCI has developed into a highly interdisciplinary field, integrating the concerns and methods of disciplines as diverse as anthropology, sociology, artificial intelligence, ethics, and policy analysis (Carroll 2000).

Through the past two decades, HCI has also increasingly focused on science-based design, integrating its interdisciplinary scientific roots with the engineering goal of improving the usability of computer systems and applications. Thus the practice of HCI is often referred to as usability engineering (Nielsen 1993, Mayhew 1999, Rosson and Carroll 2002), pointing to the important role of establishing usability targets early in design, and using evaluation results to iterate toward these targets.

These twin bodies of knowledge—diverse perspectives on humans and their use of information technology, and user-centered methods for analyzing and ensuring the design of effective systems—present a major challenge for undergraduate education. Many American universities are just beginning to integrate HCI within their curricula and offer students a single introductory course. Educators may provide a comprehensive survey of the field, ensuring that students encounter a large number of concepts and theories relevant to HCI, but at a fairly high level. An alternative—and the one that we have adopted in our own teaching—is to provide a more eclectic survey of HCI topics and methods, but ensure that exposure is deep enough that students can be expected to apply it as they continue their education and professional work.

COMPARISONS – SCENARIOS FOR MAKING USE

Scenario-Based Design (SBD) methods have become pervasive in HCI research and practice (Carroll 1995, 2000, Cooper 1999, Rosson and Carroll 2002). Scenarios are increasingly recognized as a promising representation for raising and integrating the concerns of diverse stakeholders (Benyon and Macaulay, Chapter 11 in this volume; Carroll 1995; Holtzblatt, Chapter 10 in this volume; Rosson and Carroll 2000).

Scenarios are an excellent medium for teaching a user-centered perspective on design. They are concrete stories, which make them easy for HCI novices to develop and share; at the same time are flexible, quickly elaborated, or revised. A scenario can be rendered in many forms, from a lightweight text description to an elaborate and realistic scenario-machine (Kyng 1995, Nielsen 1995, Rosson and Carroll 2002; see also Chapter 1 of this volume). Because scenarios are stories about use, they maintain a central focus on use as the goal of design. Scenarios narrate one more users' experience during an activity, which helps designers to empathize with and analyze the factors that are causing different aspects of the usage experience.

In thinking about how best to teach SBD to computer scientists, we began with the software-engineering perspective that pervades computer science curricula—software development is largely about writing code, and writing code involves an iterative cycle of design and implementation, with testing and debugging guiding progressive refinement of the software. To support our education goals, we developed a usability engineering life cycle for SBD that elaborates and systematizes the task-artifact cycle (Carroll, Kellogg and Rosson 1991, Carroll and Rosson 1992). The task-artifact cycle describes how usage analyses and visions continually point to new opportunities for system design, while users' reactions to and adaptations of software systems establish new forms of activity.

In scenario-based usability engineering, scenarios represent users' activities and a broad range of scenario analysis and design techniques are used to guide the transition from use to artifact and back again (Figure 23.1). The critical step from current practice to design is inspired by metaphor, available technologies, and theory, but design possibilities are also considered in a more systematic fashion by examining associated design rationale—the likely consequences a specific design move might have for users (Moran and Carroll, 1996). When one or more scenarios are implemented as a system, the system in use is analyzed in terms of the new scenarios it now supports or evokes. These new scenarios describe how the artifact is used but are grounded in a rich context of system stakeholders, their needs, and their evolving work practices.

The usability engineering life cycle we teach to undergraduates is similar in many respects to the software engineering perspective conveyed by Boehm's spiral model (1988). Prototyping and testing of ideas is a crucial element at all phases of the design, with many opportunities to revisit and refine earlier analyses and design products. However, we teach students usability engineering methods that emphasize a forward progression of scenario representations and associated design rationale. They work with scenarios that originate in the problem domain and learn to continuously analyze and incrementally transform these scenarios into a design that better meet the needs and preferences of system stakeholders.

A complete description of the scenario-based process we teach to students is beyond the scope of this chapter, but the diagram in Figure 23.2 summarizes key concepts (for more detailed descriptions, see Rosson and Carroll 2002, 2003). Problem scenarios are written to synthesize observations and other field data gathered during requirements analysis. These scenarios are iteratively refined through claims analysis, which identifies features of current practice that have important positive and negative consequences for users, particularly consequences that may be addressed by information technology. During conceptual design, the problem scenarios are transformed into activity scenarios that envision future practices; this crucial design step is guided by analyses of existing technologies, conceptual metaphors, user profiles, and HCI theory and guidelines. The trade-offs documented through claims analysis also play a critical role, as designers seek to enhance the design's positive impacts on usage and minimize its negative impacts.

FIGURE 23.1 Scenarios as a central representation in the task-artifact cycle

FIGURE 23.2 Phases in scenario-based usability engineering

During detailed design, the activity scenarios (which are deliberately neutral with respect to technology details) are progressively elaborated as information scenarios and later as interaction scenarios, that specify in detail how technology will be used to present and manipulate the software objects and procedures needed to support the envisioned activity. Again, claims analyses are developed in a continuing fashion to identify and reason about important trade-offs in the emerging design. As soon as possible, implementation and testing begins, supported by a range of low- and high-fidelity prototypes that instantiate some or all of the design scenarios. The evaluation materials and criteria are based on the scenarios and claims analyses created during design.

With respect to undergraduate education, placing SBD within a usability engineering process of this sort accomplishes several aims,

• Articulates the ideas and methods of SBD by decomposing the overall process into phases;
• Assumes and leverages students' prior knowledge of methods for phased software development; and
• Offers a concrete SBD process model, including analysis and design techniques and representations, for completing a class project in several phases.

Importantly, the phased approach also provides a natural structure within which to introduce HCI concepts and theories. For example, we teach students about the subtleties of work practice, tacit and explicit knowledge, artifact analysis, and so on, during the problem scenario analysis. Mental models and metaphorical reasoning are covered during activity design, while the psychological process of perception, interpretation, learning, and action are considered during information and interaction design.

USING CASES TO TEACH SCENARIO-BASED USABILITY ENGINEERING

All HCI educators recognize the crucial role of realistic examples and projects in teaching the concepts and methods of user-centered design (Hewittet al. 1990, Stronget al. 1994). However, it is a constant challenge to meet these pedagogical goals—many convenient examples are generic or piecemeal, and projects are often simplified to fit within the time constraints of a conventional course (e.g., three hours of credit over a 15-week period). Given our scenario-based usability engineering, we also have a requirement to introduce (and practice) an SBD life-cycle approach while simultaneously conveying the aspects of HCI most critical for supporting students' analysis and design projects. A case study is a narrative description of a specific and realistic instance of a process. Case studies invariably evoke trade-offs—often trade-offs that have no simple solution—and thus are an excellent vehicle for illustrating complex processes like usability engineering.

Why Case-Based?

Case-based learning enjoys status as favored pedagogy in many disciplines, but is most firmly entrenched and adopted in professional schools. Many law, business, and medical schools use cases to scaffold the experience of students as they acquire discipline-specific skills and expertise. The transition from neophyte to practitioner is not simple, and in many ways learners' exposure to “real world” issues through case analysis and discussion provides a surrogate for the experience set necessary to make expert-level decisions and judgments. The classroom becomes more than a place to learn facts and theories; it becomes a sort of simulated world in which students learn about “being in the trenches” without having to go there first.

What is it about cases and case-based learning that allows educators achieve such lofty ambitions? To understand this, it's important to fix a few characteristics of the cases themselves. While cases may appear to be isolated examples extracted from specific experiences, they are, in fact, interconnected through the principles they exemplify. To present a case argues that the present example is an instance of a broader category (Shulman 1992). A case study integrates the concepts and criteria for making decisions with the decision-making process itself. Thus we are able to construct “pedagogical case studies” for SBD that illustrate how the theories and concepts of HCI (e.g., visual perception, planning for action) can be applied through the methods of usability engineering (e.g., prototyping and user testing).

Case-based learning involves learning by doing, the development of analytical and decision-making skills, internalization of learning, and learning how to grapple with messy real-life problems. Indeed it is the messy and real-world character of case studies that leads to one their greatest strengths—providing persuasive evidence to novice designers that trade-offs are inevitable and pervasive in design, that there is never a single “right” answer, and that being able to compare the strengths and weaknesses of different solutions is evidence of a successful design process.

The Usability Case Study Library

In the course of developing a textbook for our usability engineering course, we documented a case study from our own development work—a virtual science fair within the MOOsburg environment, where students could exhibit their science projects and family and friends could visit, ask questions, and so on (Carroll 2000). In developing this textbook example, we were careful to document the case completely, from problem scenario analysis through prototyping and evaluation (see Chapters 2–8 in Rosson and Carroll 2002). However, from the start we also intended to complement this case with others drawn from commercial organizations; we obtained support from the U.S. National Science Foundation to develop a prototype library and browser for several usability case studies.

Table 23.1 summarizes three cases documented for our use in teaching scenario-based usability engineering (see also http://ucs.cs.vt.edu, which includes the virtual science fair and several other partial examples). The cases were selected to cover a range of problem domains and usability interaction techniques—a fairly conventional e-Commerce site for browsing and purchasing gardening supplies; a banking application for palmtop computers; and a telephony project. As the figure suggests, the analysis and design work for each case are organized by the main “phases” of our scenario-based usability engineering life cycle, beginning with requirements analysis and continuing through design and evaluation.

Table 23.2 provides a more detailed view of the most comprehensive case study (Garden.com), expanding each phase to reveal the individual documents in a category. As this expansion suggests, one way to view each of our prototype case studies is as a mini library of analysis and design documents, interconnected with a narrative that describes the role and consequences of each document within the project life cycle. The Garden.com case study is particularly rich in the Information Gathering portion of Requirements Analysis, providing a total of nine photographs of nursery and/or garden shopping settings. These concrete field study artifacts are effective probes for interactive class discussion, prompting questions like

TABLE 23.1 Usability engineering case studies

Case Study	Case Documents
Garden.com: a virtual store offering comprehensive services for researching and purchasing garden and lawn supplies. Customers were given personalized treatment after registering for the store. It also included community-building support, for example advice from experts, chat, and so on.	Requirements Analysis: 25 Activity Design: 9 Information Design: 12 Interaction Design: 12 Documentation: 0 Usability Testing: 24
m-Banking: an extension to existing e-banking services (Web-based) that allows mobile users to access and manipulate their bank accounts and stock portfolios using a handheld mobile device (Palm VII). Eventually the services were available for both PDAs and WAP phones, but the case study chronicles the early phases of development for PDAs.	Requirements Analysis: 11 Activity Design: 4 Information Design: 12 Interaction Design: 9 Documentation: 4 Usability Testing: 6
PhoneWriter: a proof-of-concept telephony project that explored the contributions of various user input technologies (e.g., pen-based input, voice recognition) on the note-taking, form completion and other tasks common in telephony interactions. Collaboration based on documents transferred among telephony partners is also supported.	Requirements Analysis: 9 Activity Design: 4 Information Design: 6 Interaction Design: 0 Documentation: 2 Usability Testing: 7

TABLE 23.2 Detailed content for the Garden.com case study

Requirements Analysis Planning: initial brainstorming session; market research; root concept Methods and Materials: script for interviewing nursery workers; script for interviewing customers; focus group agenda; focus group recruiting ad; screening survey questions Information Gathering: photo of highway entrance sign; photo of nursery exterior; photo of nursery interior; photo of nursery main retail display; photo of outside covered retail display; photo of outside open retail display; photo of superstore outdoor display; photo of indoor superstore display; photo of gardening customer; garden sketch; plant hardiness map; catalog index; catalog order form; interview excerpt-nursery workers; interview excerpt-customers. Synthesis: problem claims; problem scenarios

Activity Design Exploration: summary of proposed activities; conceptual metaphors; system technology Envisionment: activity scenarios; shopping model; sequence analysis; point-of-view analysis Rationale: activity claims

Information Design Exploration: department information requirements; product page information requirements; summary of proposed information offerings; information metaphors; information technology options Envisionment: sketch of map metaphor; sketch of catalog metaphor; sketch of homepage; sketch of search results; sketch of product page; information scenarios Rationale: information claims

Interaction Design Exploration: interaction metaphors; interaction technology Envisionment: online shopping script; functional study; design revision #1; design revision #2 Rationale: before/after main page; before/after department page; before/after product list page; before/after product information page; before/after wheelbarrow page; usability study #2 report

Usability Testing Planning: operational definition; usability specifications Methods and Materials: user background survey; gardening quiz; attitudes toward the Internet survey; session script; usability testing scenarios; post-scenario ease of use assessment; usability report card; strategy questions; debriefing questions; equipment list Data Collection: notes on sessions; notes on participants; post-subtask comments; post-session comments; usability report card scores Interpretation: gardening quiz scores; attitudes toward the Internet; time on task; lostness ratings; number of mouse clicks; post-subtask ratings Synthesis: design recommendations

The Garden.com case also includes a large number of usability testing documents. These are particularly useful to computer science undergraduates who have never conducted a behavioral study before and thus have little insight into the kinds of questions one might ask, behavioral measures one might collect, interpretations warranted from a set of data, how to draw recommendations for redesign, and so on.

In sum, the library of case studies provide a wealth of examples that illustrate different aspects of the process of usability engineering—the preparatory analysis activities, the design and user studies, and the more formal evaluation of progress. As a pedagogical device, case studies rely on reasoning from examples, a well-studied phenomenon in psychology and cognitive science (Kolodner 1993, Leake 1998). As examples, each case varies in the amount of detail provided for each phase, and on the related HCI content, but across the set of cases the overall process of usability engineering and the concepts and theories of HCI receive broad coverage.

Browsing the Usability Cases

The usability case studies consist of analysis and design documents threaded together with a narrative of how the problem was analyzed and a solution developed. As the summary of Garden.com suggests, the amount of material comprising a case will often be large (Garden.com case includes 82 documents), so it was important to provide a tool that would facilitate students' browsing and analysis of the cases. Our general approach was to build a hypermedia browser that presented the cases in a canonical fashion (organized by the usability engineering phases) but also included linkages among related design documents and issues.

Figure 23.3 contains a screenshot of the case browser as it is being used to compare and contrast the activity design scenarios and activity design claims in the m-Banking case study. As summarized in Figure 23.2, scenarios play a central integrating role in SBD, by emphasizing the experiences and trade-offs of use. A claim is a feature of the usage scenario (typically a designed feature) that has positive and negative consequences (identified as plus and minus signs in the figure) for the users in this or related scenarios Claims are addressed in design by finding ways to maintain or enhance positive consequences while minimizing or removing negative. The browser simplifies learners' analysis of scenarios and claims by providing direct links from the scenario text to an associated claim and vice versa.

FIGURE 23.3 Case study browser being used to investigate relations between a scenario and its claims analysis

Although the canonical use of the case browser is to start at the “beginning” of a case (i.e., at the beginning of Requirements Analysis) and work through it sequentially, the tool supports more focused and/or piecemeal investigations as well. For instance, there is a glossary of terms useful to learners unfamiliar with SBD or usability engineering in general. Students can also traverse a case study by following links from document to document; the hierarchical, phased exploration of the case study is only suggested, not required.

The tool also supports a document-type view of the case library. In this mode, learners see a classification of different analysis or design documents (e.g., a “root concept” as used in Requirements Analysis, “conceptual metaphors” as used in Activity Design, the “information scenarios” developed during Information Design). We created this document-type view to encourage learners to compare and contrast across case studies, for instance noting how metaphors are used to shape the design of two rather different products. We have found this organization to be particularly useful to students preparing to do some phase of their own project, where they can benefit from examining multiple examples of a usability engineering method or concept.

Finally, learners can use a search dialog to construct a custom list of design documents related to a specific topic of interest. For instance, if a learner types “agent” as a search string, the system would return a list of all documents that use the word agent in any way. The search mechanism supports ad hoc tasks not anticipated by the phase-based document hierarchy. The search can be restricted to a single case study (e.g., suppose the viewer wants to study the role of “George” as a stakeholder in m-Banking), or it can be applied across all of the case studies.

Case-Based Learning Activities

Applying cases in the teaching usability engineering requires more than a tool to examine the body of case material. Students need direction, challenges, and focus, in order systematically approach the body of experience represented in the case library. These cases are each extensive, spanning the scenario-based usability engineering process from beginning to end. Portions of each case are applicable, in sequence, as the SBD process unfolds throughout the semester or quarter. To better leverage the case studies, we have developed series of activities that guide students to read and analyze case materials in support of specific pedagogical goals.

The case library as it exists can support a broad range of activities for students in their first exposure to usability engineering. Most of our case-based learning activities focus on one phase of scenario-based development or on the transition between the phases. Activities can be implemented as written homework assignments, in-class discussions, and small group discussions. Some typical activities we have used include:

These activities vary in the degree of cognitive effort demanded, and we tend to assign the least demanding activities early in the class and make more demanding assignments later on. For instance, modeling and summarization are relatively simple example-based learning activities, but are good ways to encounter and reflect on the “lingo” of SBD (scenarios, claims, etc.). However, contrasting two cases with respect to a phase is a more challenging and open-ended assignment, because each case addresses rather different HCI issues and techniques.

We have used activities based on this taxonomy in several iterations of our courses in usability engineering. Most activities are specified as homework assignments, with written solutions (the case browser currently provides no specific support for learning activities). The case-based homework facilitates two important pedagogical goals,

• (1) the students contemplate the issues in depth and on their own before in-class discussions; and
• (2) the level of in-class discussion is more active and insightful because of this preparation.

Student Performance and Reactions

Measuring the learning impacts of case study activities is difficult; gathering quantifiable data is particularly challenging. In our first iterations with case study assignments, we gathered general subjective reactions and comments. As Figure 23.4 suggests, early student reactions were generally quite positive. Most students (61%) agreed that overall the case-based learning assignments were either “Excellent” or “Very Good”. No students rated them as “Below Average” or “Poor”. Table 23.3 elaborates on these general ratings with comments offered in response to questions asking students what they liked the most, the least, and what suggestions they had for improvement.

FIGURE 23.4 Students' general reactions to case-based learning activities

TABLE 23.3 Feedback received from students

Sample responses to “What did you like best about the case study activities?”

“Gave me an opportunity to use concepts in class in a hands-on environment that helped a lot when transitioning to the group projects.”

“I liked the brainstorming activities because they were a lot of fun to do in teams. It was interesting to find out what other people could come up with.”

“It was interesting to see how the strategies we learned in class are applied in a real prototype. Real life examples always make learning more interesting.”

“I liked working in groups a lot, it allowed us to combine different ideas and approaches. I also thought it gave us a window into real-life applications. Definitely continues these types of activities.”

Sample responses to “What did you like least about the case study activities?”

“Not everyone was always prepared to contribute.”

“Sometimes the questions asked were like pulling teeth. It was a stretch sometimes to answer the question.”

“I am not too fond of speaking in public, so I did not like the talking part.”

“Some of the classroom assignments were ambiguous and too open-ended to be productive. We often spent a large portion of oui' time trying to figure out what the assignment was, leaving only minutes to actually brainstorm on the topic.”

Sample responses to “What could we do to improve the case study activities?”

“I would conduct the activities with the entire class as a whole and not in small groups. When each group gets to share their thoughts many of the ideas are repeated.”

“Make the requirements and timing more clear and stick to the times assigned for each part.”

“Perhaps allow students to participate with other people in the class. We were always put in our pre-assigned groups.”

“Maybe interact with laptops more in class?”

A problem with these early evaluative efforts was their generality. Although learners' comments sometimes mentioned specific activities or features of the cases, most of their reactions were to the general idea of case-based learning. In more recent iterations of the course, we have explored the use of more specific evaluation instruments. We have adapted the concepts of perceived self-efficacy (Bandura 1993) to construct two to three “usability engineering self-efficacy” probes for each activity assigned. Table 23.4 summarizes student responses to a set of such items, where reactions were gathered on a Likert scale (from 1 = Strongly Disagree to 5 = Strongly Agree).

TABLE 23.4 Self-efficacy judgments for six case-based learning activities

Activities and Associated Efficacy Probes	Mean
Model Garden.com by creating an original problem scenario and claim.
Even if a requirements analysis was thought to be complete, I could extend it with an original scenario	4.15
If presented with a scenario in an unfamiliar problem domain, I could identify legitimate usability claims.	3.81
Overall mean for Model activity	3.98
Summarize how metaphors clarified requirements; suggest starting points for activity design.
Even though I only studied examples of metaphors, I could generate new metaphors useful for activity design.	4.18
I could use a conceptual metaphor to clarify design requirements even though some parts of the metaphor are irrelevant.	3.89
Overall mean for Summarize activity	4.04
Trace claims through activity design, information design, and interaction design phases.
Even when a design develops over a number of phases, I can identify design issues and trace how they were managed in the design process	3.85
Although problem scenarios may be very different from design scenarios, I can map problem claims onto corresponding design claims.	3.85
Even though the PhoneWriter design process was presented to me within the limits of a case study, I am confident I could investigate and report on a similar design effort in the real world.	4.15
Overall mean for Trace activity	3.95
Contrast Virtual Science Fair and Garden.com use of prototyping.
Though design and interaction prototyping can be done in a variety of ways, I could select an appropriate prototyping strategy even if I was the sole usability engineer on a project.	4.16
If I joined a project team midstream, I could generate an interaction prototyping strategy even if this required digging up or backfilling some of the information design details.	4.03
Even though there are many project-specific reasons for choosing prototyping techniques, I could compare the effectiveness of prototyping approaches in different projects.	4.10
Overall mean for Contrast activity	4.10
Analyze design trade-offs in Garden.com's interaction design
Even though I only studied interaction design issues for Garden.com, I could analyze trade-offs for other interaction designs	3.95
I could evaluate how trade-offs were resolved in an interaction design even when the design documentation did not discuss each trade-off in detail.	3.72
Overall mean for first Analyze activity	3.84
Analyze m-Banking.com with respect to a choice of models
Even in a case where documentation design blends minimalism and the systems approach, I can identify which documentation features correspond to each model	3.83
Even without any empirical data evaluating a set of documentation, I can use analytic methods to judge how well the documentation implements minimalism or the systems approach.	3.82
Overall mean for second Analyze activity	3.83

As the table conveys, students' reactions were again generally positive. The modal response for each probe was “Agree”, with more ratings of “Strongly Agree” than “Neutral”. We were surprised that the differences between activities were rather small. For example, even at the beginning of the course, students seemed to be fairly confident about their abilities to model and summarize parts of case studies. Later on in the course, when they attempted more ambitious assignments, they continued to express confidence, even for the more cognitively demanding activities of contrasting and analyzing cases.

Although self-efficacy ratings are known to be a good predictor of achievement (Bandura 1976), at this point we can only speculate about the relation of our self-efficacy probes to students' actual comprehension and facility with usability engineering concepts and methods. However, we are able to report that during the same course in which the Table 23.4 data were gathered, the instructor wrote exams that in the past had yielded quite varied scores, and was pleasantly surprised to find what appeared to be a ceiling effect in this class—it seems that either students were highly motivated to prepare for the exam or they had genuinely learned the material thoroughly.

STRENGTHS AND WEAKNESSES

Case-based discussions in class can be, and often are, lively and interesting. As the previous section shows, students seem to appreciate the “real worldness” of case-based activities and seem to get a lot out of them. And while the cost of creating case studies, and designing effective case-based learning activities is high, once such materials are in place, the cost to educators of conducting these activities is rather low. Each of the activities in Table 23.4 required students to spend considerable time reading and synthesizing case documents but resulted in relatively short and pointed assignments for review and grading.

Case-based methods of learning intentionally create contextualized appreciation. This can deepen processing of information and aid learning, but it is not clear that all learners will reach the same level of understanding on their own. Written homework assignments may be used to “force” learners to receive similar exposure to the underlying principles, whereas in class discussions some students may become passive and not participate in the exchange. We believe that by combining these teaching methods, we can ensure a basic level of exposure and increase the likelihood of participation in group discussion.

Initially, instructors may find it more difficult to use case-based teaching methods than lecturing on a topic. Case-based instruction requires more effort to select or design an effective case and to understand it thoroughly before interacting with students. It also requires discussion-facilitation skills that not all academic faculty possess. Owing to the dynamic nature of discussions, instructors must be flexible and let the learners take the lead in determining the direction for the discussion. This may mean that conversations may not go exactly where the instructor wants it to go. It takes considerable classroom presence to provide appropriate interjections that guide the discussion without stifling it.

Evaluating student learning based on case methods is also a challenge that requires careful consideration and planning. Teaching assistants who grade homework based on case methods may need extra guidance, as there is often no single answer and design arguments may need subjective evaluation. A graduate assistant asked to evaluate an undergraduate's level of understanding will need extra guidance in how to attend to the “thought process” more than the “solution” provided.

Overall, case-based learning activities seem to be quite appropriate for teaching a complex process like usability engineering. By grounding learners' exposure to HCI principles in real-world contexts, students used to the concrete world of programming and software development begin to appreciate the fuzziness that accompanies software system design. For many students, their first HCI class provides their initial exposure to the nuances introduced when considering real people doing real things in real settings. The credibility contained in cases is significant and important. Learning to reason about trade-offs when the grounds for decision-making are unclear and incomplete is a crucial skill that is often only learned on the job.

DISCUSSION AND FUTURE DIRECTIONS

Over the past 20 years, considerable effort and resources have been directed to the development and application of case studies in engineering education (Kardos 1979, Smith and Kardos 2000). One of the most refined products of this work is the NEEDS (National Engineering Education Delivery System) project (http://www.needs.org). This digital library project presents online materials collected and refined through the collaborative efforts of many engineering schools over many years; detailed case studies are an important element in the library. Our prototype project is an initial step toward analogous resources for HCI education.

While our current case studies cover a range of problem domains, we are currently working to identify and develop examples from other domains of interest to the HCI community. For example, HCI professionals have recently become heavily involved in collaborative computing, studying user interface mechanisms that promote the development of online communities. Another attractive domain is scientific computing (e.g., “problem-solving environments”), where users are likely to be considerably more sophisticated than those for the average application. Two other domains of current interest in HCI are medical informatics and entertainment.

In addition to expanding the case library, we are developing new requirements for the case-browsing tool. For example, students in usability engineering typically carry out a semester project, from requirements through evaluation. These group projects might benefit from authoring support in the tool, so that each student project can itself be documented (and shared) as a mini-case. We also hope to better integrate the learning activities with the tool support; for example, an activity that requires analysis of how metaphors are used to move from problem to design scenarios might encapsulate or index into the library to make the relevant material more evident and convenient for learners.

Our ongoing work with case studies for teaching usability engineering continues to evolve from merely including interesting cases in an HCI course to using the case method as a pedagogy for the entire course. As we have refined our usability-engineering course, it has come to rely more extensively on case-based activities as the paradigm for exposing students to key HCI concepts. We have found ourselves moving from the use of cases as illustrations of usability engineering to using them to let students experience “vicariously” the complexities and trade-offs endemic to user-centered design. We are learning how the study of cases can be, in some ways, comparable to partaking in the experience narrated by the case study. Moreover, we believe that careful and systematic exposure to meaningful case studies may yield broader and more well-founded expertise than what is obtained through the more random experience of real world projects.

At the same time we are making contact with other HCI educators to share and further explore case-based learning of scenario-based usability engineering. For example, we have recently conducted a workshop involving a dozen HCI educators from universities in the United States and Canada. Preliminary results from the workshop include

• General agreement that the case method is effective in teaching usability engineering as a process.
• Observation that student projects could be and often were formatted in a manner akin to the structure found in the case study browser.
• Acknowledgement that additional cases would be useful and that student produced cases based on their projects would be particularly interesting.
• Identification of further requirements to extend and improve the case study browser.
• Agreement to participate in and share case study exercises and evaluation of their effectiveness.

We will continue to facilitate discussion and sharing among this group of educators, as well as to begin disseminating and evaluating the cases and associated learning activities more generally.

The scenario-based case studies we have developed are comprehensive, comprising many different documents woven together via text summaries for each phase of the project. Another direction for future work is the editing of a full case to tell the story of just one or two key issues addressed in a design project. Such a treatment would remove much of the real world context that makes the overall case complex, but might serve the pedagogical goal of helping students to understand how a particular HCI concern was raised and addressed (to some extent, this is what the Trace activity was designed to do; see the section titled “Using Cases to Teach Scenario-Based Usability Engineering”. These “brief cases” might be particularly useful in teaching agile development skills (Beck and West, Chapter 13), where getting to the heart of an issue quickly is essential to making progress. Alternatively, if we collected user stories as they evolved in an extreme programming project, perhaps annotated with connections to design ideas and trade-offs, we could construct a scenario-based case study of an agile development project.

Case studies have a long and successful history in learning and education. But case studies are more than examples. Cases illustrate the process of solving a problem; exploration of case studies is particularly important in disciplines where the problems are ill-defined and complex, where solutions rely on understanding and addressing the details of a situation, where there is usually more than one right answer, and where it is impossible to crystallize an expert's problem-solving abilities into a set of guidelines or principles. In many ways, case studies can be a surrogate for joining the workforce and gaining actual project experience. Computer science students who grasp the usage complexities and trade-offs in real world usability engineering cases are well on their way to making use.

ACKNOWLEDGEMENTS

Development of the case studies and browser was supported by The National Science Foundation (DUE-0088396 and DUE-0231111)h. We thank Jennifer Thompson, Richard Bowman, Wesley Lloyd, Jiunwei Chen, and Vinoth Jagannathan for their help with the case study content, activities, tools, and evaluations. An abbreviated version of this curriculum project was presented in March 2004 at ACM SIGCSE in Richmond VA (Rosson, Carroll & Rodi, 2004).

KEYWORDS

Usability Engineering

Human–Computer Interaction (HCI)

Undergraduate Education

Scenario-Based Design (SBD)

Case Studies

Active Learning

REFERENCES

Bandura, A., Perceived self-efficacy in cognitive development and functioning, Educational Psychologist, 28, 117–148, 1993.

Bandura, A, Self-efficacy: The exercise of control, W.H. Freeman and Company, 1997.

Boehm, B., The spiral model of software development and enhancement, IEEE Computer, 21(5), 61–72, 1988.

Brooks, F., The Mythical Man-Month: Essays on Software Engineering, Addison-Wesley, 1995.

Card, S.K., Moran, T.P., and Newell, A., The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, 1983.

Carroll, J.M. (Ed.), Scenario-Based Design: Envisioning Work and Technology in System Development, John Wiley & Sons, 1995.

Carroll, J.M, Human-Computer Interaction: Psychology as a science of design, International Journal of Human-Computer Studies, 46, 501–522, 1997.

Carroll, J.M., Making Use: Scenario-Based Design of Human-Computer Interactions, MIT Press, 2000.

Carroll, J.M. and Rosson, M.B., Getting around the task-artifact cycle: How to make claims and design by scenario, ACM Transactions on Information Systems, 10(2), 181–212, 1992.

Carroll, J.M. and Thomas, J.C., Metaphors and the cognitive representation of computing systems, IEEE Transactions on Systems, Man, and Cybernetics, 12(2), 107–116, 1983.

Hewitt, T. Baecker, R., Card, S., Carey, T., Gasen J., Mantei, M., Perlman, G., Strong G., Verplank, W., ACM SIGCHI Curricula for Human-Computer Interaction, ACM Press, 1992.

Carroll, J.M., Kellogg, W.A., and Rosson, M.B., The task-artifact cycle, in J.M. Carroll (Ed.), Designing Interaction: Psychology at the Human-Computer Interface, Cambridge University Press, 1991, 74–102.

Cooper, A., The Inmates are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity, SAMS Press, 1999.

Kardos, G., Engineering Cases in the Classroom, Online paper available from Carleton University, 1979, http://www.civeng.carleton.ca/ECL/cclas.html

Kolodner, J., Case-Based Reasoning, Morgan Kaufmann, 1993.

Kyng, M., Creating contexts for design, in J. Carroll (Ed.), Scenario-Based Design: Envisioning Work and Technology in System Development, John Wiley & Sons, 1995, pp. 85–108.

Leake, D.B. (Ed.), Case-Based Reasoning: Experiences, Lessons, and Future Directions, AAAI Press, 1998.

Mayhew, D.J., The Usability Engineering Lifecycle: A Practioner's Handbook for User Interface Design, Morgan Kaufmann, 1999.

Moran, T. and Carroll, J.M. (Eds.), Design Rationale: Concepts, Methods and Techniques, Lawrence Erlbaum Associates, 1996.

NEEDS, National Engineering Education Delivery System, 2004, http://www.needs.org/

Nielsen, J., Usability Engineering, Academic Press, 1993.

Norman, D.A., Cognitive engineering, in D.A. Norman and S.D. Draper (Eds.), User Centered System Design, Lawrence Erlbaum Associates, 1986, pp. 31–61.

Reitman, W.R., Cognition and thought.: An information processing approach, John Wiley and Sons, New York, 1965.

Rosson, M.B. and Carroll, J.M., Usability Engineering: Scenario-Based Development of Human-Computer Interaction, Morgan Kaufmann, 2002.

Rosson, M.B. and Carroll, J.M., Scenario-based design, The Human-Computer Interaction Handbook, Lawrence Erlbaum Associates, 2003.

Rosson, M.B., Carroll, J.M., Rodi, C.M, Case studies for teaching usability engineering, Proceedings of Technical Symposium on Computer Science Education, 36–40, 2004.

Shulman, J.H. (Ed.), Case methods in teacher education, Teachers College Press, New York, 1992.

Simon, H.A, The structure of ill-structured problems, Artificial Intelligence, 4, 181–201, 1973

Smith, C.O. and Kardos, G., Design in Materials Courses? Naturally! Online paper available from Carleton University, 2000, http://www.civeng.carleton.ca/ECL/dsngmat.html

Strong, G. et al., New Directions in Human-Computer Interaction Education, Research, and Practice, NSF-ISP, NSF-AATP, and ARPA-SIST, 1994.

RECOMMENDED READINGS

Carroll, J.M., Making Use: Scenario-Based Design of Human-Computer Interactions, MIT Press, 2000. This monograph presents theoretical arguments for scenario-based analysis and design of interactive systems, as well as a number of examples of these techniques applied to projects involving education, design, and software development.

Harvard Business School, Teaching and the Case Method, 3^rd ed, McGraw-Hill, 1994. This is one of the classic texts on case-based methods that has been revised and updated over many years. The book introduces a variety of techniques for case-based teaching that can be applied to fields as diverse as medicine and business administration. It also includes an Instructor's Guide with teaching notes and suggestions for starting and running case-based teaching seminars.

Rosson, M.B. and Carroll, J.M., Usability Engineering: Scenario-Based Development of Human-Computer Interaction, Morgan Kaufmann, 2002. This textbook provides a comprehensive introduction to the SBD framework summarized in the section titled “Comparisons—Scenarios for Making Use”. It integrates coverage of key concepts and theories in human-computer interaction with the phases of an SBD project, and illustrates the entire lifecycle with an extended case study.