Chapter 15
Rigorous Empirical Evaluation
Running the Session
Objectives
After reading this chapter, you will:
1. Know the preliminaries and protocol issues in dealing with rigorous UX evaluation participants
2. Be prepared to generate and collect objective and subjective quantitative UX evaluation data
3. Know how to generate and collect qualitative UX data by critical incident identification, think-aloud techniques, and post-session interviews and probing
4. Be able to use special techniques for gathering emotional impact and phenomenological data
15.1 Introduction
15.1.1 You Are Here
We begin each process chapter with a “you are here” picture of the chapter topic in the context of the overall Wheel lifecycle template; see Figure 15-1. This chapter is about running a lab-based evaluation, a step of rigorous UX evaluation.
Figure 15-1 You are here, at running the session, within the evaluation activity in the context of the overall Wheel lifecycle template.
15.2 Preliminaries with participants
15.2.1 Introduce Yourself and the Lab: Be Sure Participants Know What to Expect
In this chapter, our story opens with the arrival of participants at your UX lab. If you have a separate reception room in your UX facility, this is where you meet your participants before getting down to business with evaluation. Greet and welcome each participant and thank him or her for helping. Bring them in and show them around.
Introduce them to the setup and show them the lab. If you have one-way glass, explain it and how it will be used and show them the other side—what happens “behind the curtain.” Openly declare any video recording you will do (which should have been explained in the consent form, too). Make participants feel that they are partners in their role.
Tell your participants all about the design being evaluated and about the process in which they are participating. For example, you might say “We have early screen designs for our product in the form of a low-fidelity prototype of a new system for … .” Tell them how they can help and what you want them to do.
Do your best to relieve anxiety and satisfy curiosity. Be sure that your participants have all their questions about the process answered before you proceed into evaluation. Make it very clear that they are helping you evaluate and are not evaluating them in any way. For example, “You are here to evaluate a new design for … ; you will be asked to try some tasks using the computer, to help us find places where the design is not supportive enough for you.”
15.2.2 Paperwork
While still in the reception room or as soon as the user has entered the participant room:
Have each participant read the general instructions and explain anything verbally, as needed.
Have the participant read the institutional review board consent form (Chapter 14) and explain the consent form verbally as well.
Have the participant sign the consent form (two copies); it must be signed “without duress.” You keep one signed copy and give the participant the other signed copy. Your copy must be retained for at least 3 years (the period may vary by organization).
Have the participant sign a non-disclosure form, if needed.
Have the participant fill out any demographic survey you have prepared.
A short written demographic survey can be used to confirm that each participant does, indeed, meet the requirements of your intended work activity role and corresponding user class characteristics.
15.2.3 The Session Begins
Give the participants a few minutes to familiarize themselves with the system unless walk up and use is a goal. If you are using benchmark tasks, after the preliminaries and when you both are ready to start, have the participant read the first benchmark or other task description and ask if there are any questions. If you are taking timing data, do not include the benchmark task reading time as part of the task.
Once the evaluation session is under way, interesting things can start happening quickly. Data you need to collect may start arriving in a flood. It can be overwhelming, but, by being prepared, you can make it easy and fun, especially if you know what kinds of data to collect. We will look at the possible kinds of data and the methods for generating and collecting them. But first, we need to get some protocol issues out of the way.
15.3 Protocol issues
Session protocol is about the mechanical details of session setup and your relationship with participants and how you handle them throughout each session.
15.3.1 Attitude toward UX Problems and toward Participants
Before you actually do evaluation, it is easy to agree that this UX testing is a positive thing and we are all working together to improve the design. However, once you start hearing about problems participants are having with the design, it can trigger unhelpful reactions in your ego, instincts, and pride. Proceed in your testing with a positive attitude and it will pay off.
15.3.2 Cultivating a Partnership with Participants
Take the time to build rapport with your participants. More important to the success of your UX evaluation sessions than facilities and equipment is the rapport you establish with participants, as partners in helping you evaluate and improve the product design. Once in the participant room, the facilitator should take a little time to “socialize” with the participant. If you have taken the participant on a “tour” of your facilities, that will have been a good start.
If you are using co-discovery techniques (Chapter 12), allow some time for co-discovery partners to get to know each other and do a little bonding, perhaps while you are setting things up. Starting the session as total strangers can make them feel awkward and can interfere with their performance.
15.3.3 Interaction with Participant during the Session
So far, you have done the necessary preparation for your evaluation, including preparation of benchmark task descriptions, procedures, and consent forms, as well as participant preparation. It is finally time to get an evaluation session underway. The facilitator helps ensure that the session runs smoothly and efficiently.
It is generally the job of the facilitator to listen and not talk. But at key junctures you might elicit important data, if it does not interfere with task timing or if you are focusing on qualitative data. You can ask brief questions, such as “What are you trying to do?” “What did you expect to happen when you clicked on the such-and-such icon?” “What made you think that approach would work?”
If you are focusing on qualitative data, the evaluator may also ask leading questions, such as “How would you like to perform that task?” “What would make that icon easier to recognize?” If you are using the “think-aloud” technique for qualitative data gathering, encourage the participant by prompting occasionally: “Remember to tell us what you are thinking as you go.”
Do not “blow off” problems perceived by the participant as just, for example, an anomaly in the prototype. If you think that an issue pointed out as a UX problem by a participant is actually not a genuine issue, write it down as a problem for the moment, anyway. Otherwise you will discourage them from mentioning problems that might not be as real as they seem.
If participants show signs of stress or fatigue, give them a break. Let them leave the participant room, walk around, and/or have some refreshments. Do not be too up-tight about the session schedule. It is almost impossible to set a time schedule for tasks and steps for the participant in a session. It is better to present a list of objectives and let the participant know where you both are, as a team, in working toward those goals. To the extent possible, let the participant decide when to take breaks and when to stop for lunch.
15.3.4 To Help the Participant or Not to Help the Participant?
Give hints if necessary, but direct help almost always works against the goals of the session. Sometimes when participants are not making progress, they can benefit from a hint to get them back on track so that their session again becomes useful. You want to see whether the participant can determine how to perform the task. You should not give them information about how to complete a task. So, if participants ask for help, how can you let them know you are there for them without doing some coaching? Often the best way is to lead them to answer their own questions.
For example, do not answer questions such as “is this right?” directly, but by asking your own questions, directing them to think it through for themselves. With experience, evaluators become very creative at being appropriately evasive while still helping a participant out of a problem without adversely affecting data collected. Sometimes it helps to tell the participant upfront that you will decline to answer design-related questions to see how the participant interacts with the system. Make note of those questions and answer them at the end of the session.
15.3.5 Keeping Your Participant at Ease
Remind yourself and your whole team that you should never, never laugh at anything during a UX evaluation session. You may be in the control room and think you have a sound-proof setup but laughter has a way of piercing the glass. Because participants cannot see people behind the glass, it is easy for participants to assume that someone is laughing at them.
If participants become visibly flustered, frustrated, “zoned out,” or blame themselves continually for problems in task performance, they may be suffering from stress and you should intervene. Take a short break and reassure and calm them. Remind them that “you are evaluating the design; we are not evaluating you.” If participants become so discouraged that they want to quit the entire session, there is little you can or should do but thank them, pay them, and let them go.
15.3.6 Protocols for Evaluating with Low-Fidelity Prototypes
Have your paper prototype laid out and ready to go. Well before starting a session using a paper prototype, the team should prepare for using the prototype by assembling all the parts and pieces of the prototype. To prevent the easel (Chapter 11) from moving during the action, consider taping it to the working table between the participant, the facilitator, and the executor, as shown in Figure 15-2.
Figure 15-2 Typical setup at the end of a table for evaluation with a paper prototype.
“Boot up” the prototype by putting the initial “screen” on the easel, having each moving part ready at hand and convenient to find and grab to enter it into the action. Make sure that any changes made to data and prototype internal states by previous participants are reset to original values for the current participant.
Before each participant enters, the “executor” should arrange everything necessary for running the prototype, including stacks of prototype parts and other equipment (e.g., marking pens, extra paper or plastic, tape).
Have the whole evaluation team ready to assume their roles and be ready to carry them out in the session.
Evaluation facilitator, to keep the session moving, to interact with participants, and to take notes on critical incidents (pick a person who has leadership abilities and “people” skills).
Prototype executor, to move transparencies and provide “computer” responses to user actions (pick a person who knows the design well and is good at the cold and consistent logic of “playing” computer).
User performance timer, to time participants performing tasks and/or count errors (to collect quantitative data)—the timer person may want to practice with a stopwatch a bit before getting into a real session.
Critical incident note takers (for spotting and recording critical incidents and UX problems).
Review your own in-session protocol. Some of the “rules” we suggest include:
Team members must not coach participants as they perform tasks.
The executor must not anticipate user actions and especially must not give the correct computer response for a wrong user action! The person playing computer must respond only to what the user actually does!
The person playing computer may not speak, make gestures, etc.
You may not change the design on the fly, unless that is a declared part of your process.
Figure 15-3 shows another view of a participant with a paper prototype.
Figure 15-3 Participant pondering a paper prototype during formative evaluation.
15.4 Generating and collecting quantitative UX data
If your evaluation plan calls for taking quantitative data, participants perform prescribed benchmark tasks during a session and evaluators take numeric data. For example, an evaluator may measure the time it takes the participant to perform a task, count the number of errors a participant makes while performing a task, count the number of tasks a participant can perform within a given time period, and so on, depending on the measures established in your UX targets.
15.4.1 Objective Quantitative Data
The main idea is to use participant performance of benchmark tasks as a source of objective (observed) quantitative UX data.
Timing task performance
By far the simplest way to measure time on task is by using a stopwatch manually. It is really the only sensible way for low-fidelity, especially paper, prototypes. Timing with a stopwatch is also acceptable for software prototypes and is still the de facto standard way, sufficing for all situations except those demanding the most precision. If timing manually, you usually start the timer when the participant has finished reading the benchmark task description, has no questions, and you say “please start.”
Try not to use very short benchmark tasks. It can be difficult to get accurate timings with a stopwatch on very short tasks. Something in the order of 5 minutes or more is easy to time.
If precise timing measurements are required, it is possible to embed software timers to instrument the software internally. These routines keep time stamps denoting when execution of the application software enters and exits key software modules. Software timers also free up one data collector for other jobs, such as observing critical incidents, but they do require more post-session work to compile timing data.
Counting user errors
The key to counting errors correctly is in knowing what constitutes an error. Not everything that goes wrong, not even everything a user does wrong, during task performance should be counted as a user error. So what are we looking for? A user error is usually considered to have occurred when the participant takes any action that does not lead to progress in performing the desired task. The main idea is to catch the times that a participant takes a “wrong turn” along the expected path of task performance, especially including cases where the design was inadequate to direct the interaction properly.
Wrong turns include choosing an incorrect item from a menu or selecting the wrong button, choices that do not lead to progress in performing the desired task. Other examples include selecting the wrong menu, button, or icon when the user thought it was the right one, double-clicking when a single click is needed, and vice versa.
If a participant takes a wrong turn but is able to back up and recover, an error has still occurred. In addition, it is important to note the circumstances under which the participant attempted to back up and whether the participant was successful in figuring out what was wrong. These occasions can provide qualitative data on user error recovery strategies that you might not be able to get in any other way.
The simplest way to count user errors during task performance is to use a manual event counter such as a handheld “clicker” for counting people coming through a gate for an event. Manual counters are perfect for low-fidelity, especially paper, prototypes. For software prototypes and operational software applications, if you use video to capture the interactions, you can tag the video stream with error points and the video analysis software can tally the count easily.
What generally does not count as a user error?
Typically, we do not count accessing online help or other documentation as an error. As a practical matter, we also want to exclude any random act of curiosity or exploration that might be interjected by the user (e.g., “I know this is not right, but I am curious what will happen if I click this”). Also a different successful path “invented” by the user is not really an error, but still could be noted as an important observation.
And we do not usually include “oops” errors, what Norman (1990, p. 105) calls “slips.” These are errors that users make by accident when, in fact, they know better. For example, the user knows the right button to click but clicks the wrong one, perhaps through a slip of the hand, a brain burp, or being too hasty. However, we do note an oops error and watch for it again. If it recurs, it may not be random and you should look for a way the design might have caused it. Finally, we do not usually include typing errors, unless their cause could somehow be traced to a problem in the design or unless the application is about typing.
15.4.2 Subjective Quantitative Data: Administering Questionnaires
If you are using questionnaires, the data collection activity is when you administer them. Have the participant fill out the questionnaires you have chosen per the timing you have decided, such as after a task or at the end of a session.
15.5 Generating and collecting qualitative UX data
In Chapter 12 we discussed how the goal of qualitative data collection is to identify usability problems and their causes within the design. In lab-based testing with participants, this goal is achieved primarily through observation and recording of critical incidents, often with the help of the think-aloud technique.
Critical Incident
A critical incident is a UX evaluation event that occurs during user task performance or other user interaction, observed by the facilitator or other observers or sometimes expressed by the user participant, that indicates a possible UX problem. Critical incident identification is arguably the single most important source of qualitative data.
Think-Aloud Technique
The think aloud technique is a qualitative data collection technique in which user participants verbally externalize their thoughts about their interaction experience, including their motives, rationale, and perceptions of UX problems. By this method, participants give the evaluator access to an understanding of their thinking about the task and the interaction design.
15.5.1 Collecting Critical Incident Information
The way that you collect and tag critical incident data as you go will have much to say about the accuracy and efficiency of your subsequent data analysis. Get your detailed critical incident and UX problem information recorded as clearly, precisely, and completely as you can in real time during data collection.
Do not count on going back and reviewing video recordings to get the essence of the problems. In the raw data stream, there are huge amounts of “noise,” data not relevant to UX analysis. Important events, such as critical incident occurrences, are embedded in and obscured by irrelevant data. The wheat is still within the chaff.
Finding critical incidents within this event stream by viewing the video is laborious and time-consuming, which is one important reason for using direct (real-time) critical incident observation by an evaluator as a primary data collection technique. Do as much filtering as possible at the moment of data capture.
As events unfold, it is the job of the skilled evaluator to capture as much information about each critical incident as possible, as they happen in real time. In early stages or when you do not have any software tool support, you can just take notes on the critical incidents that you spot.
In Chapter 16 we discuss UX problem instance records. It is helpful to use that as a template to support completeness in collecting critical incident data.
15.5.2 Critical Incident Data Collection Mechanisms
Video recording for critical incident data collection. In some labs, video recording is used routinely to capture all user and screen actions and facilitator and participant comments. Once you get into video recording, it is probably best to use a software support tool to control the video equipment for recording, review, and later analysis and for tagging the video with evaluator comments.
If you use video recording, the minimal video to capture is the real-time sequencing of screen action, showing both user inputs and system reactions and displays. Software support tools such as Morae™ or OVO™ can capture full-resolution video stream of screen action automatically. This is adequate for a large portion of the UX evaluation sessions we do. However, if participant physical actions, gestures, and/or facial expressions are important, as they might well be to evaluate emotional impact, digital video streams from cameras can be added and most capture software will synchronize all the video inputs automatically.
For some purposes, you can use one video camera aimed at the participant’s hands to see details of physical actions and gestures and another at the participant’s face to see expressions and, if useful, you can even have a third camera to capture a wide-angle overview of evaluator, participant, the computer, or other device being evaluated—the full context of the user experience.
Critical incident markers: Filtering raw data. Each critical incident marker created by the evaluator points to a particular place in the raw video stream, tagging the wheat as it still resides within the chaff. This real-time tagging enables evaluators to ignore the chaff in any further analysis, boosting the efficiency of the data analysis process enormously.
Tagging critical incidents is somewhat analogous to a similar kind of data tagging in crime scene data collection and analysis: little flags or tags are arranged in proximity to items that are identified as evidence so that the crime scene analysts can focus on the important items easily. Each “start” and “stop” tag (Figure 15-4) denotes a video clip, which constitutes filtered raw data representing a critical incident.
Figure 15-4 Overview of raw data filtering by tagging critical incidents and adding interpretive comments.
Critical incident comments: Interpretive data. In addition to marking critical incidents, evaluators sometimes want to make comments explaining critical incidents. The comments are linked to the corresponding video clips so that analysts subsequently can view the related clips as they read the comments. The comments are an interpretive “coating” on filtered raw data. Figure 15-4 illustrates the video stream tagged with critical incident markers and associated evaluator comments.
Working without a net: Not recording raw data. It is more efficient if you do not have to use video recording. Although video recording of interaction even with paper prototypes can be appropriate and useful, evaluators certainly do not always record video of the interaction events, especially for evaluations of early prototypes where the fast pace of iterative design changes calls for lightweight data collection and analysis.
Operating without the safety net of a video recording results in immediate loss of raw data. Evaluators depend solely on the comments, and the analysis process begins with just the interpretive accounts of data, but this if often fully adequate and appropriate for early versions of a design or when rapid methods are required for all versions.
Manual note taking for critical incident data collection. Manual note taking is the most basic mechanism and is still a useful and efficient approach in many UX labs. Evaluators take comprehensive, real-time raw critical incident notes with a laptop or with pencil and paper. When thoughts come faster than they can write, they might make audio notes on a handheld digital voice recorder—anything to capture raw data while it is still fresh.
Even if you are also making audio or video recordings, you should take notes as though you are not. It is a mistake to rely on your video recordings as a primary method of raw data capture. It is just not practical to go back and get your raw critical incident notes by reviewing the entire video. The video, however, can be a good backup, something you can look at for missing data, to resolve a question, or to clear up a misunderstanding.
15.5.3 Think-Aloud Data Collection
Although there are some points to watch for, in its simplest form this technique could not be easier. It simply entails having participants think out loud and share their thoughts verbally while they perform tasks or otherwise interact with a product or system you want to evaluate. Within the UX evaluation method you are using:
At the beginning, explain the concept of thinking aloud and explain that you want to use this technique with participants.
Explain that this means you will expect them to talk while they work and think, sharing their thoughts by verbalizing them to you.
You might start with a little “exercise” to get warmed up and to get participants acclimated to thinking aloud.
Depending on your main UX evaluation method, you may capture think-aloud data by audio recording, video recording, and/or written or typed notes.
Among the thoughts you should encourage participants to express are descriptions of their intentions, what they are doing or are trying to do, and their motivations, the reasons why they are doing any particular actions.
You especially want them to speak out when they get confused, frustrated, or blocked.
Depending on the individual, thinking aloud usually comes quite naturally; it does not take much practice. Occasionally you might have to encourage or remind the participant to keep up the flow of thinking aloud.
15.6 Generating and collecting emotional impact data
Collecting emotional impact data depends on observing and measuring indicators of emotional response through verbal communication, facial expressions, body language, behaviors, and physiological changes.
15.6.1 Applying Self-Reporting Verbal Techniques for Collecting Emotional Impact Data
Applying the think-aloud technique to evaluate emotional impact
We have already talked about using the think-aloud technique for capturing the participant’s view of interaction, critical incidents, and UX problems. The think-aloud technique is also excellent for obtaining a window into the mind of the user with respect to emotional feelings as they occur.
Depending on the nature of the interaction, emotional impact indicators may be infrequent in the flow of task performance user actions, and you may see them mainly as a by-product of your hunt for usability problem indicators. So, when you do encounter an emotional impact indicator during observation in task performance, you certainly should make a note of it. You can also make emotional impact factors the primary focus during the think-aloud technique.
When you explain the concept of thinking aloud, be sure participants understand that you want to use this technique to explore emotional feelings resulting from interaction and usage.
Explain that this means you will expect them to share their emotions and feelings while they work and think by talking about them to you.
As you did when you used the think-aloud technique to capture qualitative UX data, you may wish to begin with a little “exercise” to be sure participants are on the same page about the technique.
As before, you can capture think-aloud data by audio recording, video recording, and/or written or typed notes.
Also as before, you may have to remind participants occasionally to keep the thinking aloud flowing.
During the flow of interaction:
You can direct participants to focus their thinking aloud on comments about joy of use, aesthetics, fun, and so on.
You should observe and note the more obvious manifestations of emotional impact, such as expressions like “I love this” and “this is really cool” and “wow” expressions, annoyances, or irritation.
You also need to watch out for the more subtle expressions that can provide insights into the user experience, such as a slight intake of breath.
As a practitioner, you also must be sensitive to detecting when emotional impact goes flat, when there is no real joy of use. Ask participants about it, causes, and about how it can be improved.
Finally, a caution about cultural dependency. Most emotions themselves are pretty much the same across cultures, and non-verbal expressions of emotion, such as facial expressions and gestures, are fairly universal. But cultural and social factors can govern an individual’s willingness to communicate about emotions. Different cultures may also have different vocabularies and different perspectives on the meaning of emotions and the appropriateness of sharing and revealing them to others.
Applying questionnaires to evaluate emotional impact
Based on your project context and the type of application, use the discussion of questionnaires in Chapter 12 to select and apply a questionnaire to evaluate emotional impact.
15.6.2 Applying Direct Non-Verbal Techniques for Collecting Emotional Impact Data
Using non-verbal techniques for collecting emotional impact usually means deploying probes and instrumentation; see Chapter 12 for a discussion of physiological measurements.
15.7 Generating and collecting phenomenological evaluation data
Get ready for a study of emotional impact situated in the real activities of users over time, if possible from the earliest thinking about the product to adoption into their lifestyles. You will need to choose your data collection techniques to compensate for not being able to be with your participants all the time, which means including self-reporting.
We encourage you to improvise a self-reporting technique yourself, but you should definitely consider a diary-based technique, in which each participant maintains a “diary,” documenting problems, experiences, and phenomenological occurrences within long-term usage. Diaries can be kept via paper and pencil notes, online reports, cell-phone messages, or voice recorders.
For a diary-based technique to be effective, participants must be primed in advance:
Give your users a list of the kinds of things to report.
Give them some practice exercises in identifying relevant situations and reporting on them.
Get them to internalize the need to post a report whenever they confront a usage problem, use a new feature, or encounter anything interesting or fun within usage.
To encourage participants to use voice-mail for reporting, consider paying them a per-call monetary compensation (in addition to whatever payment you give them for participating in the study). In the Palen and Salzman study, they found that a per-call payment encouraged participants to make calls. There is a possibility that this incentive might bias participants into making some unnecessary calls, but that did not seem to happen in this study.
To perhaps get more representative data, you might choose to trigger reporting to control the timing (Chapter 12, under data collection technique for phenomenological aspects), rather than letting your participant perform reporting at when it is convenient or times when things are going well with the product. For example, you can give your participant a dedicated pager (Buchenau & Suri, 2000) to signal randomly timed “events” at which times the participant is asked to report on their usage and context.
Another way you could choose to sample phenomenological usage is by periodic questionnaires over time. You can use a series of such questionnaires to elicit understanding of changes in usage over those time periods.
You can also choose to do direct observation and interviews in simulated real usage situations (Chapter 12, under data collection technique for phenomenological aspects). You will need to create conditions to encourage episodes of phenomenological activity to occur during these observational periods.
As an example of using this technique, Petersen, Madsen, and Kjaer (2002) conducted a longitudinal study of the use of a TV and video recorder by two families in their own homes. During the time of usage, periodic interviews were scheduled in the analysts’ office, except in cases where users had difficulty in getting there and, then, the interviews were conducted in the users’ homes.
During the interviews, the evaluators posed numerous usage scenarios and had the participants do their best to enact the usage while giving their feedback, especially about emotional impact. All interviews were videotaped. The idea is to set up conditions so that you can capture the essence of real usage and reflect real usage in a tractable time frame.
Here are some tips for success:
Establish the interview schedule to take into account learning through usage by implementing a sequence of sessions longitudinally over time.
As in contextual inquiry, it is necessary to observe user activities rather than just to ask about them. As we know, the way people talk about what they do is often not the same as what they actually do.
Be cautious and discreet with videotaping in more private settings, such as the participant’s home, usually found in this kind of usage context.
As you collect data, you will be looking for indicators of all the different ways your users involve the product in their lives, the high points of joy in use, how the basic mode of usage changes, evolves, or emerges over time, and especially how usage is adapted to emerge as new and unusual kinds of usage. As said in Chapter 12, you want to be able to tell stories of usage and good emotional impact over time.
15.8 Wrapping up an evaluation session
Returning to the more traditional lab-based or similar evaluation methods, we now need to do several things to wrap up an evaluation session.
15.8.1 Post-Session Probing Via Interviews and Questionnaires
Immediately after the sessions, but while your participant is still present, is the best opportunity to ask probing questions to clear up any confusion you have about critical incidents or UX problems. Conduct post-session interviews and administrator questionnaires to capture user thoughts and feelings while they are fresh.
Clarify ambiguities about the nature of any problems. Be sure you understand the real problems and their causes. Interact with your participant as a doctor does in diagnosing a patient. If you wait until the participant is gone, you lose the opportunity to ask further questions to disambiguate the diagnoses and causes.
Facilitators often start with some kind of standard structured interview, asking a series of preplanned questions aimed at probing the participant’s thoughts about the product and the user experience. A typical post-session interview might include, for example, the following general questions. “What did you like best about the interface?” “What did you like least?” “How would you change so-and-so?” An interesting question to ask is “What are the three most important pieces of information that you must know to make the best use of this interface?”
For example, in one design, some of the results of a database query were presented graphically to the user as a data plot, the data points of which were displayed as small circles. Because most users did not at first realize that they could get more information about a particular data point if they clicked on the corresponding circle, one very important piece of information users needed to know about the design was that they should treat a circle as an icon and that they could manipulate it accordingly.
It can be even more effective to follow up with unstructured opportunistic questioning. Find out why certain things happened. Be sure to ask about any critical incidents that you are not sure about or potential UX problems for which you do not yet completely understand the causes.
15.8.2 Reset for the Next Participant
After running an evaluation session with one participant, you should clean up everything to be ready for the next participant. This means removing any effects from the previous session that might affect the participant or task performance in the next session. Often you will have to do this kind of cleanup even between tasks for the same participant.
If you are using a paper prototype, you still must reset internal states and data back to initial conditions needed by the first task using the prototype. For example, if previous participants made changes to the prototype, such as filling in information on a paper or plastic form, provide a fresh clean form. If the user made changes to a “database,” recorded anywhere that will be visible to the next participant, these have to be reset for a fresh participant.
For Web-based evaluation, clear out the browser history and browser cache, delete temporary files, remove any saved passwords, and so on. For a software prototype, save and backup any data you want to keep. Then reset the prototype state and remove any artifacts introduced in the previous session.
Delete any temporary files or other saved settings. Reset any user-created content on the prototype, such as any saved appointments, contacts, or files. Reset any other tools, such as Web-based surveys or questionnaires, to make sure one participant’s answers are not visible to the next one.
Finally, give the participant(s) their pay, gifts, and/or premiums, thank them, and send them on their way.
Exercise
See Exercise 15-1, UX Evaluation Data Collection for Your System
15.9 The humaine project
The European community project HUMAINE (Human-Machine Interaction Network on Emotions) issued a technical report detailing a taxonomy of affective measurement techniques (Westerman, Gardner, & Sutherland, 2006). They point out that there is a history of physiological and psychophysiological measurement in human factors practice since the late 1970s to detect, for example, stress due to operator overload, and an even longer history of this kind of measurement in psychological research.
In the HUMAINE report, the authors discuss the role of medicine in physiological measurement, including electroencephalograms and event-related potential, measured with electroencephalography, a technique that detects and measures electrical activity of the brain through the skull and scalp. Event-related potentials can be roughly correlated to cognitive functions involving memory and attention and changes in mental state.
As the authors say, these physiological measurements have the advantage over self-reporting methods in that they can monitor continuously, require no conscious user actions, and do not interrupt task performance or usage activity. To be meaningful, however, such physiological measurements have to be associated with time stamps on a video of user activity.
A major disadvantage, ruling the approach out for most routine UX evaluation, is the requirement for attached sensors. New, less intrusive instrumentation is being developed. For example, Kapoor, Picard, and Ivanov (2004) report being able to detect changes in user posture, for example, due to fidgeting, with pressure sensors attached to a chair.