18.1.1 No Such Thing as the “Best UX Evaluation Method”

When you seek a UX evaluation method for your project, it usually is not about which method is “better,” but which one helps you meet your goals in a particular evaluation situation. No existing evaluation method can serve every purpose and each has its own strengths and weaknesses. You need to know your goals going in and tailor your evaluation methods to suit.

And goals can vary a lot. In the very early design stages you will likely have a goal of getting the right design, looking at usefulness and a good match to high-level workflow needs, which requires one kind of evaluation. When you get to later stages of refining a design and mainly want feedback about how to improve UX, you need an entirely different kind of evaluation approach.

So much has been written about which UX evaluation method is better, often criticizing methods that do not meet various presumed standards. Too often these debates continue in absolute terms, when the truth is: It depends! We believe it is not about which method is “better,” but which one helps you meet your goals in a particular evaluation situation.

In fact, each aspect of the overall evaluation process depends on your evaluation goals, including design of evaluation sessions (e.g., focus, procedures), types of data to be collected, techniques for data analysis, approaches to reporting results, and (of course) cost. Tom Hewett was talking about evaluation goals way back in the 1980s (Hewett, 1986).

18.2.1 Goals Tied to Resources

Sometimes you just cannot afford a fully rigorous UX evaluation method. Sometimes paying less is necessary; it is a rapid or “discount” method or nothing. Some criticism of discount methods is based on whether they are as effective as more expensive methods, whether they find UX problems as well as other methods such as lab-based UX testing. This misses the point: you choose rapid or “discount” evaluation methods because they are faster and less expensive! They are not generally as effective as rigorous methods, but often they are good enough within an engineering context.

Some criticism is even aimed at whether “discount” methods provide statistically significant results. How can that be an issue when there is no intention for that kind of result? A more appropriate target for critical review of “discount” methods would be about how much you get for how much you pay. Different methods meet different goals. If statistical significance is a goal, only formal summative evaluation will do, and you have to pay for it.

18.2.2 UX Evaluation Methods and Techniques by Stage of Progress

In Table 18-1 we show some simple stages of design representations such as storyboards and prototypes and representative formative evaluation approaches appropriate to those stages.

Table 18-1 Appropriateness of various formative UX evaluation approaches to each stage of progress within the project

Design walkthroughs and other design reviews are rapid and flexible methods that employ informal demonstrations of design concepts or early versions of designs to obtain initial reactions before many design details exist. Usually at this point in a project, you have only scenarios, storyboards, screen sketches, or, at most, a low-fidelity (non-interactive) prototype.

So you have to do the “driving”; it is too early for anyone else in a user role to engage in real interaction. Walkthroughs are an important way to get early feedback from the rest of the design team, customers, and potential users. Even early lab-based tests can include walkthroughs (Bias, 1991). Also, sometimes the term is used to refer to a more comprehensive team evaluation, more like a team-based UX inspection.

By the time you have constructed a low-fidelity prototype, a paper prototype, for example, UX inspection methods and lightweight quasi-empirical testing are appropriate. Inspection methods are, in practice, perhaps the most used and most useful UX evaluation methods for this stage.

Sometimes evaluators overlook the value of critical review or UX inspection by a UX expert. Unlike most participants in lab-based testing, an expert will be broadly knowledgeable in the area of interaction design guidelines and will have extensive experience in evaluating a wide variety of interaction styles. The most popular UX inspection method (Chapter 13) is the heuristic evaluation (HE) method.

High-fidelity prototypes, including programmed prototypes and operational products, are very complete design representations that merit complete or rigorous evaluation, as by lab-based testing. The RITE method is a good choice here, too, because it is an empirical method that uses participants but is rapid. Alpha and beta testing with selected users and/or customers are appropriate evaluation methods for pre-release versions of the near-final product.

Finally, you can continue to evaluate a system or product even after deployment in the field via remote surveys and/or questionnaires and remote UX evaluation, a method that has the advantage of operating within the context of real-world usage.

18.2.3 Synthesize Your Own Hybrid Method

As you gain experience, you will synthesize hybrid approaches as you go, adapting, transforming, and combining methods to suit each unique situation. For example, in a recent UX session we observed some phenomena we thought might be indicative of a UX problem but which did not lead to critical incidents occurring with users in the lab. So after the UX lab session, we added our own analytic component to investigate these issues further.

In this particular example, the system was strongly organized and presented to the user by function. Users found it awkward to map the functions into task sequences; they had to bounce all over the interface to access parts that were closely related in task flow. It took some intense team analysis to get a handle on this and to come up with some specialized local prototypes to test further just this one issue and evaluate some alternative design solutions.

This is not the kind of UX problem that will just pop out of a critical incident with a user in the lab as does, for example, a label wording commented on by a user. Rather, it is a deeper UX problem that might have caused users some discomfort, but they may have been unable to articulate the cause. Sometimes a problem requires the attention of an expert evaluator or interaction designer who can apply the necessary abstractions. If we had not been willing to do some additional analysis, not part of the original plan, we might have missed this important opportunity to improve the quality of user experience in our design.

18.3.1 Evaluate Design Concepts before Details

When practitioners do UX testing, they often think first of low-level UX, the usability about details of button placement and label wording, and most lab-based usability evaluation ends up being aimed at this level. As we said in earlier chapters, however, the fit of the design to support work practice is fundamental. Find out how they do the work and design a system to support it. What is the point of using low-level usability evaluation to hone the details of what is basically a poor, or at least the wrong, design while blocking your ability to see broader and more useful design ideas?

Our friend and colleague Whitney Quesenbery (2009) has said:

 If there is a single looming problem in UX these days, it is that usability analysts too often get caught up in enumerating detailed observations, and spend a good deal less time than they should in thinking carefully about the underlying patterns and causes of the problems. This turns UX into a sort of QA checklist, rather than letting us usability analysts be the analysis partners we should be to the designers. Some of this, of course, is a legacy of having UX evaluation done too late, some is because there is often so much to do in fixing “obvious” problems, but some is because we have not taken seriously our role in supplying insights into human behavior.

In early stages when the concepts are young, formative evaluation should focus on high-level concepts, such as support of the work with the right work roles, workflows, the right general form of presenting the design to the user, mission-critical design features (e.g., potential “hanging chads” in the design), and safety-related features.

18.3.2 User Experience In Situ vs. User Reflections

The more directly the evaluation is related to actual usage, the more precise its indicators of UX problems will be. It is difficult to beat empirical evaluation with a “think-aloud technique” for accessing immediate and precise UX data when evaluating the design of an artifact in the context of usage experience.

Indirect and subjective evaluation such as obtained by a post-performance questionnaire can be less expensive and, if focused carefully on the important issues, can be effective. Alpha and beta testing (Chapter 13) are even less direct to the usage experience, severely limiting their effectiveness as formative evaluation methods.

At least when it comes to detailed formative evaluation, Carter (2007) nicely reminds us to think in terms of “inquiry within experience,” evaluating the design of an artifact in the context of usage experience. To Carter, a survey or questionnaire is an abstraction that removes us from the user and the usage events, sacrificing a close connection to user experience in real time while the design is being used.

User surveys or questionnaires, even when administered immediately at the end of a UX testing session, are retrospective. A questionnaire produces only subjective data and, as Elgin (1995) states, “Subjective feedback is generally harder to interpret than objective feedback in a known setting… .” More importantly, survey or questionnaire data cannot be the immediate and precise data about task performance that are essential for capturing the perishable details necessary to formative UX evaluation.

For evaluating design details, it is about getting into the user’s head with “thinking-aloud” techniques. But the further the time of UX data capture from the time of occurrence, the more abstraction and need for retrospective recall, thereby the more loss of details. We looked at techniques for capturing indicators of UX problems and for capturing indicators of success with UX as observed directly within the real-time occurrence of usage experience (Chapter 12).

18.3.3 Evaluating Emotional Impact and Phenomenological Aspects

Do not just focus on usability or usefulness in your evaluations. Remember that one of your most important evaluation goals can be to give emotional impact and phenomenological aspects attention, too. Specific evaluation methods for emotional impact were detailed in Chapter 12.

18.4.1 Keep the Flexibility to Change Your Mind

Your organization is paying for evaluation, and it is your responsibility to get the most out of it. The key is flexibility, especially the flexibility to abandon some goals in favor of others in conflict situations. Your evaluation goals can be predetermined, but you can also come up with new or different goals as a result of what happens during evaluation.

As an illustration of staying flexible within goal-directed UX evaluation, consider an evaluation session in which you are doing the typical kind of UX evaluation and something goes wrong for the user. If you stop and ask the user questions about what went wrong and why, you will lose out on gathering quantitative performance data, one of your goals. However, because that data now will not be very useful, and understanding UX problems and their causes is your most important goal, you should stop and talk.

In most formative evaluation cases, when there is a conflict between capturing quantitative (e.g., task timing) and qualitative data (e.g., usability problem identification), qualitative data should take precedence. This is because when user performance breaks down, performance data are no longer useful, but the incident now becomes an opportunity to find the reason for the breakdown.

As part of your pre-evaluation preparation, you should prioritize your goals to prepare for conflicting goal situations. As Carter (2007) says, you should retain the freedom for interruption and intervention (even when the original goal was to collect quantitative data).

18.4.2 Do Not Let UX Evaluation Dogma Supplant Experience and Expertise

Give people a process they can follow and sometimes they hang on to it until it becomes a religion. We encourage you, instead, to do your own critical thinking. You still have to use your head and not just follow a “process.” Be ready to adapt and change directions and techniques. Be ready to abandon empirical testing for thoughtful expert analytic evaluation if the situation so demands.

According to Greenberg and Buxton (2008), “evaluation can be ineffective and even harmful if naively done ‘by rule’ rather than ‘by thought.’” Instead of following a plan unquestioningly, you must make choices as you go that will help you reach your most important goals. It is your job to think about and judge what is happening and where things are going within the project. Sauro (2004, p. 31) warns against “one-size-fits-all usability pronouncements” mimed and unencumbered by the thought process.

The dogma of our usual doctrine of usability testing reveres objective observation of users doing tasks. But sometimes we can evaluate a design subjectively, applying our own personal knowledge and expertise as a UX professional. As Greenberg and Buxton (2008, p. 114) put it, “Our factual methods do not respect the subjective: they do not provide room for the experience of the advocate, much less their arguments or reflections or intuitions about a design.”

Greenberg and Buxton quote a wonderful passage from the literature of architecture about the “experienced designer-as-assessor.” The quote (from Snodgrass and Coyne, 2006, p. 123) defends design evaluation by an architect designer, illustrating the fact that just because the person is a designer does not rule them out as an evaluator.

In fact, the designer has acquired a rich understanding of architectural design principles, processes, and evaluation criteria. They make the case that just because an evaluation is done subjectively by an expert, it does not mean that the results will be wild and uncontrolled. The work of expert assessors is built on a common foundation of knowledge, fundamentals, and conventions. The striking similarity to our situation should not be surprising—it is all about design.