Chapter 7
Iterate, Iterate, Iterate!
Abstract
Getting our assumptions out on the table (whether to align the internal teams or to engage with external stakeholders) is necessary but not sufficient. Our first engagement with users will likely raise more questions than answers. To get the most out of PrD, we must move through the cycle multiple times. Iterating is key to design and a fundamental aspect of agile software development. PrD is a little different: We iterate in service of exploring the problem space as opposed to refining the solution. In Iterate, Iterate, Iterate! we argue against the notion that plans well executed are the only measure of a project’s success. As we illustrate in this chapter, abandoning initial plans is exactly what PrD is about, and to good purpose.
Keywords
iteration
risk reduction
small samples
time
good enough
The phoenix must burn to emerge.
—Janet Fitch
There is no harm in repeating a good thing.
—Plato
Overview
The principle of Iterate, Iterate, Iterate requires us to repeatedly expose our assumptions to our external stakeholders. Through each cycle, we:
• Vet and refine those assumptions.
• Build our knowledge about our stakeholders and their needs.
• Increase our knowledge about the problem space in which we’re working.
With each subsequent iteration, our assumptions, artifacts, and ideas increasingly resonate with external stakeholders.
PrD is like a magic trick. We present our canard in all seriousness. We vanish it and then make it reappear as something slightly more on-point, useful, appealing, and compelling. But, unlike a magic trick, we repeat it over and over until, finally, the canard manifests as something beautiful.
Each iteration tests the internal team’s assumptions, expressed as an artifact and task list. PrD is based on this axiom: Although our stakeholders cannot design the future, they can (and most certainly will) criticize it when it’s shown to them. The external stakeholders will tear our design and ideas apart. When they see something that just isn’t right, and we invite them to critique it, they will reveal their latent and inherent needs to us. When they do, we want as many team members as possible to observe it firsthand. There isn’t a faster and more compelling way to disseminate “ahas” across a team (Figure 7.1).
Figure 7.1 Creation Session team members brainstorming on their assumptions
In The Case of the Balking Bicyclists, the artifact used in the first iteration was thoroughly trashed by external stakeholders. By the third iteration, the team deeply understood their external stakeholders’ problem space and had honed in on possible solutions. In The Case of the Preoccupied Passersby, the first iteration of the researchers’ artifact was a low-fidelity game. The team used it to discover key requirements for the final installation, including how much time residents of the city of Chicago would take to explore ideas about the city’s arts and culture.
As the internal team progresses in its learnings, the artifacts evolve. In the first couple of Engagement Sessions, stakeholders may tear apart assumptions about what they do and how they do it. Maybe in the next few sessions they’re tearing apart assumptions about their main goals. As the internal team produces ideas with greater resonance, the focus of critique shifts—from underlying assumptions to details of the designs. This is how teams iterate from problem to solution space. At first the internal team will have to swallow its pride as stakeholders tear apart its misconceptions and ignorance of the problem space. Soon they’re tearing apart the details, until, eventually, the tearing apart becomes nitpicks. This is when we know PrD has served its purpose (Figure 7.2).
Figure 7.2 How PrD scales with project focus
Iterating Is Not Wasted Effort
Iterating entails throwing stuff in the garbage. From a waterfall (i.e., “fail-late development”) perspective, throwing stuff out and starting over sounds like added cost. It sounds expensive, and it is expensive—if it happens late in the cycle. But, as we’ve said before, ideas are cheap. With PrD, we test the validity of these ideas economically (as we discuss in Chapter 8). More importantly, by engaging with stakeholders early, those tests have their greatest impact. The more ideas we vet (and the earlier we vet them), the more confident we can be in our final solution. The key is to test and iterate ideas quickly.
Iteration reduces the real risk of downstream wasted effort. The iterative throwing out of ideas prevents early, small errors from having later, large impacts.1 In highly structured, phase-gate development practices (such as waterfall), our initial plans become the measure of our success: How well did we achieve what we said we were going to do? But PrD (as in agile-type cultures) suggests our initial plans should be treated as irrelevant—since we don’t know what the problem is, how can we trust a plan to solve it?
Iteration Begins Day One
A crucial difference between PrD and traditional user-centered design (UCD) is in PrD we iterate on Day One, before any important decisions have been made. In contrast, in traditional UCD, iteration typically occurs after many of the key decisions have long been made. A crucial moment in any development project is the point of decision: the point when the team decides what the final solution will be. (Recall from Chapter 2, the Double Diamond diagram. See Chapter 8 for more on this topic.) The time before that point is the “problem space” and the time after it the “solution space.” PrD and traditional UCD differ the most in how each process approaches the problem space.
Of course, no project has a single, isolated decision where the team agrees on the final solution. In practice the “decision point” may be encountered several times. Nevertheless, at some point the team does shift from defining the problem to refining a solution to it.
We distinguish between predecision and postdecision project work. When designers speak of “iterative design” they’re typically referring to the iterative testing and refining of prototypes past the point of decision. Iterative design normally refers to iteration in the solution space, not iteration in the project’s problem space. In traditional UCD (even in projects considered “iterative” and “agile”), the problem space is handled in a waterfall-like or stepwise fashion. During the phases preparing for the decision point, product marketing managers conduct primary research. Finance and strategists determine pricing and impacts on production and operations. These activities culminate in a product definition, a fairly lengthy document with hundreds of requirements.
Implicit in those requirements are hundreds of unspoken assumptions about what the product will do. Those assumptions aren’t tested predecision point because they aren’t perceived as having an impact on strategic decisions. By design, phase-gate development delays investment in problem solving until after the decision point. This makes sense in the phase-gate world: Before the decision point we focus solely on the elements required to get to decision. Anything after that will be handled in the postdecision phase.
The challenge with that approach is many of the requirements going into the decision point are based on unstated (and unknown) assumptions about the stakeholders’ needs (as we discussed in the prior chapter). Those assumptions have significant impact downstream as the development cycles bump into them and as external stakeholders ultimately review them.
PrD approaches both the problem and solution spaces iteratively. As a result, the team’s confidence in the outcome of the project is raised prior to the decision point.
Iterating is a Risk Reduction Strategy
PrD’s iteration of problem definition makes it an agile approach, distinguishes it from traditional UCD, and reduces risk. In such a risk-reduced project, teams are confident their offering will solve the right problems before setting out to build it the best they can. We’ve rarely experienced or heard of companies using this approach, even though it sounds logical. Those numbers are changing because market dynamics are forcing companies to change (as we discussed in Chapter 3).
Historically, management has been rewarded for efficiency (doing things right) and not effectiveness (doing the right thing).2 Time and time again, organizations push solutions weighted toward what can be built, whether they are out of the box from third-party vendors or small incremental changes to existing offerings. Defenders of this approach put up a rational argument: Keeping development costs as low as possible reduces the risk of a bad ROI. This argument plays well with the agilists: Deliver the smallest increment of work returning the highest business value.
As the Pharaoh’s story illuminated in Chapter 6, all of that is for naught if the team is focused on the wrong target. If the team is only concerned about the cost of development, it misses a much larger potential ROI. For example, incremental cost of development pales in comparison to end-user loss of productivity, time on training, customer support, and other externalities.3 And those costs only look at productivity losses. When we factor in higher adoption rates because users want to use the product, entire businesses flourish.4,5
In contrast, PrD starts by identifying the right thing to build. PrD iterates throughout the problem definition cycle beginning on Day One.
Moving fast is key. We not only start as soon as possible but also do only as many Engagement Sessions as necessary. We don’t bog ourselves down by scheduling more than a few stakeholders per iteration. We learn more by running more iterations. More iterations with fewer external stakeholders are always better than fewer iterations with larger samples. Typically, three Engagement Sessions per iteration (each with one external stakeholder) are enough; five or six are plenty.
Here’s the math (see Figure 7.3): Assume we have one hour each with 15 external stakeholders per month. The preferred approach is to complete five Engagement Sessions, create a new artifact, complete five more, rework the artifact (as needed), and then do five more. In this model we complete six iterations in two months. If, instead, we engage all 15 stakeholders in a single iteration in the first month; rework the artifact and then do 15 more Engagement Sessions the following month, it’s the same amount of time (two months), the same amount of stakeholder investment (one session per month), and the same number of stakeholders (15). But in the second approach we iterate only twice, one-third the number in the first approach.6
Figure 7.3 More iterations are better than few. Reduce the number of stakeholders per iteration
We iterate to learn. When we limit the number of iterations, we reduce the opportunity to discover how wrong we are and in what ways. Ultimately our goal is not to run bunches of Engagement Sessions; we’re trying to test our assumptions and throw lots of ideas away. How many times should we turn the crank? We iterate until the team runs out of money or time or no longer learns anything new. These three resources act as operating constraints on the process.
Time
PrD, because of its reliance on immediate engagement with external stakeholders, is relatively quick as compared to other research approaches. Still, PrD requires some minimum amount of time to complete, below which it will fail to deliver value. As we’ll discuss in greater detail in Chapter 8, time is always the enemy. The team must move with a sense of urgency. “Art is never finished, only abandoned,” as da Vinci is quoted as saying. The same can be said about products and services. They can always be improved more. Given diminishing returns on investment, the real question is always, “What’s good enough?”
For modest projects, PrD may take only a few weeks to complete (durations vary based on calendar and availability). For small projects a few days. PrD doesn’t need a lot of time.
Every team complains about schedules and a lack of enough time. Assuming the team has enough time to do at least a minimum number of iterations, PrD’s value (artifacts, interviews, and the process itself) doesn’t necessarily increase with more time. The opposite is true: When the team faces time pressures, the process delivers extraordinary results. The challenge, then, is to ensure there is enough time to fit as many iterations as possible (as we’ll see in the next chapter) and maximize learning in each engagement.
If time is a limiting factor, the team must identify the highest-value objectives they need to evaluate. These discussions occur with key internal stakeholders who help determine the value propositions for the project. In the interest of time, the team forgoes any objectives falling below the value line (we discuss objectives in detail in Chapter 11). In the event the team doesn’t have enough time to accomplish key objectives, this constitutes a checkpoint on the project, requiring sponsors to make the call for next steps.
Money
With a few exceptions, PrD is most effective when teams work face-to-face, whether during Creation Sessions or Engagement Sessions. The key is to maximize the number of iterations the team can complete with the least cost. Given the bias toward face-to-face, merely relying on digital and remote collaboration technologies won’t necessarily solve the problem. There are hidden costs of doing PrD digitally and remotely—costs that aren’t immediately obvious.
We try to have at least a few face-to-face sessions with our target stakeholders, in their context, with the artifacts in hand. There is nothing more powerful than to have external stakeholders demonstrate how awful an artifact is, in person, in the real context of their work. Any detraction from this ideal scenario reduces insights. Not interacting with the stakeholder face-to-face, not handing her a physical artifact, or not conducting the session where she does her tasks negatively impacts the results. What if the artifact is so wrong that it inspires a stakeholder to pick up something in his environment and demonstrate it as an exemplar addressing his needs (as the elderly women did in The Case of the Pushy Pillboxes)? Dramatic learnings increase when sessions are executed in person and on location.*
When we were working with sales teams in service of a large-scale PrD effort, a salesman demonstrated, in his cubicle, how he used business cards stacked in particular ways to manage his many contacts. In a separate instance, Charles observed a saleswoman leave her cubicle, go into the aisle, and describe an artifact she and her team used on a whiteboard. “Being there” has far greater ROI (in terms of capturing stakeholder needs or maximizing understanding) than snazzing up artifacts. When faced with crafting fancy artifacts versus traveling, we never flinch: Travel wins every time.
In The Case of the Rx Reminder, key learnings and observations were made by working with stakeholders in their own homes. Only by asking stakeholders to use the artifacts during their daily routine could the team have discovered the need for mobility in both the reminder and the pillbox. Similarly for The Case of the Star Trek Holodeck: Only by being present with stakeholders, in the moment, could Steve and his team have gleaned the key insights they uncovered. In The Case of the Transformed Treatment, only by being collocated with patients, therapists, and caregivers could Chris discover a key element to his innovative therapy.
Learning Nothing New
As with many field research techniques, PrD starts to “discover” the same stories over and over again. In qualitative research this is called “saturation.” Often, what was novel in the first couple of sessions becomes predictable by the third. Great facilitators learn to maintain freshness in their sessions, even if they could write the new stakeholder’s story themselves. PrD requires us to stop inquiring about the stories we have heard more than a handful of times. If we don’t, we’re not increasing our learnings. If we’ve heard the same thing three times, that may be enough. A couple more for good measure if we can afford the time or money, otherwise it’s not worth hearing again. But that is not to say we’re done! There are many more stories likely waiting to be told—just not the ones we’ve already heard. The beauty of PrD is how flexible, fluid, and adjustable the process is.
Leo was visiting several customers in the same city to save on travel costs. After the first day of meetings, the team met that night in the hotel bar to go over what they’d heard. They quickly identified topics they’d heard enough about and adjusted the artifact and prompts accordingly. The artifact had left a particular interface very ambiguous, allowing the stakeholders to tell the team what it meant to them. In almost all cases, the external stakeholders offered similar interpretations. They suggested how the interface should operate and what their underlying needs were. The team altered the artifact that night to explicitly expose that interpretation. The script for the subsequent interviews assumed stakeholders would interpret the revised artifact as the prior stakeholders had. When new stakeholders interacted with the modified artifact, the team listened carefully. If these stakeholders moved through the interface without any conflict or questions, they would consider this interpretation valid. Instead of continuing to hear the same interpretation again, the team could focus on areas they hadn’t had time to get to in prior interviews. Because the team iterated multiple times in the problem space, it increased its confidence and the quality of its product.
If the team truly isn’t learning anything new, it’s time to stop, even if they have the time and the money to keep going. Typically there is something new to learn; usually time or money comes up short before a lack of learning.
Risk Factors
Project Constraints
Anything constraining the team’s ability to do multiple iterations is considered a risk to Iterate, Iterate, Iterate! We’ve already discussed the three constraints on iteration: Time, Money, and Learning Nothing New. Sometimes time and money are so constrained that there’s not enough to do more than one iteration. This risk is serious enough to warrant a checkpoint on the program. If the program can’t afford to do PrD, the least expensive process available, it is likely not prepared to sincerely engage with stakeholders in the first place.
So, if there are insufficient resources to do any PrD iterations, we recommend finding another program to apply your talents. But what if there is some time and money, perhaps enough to do just one iteration? Would doing one iteration be better than none? In companies where UX’s value is still emerging, practitioners have learned to make excellent-tasting lemonade, so the answer is usually “Yes.” Of course, doing something is far better than nothing, but in this case, the risk is around messaging. If our sponsors think we’re going to get them the answers they need, reduce their program risk, and all of the other goodness we’ve sold them around PrD, we need to qualify those promises based on the actual budget.
Returning information from one iteration is only slightly less risky than none at all. One iteration is almost not worth doing if we’re only talking to one stakeholder. At minimum, even if we can only do one iteration, we still need a handful of individuals—three to six at minimum depending on how broad or tight the investigation is.
PrD remains the least expensive, richest method of design research we’ve encountered, but it still costs something. If the project is so constrained that it’s limited to one iteration, we will either pull the emergency stop cable and get off or lower our stakeholders’ expectations.
Stalling
As this chapter has said repeatedly, and as we’ll discuss in greater depth in the next chapter, time is of the essence. The team can be its own worst enemy by failing to move with the speed and urgency PrD demands. Stalling can come from any quarter: Engineering can raise technical concerns, program managers can declare customers unavailable, designers can demand “one more day” to polish an idea, and so on. None of these, or the dozens of other examples of delays and hurdles, do anything to help PrD. Delaying does not improve the results. We start with the proposition we are wrong (as discussed in Chapter 4) and so, why wait?
One antidote to stalling is to set unreasonable deadlines. Working with the program manager, we set impossible due dates. This forces the conversations PrD demands, and it actually works. In The Case of the Hard-Boiled Egg, when the team appeared to be entering a multiday design session to figure out what the screens should look like, Leo demanded they get results within a day or two—long before they would have completed their design effort, let alone user engagements. Similarly, in The Case of the New Case, the team had only a few days to come up with alternatives to what they had already crafted. An hour of whiteboard sketching with key engineers was sufficient to establish enough technical confidence to proceed with refined Photoshop imagery.
Summary
• PrD differs from traditional UCD in how it approaches the problem space of a project, the portion of the project work prior to the “decision point.”
• PrD distinguishes itself from iterative design by focusing on iterative problem definition. Where iterative design refers to the iterative refinement of a single solution, in PrD “iterating” refers to the iterative abandoning of ideas while still in problem space. (We’re not iterating ideas if we pursue only a single solution.)
• Time to insight is improved by doing more iterations with fewer external stakeholders versus fewer iterations with more. Larger sample sizes here are a waste of time and money.
• PrD doesn’t benefit from more time. Just the opposite is true: By creating a sense of urgency, the team is forced to offer its assumptions to external stakeholders as early as possible.
• Iterating continues until the team runs out of time, money, or new learning. If there is insufficient time or money to do more than one iteration, PrD (or any other research mechanism) will likely perform below expectations. Adjust accordingly.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.