The Distinction between Identification and Selection

The two-stage identification and selection process goes under many names. It is the essential mechanism of Darwinian evolution in biology; philosophers refer to it as universal Darwinism or evolutionary epistemology; and psychologists talk of trial-and-error learning.¹ These labels all refer to the same general two-stage process, perhaps clearest in its original evolutionary context. In the evolution of species, organisms are created with some variation that is introduced by genetic recombination and mutation mechanisms during reproduction. Then the environment selects the organisms that are the fittest. These “solutions,” the fittest organisms, are passed on to the next generation through their genetic code. Then this cycle is repeated generation after generation. Eventually, after the two-stage process—the generation (we call this “identification” in our discussion of group processes) of new organisms and then the selection of the fittest from the new generation—has repeated many times, evolution has “selected out” the organisms that are unfit for the environment. The result is “adapted,” often close-to-optimal organisms.

The identification and selection process is also the essential mechanism in the field of machine learning in modern computer science, where the process is called the genetic algorithm.² When complex learning needs to occur—suppose a program has been written to search a complex mathematical space for the solution to an equation—a basic template for a solution is set up and the program randomly tries various values. When the values generated by the random process are closer to the objectives defined for the problem, the elements of that approximate solution are rescrambled and then tested again. Solutions that are closer to the objectives are retained and rescrambled; solutions that fail are thrown away. This cycle of identification (by scrambling prior good solutions) and selection continues for thousands of trials (generations) until a good solution is found. Explorations of how to refine this identification and selection process are among the major research programs in computer science today. This same genetic algorithm machine learning approach is used in many of the breakthrough data-mining applications to find patterns in large data sets.

These same ideas are essential in psychological theories of individual creativity and concept learning. They are applied in a practical form in the methods that are used every day at IDEO, Eureka! Ranch, and many other companies that sell innovation processes to other companies.³ The recipe for innovation, used by these companies, is this same two-stage process: brainstorm to generate new solutions, then critically evaluate to select the best solutions from those identified in the first stage.

Decision-making groups do not have the time to run thousands of machine learning simulations or thousands of biological generations, but they can cycle through the two-stage identification and selection process once, twice, or three times. If conditions are favorable, even a single pass through the two separate stages produces collective solutions that are better than what can be achieved by the best individual solvers or deciders.

The identification stage focuses on the problem of generating many alternative solutions, without giving any of them a time-consuming and inhibiting evaluation. The identification stage can involve a search for solutions to add to the consideration set of possible solutions. (A company that makes computer games searches for a cutting-edge virtual reality system and acquires the company that manufactures it.) Alternatively, it could rely on refinement of a current design or the integration of several successful current designs. (The Apple iPhone is an elegant combination of the features of many precursor cell phones, each tweaked to the next level of refinement and then integrated into a dominant product.) Finally, the stage can pursue invention methods to create new solutions. (An appliance manufacturer spends hundreds of millions of dollars to reengineer the electrical motor in its products, inventing a new engine that is more powerful, weighs one-tenth as much, and operates on a magnetic pulsing principle, rather than rotating metal brushes.) For almost any problem-solving system, it is better to separate selection, with its emphasis on critical evaluations, from identification, which is best served by an uncritical, open-minded attitude. The qualities of a good selection process—critical, anxious, skeptical evaluation—are antithetical to the qualities that make for a creative, open-minded, divergent process like identification.

To introduce the conditions that make the two-stage identification and selection process work, let’s imagine a company that wants to offer a new social network product. That’s an ambitious idea, to be sure, and maybe preposterously so, but please bear with us for a bit.

How can the group solve the first problem of identifying possible networking website solutions that will be evaluated in the second selection stage? There are many examples of social networking sites out there on the internet, and a good start for the identification stage is to search for existing solutions. A quick search finds many sites: Myspace, Facebook, Google+, LinkedIn, Twitter, Meetup, VK, Pinterest.

For some companies, an existing website might be the best solution. If the company has the resources, it can identify an existing solution and then acquire it. We see this happening every day in the business news, as Google, Facebook, Microsoft, and IBM snap up smaller innovative firms. For the company in our example, the identification stage would simply need to uncover as many suitable candidate sites as possible. These would be passed on to the selection stage and scrutinized, with reference to the company’s criteria for an ideal site. Then the best already-existing site in the set of candidates would be identified and acquired.

If search is the method used to solve the identification stage, it is important that the search be broad and inclusive. You achieve breadth and inclusiveness by using many independent clues or by searching in many directions over a wide landscape of potential solutions. Many searches, independent searchers, and diverse searches are all good characteristics of a process aimed to solve the identification stage.

A second approach in the identification stage would be to refine or redesign a current site. This might mean imitating current sites, perhaps with modifications, minor or not so minor, that would produce innovative solutions. This method of finding existing good solutions and then refining them or mixing up the parts of a few of those solutions is the essence of one approach to creativity. For example, Why Not?—an insightful book on enhancing individual creativity—is focused on the use of analogies and transfers of a solution from one setting to another.⁴

If a television show is successful in Denmark, consider an American version of the same story. If a mechanism works to save money in European hotels, implement it in American motels. For our hypothetical project, consider LinkedIn. This popular (if also irritating!) networking site for workers and professionals serves career development and employee recruiting needs for its users. The invention process might consider a variation on the original website that provides networking for a specialized occupation—maybe a career networking site serving professional athletes. The site could provide special features, tuned to the needs of that profession: validation of the performance records of those seeking sponsors, contact information for sponsors seeking new athletes, coaches and training centers seeking students, and so forth.

A third approach to identification would be to invent a new solution that has not been exploited by any other sites. For example, families are the most essential human social group. So how about a site that links family members, provides family-oriented archives (photo albums, recipe files, biographies, medical records) and services (genealogy analysis, wedding planning), and attracts new members by finding connections through biological kinship links?

Sure, many of the initial solutions will be harebrained and easy to reject at first glance. Others will be rejected because they are good solutions to some problems, but not to the problem the company is trying to solve now. But if the initial identification process is productive, there will be viable, maybe even close-to-optimal solutions in the consideration set. The important characteristic of the identification phase is that it generates many relevant solutions.

These solutions then enter the selection stage of the process. During selection, members critically evaluate the solutions to select the best one. Group members usually begin this stage by reviewing the criteria that define a good solution. As noted, many of the candidates from the identification phase will be easily rejected, but every so often, there will be a surprise, in which a nonobvious solution turns out to be the fittest, or best.

To return to our example, perhaps our fictional company is looking for a strategy that would help attract Facebook users away from that dominant site. Perhaps after critical review the family-oriented website emerges as the most promising solution. Such a website might aim at the demographic that is least active on Facebook—parents and grandparents. And it might attract younger Facebook users as they age into their parental roles and become more interested in family-oriented social links. Remember, we do not think that we have invented a Facebook-beater here (though, come to think of it, a family-oriented site might not be a bad tactic), but we are using the social-website problem to illustrate how the identification and selection mechanism operates.

One secret to the success of this two-stage method is to identify the criteria for a successful solution before you even begin to identify solutions. Keeping the criteria in the back of your mind during the identification stage can guide the process in subtle, unconscious ways to find or generate better candidates. And of course, having the criteria for a good solution in hand is essential in the selection stage. There is a delicate balance here. Extensive evaluation of candidate solutions during the identification stage gums up the process, but complete disregard for the criteria can produce a massive glut of worthless candidates.

Guidelines for the Two Stages of Decision Making

Innovation companies like IDEO seem to get this balance just right in face-to-face deliberating teams. They begin the process with a thorough review of the client’s criteria for an exceptional solution, then they allow the criteria to fade into the background of consciousness and into the periphery of the group process while solutions are generated. In the selection stage, they return to the objectives to undertake a rigorous evaluation of the solutions identified.

The identification stage is where variety, independence, and diversity trump a systematic focus and even analytic reasoning. It should be obvious that a team with different perspectives, different knowledge bases, different skill sets—in short, a diverse team—is likely to generate a large and varied set of candidate solutions. And if search is the method to identify solutions, a broad search will be better. For identification, here are the important guidelines.

1. Start with well-defined objectives and criteria that will be applied later to evaluate solutions. As noted above, doing so will subtly prime the mind to produce relevant solutions that are not wildly outside the realm of acceptable solutions defined by the objectives.
2. Insulate identification from selection. It is important to discourage undue criticism and evaluation in the identification stage; those goals will be served later in the selection stage.
3. Start the group process (discussion, network-based pooling of information) with all individuals sharing their personal best solutions or best presolution concepts; then provide time between group deliberation episodes for individuals to reflect and create additional solutions stimulated by their own prior solutions and the ideas contributed by others.
4. Promote diverse solutions—within heads and between individuals—in the identification stage. In some applications, diversity is facilitated if the contributions are anonymous, thus avoiding the kinds of social influences that inhibit rigorous analysis.
5. Adopt some means of recording and remembering the solutions generated in the identification stage. Groups often do this with flip charts, Post-It-style idea icons, or electronic files. Innovation companies use visual reminders in effective ways.

For the selection phase, the guidelines look a lot different:

1. Review the criteria for an optimal solution and, if necessary, plan for concrete operational tests of the solutions passed on from the identification stage. In some applications, this is best done by adding new group members to the process who are closer to the organization’s primary leaders and who are distinguished by their focus on the team’s or organization’s mission and ultimate objectives. In other applications, the best approach might involve consulting the users of the final product of the process—for example, customers who would use or consume the solutions, citizens affected by a regulation, or others in touch with the implementation and consequences of the solution (see chapter 12).
2. Do not allow irrelevant social factors such as status, talkativeness, and likability of the sources of feedback to bias evaluations. Again, this can mean that an anonymous procedure (e.g., secret balloting) is the most effective process.
3. Adopt a decisive method of combining (independent) individual evaluations into an acceptable consensus.

The challenge of combining requirements 2 and 3 for the selection stage has led to some ingenious procedures. For example, voting systems and market mechanisms are designed to be tamper-proof (i.e., difficult to manipulate with insincere voting strategies and resistant to bubbles and “market makers”), to protect minority interests, and to maximize the chance that even those disenfranchised by the outcome of the decision will nonetheless accept it and support its implementation.⁵ Likewise, effective project review procedures provide for diverse, representative review panel membership at the same time that the actual review process is secret (and insulated from ulterior influences).

Putting the Guidelines into Practice

All this is pretty abstract. Let’s look at some concrete examples.

The first involves the title of this very book. AllOurIdeas.org is a research tool that attempts to use surveys to tap crowd wisdom. It has received a lot of attention from both public and private institutions. To take just one example, the United Nations Division for Sustainable Development has used this crowd-sourcing tool to ask people what issues should be prioritized.

As AllOurIdeas works, people are presented with a pair of options and are asked to say which they prefer. For example, should the focus be on “Getting GDP Right” or instead on “Air Quality Standards Set Forth by the World Health Organization”? People decide between them. After making that decision, they are presented with another pair of options, and then another. All of these come from a larger preexisting set of options. But users can also add their own ideas, which then appear for later viewers to evaluate. Answers are ranked in terms of the probability that they will be chosen by someone who makes a choice in comparison with a random alternative.

You should be able to see that AllOurIdeas works by facilitating both identification (people can add their own ideas) and selection (people choose). In fact, Matthew Salganik, a trailblazing sociologist at Princeton who designed the site, assures us that he sought to facilitate both goals at once. For the title of this book, our publisher put in nine ideas; people who took the survey added nine more. The titles received over two thousand votes. Our ultimate choice was not on the original list. Someone added the suggestion, it ranked near the top, and we liked it. AllOurIdeas has clearly found a clever way to combine identification and selection.

The site has a lot of success stories, and they are growing over time. For example, Catholic Relief Services, a worldwide nongovernmental organization, used the site to develop new guidelines for recruitment and career development globally.

As a kind of lark and also as a test, we added another survey on AllOurIdeas.org, asking people to identify the most serious air pollution problem in the United States. Our candidates included particulate matter, benzene, arsenic, ozone, carbon dioxide, and carbon monoxide. In our view and in the view of most experts, particulate matter should probably rank at the top of the list, with carbon dioxide (the leading greenhouse gas) being a strong competitor. But the words particulate matter do not exactly strike evoke strong reactions, and we added arsenic somewhat mischievously; it is not a serious air pollutant, but no one likes the idea of arsenic in the ambient air. Over two hundred people took the survey, and to our big surprise, particulate matter was ranked first, with carbon dioxide second, and arsenic dead last. Some crowds do turn out to be wise.

Companies frequently use in-house or outsourced brainstorming methods to identify and select good solutions for current problems. Companies like IDEO and Eureka! Ranch have provided many reports of their innovation methods.⁶ IDEO labels what we call the identification and selection mechanism the innovation funnel: the process starts broadly with open-ended, divergent thinking to generate or identify many solutions (IDEO calls this the “deep dive”) and then selects good solutions, gradually zeroing in on one or two winning solutions. The process also includes a third stage in which the initial winning solutions are prototyped, polished, and integrated to produce a final implemented solution.

At IDEO, the identification stage is accomplished by following these steps: (1) meeting with the clients to review the objectives and the limitations for an ideal solution; (2) cycling through a social brainstorming process in which solutions are generated individually, shared, and then “built on” through a constructive, uncritical, social “riffing” process; and (3) keeping track of the solutions. The selection stage includes three steps as well: (1) reviewing the objectives and sometimes adding additional objective-focused members; (2) rigorously evaluating each solution, with reference to the criteria or objectives; (3) selecting winners, usually through a simple plurality voting scheme. Sometimes only higher-ranking members of the team or the client firm participate in the final evaluation and voting, namely those who are most informed and care the most about the objectives and criteria for a good solution.

In many commercial innovation procedures, there is a third stage, design implementation, in which the best solutions from the identification and selection mechanism are prototyped, tested, and refined before final implementation. Notice that many of the most highly regarded innovations from companies like Apple and Microsoft are primarily the results of only this final refining and implementation stage. In important respects, the iPod, iPhone, and iPad were all fully “invented” before these projects were taken on by Apple, where they were refined into much more user-friendly versions by Apple’s design and engineering staffs.

Nothing at all to sneer at here—but the notion that Apple created these products is not entirely accurate. What Apple did was to select features from a universe of many similar products and then combined them and refined them into more elegant, usable objects of desire.

How Identification and Selection Processes Reduce Biases

In chapter 2, we saw many examples of how biases can undermine group performance. Fortunately, effective identification and selection processes can reduce the effects of those biases.

Some biases involve various forms of myopia—of seeking too little information or of focusing unduly on a few pieces of information. A broad, open-ended identification stage will counteract these problems. Consider availability, the bias of excessive focus on a single salient cue or item of information. An effective identification stage—in which the group looks at more than one solution and acquires information relevant to several options—is likely to reduce and eliminate availability effects, especially if there is some diversity within the group.

Framing effects are also a form of closed-minded thinking. If everyone in a group shares a single frame for the problem, it is likely that the group decision will amplify any biases produced by that perspective. Framing is one micromechanism underlying group polarization. An enhanced identification stage will increase the chances that an alternative view will be introduced into the discussion, thus reducing the tendency to polarize.

Confirmation bias can also be reduced by beginning with an open-ended group process; separating the two stages and enhancing the identification stage should improve performance further. Obviously, a rigorous identification process will reduce or eliminate egocentric biases, which are likely to remain at the group level only when there are traces of groupthink and overconformity in the first place. Hindsight bias can be viewed as a myopic focus on the outcome that actually occurred, making it difficult to regain the wide-open view of alternative possibilities that is characteristic of a realistic, future-directed view. Here too, open, unconstrained group discussion reduces the bias. An improved identification stage can help even more.

The planning fallacy is a form of myopia, focusing on one streamlined scenario. And again, a rigorous identification stage can shift the focus from one scenario to a more complex representation of the situation. At its best, rigorous identification techniques will simultaneously change the overall strategy from a so-called inside view to comparisons to relevant cases, or an outside view.⁷

Executing a separate and rigorous second selection stage can also reduce some judgment biases, especially those associated with incomplete thinking about the consequences and objectives of an action. Importantly, a thorough application of a future-oriented analysis will counteract the tendency to escalate a commitment to a losing strategy or to honor sunk costs.

We acknowledge that some biases are unlikely to be reduced by even an enhanced, two-stage process: these include those created by the representativeness heuristic, or similarity-based thinking. This powerful, intuitive habit is not simply a matter of myopic thinking and is not easily solvable with more-systematic evaluations. The only remedy that we know is training in analytic, statistical reasoning. With individual expertise and an organizational culture that promotes logical, data-based arguments, there is hope of counteracting the use of such heuristics.

Cost-Benefit Analysis

What, exactly, are more-systematic evaluations? We have suggested that they are based on more analytic thinking and on data. Within the federal government, systematic evaluation goes by a specific name: cost-benefit analysis. For regulatory decisions, President Reagan required such analysis in 1981. In particular, he said that to the extent permitted by law, regulatory agencies could not proceed with a regulation unless the benefits outweighed the costs and unless the chosen approach “maximized net benefits.”

The latter requirement means that even if an agency identifies an approach with benefits in excess of costs, the agency has to investigate whether another approach might have even higher net benefits. Reagan’s ideas applied across a spectacularly wide range, covering regulations meant to protect the environment, increase food safety, reduce risks on the highways and in the air, promote health care, improve immigration, affect the energy supply, or increase homeland security. Revealingly, Reagan’s approach won bipartisan approval, and the requirements of cost-benefit balancing and of maximizing net benefits have been part of American government for well over thirty years.

A large part of Sunstein’s job at OIRA was to promote compliance with those requirements. Within the federal government, cost-benefit balancing operates as an indispensable safeguard against behavioral biases. Suppose, for example, that bad incidents have occurred in the recent past—say, cases of food poisoning. There is a natural inclination to respond with a regulatory requirement. But would the requirement do much good? How much would it cost? Cost-benefit analysis puts a spotlight on the right questions.

When Sunstein was in government, a member of the president’s cabinet once asked him, “Cass, how can you put a price on a human life?” That’s a good question, but economists have an answer. (We’ll spare you the details, but it’s about $9 million. If that seems alarming, note that the better way to think about it is that if the government is eliminating a mortality risk of one in a hundred thousand, it would be willing to spend about $90. The reason is that evidence suggests that the average person would spend that amount to reduce a mortality risk of that magnitude.) Whether or not that answer is right, no government can or will spend an infinite amount to save a single life—which means that trade-offs are inevitable. We need to make the right ones.

Within government, cost-benefit analysis is an indispensable safeguard not only against individual biases, but also against group errors. If people are prone to availability bias, and thus too fearful (because a disaster has occurred in the recent past) or too complacent (because no such event has occurred), cost-benefit analysis imposes real discipline and helps to counteract the bias. And if group polarization leads people toward either extreme action or inaction, cost-benefit analysis imposes a reality check. Cost-benefit analysis also supplies numbers that weaken any particular framing effects. Time and again, cost-benefit analysis operates as a check both on individual error and on potentially large, even tragic mistakes at the group level. People are a lot safer, and the economy is a lot more efficient, as a result.

Often the best evidence informing the use of cost-benefit analysis comes from randomized controlled trials, by which researchers vary an aspect of a situation for one population and compare the outcomes to those for an otherwise similar population. In medicine, of course, randomized trials are the gold standard, telling us what treatments really work. Increasingly, policy analysts use such trials as well, asking (for example) whether a fee for the use of plastic bags really diminishes the use of such bags, or whether reductions in cell phone use have a beneficial effect on highway safety, or whether and to what extent people’s eating habits change when they are informed of calorie counts. For businesses, randomized controlled trials are common as well, and they can provide a lot of information about products and innovations. We should expect far more use of such trials in the future—not least because they promise to overcome some of the problems associated with group decisions.

Although cost-benefit analysis is not our main topic here, it is important to emphasize that for the private and public sector, that form of analysis is increasingly feasible and is often the best safeguard at the selection stage. Big data, coming from randomized trials, can provide a lot of information about both costs and benefits—for example, about likely consumer reactions and about possible surprises. The best check on behavioral biases, at both the individual and the group level, is to get systematic about the facts. Moneyball works in baseball, and we have only started to see its potential in medicine, business, and government.