PREFACE

A LABEL

Human error is a very elusive concept. Over the last three decades, we have been involved in discussions about error with many specialists who take widely different perspectives – operators, regulators, system developers, probability reliability assessment (PRA) specialists, experimental psychologists, accident investigators, and researchers who directly study “errors.” We are continually impressed by the extraordinary diversity of notions and interpretations that have been associated with the label “human error.” Fifteen years after the appearance of the first edition of Behind Human Error (with the subtitle Cognitive Systems, Computers and Hindsight) published by the Crew Systems Information and Analysis Center (CSERIAC), we still see organizations thinking safety will be enhanced if only they could track down and eliminate errors.

In the end, though, and as we pointed out in 1994, “human error” is just a label. It is an attribution, something that people say about the presumed cause of something after-the-fact. It is not a well-defined category of human performance that we can count, tabulate or eliminate. Attributing error to the actions of some person, team, or organization is fundamentally a social and psychological process, not an objective, technical one. This book goes behind the label “human error” to explore research findings on cognitive systems, design issues, organizational goal conflicts and much more.

Behind the label we discover a whole host of complex and compelling processes that go into the production of performance – both successful and erroneous, and our reactions to them. Research on error and organizational safety has kept pace with the evolution of research methods, disciplines and languages to help us dig ever deeper into the processes masked by the label. From investigating error-producing psychological mechanisms in the early 1980s, when researchers saw different categories of error as essential and independently existing, we now study complex processes of cross-adaptation and resilience, borrow from control theory and complexity theory, and have become acutely sensitive to the socially constructed nature of the label “human error” or any language used to ascribe credit or blame for performances deemed successful or unsuccessful. Indeed, the book examines what goes into the production of the label “human error” by those who use it, that is, the social and psychological processes of attribution and hindsight that come before people settle on the label.

The realization that human error is a label, an attribution that can block learning and system improvements, is as old as human factors itself. During the Second World War, psychologists were mostly involved in personnel selection and training. Matching the person to the task was considered the best possible route to operational success. But increasingly, psychologists got pulled in to help deal with the subtle problems confronting operators of equipment after they had been selected and trained. It became apparent, for example, that fewer aircraft were lost to enemy action than in accidents, and the term “pilot error” started appearing more and more in training and combat accident reports. “Human error” became a catch-all for crew actions that got systems into trouble. Matching person to task no longer seemed enough. Operators made mistakes despite their selection and training.

Yet not everybody was satisfied with the label “human error.” Was it sufficient as explanation? Or was it something that demanded an explanation – the starting point to investigate the circumstances that triggered such human actions and made them really quite understandable? Stanley Roscoe, one of the eminent early engineering psychologists, recalls:

It happened this way. In 1943, Lt. Alphonse Chapanis was called on to figure out why pilots and copilots of P-47s, B-17s, and B-25s frequently retracted the wheels instead of the flaps after landing. Chapanis, who was the only psychologist at Wright Field until the end of the war, was not involved in the ongoing studies of human factors in equipment design. Still, he immediately noticed that the side-by-side wheel and flap controls – in most cases identical toggle switches or nearly identical levers – could easily be confused. He also noted that the corresponding controls on the C-47 were not adjacent and their methods of actuation were quite different; hence C-47 copilots never pulled up the wheels after landing. (1997, pp. 2–3)

“Human error” was not an explanation in terms of a psychological category of human deficiencies. It marked a beginning of the search for systemic explanations. The label really was placeholder that said, “I don’t really know what went wrong here, we need to look deeper.” A placeholder that encouraged further probing and investigation. Chapanis went behind the label to discover human actions that made perfect sense given the engineered and operational setting in which they were planned and executed. He was even able to cross-compare and show that a different configuration of controls (in his case the venerable C-47 aircraft) never triggered such “human errors.” This work set in motion more “human error” research in human factors (Fitts and Jones, 1947; Singleton, 1973), as well as in laboratory studies of decision biases (Tversky and Kahneman, 1974), and in risk analysis (Dougherty and Fragola, 1990).

The Three Mile Island nuclear power plant accident in the US in the spring of 1979 greatly heightened the visibility of the label “human error.” This highly publicized accident, and others that came after, drew the attention of the engineering, psychology, social science, regulatory communities and the public to issues surrounding human error. The result was an intense cross-disciplinary and international consideration of the topic of the human contribution to risk. One can mark the emergence of this cross-disciplinary and international consideration of error with the “clambake” conference on human error organized by John Senders at Columbia Falls, Maine, in 1980 and with the publication of Don Norman’s and Jim Reason’s work on slips and lapses (Norman, 1981; Reason and Mycielska, 1982).

The discussions have continued in a wide variety of forums, including the Bellagio workshop on human error in 1983 (Senders and Moray, 1991). During this workshop, Erik Hollnagel was asked to enlighten the audience on the differences between errors, mistakes, faults and slips. While he tried to shrug off the assignment as “irritating,” Hollnagel articulated what Chapanis had pointed out almost four decades earlier: “‘human error’ is just one explanation out of several possible for an observed performance.” “Human error” is in fact, he said, a label for a presumed cause. If we see something that has gone wrong (the airplane landed belly-up because the gear instead of the flaps was retracted), we may infer that the cause was “human error.” This leads to all kinds of scientific trouble. We can hope to make somewhat accurate predictions about outcomes. But causes? By having only outcomes to observe, how can we ever make meaningful predictions about their supposed causes except in the most rigorously deterministic universe (which ours is not)?

The conclusion in 1983 was the need for a better theory of human systems in action, particularly as it relates to the social, organizational, and engineered context in which people do their work. This call echoed William James’ functionalism at the turn of the twentieth century, and was taken up by the ecological psychology of Gibson and others after the War (Heft, 1999). What turned out to be more interesting is a good description of the circumstances in which observed problems occur – quite different from searching for supposed “psychological error mechanisms” inside an operator’s head. The focus this book is to understand how systematic features of people’s environment can reasonably (and predictably) trigger particular actions; actions that make sense given the situation that helped bring them forth. Studying how the system functions as it confronts variations and trouble reveals how safety is created by people in various roles and points to new leverage points for improving safety in complex systems.

The meeting at Bellagio was followed by a workshop in Bad Homburg on new technology and human error in 1986 (Rasmussen, Duncan, and Leplat, 1987), World Bank meetings on safety control and risk management in 1988 and 1989 (e.g., Rasmussen and Batstone, 1989), Reason’s elaboration of the latent failure approach (1990; 1997), the debate triggered by Dougherty’s editorial in Reliability Engineering and System Safety (1990), Hollnagel’s Human Reliability Analysis: Context and Control (1993) and a series of four workshops sponsored by a US National Academy of Sciences panel from 1990 to 1993 that examined human error from individual, team, organizational, and design perspectives. Between then and today lies a multitude of developments, including the increasing interest in High Reliability Organizations (Rochlin, 1999) and its dialogue with what has become known as Normal Accident Theory (Perrow, 1984), the aftermath of two Space Shuttle accidents, each of which has received extensive public, political, investigatory, and scholarly attention (e.g., Vaughan, 1996; CAIB, 2003), and the emergence of Resilience Engineering (Hollnagel, Woods and Leveson, 2006).

Research in this area is charged. It can never be conducted by disinterested, objective, detached observers. Researchers, like any other people, have certain goals that influence what they see. When the label “human error” becomes the starting point for investigations, rather than a conclusion, the goal of the research must be how to produce change in organizations, in systems, and in technology to increase safety and reduce the risk of disaster. Whether researchers want to recognize it or not, we are participants in the processes of dealing with the aftermath of failure; we are participants in the process of making changes to prevent the failures from happening again.

This means that the label “human error” is inextricably bound up with extra-research issues. The interest in the topic derives from the real world, from the desire to avoid disasters. The potential changes that could be made in real-world hazardous systems to address a “human error problem” inevitably involve high consequences for many stakeholders. Huge investments have been made in technological systems, which cannot be easily changed, because some researcher claims that the incidents relate to design flaws that encourage the possibility of human error. When a researcher asserts that a disaster is due to latent organizational factors and not to the proximal events and actors, he or she is asserting a prerogative to re-design the jobs and responsibilities of hundreds of workers and managers. The factors seen as contributors to a disaster by a researcher could be drawn into legal battles concerning financial liability for the damages and losses associated with an accident, or even, as we have seen recently, criminal liability for operators and managers alike (Dekker, 2007). Laboratory researchers may offer results on biases found in the momentary reasoning of college students while performing artificial tasks. But how much these biases “explain” the human contribution to a disaster is questionable, particularly when the researchers making the claims have not examined the disaster, or the anatomy of disasters and near misses in detail (e.g., Klein, 1989).

FROM ELIMINATING ERROR TO ENHANCING ADAPTIVE CAPACITY

There is an almost irresistible notion that we are custodians of already safe systems that need protection from unreliable, erratic human beings (who get tired, irritable, distracted, do not communicate well, have all kinds of problems with perception, information processing, memory, recall, and much, much more). This notion is unsupported by empirical evidence when one examines how complex systems work. It is also counterproductive by encouraging researchers and consultants and organizations to treat errors as a thing associated with people as a component – the reification fallacy (a kind of over-simplification), treating a set of interacting dynamic processes as if they were a single object.

Eliminating this thing becomes the target of more rigid rules, tighter monitoring of other people, more automation and computer technology all to standardize practices (e.g., “…the elimination of human error is of particular importance in high-risk industries that demand reliability.” Krokos and Baker, 2007, p. 175). Ironically, such efforts have unintended consequences that make systems more brittle and hide the sources of resilience that make systems work despite complications, gaps, bottlenecks, goal conflicts, and complexity.

When you go behind the label “human error,” you see people and organizations trying to cope with complexity, continually adapting, evolving along with the changing nature of risk in their operations. Such coping with complexity, however, is not easy to see when we make only brief forays into intricate worlds of practice. Particularly when we wield tools to count and tabulate errors, with the aim to declare war on them and make them go away, we all but obliterate the interesting data that is out there for us to discover and learn how the system actually functions. As practitioners confront different evolving situations, they navigate and negotiate the messy details of their practice to bridge gaps and to join together the bits and pieces of their system, creating success as a balance between the multiple conflicting goals and pressures imposed by their organizations. In fact, operators generally do this job so well, that the adaptations and effort glide out of view for outsiders and insiders alike. The only residue left, shimmering on the surface, are the “errors” and incidents to be fished out by those who conduct short, shallow encounters in the form of, for example, safety audits or error counts. Shallow encounters miss how learning and adaptation are ongoing – without these, safety cannot even be maintained in a dynamic and changing organizational setting and environment – yet these adaptations lie mostly out of immediate view, behind labels like “human error.”

Our experiences in the cross-disciplinary and international discussions convince us that trying to define the term “error” is a bog that quite easily generates unproductive discussions both among researchers and between researchers and the consumers of research (such as regulators, public policy makers, practitioners, and designers). This occurs partly because there is a huge breadth of system, organizational, human performance and human-machine system issues that can become involved in discussions under the rubric of the term “human error.” It also occurs because of the increasing complexity of systems in a highly coupled world. The interactional complexity of modern systems means that component-level and single causes are insufficient explanations for failure. Finally, discussions about error are difficult because people tightly hold onto a set of “folk” notions that are generally quite inconsistent with the evidence that has been gathered about erroneous actions and system disasters. Not surprisingly, these folk theories are still prevalent in design, engineering, researcher and sometimes also practitioner communities. Of course, these folk notions themselves arise from the regularities in how we react to failure, but that is what they are: reactions to failure, not explanations of failure.

To get onto productive tracks about how complex systems succeed and fail – the role of technology change and organizational factors – one must directly address the varying perspectives, assumptions, and misconceptions of the different people interested in the topic of human error. It is important to uncover implicit, unexamined assumptions about “human error” and the human contribution to system failures. Making these assumptions explicit and contrasting them with other assumptions and research results can provide the impetus for a continued substantive theoretical debate.

Therefore, the book provides a summary of the assumptions and basic concepts that have emerged from the cross-disciplinary and international discussions and the research that resulted. Our goal is to capture and synthesize some of the results particularly with respect to cognitive factors, the impact of computer technology, and the effect of the hindsight bias on error analysis. While there is no complete consensus among the participants in this work, the overall result is a new look at the human contribution to safety and to risk. This new look continues to be productive generating new results and ideas about how complex systems succeed and fail and about how people in various roles usually create safety.