6
Error Analysis and Expert-Novice Differences in Medical Diagnosis
University of Manchester
Training and support for diagnosis are often based on a narrow definition of the cognitive skills involved in this critical task. The present chapter examines the complexity of medical diagnosis by analyzing errors and comparing expert and novice diagnostic strategies. The results suggest that diagnosis may involve as many as five interacting lines of reasoning. Training and decision support in complex, dynamic environments should take account of all the cognitive operations these imply.
What is diagnosis? The word entered the English language in the seventeenth century, introduced from the ancient Greek writings of Galen and Hippocrates. Its meaning was derived from the original Greek words dia (differentiation) and gnosis (knowledge) “to know the situation that confronts you well enough to differentiate it from other situations with which it might be confused.” This has remained the everyday meaning of the word up to the present time. It implies the ability to understand a problem as the basis for taking action, an essential requirement for the supervisor of any continuous process.
In many technical contexts, however, the word has acquired a much narrower meaning. In fault-finding, diagnosis usually means the localization of a specific defect which is causing the system to malfunction. In medicine, it usually means the identification of the precise pathophysiology which is causing a patient’s illness. The argument of the present chapter is that this concept of diagnosis is too narrow to serve as a basis for designing training and support in complex, dynamic environments. Diagnosing dynamic systems under conditions of uncertainty involves a much wider range of cognitive operations than the convergent search process implied by localizing a fault or naming a disease. This argument is developed by examining the case of medical diagnosis.
EXPERTISE IN MEDICAL DIAGNOSIS
We begin with a brief review of research in the psychology of medical diagnosis. Early work in this field assumed that expertise depended on skill at hypothesis testing. Many different versions of a hypothetico-deductive model of diagnostic reasoning were proposed. The following is typical, diagnosis being represented as a sequence of stages.
1. hypothesis generation (i.e., listing the diseases which are consistent with the patient’s complaint);
2. information search (i.e., gathering further data relevant to each disease hypothesis);
3. data interpretation (i.e., classifying abnormal findings as confirming (occurring in both disease and patient), non-confirming (occurring in disease but not patient), and non-contributory (occurring in patient but not disease);
4. hypothesis evaluation (i.e., accepting or rejecting disease hypotheses, or ranking them in order of probability).
The advantage of this strategy was believed to be its capacity for converting an open problem (What’s wrong with the patient?) into a set of closed problems (Has he got disease X? Has he got disease Y?) (Elstein, Shulman, & Sprafka, 1978). The hypothetico-deductive model does indeed converge rapidly on the subjectively most probable disease hypothesis. However, later in this chapter, it will be argued that this convergence can result in error.
The main building blocks of the model, disease hypotheses, were viewed as a framework for organizing the information the doctor acquired as he/she proceeded with the diagnostic investigation (Barrows & Bennett, 1972). This was regarded as crucial to expertise, because diagnostic problems generate huge quantities of information which may result in error unless the doctor finds a way of structuring and chunking it (British Medical Journal, 1977). Some studies found that doctors generated multiple hypotheses simultaneously, and some that they generated hypotheses sequentially. However, there was general agreement that the number of working hypotheses was small, generally not more than about five.
Various forms of simulation were used to explore the hypothetico-deductive model. Two major projects used high-fidelity simulation, employing actors and actresses to play sick patients who were interviewed by medical students, interns and experienced doctors as if in real-life consultations (Elstein et al., 1978; Barrows & Tamblyn, 1980). Videotaping was used to capture all aspects of these diagnostic encounters, including body language. However, most of the research employed low fidelity simulation. A frequently-used instrument of this kind was the patient-management problem, in which a patient’s medical history and presenting complaint is described and the subject is required to gather information by asking questions from a list (McGuire & Babbott, 1967). The high fidelity approach enabled qualitative data to be gathered about the way hypotheses were tested — such as their content, or how they related to the history taking or physical examination (Barrows & Tamblyn, 1980). In contrast, much of the low-fidelity research focused on information-theoretic aspects of data gathering, expertise being defined in terms of efficient search strategies which avoided irrelevant and redundant questioning.
The view that expertise in diagnosis could be equated with the hypothetico-deductive method failed to find support. Four findings were particularly significant for this conclusion. First, many studies found that diagnostic expertise was case-dependent: That is, a doctor would diagnose successfully if knowledgeable about the pathology in question; unsuccessfully if not knowledgeable (McGuire, 1976; Elstein et al., 1978). Moreover, the between-case variation in the performance of individual doctors was so great that Leaper, Gill, Staniland, Horrocks, & de Dombal (1973) concluded “the diagnostic process does not exist” (p. 569).
Second, although a difference was found in questioning strategy — experts tending to ask fewer questions, each of which produced greater payoff in terms of reduction of uncertainty (Leaper et al., 1973) — doctors with different levels of diagnostic skill did not show many other differences in the extent to which they followed the hypothetico-deductive method. Thus in Elstein et al.’s (1978) study, when a comparison was made between superior and non-superior diagnosticians (as judged by their peers), no differences were found in the point of generation of the first hypothesis, the total number of hypotheses generated, nor the number of cues acquired.
Third, the hypothetico-deductive method itself can be a source of error. For example, Elstein et al. (1978) found that hypothesis formation led to premature closure and restricted information gathering, which reduced the accuracy of the diagnosis if the initial hypothesis was not correct.
Fourth, what distinguished the best diagnosticians was the content of their hypotheses — the experts were more likely to include the correct diagnosis in the initial hypothesis set (Elstein et al., 1978; Neufeld, Norman, Feightner, & Barrows, 1981).
When it emerged that knowledge of the problem domain was a better indicator of diagnostic expertise than adherence to the hypothetico-deductive method, the attention of researchers shifted to eliciting and modeling medical knowledge. Undoubtedly, experts know more than novices — the basis of expertise was sought rather in the way that expert knowledge is organized. An early study by Wortman (1966) had failed to confirm the discrimination net as an explanatory model of the structure of doctors’ knowledge of neurological diseases, albeit in a small sample. In the search for the structure of expert medical knowledge that followed, one line of research applied the recall techniques developed by Chase and Simon (1973) for studying expertise in chess. When asked to recall the details of a patient case history, experts did so in a smaller number of “runs”, each of which contained a larger number of items, than was the case with novices. This suggests that expert medical knowledge is organized into coherent large-scale units, whereas novice knowledge comprises more isolated elements (Muzzin et al., 1982). This may explain how experts are able to extract more meaning from a set of data, for example, by recognizing syndromes. A further contribution to understanding the knowledge base in medicine is Feltovich’s (1981) suggestion that experts re-organize their knowledge of diseases into disease competitor sets (groups of similar and therefore confusable diseases).
To model the representation of disease hypotheses in memory, the idea of disease frames was developed. A disease frame is a schematic knowledge structure which depicts and organizes the pattern of patient data expected in a given disease (Pople, 1977). Through experience and exposure to the different manifestations of a disease in many patients, an expert differentiates a single disease frame into numerous variants. Diagnostic reasoning proceeds by matching patient data with the disease frames stored in memory (Johnson, Duran, Hassebrock, Moller, Priutela, Feltovich, & Swanson, 1981).
Another popular concept is the mental model. An early study by Kleinmuntz (1968) proposed that experts in neurology diagnose by reference to a stored model of the nervous system. Both Kuipers and Kassirer (1984) and Patel and Groen (1986) have since proposed that expert diagnosis depends on possession of a causal model of the problem domain.
However, knowledge is not sufficient to explain diagnosis. We also need to explain how the individual operates on this base. Patel and Groen (1986) used discourse analysis of think-aloud protocols to model expert diagnosis as a process of forward reasoning through a network of rules. Forward reasoning is data-driven, the antithesis of the hypothetico-deductive method which reasons backwards from disease hypotheses to data in an attempt to confirm or falsify the hypotheses. In the Patel and Groen study, backward reasoning was associated with less success in diagnosis than forward reasoning. Other recent work has focused on comprehension, that is the interpretation of patient data in terms of an existing knowledge base. Using techniques of conceptual analysis, many researchers — notably Feltovich, Spiro, & Coulson (1989) — have identified differences between expert and novice understanding of key disease entities, such as heart failure. They have demonstrated medical students’ misconceptions of pathology which can misdirect their diagnostic thinking. It has even proved possible to trace a direct influence for such misconceptions and errors to the primary instructional materials used by medical students (Feltovich, Johnson, Moller, & Swanson, 1984).
Thus in the space of forty years, research into the psychology of medical diagnosis has made significant progress. It has developed a hypothetico-deductive model of expertise, and has replaced it by a knowledge-based model which guides most contemporary research. Hypothetico-deductive reasoning has not been discarded completely; it accounts for some of the data found in protocols. However, today it is regarded as a “weak” problem-solving method used most frequently by novices.
The starting point for the present chapter is the concept of disease. This has been the main building block in the psychological theorizing reviewed above. Although, as we have described, the disease hypothesis (conceived as a verbal proposition to be tested against data) was replaced by the idea of a cluster of pathophysiological concepts and their relationships, the assumption that diagnostic expertise depends on the ability to operate on representations of disease entities has remained unchanged. The present chapter questions this. The argument is that the identification of a disease entity may be the end-point of diagnosis, but the process by which experts reach it involves a far wider variety of reasoning processes, among which thinking about disease entities is only one. Like the hypothetico-deductive method itself, diagnostic thinking that restricts itself to a consideration of disease entities alone is characteristic of novices, and may lead to error. The first step in the argument is to distinguish between the diagnosis of diseases and the diagnosis of patients.
DIAGNOSIS OF DISEASES AND DIAGNOSIS OF PATIENTS
Diagnosis of Diseases
Feinstein (1973) defines the diagnosis of diseases as the process of: “converting observed evidence into the names of diseases. The evidence consists of data obtained from examining a patient and substances derived from the patient; the diseases are conceptual medical entities that identify or explain abnormalities in the observed evidence” (p. 212).
Originally, the concept of “disease” simply meant dis-ease, that is feeling unwell. However, with the advent of scientific medicine, physicians began to identify different disease entities underlying different complaints of “dis-ease.” Each was viewed as the cause of a distinct set of clinical signs and symptoms. In the 19th century, anatomists succeeded in correlating patterns of clinical findings with specific anatomical disorders found in cadavers at post-mortem examination. In the view of many, it became the task of the clinical diagnostician to infer from outward appearances what inner disease entity was causing the patient’s illness.
Thus conceptualized, a disease is a causal chain: Manifestations (e.g., symptoms) are caused by disorders (gross abnormalities in the structure or function of an organ or system of the body, e.g., an increase in size), which in turn are caused by pathological processes (morphological or biochemical events such as inflammation), which are caused by etiological factors (genetic or environmental states which set the whole chain in motion). The diseases — causal chains — are ordered within taxonomies such as the International Classification of Diseases, and diagnosis becomes a process of placing a case within one (or more) of these categories. The process of causal diagnosis usually begins with an investigation of the patient’s symptoms, tracing them back down the chain until the source of the problem is confirmed
To classify a problem in this way adds information to the clinical findings by suggesting the etiology, pathogenesis, prognosis, and many other characteristics of the illness. On another level, the use of diagnostic labels in place of a diverse collection of findings facilitates the chunking of information, necessary for communication, record-keeping, and research.
This concept of disease is similar to the fault-finding model of system malfunction, where causal chains link the symptoms of a breakdown to the failure of one or more components. In both models, diagnosis tends to be regarded as the localization of the failed component on the basis of the symptoms. However, if disease is conceived on a broader basis — for instance, as a system-level problem — then an alternative approach to diagnosis becomes necessary.
Diagnosis of Patients
Before the correlation of internal lesions with external symptoms, physicians were unable to diagnose in the way outlined above. Nevertheless, they often made intelligent assessments of their patients’ problems, and selected treatments that were effective within the limits of what was available. A study of these ancient methods can be revealing. The Hippocratic school is said to have diagnosed the patient rather than the disease. That is, they diagnosed by differentiating between individuals, not between pathophysiological conditions. This involved generating a holistic picture of the patient by gathering a prodigious range of details, including the minutes details of his or her complaint, environment, appearance, and lifestyle. On the basis of this idiographic representation, the physician would predict the likely course of the particular illness and choose a treatment to divert it from its trajectory. Insofar as there existed concepts of different diseases, these were little more than names for groups of patients who tended to fall ill in much the same way. The fact that common etiological factors were subsequently discovered for many of these empirical groupings, and that causal definitions of disease names replaced symptomatic ones, does not destroy the distinction between these two ways of diagnosing. They are based on different sets of data and involve different patterns of information search. What now needs to be considered is whether, in the age of scientific medicine, the broader perspective offered by the diagnosis of patients has entirely lost its place in expert reasoning.
DIAGNOSIS IN GENERAL PRACTICE
The general practitioner is usually the first to be consulted by a patient who has experienced a symptom and who seeks medical advice. The consultation is a complex event with a high level of uncertainty — many patients report symptoms which are normal reactions to the stress of everyday living, rather than the result of organic disease. Even when organic disease is present, it may be in an early stage, before the symptoms by which it can be diagnosed have appeared. This uncertainty reduces the extent to which the doctor’s task can be represented solely as the diagnosis of diseases.
Michael Balint, an influential figure in the education of British general practitioners, was one of the first to question the adequacy of traditional concepts of diagnosis and disease for general practice. He distinguished between the “conventional diagnosis” taught in medical school, and what he called the “overall diagnosis,” defined as the understanding of people in a professional capacity. Overall diagnosis is patient-centered and holistic. It encompasses “the external pressures on the patient, his internal world, his relationship with significant people around him, and the way in which the doctor-patient relationship has developed” (Norell, 1973, p. xv).
A similarly holistic perspective on heart disease has been suggested by Nixon, who regards much of this kind of illness as a system-level failure attributable to the pursuit of a lifestyle incompatible with one’s physiological reserves. He argues the insufficiency of conventional diagnostic approaches in the following terms: “People push themselves or allow themselves to be driven beyond their physiological territory into boundary-testing. Some live with sick or inadequate systems for years on end without having the energy, the information, or the opportunity for recovery. The label ‘ME’ or ‘post-viral syndrome’ does not point out the remedy” (Nixon, 1990, pp. 460-461).
These ways of understanding patients’ problems are quite different from the localization of specific organic defects. They are more in line with the ancient method of diagnosing patients. To explore this dimension of expert thinking further, we will consider some examples of diagnostic error in general practice.
DIAGNOSTIC ERRORS IN GENERAL PRACTICE
Three illustrative cases are described here, all of which have been taken from a study of diagnostic errors made by trainees in general practice. In each case, the trainee’s diagnostic thinking is compared with that of an expert general practitioner who was providing supervision.
Case One
A trainee was called to a patient who complained of a stiff neck and muscle spasms. The trainee inquired about recent accidents and the patient reported that she had scratched herself on a rose tree when gardening — a plant which is often fertilized in England with horse dung. When the trainee examined her, he found facial stiffness. He diagnosed tetanus and telephoned the covering general practitioner for advice.
As he listened over the telephone, the more experienced doctor immediately doubted the diagnosis. He inquired whether the trainee had asked the patient what drugs she was taking. This, however, he had omitted to do. The experienced doctor went to the patient’s house, but the patient did not seem to him like a case of tetanus. In his own words, she was “not ill enough.” He went into the kitchen, found the cupboard in which she kept her personal hoard of medicines, and opened it up. In this way, he discovered that she had been dosing herself with a drug whose side effects include facial stiffness, and the diagnosis of tetanus was disconfirmed.
In comparing how the trainee and the more experienced doctor approached this problem, the most obvious difference is the breadth of their information search. The trainee focused narrowly on the abnormal physical finding of facial stiffness and the etiologically significant history of a rose-scratch. Such a convergent search is typical of the diagnosis of a disease. In contrast, the more experienced doctor set the abnormal findings within the context of the patient as a person — how ill she was in herself, and her hypochondriacal lifestyle. The broader search brought disconfirming evidence into the picture, enabling the doctor to fit the data to an alternative pattern — the anxious patient who is causing her own symptoms by self-medication. This way of reasoning was based on knowledge of human nature, and has much in common with the ancient diagnosis of patients. However, the expert doctor was doing much more than this: He also made use of his knowledge of disease entities. Having seen cases of tetanus before, he realized instantly that the patient was not ill enough for it to be tetanus. Thus in the expert, diagnosing the disease was performed in tandem with diagnosing the patient.
Case Two
A trainee just two months into his post was called out at midnight to a 29-year-old female patient who complained of a central right abdominal pain of four hours’ standing. He suspected appendicitis, although there had been no vomiting (which would have increased the probability of this diagnosis). When the patient refused further physical examination, the trainee proposed emergency admission to hospital for suspected appendicitis. When the patient refused this too, he called the covering general practitioner.
The experienced doctor arrived and immediately recognized the patient as an attention-seeking individual well known in the practice. He doubted the trainee’s diagnosis and suggested that the patient attend the morning surgery in a few hours’ time for further investigation. In the event, the trainee’s diagnosis turned out to be a false alarm.
In comparing how the trainee and the more experienced doctor approached this problem, once again a difference in the breadth of information search is apparent. The trainee focused on pathognomic symptoms and diagnosed a disease, while the experienced physician diagnosed the patient on the basis of his knowledge of her as a person. However, the expert followed this line of reasoning in parallel with diagnostic thinking of the conventional, disease-centered kind. Since the pain was only of four hours’ standing and there had been no vomiting, he reasoned that if it really was appendicitis, this was only the early stage. It would therefore be safe to wait until morning to observe further developments — for example, whether the classic pattern of signs and symptoms appeared. The trainee did not think of using time as a diagnostic tool in this way.
Case Three
A 62-year-old construction worker, who was a rare attender at the surgery, reported a sudden attack of central chest pain radiating to the neck. The trainee took the patient’s blood pressure, and on finding that it was high, diagnosed hypertension. In fact, the patient had suffered a heart attack, the symptoms being classic. The misdiagnosis was primarily due to the trainee not knowing that blood pressure can be transiently high in ischaemic attacks, that is ignorance of the relevant disease entity.
However, this was not the only source of the error. In comparing the trainee’s approach with that of the experienced general practitioner who was supervising him, it emerged that in such circumstances the latter would have asked himself why the patient had come to the surgery, and would have taken the patient’s occupation into account. This is to diagnose the patient: When it is asked why a heavy manual worker who rarely attends the surgery has come to report a chest pain, that particular symptom acquires more significance than it was given by the trainee.
EXPERTISE
The point illustrated by these cases is that diagnostic error in uncertain environments can be due to adopting a narrow perception of the problem. Such narrowness is encouraged by the traditional concept of disease entities, which tends to focus attention on pathognomic signs and symptoms to the exclusion of the patient as a whole. In contrast, the experienced doctors viewed the cases in broader terms. Their perceptual field was the patient, conceived as an individual whom they expected to behave in particular ways, with the physical signs and symptoms only part of this larger picture.
To understand the advantage of the expert diagnostic strategy, it is necessary to appreciate the high level of uncertainty in this particular task environment. A very wide range of problems can walk through a general practitioner’s door, but the resources for investigating them by the methods of hospital medicine are limited. The broad purview of experienced general practitioners can be interpreted as a way of solving the problem of lack of fidelity in the signal by increasing the bandwidth of their search. In other words, construing ambiguous signs and symptoms in the context of expectations about the patient’s behavior can clarify their meaning.
This strategy has much in common with the diagnosis of patients in prescientific days, and with Balint’s concept of overall diagnosis. However, it differs in important ways. It differs from the ancient diagnosis of patients because it is not a substitute for reasoning about pathophysiological entities, but an overarching strategy which includes the latter as a subroutine. It differs from the Balint approach by including in its scope illnesses that are organic in origin, whereas Balint was primarily concerned with the minor psychiatric illnesses that produce physical symptoms.
If diagnosis is represented as identifying specific disease entities, then the reasoning process is relatively simple. The knowledge base consists of specific pathophysiological conditions and how they manifest themselves. However, when problems are more complex, expert diagnostic reasoning is based on a more complex set of reasoning processes and on a broader knowledge base. Current research focuses on disease knowledge, but the cases discussed above suggest that the expert knowledge base also includes an understanding of patients as people. The remainder of this chapter describes five lines of reasoning which may be interwoven in this kind of thinking.
FIVE LINES OF DIAGNOSTIC REASONING
Understanding Normal Variability
The diagnostician must differentiate normal system states from abnormal ones. Abnormality is an ambiguous concept, that has both causal and stochastic meanings: It may be construed either as a malfunction or as a deviation from average behavior. A source of uncertainty in many diagnostic tasks is that particular systems, whether human beings, blast furnaces, computer installations or flexible manufacturing plants, have idiosyncratic ways of behaving. Consequently, knowledge of the range of behavior manifested by a particular system in a healthy state can assist in understanding whether or not its abnormal behavior indicates a defect. This line of reasoning is illustrated by the expert doctor in Case One. An important part of the knowledge base is familiarity with the specific environment in which the problem has occurred — in general medical practice, this may include personal knowledge of individual patients.
Attributing Abnormal Signs and Symptoms to the System’s Normal Response to Adverse Operating Conditions
Whereas the first line of reasoning sets the boundaries of normal and abnormal behavior for a particular system, this one draws causal links between the behavior of a system and the conditions under which it is operating. It enables the diagnostician to interpret supposed abnormalities as the normal responses to the way an intrinsically healthy system reacts to adverse operating conditions. The knowledge base comprises how the system functions in a healthy state, and which environmental factors may disrupt it. However, when training is based on the narrow view that diagnosis is the localization of a specific defect, it tends to neglect the latter.
Assessing the Response of the System to an Intervention Designed to Remove the Symptoms
Diagnosis is sometimes represented as a discrete event which precedes treatment, the latter being selected in the light of the former. In complex, dynamic environments, however, attempts to understand the problem may alternate with attempts to solve it. Symptomatic treatment is a powerful diagnostic tool in uncertain environments: Whether or not the symptom disappears may generate crucial information. So we may define a third line of reasoning — assessing how the system is responding to attempts to control its abnormal behavior by purely symptomatic treatment, before any physical defect has been identified. Assessment of this kind probably takes up more of the general practitioner’s time than efforts to diagnose diseases in the conventional way.
These three lines of reasoning support the broader diagnostic strategy demonstrated by the experts whose decision making was discussed earlier. However, they operate in parallel with the traditional disease-oriented approach to diagnosis, so for completeness we must include the two other lines of reasoning which relate to that way of thinking.
Identifying a Specific Defect That is Causing Symptoms
This is the identification of a causal link between symptom and etiological entity, which (we have argued) is often wrongly assumed to be the sole line of reasoning in diagnosis. The dangers of this assumption have already been emphasized — increased chances of error in uncertain environments, and the delay of intervention until the fault has caused sufficient damage to generate the data by which it can be diagnosed.
Predicting How a Defective System Will Continue to Malfunction
After a disease has been diagnosed, a prognosis of its future course may be made. This is distinct from the prediction of normal system functioning defined above. It may be a crucial line of reasoning where a specific fault has been found and the main concern is damage limitation.
KNOWLEDGE FOR TRAINING AND SUPPORTING DIAGNOSIS
The practical purpose of defining these five lines of diagnostic reasoning is to identify the knowledge which should underpin training and decision support. The technical knowledge required for the last two lines of reasoning has been extensively discussed elsewhere. The main point to be made about the first three is that they require a broadening of the conventional knowledge base. One direction this broadening must take is towards the representation of whole systems interacting with their environments. Another is the incorporation of idiographic knowledge of the peculiarities of individual systems. Unfortunately, the training of diagnosticians, insofar as it is based on “fault localization” models, gives insufficient emphasis to holistic, contextualized, and idiographic knowledge. Similar concerns must be expressed about many conventional approaches to decision support.
REFERENCES
Barrows, H.S., & Bennett, K. (1972). The diagnostic (problem-solving) skill of the neurologist: Experimental studies and their implication for neurological training. Archives of Neurology, 26, 273–277.
Barrows, H.S. & Tamblyn, R.M. (1980). Problem based learning. An approach to medical education. New York: Springer.
British Medical Journal (1977). Reducing doctors’ errors (editorial). British Medical Journal, 1977, 1, 1178–1179.
Chase, W.G., & Simon, H.A. (1973). Perception in chess. Cognitive Psychology, 1, 55–81.
Elstein, A.S., Shulman, L.S., & Sprafka, S.A. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press.
Feinstein, A. R. (1973). An analysis of diagnostic reasoning. I. The domains and disorders of clinical macrobiology. Yale Journal of Biology and Medicine, 46, 212–232.
Feltovich, P.J. (1981). Knowledge based components of expertise in medical diagnosis (Tech. Rep. No. PDS 2). Pittsburgh: University of Pittsburgh, Learning Research and Development Center.
Feltovich, P.J., Johnson, P.E., Moller, J.H., & Swanson, D.B. (1984). DCS: The role and development of medical knowledge in diagnostic expertise. In W.J. Clancy & E.H. Shortcliffe (Eds.), Readings in medical artificial intelligence: The first decade (pp. 55–65). Reading, MA: Addison-Wesley.
Feltovich, P.J., Spiro, R.J., & Coulson, R.L. (1989). The nature of conceptual understanding in biomedicine: The deep structure of complex ideas and the development of misconceptions. In D.A. Evans & V.L. Patel (Eds.), Cognitive science in medicine (pp. 102–120). Cambridge, MA: The MIT Press.
Johnson, P.E., Duran, A.S., Hassebrock, F., Moller, J., Prietula, M., Feltovich, P.J., & Swanson D.B. (1981). Expertise and error in diagnostic reasoning. Cognitive Science, 5, 235–283.
Kuipers, B., & Kassirer, J. (1984). Causal reasoning in medicine: Analysis of a protocol. Cognitive Science, 8, 363–385.
Leaper, D.J., Gill, P.W., Staniland, J.R., Horrocks, J.C., & de Dombal, F.T. (1973). Clinical diagnostic process: An analysis. British Medical Journal, 3, 569–574.
McGuire, C.H. (1976). Simulation technique in the teaching and testing of problem-solving skills. Journal of Research in Science Teaching, 13, 89–100.
McGuire, C.H., & Babbott, D. (1967). Simulation technique in the measurement of problem-solving skills. Journal of Educational Measurement, 4, 1–10.
Muzzin, L.J., Norman, G.R., Jacoby, L.L., Feightner, J.W., Tugwell, P., & Guyatt, G.H. (1982, September). Manifestations of expertise in recall of clinical protocols. Paper presented at the 21st Annual Conference on Research in Medical Education, Washington DC.
Neufeld, V.R., Norman, G.R., Feightner, J.W., & Barrows, H.S. (1981). Clinical problem-solving by medical students: a longitudinal and crosssectional analysis. Medical Education, 15, 315–327.
Nixon, P. (1990). Contribution to discussion. Philosophical Transactions of the Royal Society of London, B327, 449–462.
Norell, J. S. (1973). Introduction. In E. Balint & J. S. Norrell (Eds.), Six minutes for the patient (pp. ix-xxi). London: Tavistock/Routledge.
Patel, V.L., & Groen, G. (1986). Knowledge based solution strategies in medical reasoning. Cognitive Science, 10, 91–116.
Pople, H.E. (1977, August). The formation of composite hypotheses in diagnostic problem-solving: An exercise in synthetic reasoning. Paper presented at the Fifth International Joint Conference on Artificial Intelligence. Pittsburgh, PA.
Wortman, P.M. (1966). Representation and strategy in diagnostic problem-solving. Human Factors, 8, 48–53.