Chapter 10

Using Representation

As I complete this survey of ways to represent knowledge, I want to suggest some general lessons for the study of the mind. In this book, I did not advocate a particular type of representation as the way that information is stored. Indeed, it should be obvious by now that there is much debate over how to think about knowledge representation in explanations of the range of behaviors that people exhibit. Instead, my point was to introduce a variety of formats for thinking about representation. Every format has produced insights about how human cognition works, and all of them are likely to be the source of future insights. Thus, rather than advocating a particular type of representation, I have tried to describe what the different approaches are good for. In this chapter, I give seven proposals for the use of representation in cognitive models, proposals that I think are important for research in cognitive science.

These issues are listed in Table 10.1 (see also Markman & Dietrich, 1998). The first three proposals are about representations themselves. Proposal 1 implies that representations must actually satisfy the definition presented earlier: They must represent. Proposals 2 and 3 explicitly admit that there are many types of representations and that many different types may need to be integrated in cognitive models. Proposal 4 focuses on the processes that act on representations. Proposals 5 through 7 stress the relationship between models of representation and the psychological phenomena to be explained: Cognitive models must deal with details, context, and relations between people and the world.

TABLE 10.1
Seven Proposals for the Use of Representation in Cognitive Models

PROPOSAL 1: THE NEED FOR REPRESENTATIONS THAT ACTUALLY REPRESENT

The first proposal in Table 10.1 is that cognitive models must be based on representations that actually represent. Although this statement may seem trivial, most psychological models are simplified to effectively illustrate a particular theoretical position, and it is unclear whether they can scale up to real domains. In these systems, the representational scheme lacks all the components in the definition of a representation given in chapter 1. As defined in this book, representation has four requirements: (1) a represented world, (2) a representing world, (3) a consistent set of relationships between the represented and representing worlds, and (4) processes that make use of the information in the representing world. Most forms of representation described in this book have focused on the structure of the representing world and on the processes using the information in the representing world. The representing world may be a space, with procedures for measuring distance in the space, or a structured relational representation with procedures for comparing pairs of such representations to find their commonalities and differences.

The represented world has often been omitted from these discussions. Investigators have simply assumed that the components of the representing world refer to some represented world. Often, the represented world is denoted by labels given to aspects of the representation. For example, the points in a multidimensional space may be labeled, or the nodes in a semantic network may have words associated with them. In many cases, the representing worlds are actually connected to the represented world indirectly through a user who can interpret the labels on a representation (that is, they have a user semantics).

In many practical cases, having representations that actually represent (i.e., having both a represented and a representing world) is not so important. If a researcher wants to predict people’s behavior in a particular psychological task, he or she may construct an overly simple model of the task to demonstrate the consequences of a theory. For example, Markman and Gentner (1993b) tried to predict people’s behavior in a mapping task. Figure 10.1 shows a sample pair of pictures from these studies. Each picture pair contained a cross-mapping, which occurs when objects in each of a pair of scenes look similar but play different relational roles. When a pair containing a cross-mapping is compared, there is a tension between seeing the pictures as similar because of the objects that look alike and seeing the pictures as similar because of the similarities in relations. In the mapping task, people first rated the similarity of the pair of pictures on a 9-point scale. Then, the experimenter pointed to one of the cross-mapped objects (e.g., the woman in the top picture in Figure 10.1) and asked the subject to point to the object in the bottom scene which “goes with” that object. Typically, people selected either the object that looks similar (e.g., the woman in the bottom picture) or the object that plays the same relational role (e.g., the squirrel receiving food in the bottom picture). In general, subjects were more likely to select the object playing the same relational role when they had just made a similarity judgment than when they had not. Workers have taken this finding as evidence that judging the similarity of a pair of scenes involves a process of structural alignment (like that described in chap. 5), which promotes attention to relational commonalities between the pair.

To provide an explanation for people’s performance in this task, structured relational representations for the scenes in Figure 10.1 were created. These representations, shown in Figure 10.2, embody several assumptions about people’s representations critical to structure-mapping theory. First and foremost, the representations are structured. Next, the main story in the pictures is represented by the deepest and most connected relational structure (i.e., it is most systematic). Objects that look similar (like the women in the top and bottom pictures) are represented by sets of overlapping attributes. These representational assumptions are the ones taken to be important psychologically.

To construct a model of a cognitive process, other assumptions must be made as well. For example, the relation describing the event of the woman giving food to the squirrel is represented as the three-place predicate:

The commitment in this model is to a structured representation of this event. It is possible that people have a single predicate with three arguments that represent the event of giving, but it is also possible that this event is represented by using a fully decomposed representation of the transfer as discussed in chapter 7. Likewise, one assumes that the objects in the pictures are represented with sets of attributes, but no one knows which attributes people use. This is not to say that how people represent events of giving or how they represent particular objects is unimportant. Rather, too little is known about the fine details of people’s representations to describe their specific content with any certainty. There is no deep commitment to the assumptions made simply to fill out the details of the representation.

FIG. 10.1. Pair of pictures containing a cross-mapping. From A. B. Markman and D. Gentner (1993). Copyright © 1993 by Academic Press. Reprinted with permission.

FIG. 10.2. Structured representations of the scenes in Figure 10.1 given to the structure-mapping engine in a simulation of how people compare scenes with cross-mappings.

These representations were given to the structure-mapping engine (SME; Falkenhainer, Forbus, & Gentner, 1989), which implements the structural alignment process described in chapter 5. This model takes pairs of structured representations as input and yields sets of structurally consistent correspondences between representations. The program suggested two possible sets of correspondences, shown in Figure 10.3. The preferred interpretation, shown in Figure 10.3A, placed the objects in the pictures in correspondence on the basis of relational similarities. In this interpretation, the woman receiving food in the top scene is placed in correspondence with the squirrel receiving food in the bottom scene. A less preferred interpretation, shown in Figure 10.3B, placed the objects in correspondence on the basis of shared attributes of perceptually similar items. In this interpretation, the woman in the top scene is placed in correspondence with the woman in the bottom scene. Thus, as observed in the data, the relational correspondence was preferred to the object correspondence.

FIG. 10.3. Two interpretations that emerge from a comparison of the representations in Figure 10.2. A: An interpretation that preserves relational similarities. B: An interpretation that preserves object similarities.

This example makes two important points. First, it can be quite useful to use representations that do not actually represent. The predicate structures fed to SME were assumed to have properties similar to those of people’s representations of the same situation. People were assumed to represent conceptual relationships by using a representation with explicit connections between relations and the things they relate. The representations of perceptually similar objects were assumed to be described by similar sets of representational elements. There was, however, no commitment to the actual predicates used in the representation in Figure 10.2. Instead, these representations embodied a set of assumptions about cognitive representations as well as other less crucial assumptions needed to get the model to work. Presumably, as investigators learn more about the way people represent objects and events, firm commitments can be made about these aspects of representations as well.

Although much research can use this strategy, it is important to make progress on the issue of how people actually represent information. There are at least three reasons why it is important to use representations that actually have both a represented world and a representing world. First, much research in psychology focuses on narrow problems. Research on logical reasoning has little contact with research on social reasoning, and research on classification has little contact with research on other aspects of memory. Part of the reason for this narrowness is that cognitive models are often built on minimal representations that are sufficient for understanding the particular task being modeled, but not for other related tasks. By thinking about how a representation may actually connect to something in the outside world, one can arrive at representations that are easily used for more than one task. Creating representations that serve more than one task may place constraints on the structure and content of representations; there is no room for idiosyncratic representational assumptions that satisfy the constraints of only a single task.

A second reason that it is important to think about representations that actually represent is that perceptual and motor tasks demand it. People cannot only reason about complex problems; they can also make their way through the world. Researchers have made significant progress in thinking about conceptual processing by using representations that are mere toys (as in the previous similarity example), but it is increasingly difficult to make process in research on perception and action without thinking about how the information in the world is actually represented. Researchers have complained that symbols and structured representations are an inappropriate basis for models of human cognition (Gibson, 1986; Port & Van Gelder, 1995; Thelen & Smith, 1994; Van Gelder & Port, 1995). Many such attacks have come from people who study perception and motor control. These researchers have abandoned symbols because of difficulties in attaching them in a representing world to actual aspects of the represented world. If psychologists are to develop a more complete understanding of human cognition, they must bridge the gap between higher cognitive processes (like categorization and reasoning) and lower level abilities (like perception and motor control). Accomplishing this task requires finding representing worlds that actually connect to their represented worlds.

A third reason to consider representations with an actual represented world arises from the symbol-grounding problem (discussed in chap. 1). This problem characterizes systems that use a conceptual role semantics (in which many representational elements are given meaning by their relation to other representational elements). Lacking a represented world, a representational system with a conceptual role semantics has no grounded symbols and thus no meaningful symbols.

Of course, not all cognitive models are based on representations without a representing world. Forbus’s (1984) qualitative process (QP) theory—a representational system for reasoning about physical systems (described in chap. 9)—has been used to represent and reason about a variety of different domains. Researchers have developed programs that create representations of physical systems from other information about them. In this way, the notations of QP theory have been embedded in models of other tasks. By using QP theory in models that must actually do reasoning about physical systems, a representational scheme can be evaluated with respect to an actual represented world. This kind of explicit attempt to connect representational schemes to specific represented worlds is important for validating particular approaches to knowledge representation.

PROPOSAL 2: MULTIPLE APPROACHES TO REPRESENTATION

If the cognitive system represented information in only one way, this book would read like a pamphlet by Marx and Engels: There would be an orderly progression of proposals for the nature of mental representation culminating in the ultimate form of representation that is right for all models. On this view, previous theories of mental representation would be mere scientific stepping-stones along the way to the one form of representation that serves as the basis of cognitive processing.

There are two reasons why this book does not follow such a progression. First, current cognitive science does not yet point to a single representation scheme as a good candidate for the one true form of representation. From a pragmatic standpoint, cognitive models need to consider a variety of representational types to judge their goodness as the basis of cognitive models. As discussed throughout this book, different approaches to representation have different strengths and weaknesses. When faced with a problem that requires a particular set of characteristics, one should be able to reach into the representational toolbag to pull out a representation that is appropriate for the particular task. Second and more deeply, however, there is little reason to believe that one true form of representation is the only basis of cognitive processing. People display varied abilities, from perception and motor control to language and problem solving. People’s behaviors in these abilities do not all seem to be reasonably described by processes acting over a single form of representation.

In chapter 2, I discussed representations that use mental spaces in which items are represented as points or vectors in space. Processing involves geometric operations like measuring distance or projecting one vector onto another. These models have the advantage of procedures (like connectionist learning algorithms) that can create such spaces, as well as efficient processes (like projecting one vector on another) for determining the similarity of pairs of representations. In many spatial models, it is easy to create representations that exist for only short durations. For example, the activation values in a connectionist state vector can change from moment to moment. Thus, spatial representations may be particularly good for models in which the representational states are not expected to be enduring (Port & Van Gelder, 1995). In chapter 1, I discussed Watt’s steam engine governor, which represents the pressure in a steam engine by the speed with which the governor spins. The spinning of the governor is a transient representation of pressure. The behavior of mechanical systems like the governor can be described by using systems of differential equations known as dynamic systems (Port & Van Gelder, 1995). When the equations are solved, a graph that relates the behavior of quantities in these equations can be constructed. Such a graph, called a state space, can define the behavior of a complex system by the way it traverses through this state space. Thus, dynamic systems can be construed as yet another type of spatial system (like those described in chap. 2).

One weakness of spatial models is that they cannot explicitly represent information that requires bindings between two or more elements. A second weakness is that they have no good procedures for accessing the commonalities and differences of representations that are compared. Thus, spatial representations can be very useful when efficient processing is needed and when the representational states are expected to be transient, but not when access to specific properties of representations is called for.

Featural representations (described in chap. 3) address some shortcomings of mental space representations. They allow access to specific properties of the representations, because the features are discrete symbols. Processing with these representations can be efficient; the features are independent. For example, genetic algorithms allow search through a massive space of possibilities by allowing recombinations of elements. Because all the representational elements are independent, these recombinations are possible. Thus, for situations in which access to specific elements is needed but relational binding is not, featural representations are useful.

Semantic networks (discussed in chap. 4) are excellent representations for models that require a spread of activation through a system of information. These models are well suited to explanations of situations in which the presence of one concept primes the presence of another. Network representations are also useful for implementing parallel constraint satisfaction models, which allow many different (and possibly conflicting) constraints to be applied to a problem simultaneously. In both cases, the structure of the network guides the automatic spread of activation. Semantic networks need additional processing mechanisms to be used as models of controlled cognition and reasoning. For example, Anderson’s ACT system (discussed in chap. 5) uses production rules along with semantic networks to model behaviors like the solution of addition problems.

Structured representations (see chaps. 5–7) provide a way to capture the binding between representational elements. For situations in which it is important to know which features are connected to other features, structured representations are crucial. Applications like understanding sentences, representing the premises of reasoning problems, and solving analogies all seem to involve these kinds of structured representations. This structure comes at a price, however: The processes that act over structured representations are expensive to operate. Even a comparison process like structure mapping requires mechanisms that ensure one-to-one mapping and parallel connectivity (see chap. 5). Thus, structured representations are not appropriate for situations in which fast processing is required or for which a computationally intensive process is not feasible.

It is one thing to argue for diversity and another to actually construct models that use more than one type of representation. Generating models that reconcile different representational formats can be quite difficult; therefore, this approach has not had widespread appeal. An example of both structured and unstructured representations incorporated into a single model is Forbus, Gentner, and Law’s (1995) MAC/FAC (Many are called/Few are chosen) model of analogical retrieval. The model is designed to understand how one situation reminds people of another. For example, imagine seeing a man running around his house, trying various methods to rid the house of a bug. On seeing this event, a person may be reminded of the cartoons in which Wile E Coyote futilely chases the Roadrunner, trying various elaborate schemes that fail. Noticing this similarity involves analogical reminding, because the first situation prompted the recall of a second situation that was similar to the first primarily in the relations between the objects (i.e., X chases Y unsuccessfully).

To model analogical retrieval, the MAC/FAC model represents every situation in memory in two ways. First, each situation has a structured predicate calculus representation that links the actors and objects in the event. Second, each situation also has a feature vector representation that encodes only which predicates are incorporated into the representation without taking into account any relational bindings. MAC/FAC posits that retrieval involves two stages. In the first stage (MAC), a new item serving as a retrieval cue is represented as a flat feature representation; the MAC stage is like the exemplar models of memory described in chapter 8. At this stage, the featural representation is quickly compared to featural representations of everything stored in memory, and items with a large number of overlapping features (not counting differences) are passed along to the second stage. At the second stage of processing (FAC), structured representations of the memory items taken from Stage 1 are compared to the structured representation of the memory probe by using the structural alignment procedure in the structure-mapping engine (described in chap. 5; Falkenhainer et al., 1989). This process is more computationally intensive than is the feature comparison, but it allows the model to be sensitive to relational similarities between the cue and items in memory. Any memory items with a sufficient degree of structural similarity to the cue are then retrieved from memory and can be used for subsequent processing.

A primary virtue of this system is that it uses both a featural and a structured representation. The featural representation is used as a first pass to search through a large number of memory items in a small amount of time. The speedy feature comparison process can result in passing along some elements that are not at all structurally similar to the cue. These items are weeded out by the structural comparison process, which performs the computationally intensive structural alignment between two representations on only a small number of representations.

In short, having many different kinds of representations is a boon for cognitive science. Much work has demonstrated the strengths and weaknesses of different representational systems. Indeed, this book is a summary of this effort. Comparatively little research has, however, been devoted to understanding how representational systems can be combined to form powerful models of cognitive processing. This area is important for future research.

PROPOSAL 3: REPRESENTATIONS AT MULTIPLE GRAIN SIZES

How much information is incorporated into each representational element? As discussed in chapter 8, if a representational system has a symbol for bird, this element has a wide scope that refers to any item that is a bird. Even more specific features like red and beak are still fairly abstract. People believe that the red associated with fire engines is different from the red associated with hair or with the breast of a robin. Each of these reds can be represented with the symbol red, but something else is clearly needed to account for the fact that people acknowledge the differences between these reds as well as their commonality. Likewise, birds have beaks, but the beak of a chicken looks different from the beak of a toucan. One possibility is that this context sensitivity of features requires using only fine-grained features in representations. Fine-grained features allow different manifestations of a general concept to emerge in different contexts. For example, when representing a beak in the context of a chicken, the fine-grained features describing its shape can differ from those representing a beak in the context of a toucan. This proposal permits a concept (like beak) to vary as a function of context but does not explain how people also notice fundamental commonalities between different things called beaks.

One solution to this problem is to assume that cognitive representations for the same item exist at many different grain sizes. Some representational elements may be coarse grained, roughly equivalent to the symbol beak, and may represent beaks in general. Other elements may be fine grained and may represent subtle properties that are true for some beaks but not for others. Thus there is more than one way to represent the same item. This redundancy allows models to deal with cases in which context sensitivity is important and also to deal with cases in which invariance across context is important.

One example of how multiple grain sizes can be incorporated into a single model is Hummel and Holyoak’s (1997) LISA, a connectionist model of analogical reasoning. LISA uses both distributed vector representations (see chap. 2) and local representations of concepts and structure (see chaps. 4–5). For distributed representations, there is a pool of features for a concept, and a concept is active when some set of its features has a high level of activation. For local representations, labeled nodes correspond to different objects and relations. For example, the concept John can be accessed in its general form by activating its node. Activating this node, in turn, activates features associated with this concept. The set of features that is activated is different in different contexts. Thus, LISA is able both to respond to the context of a situation and to abstract from the fine details of a particular situation when necessary. This representational strategy seems appropriate for modeling the way that people process analogies.

The idea that cognitive models need representations at a variety of grain sizes is also consistent with neurobiological evidence of song production in the zebra finch. Behavioral evidence has suggested that these birds’ songs have both syllables (typical patterns of notes) and specific notes. Yu and Margoliash (1996) have found that activity in one brain region in zebra finches—the HVc nucleus of the forebrain—is correlated with production of song syllables, whereas activity in a second brain region—the robustus archistriatalis—is correlated with production of individual notes of the zebra finch song. These results suggest a hierarchical control for song in the zebra finch. The fluent operation of the song system requires representations of different grain sizes in a single system. This general principle is likely to operate over a wide range of psychological processes.

PROPOSAL 4: CLEAR SPECIFICATION OF PROCESSES THAT OPERATE OVER REPRESENTATIONS

A central part of the definition of a representation is that some process must extract and use information from it. One reason that specifying the processing assumptions for a given representation is important is that the implications of a representational formalism for psychological models are not clear until the processing assumptions have been laid out. For example, clearly enumerated processing assumptions demonstrate how cognitive systems deal with the holism problem raised in chapter 1. Holism is the idea that the meaning of one concept is defined in part by its relations to other concepts, and all the information in someone’s knowledge base is necessary to fix the meaning of any given concept. Most cognitive models deal with this problem by having only a subset of the information in memory active at any given time. This point was made explicitly in chapter 4, in which semantic networks were assumed to have a limited amount of activation flowing through them. In this case, only the active information fixes the meaning of a given symbol at any moment.

This way of dealing with holism is not a philosophical solution but a psychological one. Cognitive systems use methods to constrain the information taking part in a given process under the assumption that only the information available at a particular moment is relevant for processing the current situation. For example, two people may know different things about the 1986 New York Giants, but this difference in knowledge is usually not relevant to the way they process the concept cat. Thus, knowledge about the 1986 New York Giants is not normally activated during processing of the concept cat.

This issue was also discussed relative to the frame problem in chapter 9. The frame problem involves the difficulty of reasoning about what changes in a new situation. When people want to reason about changes in a physical situation, they should check to see whether every element in the represented world has changed. Because it is not computationally feasible to do this check, systems that reason about the physical world adopt a strategy for limiting the objects examined for change. Solutions to the holism and frame problems both introduce potential errors in reasoning. Communication may become derailed if a speaker activates different knowledge about a concept than does a listener. Likewise, failing to reason about a particular object in the represented world may lead to a mistaken understanding of change in that represented world. A successful cognitive model makes few errors in reasoning, and its pattern of errors mirrors that made by people.

The specification of processes is also necessary because in an important sense, without specifying processing assumptions, there is no representation at all (there is at best only representation potential), so all further discussion begs many important questions. The processing assumptions associated with a model of representation are the properties that decide whether the representation can be applied rigidly to only one situation or can be used flexibly across a variety of situations.

For example, Smolensky (1991) discussed a possible context-dependent representation for a cup. He suggested that representations consist of fine-grained features that he called micro-features; the manifestations of a concept in different circumstances differ subtly because of the micro-features that are active. For example, the micro-features of a cup active in the context of a cup with coffee (which may include insulating properties of a cup) may be very different from the micro-features of a cup active in other situations. On the surface, this proposal seems to make clear that a distributed representation can be used as a model of the context dependence of concepts and can provide flexible context sensitivity in representation. Yet Smolensky’s proposal does not carry with it any assumptions about how the features are activated or about how a micro-feature representation is used by other cognitive processes (or for that matter how micro-features manage to represent a cup in the first place). On its own, it is not clear what such a representation is capable of doing, and hence it is not clear that it can live up to Smolensky’s claims for it. Indeed, Markman, Gentner, and Wisniewski (1998) suggested that vector representations like those used in many distributed connectionist models, when combined with the dot product as a mechanism for comparing vector pairs (Equation 2.7), are not sufficient to model the complexity of human similarity processing. Simple vector representations and dot-product comparisons cannot account for the pattern of similarity judgments displayed in Figure 5.4. As discussed in chapter 5, these data strongly suggest that the comparison process operates over structured relational representations.

Representations like activation vectors are appealing because they seem to embody the flexibility often observed in cognitive processing, but a representation is fluid only if a process allows it to be used flexibly. No representation is fluid by itself. As discussed in chapter 2, vectors are spatial representations; a dot-product comparison process allows only a computation of proximity between two vectors and thereby limits a model’s flexibility rather than increases it. Conversely, applying suitable processes to a highly structured representation can make it very flexible. Indeed, the appeal of a universal grammar in linguistics is that it allows infinite productivity from a finite number of elements and rules that contain discrete elements.

One demonstration of the flexibility of using structured representations comes from structured imagination tasks (Karmiloff-Smith, 1990; Ward, 1994, 1995). Researchers in an important area in cognitive science have examined the creative extension of existing concepts (Finke, Ward, & Smith, 1992). How can flexibility in concepts be achieved? People seem to constrain their creativity by extending existing ideas to novel situations. For example, Ward (1994) asked college students to draw novel animals that might live on an alien planet. The animals that people drew tended to maintain the basic structure of animals on Earth. Most constructions were bilaterally symmetric, and many even had the familiar sense organs of known animals. The animals might differ in the number or shape of their parts, but they tended not to diverge strongly from known animals. Even an analysis of alien animals in science fiction revealed that most creatures were structured similarly to animals on Earth.

Structured representations can also exhibit flexibility when they are involved in analogical reasoning. Indeed, Barnden (1994) suggested that analogical reasoning may allow symbol systems to exhibit some of the flexibility often associated with connectionist models that use spatial representations. For example, as discussed in chapter 5, structure-mapping models of analogy and similarity have assumed that the arguments of relations placed in correspondence are themselves also placed in correspondence (Gentner, 1983, 1989; Gentner & Markman, 1997; Holyoak & Thagard, 1995). This principle of parallel connectivity allows nonidentical items to be matched by allowing the arguments of corresponding predicates to be placed in correspondence. Thus, symbol systems are not rigidly required to permit correspondences only between elements with identical symbol names. Just as connectionist models allow responses to vectors that are not identical to those presented during training (see chap. 2), models of analogy allow the recognition of similarities between nonidentical pairs of situations. Missing information in one domain of an analogical correspondence can be filled in on the basis of the structure of the second domain by carrying over relations connected to already matching structure (Clement & Gentner, 1991; Markman, 1997; Spellman & Holyoak, 1996).

The specification of processing assumptions can bring about unforeseen flexibility in other ways. Models of analogical reasoning are able to form multiple interpretations of a single match. For example, people given a “double” metaphor (Gentner, 1988) like “A cloud is a sponge” can generate more than one interpretation (e.g., “Both are fluffy” or “Both can hold water”). Likewise, when given structured representations of this metaphor, models of analogy like SME (Falkenhainer et al., 1989), and LISA (Hummel & Holyoak, 1997) can form both interpretations. These models do so because they can enforce a set of constraints (e.g., structural consistency in the case of analogy) and then start over and form a different interpretation. Interestingly, models that do not strictly enforce constraints, like the ACME model, which uses a process of parallel constraint satisfaction (as discussed in chap. 4), are unable to form multiple interpretations of a comparison (Holyoak & Thagard, 1989). Parallel constraint satisfaction models attempt to form a single interpretation that best satisfies all constraints; hence they cannot form more than one interpretation.

As I have emphasized throughout the book, representations can be discussed only in the context of the processes operating over them. It is not enough to look at a representation and decide that it does (or does not) permit some type of processing. There must be an explicitly designed process that actually makes use of the information in the representation. A benefit of attention to processing is that representations that seem highly inflexible (like structured relational representations) can exhibit quite a bit of flexibility when accompanied by the right kind of process (e.g., an analogical comparison process).

PROPOSAL 5: DETAILS AND GROSS FORM OF BEHAVIOR

Cognitive processing is often much messier and therefore more complicated than theories reflect. Most psychological theories identify a few factors that contribute to the observed behavior under study. In many theories, these factors are represented and combined in relatively simple ways. Such theories agree with good scientific practice; scientists must first see whether simple models suffice before offering complicated explanations. Although this methodology is good, its result is that most psychological models are often not robust enough to explain variability in human performance. For example, if I am asked to list all the words associated with the word fish, my list will probably be different from a list that I would give next week (Barsalou, 1989). The specific words reflect my underlying knowledge structures as well as the way that concepts are activated (perhaps through a semantic network, like those discussed in chap. 4). The eventual success of cognitive models rests on the degree to which they can account both for the gross pattern of data observed in studies (e.g., to predict the words most people associate with fish) and the fine details of particular patterns of data (e.g., to determine how an individual lists a specific set of words).

As Bruner (1990; L. B. Smith & Sera, 1992) pointed out, cognitive psychologists often study average behavior. They conduct experiments on groups of individuals and average the data that they accumulate. The assumption of such work is that a cognitive central processor governs behavior, and individual variability is less important than is average performance across individuals. Contextual constraints and background differences that give rise to variation in performance between individuals are also important details of cognitive processing, however. As discussed in chapter 1, data that are treated as noise (like variability in performance across individuals) can be meaningful with another set of representational assumptions, as when one is trying to model the fine details of human performance.

Variability in human performance is an object of study in work on expertise. Three facets of expertise bear importantly on discussions of representation. First, expertise is domain specific. This fact is perhaps so obvious that it can be overlooked, but it is crucial from the standpoint of knowledge representation. Experts are experts in a domain, not experts at all things. Thus, experts have learned information and procedures that are selectively helpful in their areas of expertise.

Second, experts are efficient in their ability to represent aspects of their domain relative to novices. Classic studies demonstrating this point have focused on memory in chess experts (Chase & Simon, 1973). Chess experts require less time to reconstruct a configuration of pieces on a chess board than do novices, and experts can remember more pieces from a configuration than can novices. This ability holds for configurations that reflect possible situations from a game, but not for random configurations of pieces. Experts seem to organize chess pieces on the board by information relative to a game (e.g., pieces that are attacking or defending). This organization does not exist for random configurations. Similar findings have been obtained in other domains as well. For example, expert field hockey players are better able to remember the position of field hockey players in a picture than are novices (Allard & Starkes, 1991). These findings suggest that experts can impose a conceptual structure on what they see in their domain of expertise.

Third, experts know different things about their domain of expertise than do novices. Again, this fact may seem too obvious to note, but it is important for understanding representation. Many studies in cognitive psychology have drawn their data from college undergraduates. The results of these studies are taken as evidence of fundamental cognitive processes at work, but these data also reflect college students’ level of knowledge about a variety of topics. Systematic differences in performance between college students and experts may reflect differences in knowledge without reflecting differences in processing.

As a demonstration of this point, recent studies have explored how people with different types of expertise classify and reason about trees (Medin, Lynch, Coley, & Atran, 1997). This work contrasted three groups of experts: landscape workers, parks maintenance workers, and taxonomists (i.e., those with scientific training relevant to trees). When asked to sort trees into groups, taxonomists tended to sort into groups based on scientific taxonomies. To a lesser degree, maintenance workers also sorted into groups of taxonomically related trees, but they were less likely to do so than were taxonomists. Finally, landscape workers sorted trees on the basis of their uses in landscaping (i.e., ornamental trees vs. trees to be planted near the street vs. weed trees). Despite these differences in the groupings formed by different types of experts, all were still able to form groups that were consistent with the knowledge relevant to their particular domain of expertise, and all groups of experts could use their categories to reason about trees. For example, they could answer a question like “If a white poplar is susceptible to a new disease, how likely is a weeping willow to be susceptible to this disease?” Thus, these experts seemed to differ primarily in their specific knowledge about trees rather than in their ability to process information in this domain.

In sum, expertise is one area in which variability across people has been treated as an important aspect of research. Studies have demonstrated that knowledge is local to the domain of expertise and that experts can use knowledge to structure their perception of events in the domain. The study of experts enables researchers to tease apart aspects of behavior that reflect workings of the cognitive system from those that reflect differences in knowledge. This analysis simply reinforces the importance of looking both at the average performance of members of groups and at the performance of individuals.

PROPOSAL 6: IMPORTANCE OF THE SOCIAL WORLD

Nearly all the cognitive models described in this book focus on the individual and on the way that individuals represent the world around them, but there is more to human cognition than individual behavior. Humans operate in a dynamic social environment, which influences cognitive processing in a variety of ways, two of which I focus on here: communication and cognitive scaffolding.

One cognitive function that has received substantial attention from cognitive scientists is language. This attention is hardly surprising; the use of language is one of the most striking differences that separate people from other animals. In this book I have discussed how representations of prepositions may be related to the way people represent space (chap. 6) and how verbs may be related to the way people represent events (chap. 7). Each of these discussions focuses largely on the individual and the way that an individual’s representations are related to language. I have not been concerned with the primary function of language, to transmit information from one person to another and to establish joint reference to things in the world (H. H. Clark, 1992, 1996; Garrod & Doherty, 1994; Krauss & Fussell, 1996; Nelson, 1988; Tomasello, 1995).

The communicative function of language places strong constraints on how information is represented. For example, establishing joint reference to objects in the world requires attention both to commonalities and to differences related to those commonalities (Markman & Makin, in press; Markman, Yamauchi, & Makin, 1997). People must attend to commonalities because the same word often refers to many similar objects. People must also attend to differences related to the commonalities because establishing reference to unique individuals or specific categories often involves first specifying a category and then modifying the category label to distinguish between members of the category. The same object can refer to many different categories at different levels of abstraction (e.g., poodle vs. dog vs. animal) and with different roles (e.g., animal vs. pet; Brown, 1958). This use of language may also affect how concepts are represented. Thus, things as fundamental as the structure of our categories are likely to be affected by the social context in which the categories are acquired. This is not to endorse a strong Sapir-Whorf view that the structure of the particular language spoken by individuals influences their concepts, but rather to make the more general point that the fact that concepts are developed in a social environment in which they must be communicated has an impact on category structure. The communicative context provides an important force that helps standardize concepts across individuals (Freyd, 1983).

Because people must communicate with other members of their linguistic communities, everyone’s concept representations must overlap substantially. Consistent with this suggestion, Garrod and Doherty (1994) gave people a task in which they had to communicate with someone else about locations in a maze. There are many different ways to establish reference to locations in this setting, such as using row and column coordinates or specifying paths. When each person communicated with a number of different people from the same group in this task, the group settled on a single scheme that all members used to talk about locations. Thus, even people who never spoke to each other ultimately used the same communication scheme because they spoke to other people in the same group.

The act of communicating is a joint action pursued by a set of individuals (H. H. Clark, 1996). All participants in communication have an influence on what an utterance means. A speaker may have one intention when speaking, but if the meaning is not adopted by a hearer and ratified in the conversation, the speaker has the option of repairing the flawed communication or letting the meaning taken by the listener stand. Thus, during a conversation, both parties must represent the discourse in a discrete way that allows them to reflect on the match between the intended meaning and the construal of an utterance.

Participants in a discourse may also store a model of what they believe the other participants know. This representation of the other conversational participants allows utterances to be formulated for others appropriately and is also important for other communicative acts like lies. People can lie only if they believe that the other person does not already know the truth and if they think that the other person believes the lie. Through the joint action of conversation, people achieve a shared reality that serves to influence how they represent the interaction between them (Higgins, 1992). Of importance here is that meaning is arrived at dynamically during conversation, but this dynamics is likely to be mediated by discrete representational states that are analyzed by the participants during the discourse.

A second role of the social setting is cognitive scaffolding (Vygotsky, 1986). Typical studies in cognitive psychology have required participants to do a substantial amount of learning on their own. For example, in the standard classification paradigm, participants are presented with objects (or descriptions of objects) and asked to respond with the category to which the object belongs. Participants are given feedback on each trial and eventually learn to distinguish between the categories they are supposed to learn. In contrast, when learning in social settings, people are often given more information (see Markman & Makin, in press, for more discussion of this point). When a parent teaches something to a child, the parent often sets up a situation that is slightly more difficult than the child can master alone, and then the parent guides the child through the learning process (Rogoff, 1990; Vygotsky, 1986). Even when adults learn, there is strong social scaffolding. Hutchins (1995) described how sailors learn the task of navigation. He observed that sailors first take part in simple tasks like reading water depth from a meter. From this vantage point, they can hear the interactions of other crew members and begin to get a feel for the overall navigation task. Over time, they are exposed to other navigation tasks until they are capable of directing the task and carrying out all the separate jobs themselves. As these examples demonstrate, people are rarely called on to develop complex representations in isolation.

Cognitive scaffolding between parents and children is also evident in concept acquisition (Callanan, 1985, 1990). Callanan (1985) examined the way that parents teach children about categories at different levels of abstraction. She found that parents teaching their children about basic-level categories (e.g., dog), or specific subordinate categories (e.g., poodle) tended simply to point to an object and name it. Parents seem to assume that children think a label applied to an object refers to the object and others generally similar to it. In contrast, when teaching children about a general superordinate category (e.g., animal), parents tended to use a strategy in which they first labeled the object with a specific label and then pointed out that it was a member of a general group. For example, parents would say, “See this dog? It is an animal.” This strategy suggests that children may have difficulty finding referents for labels that refer to a diverse group of objects and that they must start by giving a specific label to the object. In order for this scaffolding strategy for teaching superordinate categories to work, parent and child must both be able to fix their reference on the basic-level category.

There are two reasons for incorporating the social setting of cognitive processing into models. First, the social setting places constraints on the form of representations. In the case of tasks like communication, representations with discrete symbols that people can reflect on during conversation are necessary to establish meaning with other participants in a dialogue. Second, people often learn concepts in situations in which others work to facilitate this learning. Learning situations are structured so that new representations are constructed collaboratively, often in conjunction with someone else who already has experience in the domain being learned. In this way, people can rely on others to help them determine which aspects of the environment are salient. Because many cognitive models focus only on the individual, the models may assume that the problem of learning new concepts is more difficult than the one that must actually be solved by people in a social environment (Callanan, 1985, 1990; Nelson, 1988, 1996).

PROPOSAL 7: RELATIONSHIP BETWEEN THE INDIVIDUAL AND THE WORLD

In the previous section, I suggested that by ignoring the social setting in which cognition takes place, some cognitive models focus on tasks that subjects rarely perform in their actual environments. This omission leads to weaker, less explanatory models. In another sense, cognitive models may typically be overly powerful, because they typically ignore the relationship between individuals and their world. As a result, modelers are forced to deploy enduring robust representations for cognitive capacities for which they are not needed.

Considering the relationship between an individual and the world may lead to a different way of formulating the problem that the cognitive system must solve. For example, the models of visual representation discussed in chapter 6 assume that the goal of visual processing is to construct a representation of the image reflected in the pattern of light striking the retina. This representation may then be used for a variety of purposes such as moving through the world, grasping objects, or identifying things in the world. An alternative view was proposed by Gibson (1950, 1986), who had the fundamental insight that the purpose of vision is to provide an organism with information about environmental objects related to the organism’s goals. Thus, objects are of interest to an organism primarily because they may be useful for satisfying these goals, or, in Gibson’s terminology, objects have affordances. According to Gibson’s view, the visual system may not need to provide an elaborate representation of the visual world to give information about affordances of objects. Instead, affordances may be perceived directly from visual input.

As an example, imagine a saucepan sitting on a counter in a kitchen. The saucepan has a handle. Given an image of the saucepan, most models of visual processing would try to find the edges in the object and perhaps to describe the saucepan in terms of parts (e.g., a handle, a body) and spatial relations between them (as in chap. 6). In contrast, Gibson suggested that the visual system provides information about the affordances of the object, for instance, that the pot’s handle can be grasped.

Gibson’s insight is based on the fact that humans (and all other creatures) have bodies and are embedded an environment. Inspired by this approach, several researchers in cognitive science have adopted a “situated action” perspective, which assumes that people need not have a complete representation of the world around them because the world remains in existence and can itself act as an external representation (Agre & Chapman, 1987; Clancey, 1993; Hutchins, 1995; Suchman, 1987). This strategy is likely to be a good one for models of some cognitive processes: All stages of complex cognitive processes need not have explicit and enduring representations when these cognitive processes take place in a complex environment (O’Regan, 1992). Cognitive systems can use information existing in the world to aid in processing.

For example, a cook need not have a complete plan in mind for preparing a dish. Instead, the cook can look at the stove and counters to see where he or she is in the preparation of the current recipe. The presence of dishes, pots, pans, and spices in the kitchen serve as reminders about what steps need to be carried out next. The cook need not remember how long a dish should stay in the oven but can rely on timers or can look at the dish in the oven to see whether it looks like it is done. In all these ways, a cook uses information available in the world to guide the process of preparing a meal without having to represent much information at all.

The importance of embodied cognition is the focus of Glenberg’s (1997) work on memory. He suggested that traditional memory tasks studied by cognitive psychologists are flawed because they do not take into account why the human organism needs a memory. On this view, human memory is designed not to recall lists of arbitrary items, but rather to remember information like locations of objects relative to the body. Thus, humans have a capacity to recall information and to structure representations of new experiences to facilitate interactions with the world. Memory is designed to be useful for bodily actions like reaching in space. Thus, understanding the capacities of human memory first requires determining the purposes for which the memory system was developed and the relationship between memory and the body.

Hutchins’ (1995) work on navigation discussed earlier, which has focused on the way that cognition develops in a social setting, has also examined how cognition develops in combination with sets of tools designed to aid in the cognitive tasks that are carried out. Hutchins pointed out that a navigator’s particular actions when plotting a course for a ship are carried out by using maps, protractors, calculators, and pencils. Given the ship’s bearings to fixed landmarks, the navigator plots lines that assess the current position of the ship in water. The time between readings of location and the distance between location points on the chart can be used to calculate the rate of the ship. The tools used for navigation may carry out low-level aspects of the task. For example, the navigator may use a protractor and ruler simply because the task requires drawing a line at a particular angle, but Hutchins argued that, taken as a whole, the set of tools used for navigation actually defines the task that the navigator believes must be solved. For Western navigators, the task is fundamentally assumed to be about determining the location of a moving ship in a two-dimensional plane (viewed from above) with reference to fixed locations in the world. This view of the task is supported by the range of tools that navigators have at their disposal. Hutchins’ observation raises two interesting questions for further research. First, what is it about human cognition that makes the particular sets of tools developed effective? Second, in what ways do people represent the tools they use in the course of ordinary processing?

These questions are important for developing accurate models of human cognitive processing. Under normal conditions, people are not cut off from the world around them but use strategies (implicitly or explicitly) for gathering information about the world around them to carry out cognitive tasks. In many laboratory experiments, subjects are studied without the tools they habitually use. This research runs the risk of focusing on cognitive strategies that are not frequently needed—namely, those useful in unfamiliar situations. This limitation may then lead researchers to form models of cognition that are overly dependent on domain-general representations. Cognitive models must preserve the insight that cognition may have few domain-general representations and processes (as discussed in chap. 8). Organisms have bodies and live in environments, and psychologists must take these facts seriously.

A JOURNEY ENDED AND A JOURNEY BEGUN

This concludes the tour of modes of representation. Proposals for the nature of representation are tools needed to construct cognitive models. All theories of cognitive processing make assumptions about how representations are organized. The representations and associated processes determine which things that a cognitive model posits are easy for people to do and which are hard. For this reason, psychologists must choose representational assumptions with care. Cognitive scientists must be careful not to make arbitrary representational choices that may have unintended consequences down the line.

This final chapter provided a set of issues that should be kept in mind when developing a cognitive model. It is not necessary (or even appropriate) to worry about all of them when trying to decide how to model a new task. The social or physical environment may not be relevant when thinking about a particular aspect of a system. Likewise, it may not be necessary to think about how people actually represent a situation. It may be enough to make proposals about the general structure of their representations (Markman, 1997; Ohlsson, 1996). Nevertheless, it is important to step back and think about these issues periodically. Investigators can easily lose sight of their fundamental assumptions when developing a model. By thinking about the broad context in which a model is embedded, we are forced to reconsider these assumptions to ensure that we still believe them to be appropriate. Focusing on these issues is one way to recognize the hidden assumptions that can creep into a research program and that can unwittingly influence the behavior of cognitive models.