7.1.1   Biological Fusion Metaphor

Sensory fusion in biological systems provides a natural metaphor for studying artificial data fusion systems.¹ As with any good metaphor, consideration of a simpler or more familiar phenomenon can provide valuable insight into the study of more complex or less familiar processes.

Even the most primitive animals sense their environment, develop some level of situation awareness, and react to the acquired information. By assisting in the acquisition of food and the avoidance of threats, situation awareness directly supports survival of the species. A barn owl, for instance, fuses visual and auditory information to help accurately locate mice under very low light conditions, whereas a mouse responds to threatening visual and auditory cues to avoid becoming the owl’s dinner.

In general, natural selection has tended to favor the development of more capable senses (sensors) and more effective utilization of the derived information (exploitation and fusion). Color vision in humans, for instance, is believed to have been a natural adaptation that permitted apes to find ripe fruit in trees. Situation awareness in animals can rely on a single, highly developed sense, or on multiple, often less capable, senses. A hawk depends principally on a highly acute visual search and tracking capability, whereas a shark relies primarily on its sense of smell when hunting. Sexual attraction can depend primarily on sight (plumage), smell (pheromones), or sound (mating call). For humans, sight is arguably the most vital sense, with hearing a close second. Dogs, however, rely more heavily on the senses of smell and hearing, with vision playing the role of a secondary information source.

Sensory input in biological organisms typically supports both sensory cueing and situation awareness development. Sounds cue the visual sense to the presence and the general direction of a potentially relevant event. Information gained by the aural sense (i.e., direction, speed, and tentative object classification) is then combined (fused) with the information gathered by the visual system to produce a more complete, higher-confidence, or higher-level situation awareness.

In many cases, sensory fusion can be critical to successful decision making. Food that looks appetizing (sight) might be extremely salty (taste), spoiled (smell), or too hot (touch). At the other extreme, sensory fusion might be unnecessary if the various senses provide redundant, as well as highly reliable, information. Bacon frying in a pan, for instance, need not be seen, smelled, and tasted because any of the senses taken alone would suffice.

Although it might seem prudent to simply ignore apparently redundant information, such information can help sort out conflicts, both intentional (deception) and unintentional (confusion). Whereas single-source deception is reasonably straightforward to perpetrate, deception across multiple senses (sensor modalities) is considerably more difficult. Successful hunting and fishing depend, to a large degree, on effective multisource deception. Duck hunters use both decoys and mating calls to simultaneously provide deceptive visual and auditory information. Because deer can sense danger through the sense of smell, sound, and sight, the shrewd hunter must mask his scent (or stay down-wind), make little or no noise, and remain motionless if the deer looks in his direction. In nonadversarial applications, data fusion provides a natural means of resolving unintentional conflicts that arise due to uncertainty in both the measurement and the decision spaces.

Sensory fusion need not be restricted to the familiar five senses of sight, sound, smell, taste, and touch. Internal signals, such as acidity of the stomach, coupled with visual cues, olfactory cues, or both, can trigger hunger pains. The fusion of vision, inner-ear balance information, and muscle feedback signals facilitates motor control. Measurement and signature intelligence (MASINT) in tactical applications likewise focuses on a wide range of “nontraditional” information classes.

7.1.2   Puzzle-Solving Metaphor

Because situation awareness development requires the production and maintenance of an adequate multiple level-of-abstraction picture of a (dynamic) situation, the data fusion process can be compared to assembling a jigsaw puzzle for which no picture of the completed scene exists. Although assembling puzzles that contain hundreds of pieces (information fragments) can challenge an individual’s skill and patience, the production of a comprehensive situational picture, created by fusing disparate and fragmentary sensor-derived information, typically represents a far more challenging task. Despite the fact that a jigsaw puzzle represents a static product, the assembly (fusion) strategy evolves over time once patterns begin to develop. In tactical situation awareness applications, time represents a key problem dimension, with both the information fragments and the fused picture constantly evolving.

The partially completed puzzle (fused situation awareness product) illustrated in Figure 7.1 contains numerous aggregate objects (i.e., forest and meadow), each composed of simpler objects (i.e., trees and ground cover). Each of these objects, in turn, has been assembled from multiple puzzle pieces, some representing a section of bark on a single tree trunk, whereas others might be a grassy area associated with a meadow. In general, sensor-derived information (individual puzzle pieces) can run the gamut from providing just color and texture to depicting higher level-of-abstraction objects such as trees and buildings.

At the outset of the puzzle construction process, problem solving necessarily relies on rather general problem-solving strategies (e.g., locate border pieces). Because there exists little context to direct either puzzle piece selection or placement, simple, brute-force pattern matching strategies are used at the early stages of the process. A blue-colored piece, for example, might represent either sky or water with little basis for distinguishing between the two interpretations during the early stages of the analysis. If the pieces came from a previously opened box, additional complications arise because some pieces may be missing (no available sensor data) whereas pieces from another puzzle may have inadvertently been dumped into the box (irrelevant or erroneous sensor reports). Once portions of the scene begin to fill in, the assembly process becomes much more goal-directed.

Slipping a single puzzle piece into the correct spot reduces entropy and supports higher level-of-abstraction scene interpretation at the same time. As regions of the puzzle begin to take form, identifiable features in the scene emerge (e.g., trees, grass, and cliffs), permitting higher-level interpretations to be developed (e.g., forest, meadows, and mountains). By supporting the placement of the individual pieces, as well as the goal-driven search (sensor resource management) for specific pieces, context provided by the developing scene (situation awareness product) helps further focus the construction process (fusion process optimization).

Just as duplicate or erroneous pieces significantly complicate puzzle assembly, redundant and irrelevant sensor-derived information similarly burdens machine-based situation development. Thus, goal-directed information collection offers a twofold benefit: critical information requirements are satisfied, and the collection (and subsequent analysis) of unnecessary information is minimized. Although numerous puzzle pieces may be yet unplaced (undetected objects) and some pieces might actually be missing (information not collectible by the available sensor suite), a reasonably comprehensive, multiple level-of-abstraction understanding of the overall scene (situation awareness) gradually emerges.

Three broad classes of knowledge are apparent in the puzzle reconstruction metaphor:

1. Individual puzzle pieces—collected information fragments, that is, sensor-derived knowledge

2. Puzzle-solving strategies, such as edge piece-finding and pattern matching—a priori reasoning knowledge

3. World knowledge, for example, the relationship between meadows and grass—domain context knowledge

To investigate the critical role that each of these knowledge forms plays in fusion product development, we recast the analysis process as a building construction metaphor. Puzzle pieces (sensor input) represent the building blocks required to assemble the scene (fused situation awareness product). A priori reasoning knowledge represents construction knowledge and skills, and context provides the nails and mortar that “glue” the sensor input together to form a coherent product. When too many puzzle pieces (or building blocks) are missing (inadequate sensor-derived information), constructing a scene (or a building) becomes impossible.

A simple example illustrates the importance of adequate input information. Figure 7.2a illustrates a cluster of azimuth and elevation measurements associated with two separate groups of air targets. Given the spatial overlap between the data sets, reliable target-to-group assignment may not be possible, regardless of the selected analysis paradigm or the extent of algorithm training. However, with the addition of range measurements (increased measurement space dimensionality), two easily separable clusters become readily apparent (Figure 7.2b). Because the information content of the original 2D data set was fundamentally inadequate, even sophisticated clustering algorithms would be unable to discriminate between the two target groups. However, with the addition of the third measurement dimension, even a crude clustering algorithm easily accomplishes the required decision task.

A spectrum of problem-solving paradigms can be used to implement reasoning knowledge (e.g., rules, procedures, and statistic-based algorithms), evidence combination strategies (e.g., Bayes, Dempster–Shafer, and fuzzy set theory), and decision-making approaches (e.g., rule instantiation and parametric algorithms). In general, the process of solving a complex puzzle (or performing automated situation awareness) benefits from both bottom-up (deductive-based) and top-down (goal-directed) reasoning that exploits relationships among the hierarchy of domain entities (i.e., primitive, composite, aggregate, and organizational).

In the puzzle-solving metaphor, context knowledge refers to relevant domain knowledge not explicitly contained within a puzzle piece (nonsensor-derived knowledge). Humans routinely apply a wide range of contextual knowledge during analysis and decision making.* For example, context-sensitive evaluation of Figure 7.1 permits the conjecture that the picture is a summer scene taken somewhere in the western United States. The season and location are deduced from the presence of deciduous trees in full leaf (summer) in the foreground and jagged snow-capped mountain peaks in the distance (western United States). In a similar fashion, the exploitation of context knowledge in automated fusion systems can promote more robust interpretations of sensor-derived information.

During both puzzle assembly and automated situation development, determining when an adequate situation representation has been achieved can be difficult. In the puzzle reconstruction problem, although the general landscape characteristics might be evident, missing puzzle pieces could depict denizens of the woodland community that can be hypothesized, but for which no compelling evidence yet exists. Individual puzzle pieces might contain partial or ambiguous information. For example, the presence of a section of log wall in the evolving scene suggests the possibility of a log cabin. However, additional evidence is required to validate this hypothesis.

Recasting the puzzle-solving metaphor in terms of the more dynamic task of weaving a tapestry from a large number of multicolored threads can be instructive. Individual sensors (e.g., radar) might produce just a portion of a single thread (red portions for moving vehicles and yellow portions for stationary vehicles), whereas other sensors (e.g., imagery) might produce different colored “threadlets” or even higher-level patterns. Given adequate sensor diversity and effective sensor management (level 4 fusion), threads (fragmentary object tracks) can be pieced together (level 1 fusion) and then woven (fusion levels 1 and 2) to generate a “picture” of a scene that supports higher-level situation understanding (level 3 fusion).

In the tapestry metaphor, threads represent a sequence of entity states that implicitly occur as a function of time. Entities can represent individuals, vehicles, organizations, or abstract objects. Entity states can be quite general and include attributes such as status, location, emissions, kinematics, and activity. Entities might be related if they share common states or possess more abstract relationships.

7.1.3   Command and Control Metaphor

The game of chess provides an appropriate metaphor for military command and control (C²), as well as an abstract metaphor for any system that senses and reacts to its environment. Both chess players and battlefield commanders require a clear picture of the “playing field” to properly evaluate the options available to them and their opponents. In both chess and C², opposing players command numerous individual resources (i.e., pieces or units) that possess a range of characteristics and capabilities. Resources and strategies vary over time. Groups of chess pieces are analogous to higher-level organizations on the battlefield. The chessboard represents domain constraints to movement that are similar to constraints posed by terrain, weather, logistics, and other features of the military problem domain. Player-specific strategies are analogous to tactics, whereas legal moves represent established doctrine. In both domains, the overall objective of an opponent may be known, whereas specific tactics and subgoals must be deduced.

Despite a chess player’s complete knowledge of the chessboard (all domain constraints), the location of all pieces (own and opponent-force locations), all legal moves (own and opponent-force doctrine), and his ability to exercise direct control over all of his own assets, chess remains a highly challenging game. Tactical situation development possesses numerous complicating factors that make it far more of a challenge.

First, battlefield commanders rarely possess a complete or fully accurate picture of their own forces let alone those of their adversaries. If forced to deal with incomplete and inaccurate force structure knowledge, as well as location uncertainty, chess players would be reduced to guessing the location and composition of an adversary’s pieces, akin to playing “Battleship,” the popular children’s game.

Second, individual sensors provide only limited observables, coverage, resolution, and accuracy. Thus, the analysis of individual sensor reports tends to lead to ambiguous and rather local interpretations. Third, domain constraints in tactical situation awareness are considerably more complex than the well-structured (and level) playing field in chess. Fourth, doctrinal knowledge in the tactical domain tends to be more difficult to exploit effectively and far less reliable than its counterpart in chess.

Application-motivated metaphors can be useful in terms of extending current fusion applications. For example, data fusion seems destined to play a significant role in the development of future “smart highway” control systems. The underpinning of such a system is a sophisticated control capability that optimally resolves a range of conflicting requirements, such as (1) to expedite the movement of both local and long-distance traffic, (2) to ensure maximum safety for all vehicles, and (3) to create the minimum environmental impact. The actors in the metaphor are drivers (or automated vehicle control systems), the rules of the game are the “rules of the road,” and domain constraints are the road network and traffic control means. Players possess individualized objectives and tactics; road characteristics and vehicle performance capabilities provide physical constraints on the problem solution.

7.1.4   Evidence Combination

Because reliance on a single information source can lead to ambiguous, uncertain, and inaccurate situation awareness, data fusion seeks to overcome such limitations by synergistically combining relevant (and available) information sources leading to the generation of a potentially more consistent, accurate, comprehensive, and global situation awareness. A famous poem by John Godfrey Saxe² written more than a century ago, aptly demonstrates both the need for and the challenge of effectively combining fragmentary information.

The poem describes an attempt by six blind men to gain a first-hand understanding of an elephant. The first man happens to approach the elephant from the side and surmises that an elephant must be similar to a wall. The second man touches the tusk and imagines an elephant to be like a spear. The third man approaches the trunk and decides an elephant is rather like a snake. The fourth man reaches out and touches a leg and decides that an elephant must be similar to a tree. The fifth man chances to touch an ear and imagines an elephant must be like a fan. The sixth man grabs the tail and concludes an elephant is similar to a rope. Although each man’s assessment is entirely consistent within his own limited sensory space and myopic frame of reference, a true picture of an elephant emerges only after these six observations are effectively integrated (fused).

Among other insights, the puzzle-solving metaphor presented earlier illustrated that (1) complex dependencies can exist between the information fragments and the completed situation description and (2) determining whether an individual puzzle piece actually belongs to the scene being assembled can be difficult. Even when the collected information is known to be relevant, based on strictly local interpretations, it might not be possible to determine whether a given blue-colored piece represents sky, water, or some other feature class.

During criminal investigations, observations are assembled, investigators hunt for clues, and motives are evaluated. Tactical data fusion involves a similar approach. Although the existence of a single strand of hair might appear insignificant at the outset of a criminal investigation, it might prove to be the key piece of evidence that discriminates among several suspects. Similarly, seemingly irrelevant pieces of sensor-derived information might ultimately link observations with motives, or provide other significant benefits.

Thus, both the information content (information measure) of a given piece of relevant data and its relationship to the overall fusion task are vital considerations. As a direct consequence, the development of a simple information theoretic framework for the data fusion process appears problematic. Assessing the utility of information in a given application effectively demands a top-down, holistic analysis approach.

7.1.5   Information Requirements

Because no widely accepted theory exists for determining when adequate information has been assembled to support a given fusion task, empirical measures of performance must generally be relied upon to evaluate the effectiveness of both individual fusion algorithms and an overall fusion system. In general, data fusion performance can be enhanced by

• Technical improvements in sensor measurements (i.e., longer range, higher resolution, improved signal-to-noise ratio, better accuracy, higher reliability)

• Increasing measurement space dimensionality by employing heterogeneous sensors that provide at least partially independent information

• Spatially distributing sensors to provide improved coverage, perspective, and measurement reliability

• Incorporation of relevant nonsensor-derived domain knowledge to constrain information combination and the decision-making process

In general, effective data fusion automation requires the development of robust, context-sensitive algorithms that are practical to implement. Robust performance argues for the use of all potentially relevant sensor-derived information sources, reasoning knowledge, and maximal utilization of relevant nonsensor-derived information. However, to be practical to implement and efficient enough to employ in an operational setting, the algorithms may need to compromise fusion performance quality. Consequently, system developers must quantify or otherwise assess the value of these various information sources in light of system requirements, moderated by programmatic, budgetary, and performance constraints (e.g., decision time-line and hardware capability). The interplay between achieving optimal algorithm robustness and context sensitivity on the one hand, and a practical implementation on the other, is a fundamental tension associated with virtually any form of machine-based reasoning directed at solving complex, real-world problems.

7.1.6   Problem Dimensionality

Effective situational awareness (with or without intentional deception) generally benefits from the collection and analysis of a wide range of observables. As a result of the dynamic nature of many problem domains, observables can change with time and, in some cases, may require continuous monitoring. In a tactical application, objects of interest can be stationary (fixed or currently nonmoving), quasi-stationary (highly localized motion), or in motion. Individual objects possess characteristics that constrain their behavior. Objects emit different forms of electromagnetic energy that vary with time and can indicate the state of the object. Object emissions include intentional or active emissions, such as radar, communications, and data link signals, as well as unintentional or passive emissions, such as acoustic, magnetic, or thermal signatures generated by internal heat sources or environmental loading. Patterns of physical objects and their behavior provide indications of organization, tactics, and intent, as can patterns of emissions, both active and passive. For example, a sequence of signals emitted from a surface-to-air missile radar system that changes from search, to lock-on, to launch clearly indicates hostile intent.

Because a single sensor modality is incapable of measuring all relevant information dimensions, multiple sensor classes may be needed to detect, track, classify, and infer the likely intent of a host of objects from submarines and surface vessels, to land-, air-, and space-based objects. Certain sensor classes lend themselves to surveillance applications, providing both wide-area and long-range coverage, plus readily automated target detection. Examples of such sensor classes include signals intelligence (SIGINT) for collecting active emissions, moving target indication (MTI) radar for detecting and tracking moving targets against a high clutter background, and synthetic aperture radar (SAR) for detecting stationary targets. Appropriately cued, other sensor classes that possess narrower fields of view and typically operate at a much shorter range, can provide information that supports refined analysis. Geospatial and other intelligence databases can provide the static domain context for interpreting target-sensed data, whereas environmental sensors can generate dynamic estimates of the current atmospheric conditions.

7.1.7   Commensurate and Noncommensurate Data

Although the fusion of similar (commensurate) information would seem to be more straightforward than the fusion of dissimilar (noncommensurate) information, that is not necessarily the case. Three examples are offered to highlight the varying degrees of difficulty associated with combining multiple-source data. First, consider the relative simplicity of fusing registered electronic intelligence (ELINT) data and real-time SAR imagery. Although these sensors measure dramatically different information dimensions, both sources provide reasonably wide area coverage, relatively good geolocation, and highly complementary information. As a consequence, the fusion process tends to be straightforward. Even when an ELINT sensor provides little more than target line-of-bearing, overlaying the two data sets might be instructive. If the line-of-bearing intercepts a single piece of equipment in the SAR image, the radar system class, as well as its precise location, would be known. This information, in turn, can support the identification of other nearby objects in the SAR image (e.g., missile launchers normally associated with track-while-scan radar).

At the other end of the spectrum, the fusion of information from two or more identical sensors can present a significant challenge. Consider, for example, fusing data sets obtained from spatially separated forward-looking infrared (FLIR) radars. Although FLIR imagery provides good azimuth and elevation resolution, it does not directly measure range. Because the range and view angles to targets will be different for multiple sensors, combining such data sets demands some form of range estimation, as well as sophisticated registration and normalization techniques.

Finally, consider the fusion of two bore-sited sensors: light-intensified and FLIR. The former device amplifies low-intensity optical images to enhance night vision. When coupled with the human’s natural ability to separate moving objects from the relatively stationary background, such devices permit visualization of the environment and detection of both stationary and moving objects. However, such devices offer limited capability for the detection of stationary personnel and equipment located in deep shadows or under extremely low ambient light levels (e.g., heavy cloud cover, no moon, or inside buildings). FLIR devices, however, detect thermal radiation from objects. Consequently, these devices support the detection of humans, vehicles, and operating equipment based on their higher temperature relative to the background. Consequently, with bore-sighted sensors, pixel-by-pixel combination of the two separate images provides a highly effective night vision capability.

7.3.1   All-Source

Because individual data sources provide only partial characterizations of observed behavior, fusing data from multiple sources offers the promise of enhanced situational awareness. Although the term all-source has been in common usage for decades, only recently have automated fusion approaches included unstructured text input. We loosely classify all human-generated information as HUMINT. Although the addition of HUMINT potentially adds valuable problem dimensions, it presents new challenges, as well.

Whereas traditional sensors generally provide some degree of “continuous monitoring” and possess well-understood technical capabilities in terms of coverage, resolution, environmental sensitivity, and probability of detection, HUMINT tends to be highly asynchronous and opportunistic. For applications that rely heavily on HUMINT (e.g., tracking terrorists in urban environments where traditional intelligence, surveillance, and reconnaissance (ISR) sensors offer limited support), effective integration of HUMINT with traditional sensors requires that both the strengths and weaknesses of all data sources be fully understood.

HUMINT provides potentially unequaled entity description and problem domain insights. At the same time, the information content tends to be far more subjective than traditional sensor reports. Whereas conventional sensor data primarily feeds level 1, HUMINT can have relevancy across all levels of the Joint Director of Laboratories (JDL) fusion model. With traditional sensors, data latency tends to be small. By comparison, HUMINT information can be hours (e.g., CNN reports) or even days old (e.g., counterintelligence reports) by the time it reaches an automated fusion system. HUMINT is clearly not “just another Intel sensor.”

A comprehensive discussion of the many challenges associated with incorporating HUMINT into the all-source fusion process is beyond the scope of this chapter. Robust machine-based natural language understanding, however, tops the list of issues. The technical issues associated with natural language processing are well documented in the literature. Related challenges include such considerations as effective exploitation of cultural, regional, and religious biases, extraction of hidden meaning, and assessment of information validity. For our present purposes, we sidestep the many technical challenges and focus instead on the pragmatic aspects of incorporating HUMINT into the all-source fusion process.

Whereas conventional sensors typically produce highly structured outputs, HUMINT tends to be partially structured at best. To fuse human-generated level 1 “messages” with traditional multisource data, unstructured text must first be “normalized.” At a minimum, this implies that an entity’s triple attribute set (i.e., name, location, time) must be extractable along with other relevant attributes (e.g., action, direct and indirect objects, explicitly indicated relationships).

To facilitate organization discovery (level 2) and higher level situation understanding-oriented processes (level 3), preprocessing may likewise be required to create a more standardized information base that lends itself to effective machine-based information exploitation.

7.3.2   Entity Tracking

Stripped to its essence, a track is little more than a temporal sequence of states derived through physical observation, detection of relevant transactions, indirect evidence, or some other means. At least conceptually, tracks can be developed for individual entities (tanks, trucks, individuals, bank accounts), composites (tank companies, battalions, and Al Queda cells), as well as more abstract concepts such as social movements, political mood, and financing methods for illicit operations.

Historically, tracking has focused on detecting and following the movement of aircraft, ships, and ground-based vehicles using a variety of active and passive sensing systems. When kinematics plays a major role and data latencies are relatively low, state-of-the-art multiple hypothesis trackers perform reasonably well.

In contrast to traditional space–time tracking applications, following individuals in urban settings is likely to involve at least some high-latency data sources. Depending on the mix of sources, traditional kinematics-based trackers will play a diminished or nonexistent role. Figure 7.7 shows the track of an individual based on HUMINT and SIGINT and Figure 7.8 demonstrates how time-late information modified that track (the smaller the error ellipse, the higher the location accuracy of the source data).

Fusion systems that rely on HUMINT must effectively manage uncertainty across all sensing modalities. Whereas traditional sensor systems (radar, EO/infrared (IR), unattended ground sensors (UGS)) tend to have relatively well-defined operating characteristics, human-generated information can be far harder to characterize due to ambiguity associated with natural language, perceptual coloring, and nonsystematic errors. Perceptual coloring includes personal biases, training limitations, and experience base. Reporting errors can be either unintentional or deliberately introduced (e.g., interrogations of individuals having questionable motives).

7.3.3   Track Coincidence

Relationships between entities can be either explicit or implicit. A cell phone call between two individuals or an observation that places an individual in a vehicle with a known person of interest establishes an explicit link. Implicit links are relationships that must be deduced. Fusing multisource tracks provides an important mechanism for discovering implicit links between entities. For example, two individuals found to be (1) in the vicinity of each other at approximately the same time, (2) eating in the same restaurant at overlapping times, (3) staying in the same building on different days, and (4) driving a vehicle owned by another individual may be connected. In general, implicit links tend to have lower confidence than explicit links.

7.3.4   Key Locations

Any location that possesses particular significance (historical, tactical, strategic) can be considered a key location. Embassies, police stations, suspected safe houses, isolated sections of town, and an abandoned warehouse might all be key locations in a certain application. Specific homes, businesses, religious institutions, neighborhoods, districts, or regions might also qualify. Any location associated with high-value individuals (HVIs) could automatically be added to the key location database. Because key locations need not be a literal feature (e.g., “the region between the park and Mosque C”), accommodations must be made for handling both crisp and fuzzy spatial location descriptions.

Key locations can be rated based on their estimated level of significance. Certain locations might be important enough to fire imbedded triggers whenever certain individuals or classes of tracks intercept these regions. Any track or activity that associates (however loosely) with such locations might raise the interest level of the track, justifying a more detailed analysis or even operator intervention.

7.3.5   Behavior

Ideally, tracks capture at least a partial representation of entity behavior. Interpreting/understanding behavior can involve something as simple as associating a vehicle’s trajectory with a road network or as complex as predicting an entity’s goal state. Fusing individual tracks helps link previously unassociated entities, as well as identify higher level-of-abstraction entities and patterns.

Fusing multiple entity tracks helps uncover more complicated relationships (e.g., construction and emplacement of an improvised explosive device (IED)). The feasibility of assessing “apparent coordination” among a group of entities tends to increase with the number of problem dimensions. Behavior patterns of interest are likely to involve apparent coordination and coincidence patterns among individuals, each serving specific functional roles within an organization. In general, exploitation of additional information sources potentially leads to a more robust recognition of both general, as well as specific, behaviors.

Treating tracks from a more global perspective, one that spans levels 1–3 can lead to a more robust behavioral understanding. Suppose, for example, the first portion of a track of a military vehicle represents “advancing to contact” while the next portion of the same track involves little movement but a great deal of communications or other activities (e.g., coordinating activity, firing weapons). Viewed as disconnected (local) activities, no intrinsic information in the track states explains these distinctly different behavior modes. Only when viewed at a higher level of abstraction (in conjunction with levels 2 and 3) can the two separate lower-level “behavior” patterns be “understood” as phases of battle.

As a second example, consider the track of an individual that begins when he leaves his home in the morning. The man is seen at a mosque several hours later, and then at the market in the early afternoon. The man finally returns home around 5 p.m. This particular individual may have visited many other locations and engaged in numerous unobserved activities. On the basis of the currently available information, however, this is all we know.

Given our limited knowledge, the man’s known “track” might have a vast number of interpretations. Assuming intentional behavior, we are left to presume that this series of states has a definite (albeit unknown) purpose. When and if more information becomes available to “fill in the blanks,” the information can be added to the existing track file and exploited by the higher-level fusion algorithms.

If significant portions of the man’s track (pattern of activity) are repeated at predictable intervals (e.g., daily), variations in his routine can be detected. In some cases, such anomalies might be of more interest than the individual’s nominal behavior. Variations could include missing a regular prayer meeting or remaining at a particular location for a much shorter or much longer time than usual. A track may warrant additional scrutiny if the individual happens to be an HVI, the mosque he visited is a suspected meeting place for insurgents, and he remained there long after the noon prayer meeting ended.

Similarly, if an individual who normally sends just a few e-mails per day suddenly writes dozens of them, this change in “expected” behavior might be significant. If an individual that normally remains within a few miles of his home suddenly appears in a different portion of the city or in a completely different city, this anomaly might justify a more detailed analysis or even trigger an alert.

7.3.6   Context

Humans, often subconsciously, incorporate a wide range of contextual knowledge during the analysis of sensor-derived data. To achieve comparable robustness, automated approaches must incorporate such “external” knowledge, as well. Consider the following message:

“Individual A met Individual B at XYZ” where XYZ represents specific global positioning system (GPS) coordinates

Although the message clearly links individual A and individual B, it fails to reveal that (1) the meeting took place in front of Mosque C, (2) it happened in a section of town where there has recently been considerable sectarian violence, and (3) Mosque C is a suspected insurgent-controlled facility, all of which places the sensor-derived information within a larger framework. Global information system (GIS) systems, organization charts, existing databases, and numerous other data sources maintain domain knowledge that, if properly exploited, supports more effective use of sensor-derived, as well human-generated information.

Applying relevant a priori domain knowledge potentially increases information content (and thereby the dimensionality of the decision-making process) by either adding relevant supporting features or constraining the decision process. Interpreting the above-mentioned message “out of context” (i.e., without knowing about the mosque or the recent sectarian violence in the area) is analogous to solving a set of linear equations when there exist more unknowns than equations. In both cases, the problem is under-constrained.

As mentioned earlier, adding HUMINT to the fusion process introduces a number of challenges. Dealing with semantic descriptions of location is one such challenge. To correlate on location, the “location” descriptions must be comparable to those provided by conventional sensors.

The example shown in Figure 7.9 highlights the need to translate from semantic location descriptions to more formal “mathematical” representations. In this case, all eight locations listed in the left-hand box effectively refer to the same general region of space. Thus, all are semantically “similar,” something that would be difficult to deduce using a strict semantic-based reasoning approach.

Chapter 24 discusses “semantic location translation” in more detail. For the present, we merely illustrate that it is feasible to relate crisp (Boolean) and semantic (fuzzy) spatial representations. Figure 7.10 depicts the product of intersecting near a named road (fuzzy feature) with all buildings (Boolean features).

7.3.7   HUMINT and the JDL Fusion Model

Whereas traditional sensors provide a relatively low level-of-abstraction input to the fusion process (before entity extraction), humans intuitively combine and abstract sensory and nonsensory information, generating distilled results that potentially fall across all levels of the JDL fusion model. For example, a text-based message might (1) report on the current location of a suspected terrorist (level 1 input), (2) describe the organization to which an individual is associated (level 2 input), or (3) indicate a specific threat an individual or an organization might pose (level 3 input).

In more traditional fusion applications, raw sensor data must undergo level 1 processing before data exists to link and organize in level 2. Once HUMINT becomes a viable information source, however, vast stores of both text-based open source (e.g., TV, radio, newspapers, and web sites) and classified data (e.g., messages, reports, databases) exist that can be directly mined by higher-level fusion processes to help discover organizations and develop threat awareness products. Thus, once HUMINT is included as a data source, mixed-initiative bottom-up (feed forward) and top-down (feedback) reasoning across all levels of the JDL fusion model becomes feasible.

In a level 2 counterinsurgency application, for instance, both classified (messages, reports, studies, briefings) and open source information (Internet, newspapers, TV, radio) may provide direct support to higher-level fusion applications. Relevant information about specific terrorist organizations might come from reports, web sites, and media sources. Whereas some organizations remain relatively static over time, the loose organizational structures favored by many terrorist organizations tend to evolve, as well as dramatically change, over time.

Full automation of organization construction and maintenance based on digesting a variety of unstructured text sources represents an extremely challenging problem. During the early stages of the analysis process, documents are typically read line by line. Syntactic and lexical analysis algorithms tag parts of speech and assign some level of meaning to individual sentences. Word sense disambiguation and syntactic ambiguity require consideration of implicit sentence context (ranging from fixed and narrowly defined to very general).

Extracting valid relationships between tagged words found in different sentences of the same document presents many challenges. Because different sources might express the same information in different ways or may contain conflicting information, combining information from different information sources presents a far greater challenge. Even when all available information sources are fully exploited, it might not be possible to resolve all uncertainty. In the short term, at least, it is safe to say there must be a human in the loop to help guide automated processes, draw their own conclusions, and remove erroneous deductions.

7.3.8   Context Support Extensions

Existing context-sensitive fusion algorithms tend to be highly specialized and tuned to a specific application. To provide the appropriate contextual information, developers must tap an array of information sources (databases, GIS, and other knowledge repositories), as well as write code for required intermediate processing. Rather than tightly coupling context support to an application, building general services that support a broader range of applications appears to have considerable merit. Given the increasing importance of context in data fusion applications, adding a Context Support module to the existing JDL fusion model would both acknowledge the importance of nonsensor-derived information to the fusion process, as well as likely foster the development of reusable functionality that is not tied to a specific program, developer, or data source.

7.4.1   Observation 1

For an effective automation of key elements of the fusion process, it may be necessary to emulate more of the strengths of human analysts. In general, humans

• Are adept at model-based reasoning (which supports robustness and extensibility)

• Naturally employ domain knowledge to augment formally supplied information (which supports context sensitivity)

• Update or modify existing beliefs to accommodate new information as it becomes available (which supports dynamic reasoning)

• Intuitively differentiate between context-sensitive and context-insensitive knowledge (which supports maintainability)

• Control the analysis process in a highly focused, often top-down fashion (which enhances efficiency)

7.4.2   Observation 2

Global phenomena naturally require global analysis. Analysis of local phenomena can benefit from a global perspective as well. The target track assignment process, for instance, is typically treated as a strictly local analysis problem. Track assignment is typically based on recent, highly local behavior (often assuming a Markoff process). For ground-based objects, a vehicle’s historical trajectory and its maximum performance capabilities provide rather weak constraints on future target motion.

Although applying nearby domain constraints could adequately explain the local behavior of an object (e.g., constant velocity travel along a relatively straight, level road), a more global viewpoint is required to interpret global behavior. Figure 7.11 demonstrates local (i.e., concealment, minimum terrain gradient, and road seeking), medium-level (i.e., river-crossing and road-following), and global (i.e., reinforce at unit) interpretations of a target’s trajectory over space and time. The development and maintenance of such a multiple level-of-abstraction perspective is critical to achieving robust automated situation awareness.

7.4.3   Observation 3

A historical view of fusion approaches can be instructive. Production systems have historically been found to perform better against static, well-behaved, finite-state diagnostic-like problems than against problems possessing complex dependencies and exhibiting dynamic, time-varying behavior. In general, any system that relies on rigid, single level-of-abstraction control would be expected to exhibit such characteristics. Despite this fact, during the early 1990s, expert systems were routinely applied to dynamic, highly context-sensitive problems, often with disappointing results.

The lesson to be learned from this and other similar examples is that both the strengths and limitations of a selected problem-solving paradigm must be fully understood by the algorithm developer from the outset. Interestingly, when an appropriately constrained task was successfully automated using an expert system, developers frequently discovered that the now well-understood problem could be more efficiently implemented using a different paradigm.

When an expert system proved inadequate for a given problem, artificial neural systems were often seen as an alternative or even preferred approach. Neural networks require no programming, suggesting that the paradigm would be ideal for handling ill-defined or poorly understood problems. Whereas expert systems could have real-time performance problems, artificial neural systems promised high-performance hardware implementations. In addition, the adaptive nature of the neural net learning process seemed to match real world, dynamically evolving problem-solving requirements. However, most artificial neural systems operate more like a statistical or fuzzy pattern recognizer than a sophisticated reasoning system capable of generalization, reasoning by analogy, and abstract inference. In addition, network training can be problematic for realistic applications that involve complex behaviors evolving over time.

7.4.4   Observation 4

Traditional air target radar trackers employed a single statistic-based algorithm regardless of whether an aircraft was flying at an altitude of 20 km or just above tree-top level. Likewise, the algorithms were generally insensitive with regard to whether the target happened to be a high-performance fighter aircraft or a relatively low speed helicopter. Suppose a nonfriendly high-performance reconnaissance aircraft is flying just above a river that snakes through a mountainous region. There exists a wide range of potential problems associated with tracking such a target, including dealing with high clutter return, terrain masking, and multipath effects. In addition, an airborne radar system may have difficulty tracking the target due to high acceleration turns associated with an aircraft following a highly irregular surface feature. The inevitable track loss and subsequent track fragmentation errors typically require intervention by a radar analyst. Tracking helicopters can be equally problematic. Although they fly more slowly, such targets can hover, fly below tree-top level, and execute rapid directional changes.

Tracking performance can potentially be improved by making the tracking analysis sensitive to target class-specific (model-based) behavior, as well as to constraints posed by the domain. For example, the recognition that the aircraft is flying just above the terrain suggests that surface features are likely to influence the target’s trajectory. When evaluated with respect to “terrain feature-following models,” the trajectory would be discovered to be highly consistent with a “river-following flight path.” Rather than relying on past behavior to predict future target positions, a tracking algorithm could anticipate that the target is likely to continue to follow the river.

In addition to potentially improving tracking performance, interpreting sensor-derived data within context permits more abstract interpretations. If the aircraft were attempting to avoid radar detection by one or more nearby surface-to-air missile batteries, a nap of the earth flight profile could indicate hostile intent.

Even more global interpretations can be hypothesized. Suppose a broader view of the “situation picture” reveals another unidentified aircraft operating in the vicinity of the river-following target. By evaluating the apparent coordination between the two aircraft, the organization and mission of the target group can be conjectured. For example, if the second aircraft begins jamming friendly communication channels just as the first aircraft reaches friendly airspace, the second aircraft’s role can be inferred to be “standoff protection for the primary collection or weapon delivery aircraft.” The effective utilization of relevant domain knowledge and physical domain constraints offers the potential for developing both more effective and higher level-of-abstraction interpretations of sensor-derived information.

7.4.5   Observation 5

Indications and warnings, as well as many other forms of expectation-based analysis, have traditionally relied on relatively rigid doctrinal and tactical knowledge. Contemporary data fusion applications, however, often must support intelligence applications where flexible, ill-defined, and highly creative tactics and doctrine are employed. Consequently, the credibility of any analysis that relies on rigid expectation-based behavior needs to be carefully scrutinized. Although the lack of strong, reliable a priori knowledge handicaps all forms of expectation-based reasoning, the effective application of relevant logical, physical, and logistical context at least partially compensates for the lack of more traditional problem domain constraints.