3.1.1   Background

The data fusion model, developed in 1985 by the U.S. Joint Directors of Laboratories (JDL) Data Fusion Group, with subsequent revisions, is the most widely used system for categorizing data fusion-related functions. The goal of the JDL data fusion model is to facilitate understanding and communication among acquisition managers, theoreticians, designers, evaluators, and users of data fusion technology to permit cost-effect system design, development, and operation.^1,2

The stated purpose for that model and its subsequent revisions have been to

• Categorize different types of fusion processes

• Provide a technical architecture to facilitate reuse and affordability of data fusion and resource management (DF&RM) system development

• Provide a common frame of reference for fusion discussions

• Facilitate understanding of the types of problems for which data fusion is applicable

• Codify the commonality among problems

• Aid in the extension of previous solutions

• Provide a framework for investment in automation.

It should be emphasized that the JDL model was conceived as a functional model and not as a process model or as an architectural paradigm.³ An example of the latter is the dual node network (DNN) technical architecture developed by Bowman⁴ and described in Chapter 22.

3.1.2   Role of Data Fusion

The problem of deriving inferences from multiple items of data pervades all biological cognitive activity and virtually every automated approach to the use of information. Unfortunately, the universality of data fusion has engendered a profusion of overlapping research and development in many applications. A jumble of confusing terminology (illustrated in Figure 3.1) and ad hoc methods in a variety of scientific, engineering, management, and educational disciplines obscures the fact that the same ground has been covered repeatedly.

The role of data fusion has often been unduly restricted and its relevance limited to particular state estimation problems. For example, in military applications such as targeting or tactical intelligence, the focus is on estimating and predicting the state of specific types of entities in the external environment (e.g., targets, threats, or military formations). In this context, the applicable sensors or sources that the system designer considers are often restricted to sensors that directly collect observations of targets of interest.

Ultimately, however, such problems are inseparable from other aspects of the system’s assessment of the world. In a tactical military system, these will involve estimation of the state of one’s own assets in relation to the relevant external entities: friends, foes, neutrals, and background. Estimation of the state of targets and threats cannot be separated from the problems of estimating one’s own navigation state, of calibrating one’s sensor performance and alignment, and of validating one’s library of target models. The data fusion problem, then, becomes that of achieving a consistent, comprehensive estimate and prediction of some relevant portion of the world state. From such a view, data fusion involves exploiting all available sources of data to solve all relevant state estimation or prediction problems; where relevance is determined by utility in forming plans of action.

The data fusion problem therefore encompasses a number of interrelated problems: estimation and prediction of states of entities, both external and internal (organic) to the acting system, and the interrelations among such entities. Evaluating the system’s models of the characteristics and behavior of all of these external and organic entities is, likewise, a component of the overall problem of estimating the actual world state.

The complexity of the data fusion system engineering process is characterized by difficulties in

• Representing the uncertainty in observations and in models of the phenomena that generate observations

• Combining noncommensurate information (e.g., the distinctive attributes in imagery, text, and signals)

• Maintaining and manipulating the enormous number of alternative ways of associating and interpreting large numbers of observations of multiple entities.

Deriving general principles for developing and evaluating data fusion processes—whether automatic or manual—can benefit from a recognition of the similarity in the underlying problems of associating and combining data that pervade system engineering and analysis, as well as human cognition. Furthermore, recognizing the common elements of diverse data fusion problems can provide extensive opportunities for synergistic development. Such synergy—enabling the development of information systems that are cost-effective and trustworthy—requires (a) common performance evaluation measures, (b) common system engineering methodologies, (c) common architecture paradigms, and (d) multispectral models of targets and of information sources.

3.1.3   1998 Revision

In 1998, Steinberg et al.⁵ published a article formally addressing various extensions to the model. That article began by revisiting the basic definition(s) of data fusion both conceptually and in terms of the levels that are characterized in the original JDL model. This article was republished in 2000 with minor revisions.

Figure 3.2 depicts the 1998 version of the JDL data fusion model. The goals of the 1998 revision—as in the present proposed revision—were to

• Clarify some of the concepts that guided the original model

• Refine the definition of the JDL data fusion levels to better reflect different classes of data fusion problems—that is, systematic differences in types of inputs, outputs and techniques

• Broaden the definitions of fusion concepts and functions to apply across as wide a range of problems as possible, beyond the initial focus on military and intelligence problems

• Maintain a degree of continuity by deviating as little as possible from the usage of concepts and terminology prevailing in the data fusion community

The 1998 revision included the following actions:

1. The definitions of data fusion levels 1–3 were broadened to accommodate fusion problems beyond the military and intelligence problems that had been the focus of earlier versions of the JDL model.

2. A level 0 was introduced to address problems of detecting and characterizing signals, whether in a one-dimension time-series or transform spaces or in multiple spatial dimensions (as in imagery or video feature extraction).

3. The 1998 revision article emphasized that process refinement, as originally conceived, involves not only data fusion functions but planning and control functions as well. Given the formal duality between estimation and control and the similar duality between data association and planning, as discussed in Section 3.6, the present recommended revision clarifies these distinctions: (a) it retains within level 4 fusion the process assessment functions such as performance and consistency assessment and (b) it treats the process refinement management functions as a part of the resource management functional partitioning.

4. The database management system (DBMS) is treated as a support application accomplished within a layer of the common operating environment below the applications program interface.

5. The notion of estimating informational and psychological states in addition to the familiar physical states was introduced, citing the work of Waltz.⁶

6. An approach to standardization of an engineering design methodology for data fusion processes was introduced, citing the earlier works of Bowman,⁷ Steinberg and Bowman,⁸ and Llinas et al.⁹ in which engineering guidelines for data fusion processes were elaborated.

3.1.4   Definition of Data Fusion

The initial JDL data fusion lexicon (1985) defined data fusion as

A process dealing with the association, correlation, and combination of data and information from single and multiple sources to achieve refined position and identity estimates, and complete and timely assessments of situations and threats, and their significance. The process is characterized by continuous refinements of its estimates and assessments, and the evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.¹

As theory and applications have evolved over the years, it has become clear that this initial definition is rather too restrictive. A definition was needed that could capture the fact that similar underlying problems of data association and combination occur in a very wide range of engineering, analysis, and cognitive situations. Accordingly, the initial definition requires a number of modifications:

1. Although the concept combination of data encompasses a broad range of problems of interest, correlation does not. Statistical correlation is merely one method for generating and evaluating hypothesized associations among data.

2. Association is not an essential ingredient in combining multiple pieces of data. Work in random set models of data fusion provides generalizations that allow state estimation of multiple targets without explicit report-to-target association.^{6,7 and 8}

3. ‘Single or multiple sources’ is comprehensive; therefore, it is superfluous in a definition.

4. The reference to position and identity estimates should be broadened to cover all varieties of state estimation.

5. Complete assessments are not required in all applications; ‘timely’, being application-relative, is superfluous.

6. ‘Threat assessment’ limits the application to situations where threat is a factor. This description must also be broadened to include any assessment of the impact (i.e., the cost or utility implications) of estimated situations. In general, data fusion involves refining and predicting the states of entities and aggregates of entities and their relation to one’s own mission plans and goals. Cost assessments can include variables such as the probability of surviving an estimated threat situation.

7. Not every process of combining information involves collection management or process refinement. Thus, the definition’s second sentence is best construed as illustrative, not definitional.

Pruning these extraneous qualifications, the model revision proposes the following concise definition for data fusion:⁵

The process of combining data or information to estimate or predict entity states.

Data fusion involves combining data—in the broadest sense of ‘data’—to estimate or predict the state of some aspect of the universe. Often the objective is to estimate or predict the physical state of entities: their identity, attributes, activity, location, and motion over some past, current, or future time period.

However, estimation problems can also concern nonphysical aspects. If the job at hand is to estimate the state of people (or any other sentient beings), it may be important to estimate or predict the informational and perceptual states of individuals or groups and the interrelation of these with physical states.

Arguments about whether ‘data fusion’ or another label—say, some other label shown in Figure 3.1—best describes this very broad concept are pointless. Some people have adopted terms like ‘information integration’ in an attempt to generalize earlier, narrower definitions of ‘data fusion’ (and, perhaps, to dissociate themselves from old data fusion approaches and programs).

Nonetheless, relevant research should not be neglected simply because of shifting terminological fashion. Although no body of common and accepted usage currently exists, this broad concept is an important topic for a unified theoretical approach and, therefore, deserves its own label.

3.1.5   Motivation for Present Revision

Like ‘data fusion’ itself, the JDL data fusion levels are in need of definitions that are at once broadly applicable and precise. The suggested revised partitioning of data fusion functions is designed to capture the significant differences in the types of input data, models, outputs, and inferencing appropriate to broad classes of data fusion problems. In general, the recommended partitioning is based on different aspects of a situation for which the characterization is of interest to a system user. In particular, a given entity—whether a signal, physical object, aggregate, or structure—can often be viewed either (a) as an individual whose attributes, characteristics, and behaviors are of interest or (b) as an assemblage of components whose interrelations are of interest.* For effective fusion system implementation, the fusion system designer needs to determine what would be the types of basic entities of interest from which the relationships of interest can be defined.

Sections 3.2 and 3.3 respectively present and discuss the recommended data fusion level definitions. Section 3.4 discusses the system integration of data fusion processes across the levels. Section 3.5 examines some prominent alternatives and extensions to the JDL model. Sections 3.6 and 3.7 extend the revised JDL data fusion levels to corresponding dual levels in resource management, which together are used to decompose DF&RM problems and processes for effective system design. The revised level 4 of data fusion retains the “process assessment” fusion functions in the original JDL fusion level 4 and includes process and sensor management as part of the corresponding dual resource management levels. It is proposed that these extended DF&RM levels will serve much the same purpose for resource management problems as the original JDL levels do for data fusion problems: facilitating understanding, design and integration.¹⁰

3.3.1   Level 0: Signal/Feature Assessment

The 1998 revision⁵ recommended defining a level 0 fusion to encompass various uses of multiple measurements in signal and feature processing. These include processes of feature extraction from imagery and analogous signal detection and parameter estimation in electro-magnetic, acoustic, or other data. In the most general sense, these problems concern the discovery of information (in Shannon’s sense) in some region of space and time.

Level 0 processes include inferences that do not require assumptions about the presence or characteristics of entities possessing such observable features. They concern the structure of measurement sets (their syntax), not their cause (i.e., their semantics). In cases where signal characteristics are conditioned on the presence and characteristics of one or more presumed entities, the latter are treated as a context, or situation, in which the signals or features are inferable, for example, through a likelihood function λ(Z|x).

These functions include DAI/FEO (data in/features out) and FEI/FEO (features in/features out) functions as defined in Dasarathy’s model,¹¹ as well as DEI/FEO (decisions in/features out) in our expanded version of that model.*¹²

Level 0 processes are often performed by individual sensors, or with the product of individual sensors. That certainly is the case with most tactical systems. However, when communications bandwidth and processing time allow, much can be gained by feature extraction or parameter measurement at the multisensor level. Image fusion characteristically involves extracting features across multiple images, often from multiple sources. Multisensor feature extraction is also practiced in diverse systems that locate targets using multistatic or other multisensor techniques.

3.3.2   Level 1: Entity Assessment

The notion of level 1 data fusion—called variously object assessment or entity assessment—was originally conceived as encompassing the most prominent and most highly-developed applications of data fusion: detection, identification, location, and tracking of individual physical objects (aircraft, ships, land vehicles, etc). Most techniques involve combining observations of a target of interest to estimate the states of interest. Thus, it was convenient to conflate process, target and product in distinguishing level 1 fusion. However, states of physical objects can be estimated using indirect, non-attributive information. Also, non-physical objects are amenable to attributive state estimation. Accordingly, we prefer to construe level 1 data fusion as the estimation of states of entities considered as individuals. JDL level 1, as in earlier versions, is concerned with the estimation of the identity, classification (in some given taxonomic scheme), other discrete attributes, kinematics, other continuous parameters, activities, and potential states of the basic entities of interest for the fusion problem being solved.

Individuation of entities implies an inclusion, or boundary, function such as is found in such paradigmatic entities as

• Biological organisms

• Purpose-made entities; for example, a hammer, an automobile, or a bird’s nest

• Other discriminated spatially contiguous uniformities that persist over some practical time interval, for example, a mountain, coral reef, cloud, raindrop, or swarm of gnats

• Societally discriminated entities, for example, a country, tract of land, family, or other social, legal or political entity.

A level 1 estimation process can be thought of as the application of one-place relations R⁽¹⁾(a) = x, where x may be a discrete or a continuous random variable, or a vector of such variables, so that R⁽¹⁾(a) may be the temperature, location, functional class, an attribute, or activity state of an entity a.

3.3.3   Level 2: Situation Assessment

Level 2 data fusion concerns inferences about situations. Devlin¹³ provides a useful definition of a situation as any structured part of reality. We may parse this in terms of a standard dictionary definition of a structure as a set of entities, their attributes, and relationships (Merriam-Webster Dictionary, 3rd edition). Once again, data fusion is the estimation of the state of some aspects of the world on the basis of multiple data.

Abstract situations can be represented as sets of relations; actual situations can be represented as sets of instantiated relations (or, relationships).* Thus, situation assessment involves inferencing from the estimated state(s) of one entity in a situation to another and from the estimated attributes and relationships of entities to situations. Attributes of individuals can be treated as one-place relationships.

Methods for representing relationships and for inferring entity states on the basis of relationships include graphical methods (e.g., Bayesian and other belief networks).

Situation assessment, then, involves the following functions:

1. Inferring relationships among (and within) entities

2. Recognizing/classifying situations basis estimates of constituents (entities, attributes, and relationships)

3. Using inferred relationships to infer entity attributes, to include attributes (a) of entities that are elements of a situation (e.g., refining the estimate of the existence and state variables of an entity on the basis of its inferred relationships) and (b) of the entities that are themselves situations (e.g., in determining a situation to be of a certain type)

These functions implicitly include inferring and exploiting relationships among situations, for example, predicting future situation basis current and historical situations.

As noted above, an entity—whether a signal, physical object, aggregate, or structure—can often be viewed either (a) as an individual whose attributes, characteristics, and behaviors are of interest or (b) as an assemblage of components whose interrelations are of interest. From the former point of view, discrete physical objects (the paradigm targets of level 1 data fusion) are components of a situation. From the latter point of view, the targets are themselves situations, that is, contexts for the analysis of components and their relationships.

A more detailed discussion on situation assessment and its relationship to entity and impact assessment is given in Chapter 18.

3.3.4   Level 3: Impact Assessment

It must be admitted that level 3 data fusion has been the most troublesome to define clearly and consistently with common usage in the field. The initial name of “threat assessment” is useful and relates to a specific, cohesive set of operational functions, but only within the domain of tactical military or security applications. The 1998 version attempted to broaden the concept to nonthreat domains; with a change of names to impact assessment. This was defined as the estimation and prediction of effects on situations of planned or estimated/predicted actions by the participants (e.g., assessing susceptibilities and vulnerabilities to estimated/predicted threat actions, given one’s own planned actions).

We may simplify this definition by taking impact assessment to be conditional situation estimation. Questions to be answered include those of the following form:

“If entities x_i ∈ X follow courses of action α_i, what will be the outcome?”

Many problems of interest involve a given agent x operating in a reactive environment (i.e., one that responds differentially to x’s actions). Furthermore, such reaction is often the result of one or more responsive agents, often assumed to be capable of intentional activity. As such, impact assessment problems can be amenable to game theoretic treatment, when the objective functions of the interacting agents can be specified.

In military threat assessment, we are usually concerned with the state of a set of such agents that constitute our forces as affected by intentional actions and reactions of hostile agents. In other words, we are concerned with inferring the concept:

“If we follow course of action α, what will be the outcome (and cost)?”

Impact assessment so construed can also include counterfactual event or situation prediction:

“If x had followed this course of action, what would have been the outcome?”

In general then, level 3 fusion involves combining multiple sources of information to estimate conditional or counterfactual outcome and cost. It is amenable to cost analysis, whether it is Bayesian-based or otherwise.

Because the utility of a fused state in supporting a user generally needs to be predicted based on the estimated current situation, known plans, and predicted reactions, level 3 fusion processing typically has different inputs and different outputs (viz. utility predictions) than the other fusion levels.

An example of a level 3 type of association problem is that of determining the expected consequence of system plans or courses of action, given the current estimated signal, entity, and situational state. An example of a level 3 type of estimation problem is the prediction of the impact of the estimated current situational state on the mission utility.

Level 3 issues and methods are discussed further in Chapter 18.

3.3.5   Level 4: Process Assessment

In early versions of the JDL model, level 4 was designated as process refinement. This was meant to encompass both assessment and adaptive control of the fusion process and—in many interpretations—in the data collection process as well. We have argued (a) that a more fundamental partitioning of functionality involves a distinction between data fusion functions (involving data association and state estimation) and resource management functions (involving planning and control), and (b) that functions that determine how the system resources should respond are clearly resource management functions.^5,8,12

There is, however, a need for a system-level category of level 4 fusion processes, one that is involved in associating system states and actions to desired system states and estimating or predicting the performance of the system. We therefore propose a data fusion level 4, concerning this process performance. Level 4 data fusion functions combine information to estimate a system’s measures of performance (MOPs) and MOEs based on a desired set of system states and responses. In the case of a data fusion system, MOEs can involve correlation of system state estimates to truth.

In level 4 data fusion, the primary association problem to be solved is that of determining which system outputs correspond to which of the desired goal states.

In the case of a data fusion system, the level 4 association problem may be that of determining a correspondence between system tracks and real-world entities.

In level 4 fusion, estimation may include such MOPs as sensor calibration and alignment errors, track purity and fragmentation, etc. MOEs could include utility metrics on target classification and location accuracies. Such estimates may be made on the basis of the system’s fusion multisensor product, or by an external test-bed system on the basis of additional fiducial information.

For an integrated DF&RM system, there can be several levels of planning and control (discussed in Sections 3.6 and 3.7).

RM association problems may involve determining a correspondence between resource actions and desired responses. A hierarchy of MOPs and MOEs can be estimated based on these associations.

3.5.1   Level 5: User Refinement

A process identified as “user refinement” has been proposed variously by Hall et al.¹⁴ and by Blasch and Plano.¹⁵ The latter give the following definition:

Level 5: Visualize. This process connects the user to the rest of the fusion process so that the user can visualize the fusion products and generate feedback or control to enhance/improve these products.¹⁵

Hall and McMullen¹⁶ define level 5 data fusion as cognitive refinement and human–computer interaction. Cognitive refinement includes the transformation from sensor data to a graphics display, but encompasses all methods of presenting information to human users. User refinement processes can involve adaptive determination of which users have needs for information and which have access to information. User refinement can also include knowledge management to adaptively determine the data that is most needed to be retrieved and displayed to support cognitive decision making and actions.

While there is no denial of the essential character of such functions in many systems involving both human and automated processes, it can be argued that these are not specifically data fusion functions. In fact they usually need to be managed by higher resource management processes such as objective management and resource relationship management. As a functional model, the JDL data fusion model is partitioned according to the functions performed, not by the performance and response management mechanisms, whether automatic, human, or other mechanisms.

The level 0–4 data fusion processes as we have defined them—involving the combination of information to estimate the state of various aspects of the world—can and have been performed by people, by automated processes, and by combinations thereof.*

We therefore argue against the inclusion of such a level 5 in the data fusion model. Like data archiving, communications, and resource management, such functions as data access, data retrieval, and data presentation are ancillary functions that often occur in systems that also do data fusion. As these descriptions indicate, some of these functions are resource management functions that are formal duals of certain data fusion functions.

3.5.2   Dasarathy’s Input/Output Model

The partitioning basis process products (as distinct, e.g., from process types) allows a direct mapping from Dasarathy’s input/output (I/O) model,¹¹ as extended in Chapter 2 of the previous edition of this handbook.¹² Accordingly, the latter can be seen as a refinement of the JDL model.

Dasarathy¹¹ categorized data fusion functions in terms of the types of data/information that are processed and the types that result from the process. Table 3.2 illustrates the types of I/O considered. Processes corresponding to the cells in the highlighted matrix region are described by Dasarathy, using the abbreviations DAI/DAO, DAI/FEO, FEI/FEO, FEI/DEO, and DEI/DEO.^† A striking benefit of this categorization is the natural manner in which technique types can be mapped into it.

In the previous edition of this handbook,¹² this categorization was extended by

• Adding labels to these cells, relating I/O types to process types

• Filling in the unoccupied cells in the original matrix

Note that Dasarathy’s original categories represent constructive, or data-driven, processes, in which organized information is extracted from less organized data. Additional processes—FEI/DAO, DEI/DAO, and DEI/FEO—can be defined that are analytic, or model-driven, such that organized information (a model) is analyzed to estimate lower-level data (features or measurements) as they relate to the model. Examples include predetection tracking (an FEI/DAO process), model-based feature extraction (DEI/FEO), and model-based classification (DEI/DAO). The remaining cell in Table 3.2—DAI/DAO—has not been addressed in a significant way (to the authors’ knowledge), but could involve the direct estimation of entity states without the intermediate step of feature extraction.

In Ref. 12, we extended Dasarathy’s categorization to encompass level 2, 3, and 4 processes, as shown in Table 3.3. Here rows and columns have been added to correspond to the object types listed in Figure 3.5.

With this augmentation, Dasarathy’s categorization is something of a refinement of the JDL levels. Not only can each of the levels (0–4) be subdivided basis of input data types, but level 0 can also be subdivided into detection processes and feature-extraction processes.

Of course, much of Table 3.3 remains virgin territory; researchers have only seriously explored its northwest quadrant, with tentative forays southeast. Most likely, little utility will be found in either the Northeast or the Southwest. However, there may be gold buried somewhere in those remote stretches.

3.5.3   Other Data Fusion Models

Bedworth and O’Brien²¹ provide a synthesis of data fusion models. Their resultant Omnibus model maps the early JDL and Dasarathy’s models into an OODA (observe, orient, decide, act) model.

It should be noted that the omnibus model—like the OODA loop—is a process model, whereas the JDL model and Dasarathy’s I/O model are functional models.* Among the most influential variants and alternatives to the JDL model are those presented by Endsley,²² Salerno,²³ and Lambert.²⁴ These are discussed in some detail in Chapter 18.