19.2.1   User Roles

IFSs are initiated, designed, and fielded based on a need: such as data compression,⁵ spatial coverage, and task assistance. Inherently behind every fusion system design is a user in question. Users naturally understand the benefits of fusion such as multisensor communication with their eyes and ears,⁶ multisensor transportation control of vehicles that augment movement,⁷ and multisensor economic management of daily living through computers, transactions, and supply of goods and services.⁸ Although opportunities for successful fusion abound, some of the assumptions are not well understood. For efficient communication, a point-to-point connection is needed. For transportation control, societal protocols are understood such as roadways, vehicle maintenance, and approved operators. For economic management, organization and business operations are contractually agreed on. On the basis of these examples, IFS designs also require individual, mechanical, and economic support dependent on operational, prototype, and business users.

Fusion designs can be separated into two modes: micro and macro levels. The micro level involves using a set of defined sensors within a constrained setting. Such examples include a physician requiring a fused result of a PET–MRI image, a driver operating a car, or a storeowner looking at the financial report. In each of these cases, the settings are well defined, the role of the fusion systems is constrained, and the decisions are immediate. The more difficult case is the macro level that deals with military decision support systems,⁹ political representatives managing a society, or a corporate board managing a trans-continental business. The complexity of the macro level generally involves many users and many sensors operating over a variety of data and information sources with broad availability. An example of an aggregation of micro- and macro-level fusion concepts can be seen by the multilevel display in Figure 19.1. As shown, individual users sought to spatially minimize the information by placing the displays together. Although this is not the ideal “information-fusion” display, it does demonstrate that “business users” afforded the resources, “operational users” trusted the system, and that “prototype users” were able to provide the technology. To manage large, complex systems and hope that IF can be a useful tool to augment the macro-level systems, the appropriate constraints, assumptions, and defined user roles must be subscribed up front. One goal is to reduce the complexity of fusion at the macro level by creating a common operational picture¹⁰ across sensors, targets, and environments.

The user (level 5) can take on many roles: such as a prioritizer, sensor, operator, controller, and planner, which can design, plan, deploy, and execute an IF design. Although there are a host of roles that could be determined, one taxonomy is presented in Table 19.1. This table is based on trying to field a typical IFS such as a target tracker and an identifier. The first role is that of a prioritizer, which determines what objects are of interest, where to look, and what supporting database information is needed. Most fusion algorithms require some initializing information such as a priori probabilities, objective functions, processing weights, and performance parameters. The second role is that of a sensor, which includes reporting of contextual and message reports, making decisions on data queries, and drafting information requests. The third role is that of an operator, which includes data verification such as highlighting images, filtering text reports, and facilitating communication and reporting. Many times the operator is trained for a specific task and has a defined role (which distinguishes the role vs. that of a sensor). A controller is a “business” operator who routes sensors and platforms, determines the mission of the day, and utilizes the IF output for larger goals. Finally, the planner is not looking at today’s operations, but tomorrow’s (e.g., how can the data gathering be better facilitated in the future to meet undefined threats and situations forthcoming).

The most fundamentally overlooked opportunities for successful fusion design is in the conceptual phase. Engineers take advantage of the inherent benefits of data fusion within the initial design process: developing, for instance, Bayesian rules that assume that exemplar data, a prior probabilities, and sensors will be available. Without adequate “user refinement” of the problem through constraints, operational planning, and user-augmented algorithms; the engineering design has little chance of being deployed. By working with a team of users, as in Table 19.1, we can work within operational limits in terms of available databases, observation exploitation capabilities, task allocations, platforms, and system-level opportunities. The interplay between the engineers and the users can build trust and efficacy in the system design.

In the design phase, users can aid in characterizing fusion techniques that are realizable. Two examples come to mind—one is object detection and the other is sensor planning. Because no automatic target recognition (ATR) fusion algorithm can reduce the false alarm rate (FAR) to zero, although simultaneously providing near-perfect probability of detection, individual users must somehow determine acceptable thresholds. Because operators are burdened with a stream of image data, the fusion algorithm can reprioritize images important to the user for evaluation (much like a prescreener). Another example includes sensor planning. IF requires the right data to integrate, and thus, having operators and controllers present in the design phase can aid in developing concept-of-operations constraints based on estimated data collection possibilities.

For macro-level systems, deployment, execution, and reporting within IF designs are currently notional since complexity limits successes. One example is in training,⁵ where prototype and operational users work with the system designers to better the interface. Lessons from systems engineering can aid in IF designs. Systems engineering focuses on large, complex, and multiple levels of uncertainty systems. Many concepts such as optimization, human factors, and logistics focus on system’s maintenance and sustainability.

Figure 19.2 highlights the concept that systems engineering costs escalate if users at all levels are not employed up front in the IF design. At the information problem definition stage, the research team has to determine what types of information is observable and available, and which information can be correlated with sensor design, data collection, and algorithm data exploitation. Thus, information needs and user tasks drive the concept design. During the development phase, which includes elaboration and construction, design studies, trade-offs, and prototypes need to be tested and refined (which includes the expected user roles). Llinas¹¹* indicates that many of the design developments are prototypes; however, one of the main concerns in the development phase is to ensure that the user be a part of the fusion process to set priorities and constraints. One emerging issue is that IF designs are dependent on software developers. To control costs of deployment and maintenance, software developers need to work with users to be able to include operator controls such as data filtering, algorithm reinitialization, and process initiation.

19.2.2   Prioritization of Needs

To reduce information to a dimensionally minimal set requires the user to develop a hierarchy of needs to allow one to perform active reasoning over collected and synthesized data. Some needs that can be included in such a hierarchy are:

Things. Objects, number and types; threat, whether harmful, passive, or helpful; location, close or far; basic primitives, features, edges, and peaks; existence known or unknown, that is, new objects; dynamics, moving or stationary.

Processes. Measurement system reliability (uncertainty); ability to collect more data; delays in the measurement process; number of users and interactions.

The user must prioritize information needs for designers. The information priority is related to the information desired. The user must have the ability to choose or select the objects of interest and the processes from which the raw data is converted into the fused information. One of the issues in the processing of fused information is related to the ability to understand the information origin or pedigree. To determine an area of interest, for instance, a user must seek answers to basic questions:

Who—is controlling the objects of interest (i.e., threat information)

What—are the objects and activities of interest (i.e., diagnosis, products, and targets)

Where—are the objects located (i.e., regions of interest)

When—should we measure the information (i.e., time of day)

How—should we collect the information (i.e., sensor measurements)

Which—sensors to use and the algorithms to exploit the information

To reason actively over a complex prioritized collection, a usable set of information must be available along with collection uncertainties on which the user can act.¹² Ontology, the seeking of knowledge,¹³ indicates that goals must be defined, such as reducing uncertainty. The user has to deal with many aspects of uncertainty, such as sensor bias, communication delays, and noise. Heisenberg uncertainty principle exemplifies the challenge of observation and accuracy.¹⁴ Uncertainty is a measure of doubt,¹⁵ which can prevent a user from making a decision. The user can supplement a machine-fusion system to account for deficiencies and uncertainties. Jousselme et al. presented a useful representation of the different types of uncertainty.¹⁶ A state of ignorance can result in an error. An error can occur between expected and measured results caused by distortion of the data (mis-registered) or incomplete data (not enough coverage). With insufficient sensing capabilities, lack of context for situational assessment, and undefined priorities, IFSs will be plagued by uncertainty. Some of the categories of uncertainty are

Vague—not having a precise meaning or clear sensing¹⁷

Probability—belief in element of the truth

Ambiguity—cannot decide between two results

Priority of information needs and uncertainty reduction is based on context, which necessitates the user to augment IF designs with changing priorities, informal data gathering, and real-time mission planning.

19.2.3   Contextual Information

Context information, as supplied by a user, can augment object, situational, and threat understanding. There are certain object features the user desires to observe. By selecting objects of interest, target models can be called to assess feature matches.

Modeled target features can be used to predict target profiles and movement characteristics of the targets for any set of aspect angles. Object context includes a pose match between kinematic and feature relationships. Targeting includes selecting priority weights for targets of interest, and information modifications based on location (i.e., avenues of approach and mobility corridors). By properly exploiting the sensor, algorithm, and target, the user can select the ideal perspective from which to identify a target. Examples of contextual information aids include tracking,¹⁸ robotics,¹⁹ and medicine.

SA usually means that the user’s team has some a priori world information. Aspects of context in situational assessment could include the sociopolitical environment of interest, a priori historical data, topology, and mission goals. Contextual information such as road networks, geographical information, and weather is useful.²⁰ In a military context, targets can be linked by observing selected areas, segmented by boundaries as shown in Figure 19.3. If there is a natural boundary of control, then the targets are more likely to be affiliated with the group defending that area. Types of situation contexts include

• Sociopolitical information

• Battlefield topography and weather

• Adjacent hostile and friendly force structures

• Tactical or strategic operational doctrine

• Tactical or strategic force deployments

• Supporting intelligence data

From Figure 19.3, we see that if targets are grouped near other targets of the same allegiance, we could expect the classification of a target to the same as that of other spatially adjacent targets. Likewise, we could subscribe to the fact that trucks are on roads and only tanks can be off-road. Thus, information context plays a role in classifying the objects, developing situation understanding, and assessing threat priorities. The method by which a user integrates contextual information to manage the fusion process is cognitive.

19.2.4   Cognitive Fusion

Formally, IF integrates the machine and cognitive fusion functions:

IF is a formal mathematics, techniques, and processes to combine, integrate, and align data originating from different sources through association and correlation for decision making. The construct entails obtaining greater quality, decreased dimensionality, and reduced uncertainty for the user’s application of interest.

We can subdivide the field of fusion technology into a number of categories:

Data fusion. Combining and organizing numerical entities for analysis

Sensor fusion. Combining devices responding to a stimulus

IF. Combining data to create knowledge

Display fusion. Presenting data simultaneously to support human operations

Cognitive fusion. Integrating information in user’s mental model.

Display fusion is an example of integrating cognitive and machine fusion. Users are required to display machine-correlated data from individual systems. Figure 19.1 shows a team of people coordinating decisions derived from displays. Since we desire to utilize IF where appropriate, we assume that combining different sets of data from multiple sources would improve performance gain (i.e., increased quality and reduced uncertainty). Uncertainty about world information can be reduced through systematic and appropriate data combinations, but how to display uncertainty is not well understood. Because the world is complex (based on lots of information, users, and machines) we desire timely responses and extended dimensionality over different operating conditions.²¹ The computer or fusion system is adept at numerical computations whereas the human is adept at reasoning over displayed information, so what is needed is an effective fusion of physical, process, and cognitive models. The strategy of “how to fuse” requires the fusion system to display a pragmatic set of query options that effectively and efficiently decompose user information needs into signal, feature, and decision fusion tasks.¹² The decomposition requires that we prioritize selected data combinations to ensure a knowledge gain. The decomposition of tasks requires fusion models and architectures developed from user information needs. Information needs are based on user requirements that include object information and expected fusion performance. To improve fusion performance, management techniques are necessary to direct machine or database resources based on observations, assessments, and mission constraints, which can be accepted, adapted, or changed by the user. Today, with a variety of users and many ways to fuse information, standards are evolving to provide effective and efficient fusion display techniques that can address team requirements for distributed cognitive fusion.²²

Users are important in the design, deployment, and execution of IFSs. They play active roles at various levels of command as individuals or as teams. UR can include a wide range of roles such as (1) prioritizer with database support, (2) sensor as a person facilitating communication and reporting, (3) an operator who sits at an human–computer interface (HCI) terminal monitoring fusion outputs, (4) controller who routes sensors and platforms, and (5) planner who looks at the system-level operations. Recognizing the importance of the roles of the user, the DIFG sought to update the JDL model.

19.3.1   Sensor Management

The DFIG model updates include acknowledging the resource management of (1) sensors, (2) platforms that may carry one or multiple sensors, and (3) user control; each with differing user needs based on levels of command. The user not only plans the platform location but can also reason over situations and threats to alter sensing needs. For a time-line assessment, (1) the user plans days in advance for sensing needs, (2) platforms are routed hours in advance, and (3) sensors are slewed in seconds, to capture the relevant information.

Level 4, process refinement, includes sensor management and control of sensors and information. To utilize sensors effectively, the IFS must explore service priorities, search methods (breadth or depth), and determine the scheduling and monitoring of tasks. Scheduling is a control function that relies on the aggregated state-position, state-identity, and uncertainty management for knowledge reasoning. Typical methods used are (1) objective cost function for optimization, (2) dynamic programming (such as neural-network [NN] methods and reinforcement-learning based on a goal), (3) greedy information-theoretic approaches for optimal task assignment,³⁰ or (4) Bayesian networks that aggregate probabilities to accommodate rule constraints. Whichever method is used, the main idea is to reduce the uncertainty.

Level 4 control should entail hierarchical control functions as well. The system is designed to accommodate five classes of sensing (static, event, trigger, trend, and drift), providing a rich source of information about the user’s information needs. The most primitive sensor class, state, allows the current value of an object to be obtained, whereas the event class can provide notification that a change of value has occurred. A higher-level sensor class, trigger, enables the measurement of the current value of a sensor when another sensor changes its value, that is, a temporal combination of state and event sensing. The design supports the logging of sensor data over time, and the production of simple statistics, known as trend sensing. A fifth sensor class, drift, is used for detecting when trend statistics cross a threshold. This can be used to indicate that a situation trends toward an undesirable condition, perhaps a threat. There is also a need for checking the integrity, reliability, and quality of the sensing system.

Issues must also involve absolute and relative assessment as well as local and global referencing. Many IF strategies are embedded within systems; however the user-IFS must be rigorously evaluated by a standardized approach that assesses a wide range of locations, changing targets, differing sensor modalities, and IF algorithms.

19.3.2   User Interaction with Design

The IF community has seen numerous papers that assumed a data-level, bottom-up, level 1 object refinement design, including target tracking and ID. Some efforts explored level 1 aggregation for SA.^31,32 However, if the fusion design approach was top-down, then the community would benefit to start the IF designs by asking the customer what they need. The customer is the IFS user, whether the business person, analyst, or commander. If we ask the customers what they want, they would most likely request something that affords reasoning and DM, given their perceptual limitations.³³ Different situations will drive different needs; however, the situation is never fully known a priori. There are general constructs that form an initial set of situation hypotheses, but in reality not all operating conditions will be known. To minimize the difficulty of trying to assess all known/unknown a priori situational conditions, the user helps to develop these a priori conditions that help to define the more relevant models. In the end, IFS designs would be based on the user, not strictly the machine or the situation.

Management includes business, social, and economic effects on the fusion design. Business includes managing people, processes, and products. Likewise IF is concerned with managing people, sensors, and data. The ability to design an IFS with regard to physical, social, economic, military, and political environments would entail user reasoning about the data to infer information. User control should be prioritized as user, mission, and sensor management. For example, if sensors are on platforms, then the highest ranking official coordinating a mission determines who is in charge of the assets (not under automatic control). Once treaties, air space, insurance policies, and other documentation are in place, only then would we want to turn over control to an automatic controller (e.g., sensor).

UR, level 5 fusion could form the basis to (1) advocate standard fusion metrics, (2) recommend a rigorous fusion evaluation and training methodology, (3) incorporate dynamic decision planning, and (4) detail knowledge presentation for fusion interface standards. These issues are shown in Figure 19.6.

19.4.1   User Action

User actions have many meanings that could be conveyed by SA and assessment needs. One of the key issues of SA (defined in Section 19.5) is that the IF must map to the user’s perceptual needs: spatial awareness,³⁴ neurophysiological,³⁵ perceptual,³⁶ and psychological,³⁷ and that those needs combine perceptually and cognitively.^6,38,39 If the display/delivery of information is not consistent with the user expectations, all is lost. The machine cannot reason, as Brooks stated in “Elephant’s don’t play chess.”⁴⁰ A machine cannot deal with unexpected situations.⁴¹ For this reason, there are numerous implications for (1) incorporating the user in the design process, (2) gathering user needs, and (3) providing the user with available control actions.

Users are not Gaussian processors, yet they do hypothesis confirmation. In hypothesis reasoning, we can never prove the null condition, but we can disprove the alternative hypothesis. Level 5 is intended to address the cognitive SA that includes knowledge representation and reasoning methods. The user defines a fusion system, for without a user, there is no need to provide fusion of multisensory data. The user has a defined role with objectives and missions. Typically, the IF community has designed systems that did not require user participation. However, through years of continuing debate, we find that the user plays a key role in the design process by

Level 0: determining what and how much data value to collect

Level 1: determining the target priority and where to look

Level 2: understanding scenario context and user role

Level 3: defining what is a threat and adversarial intent

Level 4: determining which sensors to deploy and activate by assessing the utility of information

Level 5: designing user interface controls (Figure 19.1)

Kokar and coworkers⁴² stress ontological and linguistic⁴³ questions concerning user interaction with a fusion system such as semantics, syntactics, efficacy, and spatiotemporal queries. Developing a framework for UR requires semantics or interface actions that allow the system to coordinate with the user. Such an example is a query system in which the user seeks questions and the system translates these requests into actionable items. An operational system must satisfy the users’ functional needs and extend their sensory capabilities. A user fuses data and information over time and space and acts through a perceived world reference (mental) model—whether in the head or on graphical displays, tools, and techniques.⁴⁴ The IFS is just an extension of the user’s sensing capabilities. Thus, effective and efficient interactions between the fusion system and the user should be greater than the separate parts.

The reason why users will do better than machines is that they are able to reason about the situation, they can assess likely routes a target can take, and they bring in contextual information to reason over the uncertainty.¹⁶ For example, as target numbers increase and processing times decrease, a user will get overloaded. To assist the user, we can let routine calculations and data processing be performed by the computer and relegate higher-level reasoning to the user.

Process refinement of the machine controls data flow. UR guides information collection, region of coverage, situation and context assessment as well as process refinement of sensor selections. Although each higher level builds on information from lower fusion levels, the refinement of fused information is a function of the user. Two important issues are control of the information and the planning of actions. There are many social issues related to centralized and distributed control such as team management and individual survivability. For example, the display shown in Figure 19.1 might be globally distributed, but centralized for a single commander for local operations. Thus, the information displayed on an HCI should not only reflect the information received from lower levels, but also afford analysis and distribution of commander actions, directives, and missions. Once actions are taken, new observable data should be presented as operational feedback. Likewise, the analysis should be based on local and global operational changes and the confidence of new information. Finally, execution should include updates including orders, plans, and actions for people carrying out mission directives. Thus, process refinement is not the fusing of signature, image, track, and situational data, but that of decision information fusion (DEC-IF) for functionality.

DEC-IF is a refined assessment of fused observational information, globally and locally. DEC-IF is a result on which a user can plan and act. Additionally, data should be gathered by humans to aid in deciding what information to collect, analyze, and distribute to others. One way to facilitate the receipt, analysis, and distribution of actions is that of the OODA loop. The OODA loop requires an interactive display for functional cognitive DM. In this case, the display of information should orient a user to assess a situation based on the observed information (i.e., SA). These actions should be assessed in terms of the user’s ability to deploy assets against the resources, within constraints, and provide opportunities that capitalize on the environment.

Extended information gathering can be labeled as action information fusion (ACT-IF) because an assessment of possible/plausible actions can be considered. The goal of any fusion system is to provide the user with a functionally actionable set of refined information. Taken together, the user-fusion system is actually a functional sensor, whereas the traditional fusion model just represents sensor observations. The UR process not only determines who wants the data and whether they can refine the information, but also how they process the information for knowledge. Bottom-up processing of information can take the form of physically sensed quantities that support higher-level functions. To include machine automation as well as IF, it is important to afford an interface that allows the user to interact with the system (i.e., sensors). Specifically, the top-down user can query necessary information to plan and focus on specific situations while the automated system can work in the background to update on the changing environment. Thus, the functional fusion system is concerned with processing the entire situation while a user can typically focus on key activities. The computer must process an exhaustive set of combinations to arrive at a decision, but the human is adept at cognitive reasoning and is sensitive to the changes in situational context.

The desired result of data processing is to identity entities or locally critical information spatially and temporally. The difficulty in developing a complete understanding is the sheer amount of data available. For example, a person monitoring many regions of interest (ROIs) waits for some event or threat to come into the sensor domain. Under continuously collecting sensors, most processed data are entirely useless. Useful data occurs when a sensed event interrupts the monitoring system and initiates a threat update. Decision makers only need a subset of the entire data to perform successful SA and impact assessment to respond to a threat. Specifically, the user needs a time-event update on an object of interest. Thus, data reduction must be performed. A fusion strategy can reduce this data over time as well as space, by creating entities. Entity data can be fused into a single common operating picture for enhanced cognitive fusion for action; however, user control options need to be defined.

19.4.2   User Control

Table 19.2 shows different user management modes that range from passive to active control.⁴⁵ In the table, we see that the best alternative for fusion systems design is “management by interaction.” Management by interaction can include both compensatory and noncompensatory strategies. The compensatory strategies include linear additive (weighting inputs), additive difference (compares alternative strategies), or ideal point (compare alternative against desired strategies) methods. The noncompensatory alternative selective strategies include dominance (strategy alternatives equal in all dimensions except one), conjunctive (passes threshold on all dimensions), lexicographic (best alternative strategy in one dimension, if equal strategies are based on second dimension), disjunctive (passes threshold on one dimension), single feature difference (best alternative strategy over alternative dimensions of greatest difference), elimination by aspects (choose one strategy based on probabilistic elimination of less important dimensions), and compatibility test (compare alternative strategies based on weighted threshold dimensions) strategies. In the strategies presented, both the user and the DFS require rules for DM. Because the computer can only do computations, the user must select the best set of rules to employ.

One method to assess the performance of a user or fusion process for target identification (ID) (for instance) is through the use of receiver operator curves (ROC). Management by interaction includes supervisory functions such as monitoring, data reduction and mining, and control. An ROC plots the hit rate (HR) versus the FAR. The ROC could represent any modality such as hearing, imaging, or fusion outputs. As an example, we can assess the performance of a user-fusion system by comparing the individual ROC to the performance of the integrated user-fusion system. The user may act in three ways (1) management-by-ignorance—turn off fusion display, do manual manipulation (cognitive-based); (2) management-by-acceptance—agree with the computer (i.e., automatic); or (3) management-by-interaction—adapt with output, reasoning (i.e., associative). What is needed is the reduction of data that the user needs to process.

For instance, consider that the human might desire acceptance—such as the case when the HR is high and independent of the FAR; or the user might desire ignorance—when we want FAR to be low, but track length high. However an appropriate combination of the two approaches would lead to a case of interaction—adjust decisions (hit [H]/false alarm [FA]) based on fusion ROC performance for unknown environments, weak fusion results, or high levels of uncertainty. From Figure 19.8 we see there is the possibility for fusion gain (moving the curve to the upper left) is associated with being able to process more information. One obvious benefit of the user to the fusion system is that the user can quickly rule out obvious false alarms (thereby moving the curve left). Likewise, the fusion system can process more data, thereby cueing more detections (moving the curve up). Operating on the upper left of the 2D curve would allow the user to increase confidence in DM over an acceptable FAR.

We have outlined user control actions; now let us return to estimation.

19.4.3   User Interaction with Estimation

It is important to understand which activities are best suited for user engagement with a DFS. We need to understand the impact of the user on fusion operations, communication delays, as well as target density. For example, UR includes monitoring to ensure that the computer is operating properly. Many UR functions are similar to business management where the user is not just operating independently with a single machine but operating with many people to gather facts and estimate states. Typically, one key metric is reaction time⁴⁴ that determines which user strategy can be carried out efficiently.

UR can impact near- and far-term analyses over which the human must make decisions. For the case of a short-term analysis, the human monitors (active vs. passive) the system by being engaged with the system or just watching the displays. For the far-term analysis, the human acts as a planner⁴⁶ (reactive vs. proactive) in which the user estimates long-term effects and consequences of information. The information presented includes not only the time span associated with collecting the data, but also the time delay needed to display the data (latent information), as well as the time between the presentation of information on the screen to the user’s perception and reaction time. User reaction time includes physical responses, working associations, and cognitive reasoning. User focus and attention for action engagement can be partitioned into immediate and future information needs. Engagement actions can be defined as

Predictive actions. Projective, analytical approach to declare instructions in advance

Reactive actions. Immediate, unthinking response to a stimulus

Proactive actions. Active approach requiring an evaluation of the impact of the data on future situations

Passive actions. Waiting, nonthought condition allowing current state to proceed without interaction

The user’s role is determined by both the type of DFS action and user goals. In Table 19.3, we see that the action the user chooses determines the time horizon for analysis and a mapping of the user actions.

Other user data metrics needed in the analysis and presentation of data and fusion information include integrity—formal correctness (i.e., uncertainty analysis); consistency—enhancing redundancy, reducing ambiguity, or double counting (i.e., measurement origination); and traceability—backward inference chain (i.e., inferences assessed as data is translated into information).

Integrity, consistency, and traceability map into trust, confidence, and assurance that the fused data display is designed to meet user objectives. To be able to relate the user needs for the appropriate design action, we use the automatic, associative, and cognitive levels of IF. To perform the actions of planning, prediction, and proactive control for fusion assessment, the user must be able to trust the current information and be assured that the fusion system is adaptable and flexible to respond to different requests and queries. Typical HCI results discussed in the literature offer a number of opportunities for the fusion community: user models, metrics, evaluation, decision analysis, work domain assessment, and display design, as detailed in Section 19.5.

19.5.1   User Models in Situational Awareness

Situation assessment is an important concept of how people become aware of things happening in their environment. SA can be defined as keeping track of prioritized, significant events, and the conditions in one’s environment. Level 2 SA is the estimation and prediction of relations among entities, to include force structure and relations, communications, etc., which require adequate user inputs to define these entities. The human-in-the-loop (HIL) of a semiautomated system must be given adequate SA. According to Endsley, SA is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.^{48, 49 and 50} This now-classic model, shown in Figure 19.9, translates into three levels:

• Level 1 SA: Perception of environment elements

• Level 2 SA: Comprehension of the current situation

• Level 3 SA: Projection of future states

Operators of dynamic systems use their SA to determine actions. To optimize DM, the SA provided by an IFS should be as precise as possible with respect to the objects in the environment (level 1 SA). An SA approach should present a fused representation of the data (level 2 SA) and provide support for the operator’s projection needs (level 3 SA) to facilitate operator’s goals. From the SA model presented in Figure 19.9, workload⁵¹ is a key component of the model that affects not only SA, but also user decision and reaction time.

To understand how the human uses the situation context to refine the SA, we use the recognition primed decision-making (RPD) model,⁵² shown in Figure 19.10. The RPD model develops user DM capability based on the current situation and past experience. The RPD model shows the goals of the user and the cues that are important. From the IFS, both the user and the IFS algorithm can cue each other to determine data needs. The user can cue the IFS by either selecting the data of interest or choosing the sensor collection information.

As another example, the fusion SA model components^53,54 developed by Kettani and Roy show the various information needs to provide the user with an appropriate SA. To develop the SA model further, we note that the user must be primed for situations to be able to operate faster and more effectively and work against a known set of metrics for SA.

19.5.2   Metrics

Standardized performance and quality of service (QoS) metrics are sine qua non for evaluating every stage of data processing and subsystem hand off of data and state information. Without metrics, a proper scientific evaluation cannot be made that takes into account the diversity of proposed systems, and their complexity and operability. A chief concern is adequate SA. The operator is not always automatically guaranteed SA when new fusion systems are deployed. Even though promises are made for these systems to demonstrate increases in capacity, data acuity (sharpened resolution, separating a higher degree of information from noise backgrounds), and timeliness among the QoS metrics, the human cognitive process can be a bottleneck in the overall operation. This occurs most frequently at the perception level of SA, with level 1 accounting for 77% of SA-related errors,⁵¹ and indirectly affecting comprehension and projection DM acuity in levels 2 and 3.

Dynamic DM requires: (1) SA/image analysts (IA), (2) dynamic responsiveness to changing conditions, and (3) continual evaluation to meet throughput and latency requirements. These factors usually impact the IFS directly, require an interactive display to allow the user to make decisions, and must include metrics for replanning and sensor management.⁵⁵ To afford interactions between future IF designs and users’ information needs, metrics are required. The metrics chosen include timeliness, accuracy, throughput, confidence, and cost. These metrics are similar to the standard QoS metrics in communication theory and human factors literature, as shown in Table 19.4.

To plan a mission, an evaluation of fusion requires an off-line and on-line assessment of individual sensors and integrated sensor exploitation (i.e., performance models). To achieve desired system management goals, the user must be able to develop a set of metrics based on system optimization requirements and performance evaluation criteria. Both of which define the quality to provide effective mission management and to ensure an accurate and effective capability to put the right platforms and sensors on station at the correct time, prepare operators for the results, and determine pending actions based on the estimated data.

19.5.3   User Evaluation

IF does not depend only on individuals making decisions, or single subsystems generating analyses. Rather, team communication and team-based decisions have always been integral to complex operations, whether civilian or military, tactical or strategic, municipal, federal, or intergovernmental. Team communication takes many forms across operational environments and changes dynamically with new situations. Communication and DM can be joint, allocated, shared, etc., and this requires that the maintenance of any team-generated SA be rigorously evaluated.

As computer processing loads increase, both correct and incorrect ways are available to design IFSs. The incorrect way is to fully automate the system, providing no role for the user. The correct way is to build the role of the operator into the design at inception and test the effectiveness of the operator to meet system requirements. Fusionists are aware of many approaches, such as level 1 target ID and tracking methods that only support the user, but do not replace the user. In the case of target ID, a user typically does better with a small number of images. However, as the number of collected images and throughput increases, a user does not have the time to process every image. One key issue for fusion is the timeliness of information.

Because every stage of IF DM adds to overall delays—in receiving sensor information, in presenting fused information to the user, and in the information user’s processing capacity—overall system operation must be evaluated in its entirety. The bottom-level component in the human-system operation involves data from the sensors. Traditionally, the term data fusion has referred to bottom-level, data-driven fusion processing, whereas the term information fusion refers to the conversion of already-fused data into meaningful and preferably relevant information useful to higher cognitive systems, which may or may not include humans.

Cognitive processes—human or machine equivalent “smart” algorithms—require nontrivial processing time to reach a decision, at both individual and team levels. This applies, analogously, to individual components and intercomponent/subsystem levels of machine-side DM. DM durations that run shorter than the interarrival time of sensor data lead to starvation within the DM process; DM durations that run longer than the interval between new data arrivals lead to dropped data. Both cases create situations prone to error, whether the DM operator is human or machine. The former case has consequences of starving the human’s cognitive flow, causing wandering attention spans, and loss of key pieces of information held in short-term memory, which could have been relevant to later-arriving data. In the latter case, data arrives while the information is being processed, or the operator is occupied, so this data often passes out of the system without being used by the operator. This occurs when systems are not adequately designed to store excess inputs to an already overburdened system. These excess inputs are dropped if there is no buffer in which to queue the data, or if the buffer is so small as to overwrite data. The operator and team must have the means to access quality information and sufficiently process the data from different perspectives within a timely manner. In addition to the metrics that establish core quality (reliability/integrity) for information, there are issues surrounding information security. Precision performance measures may be reflected by content validity (whether the measure adequately measures the various facets of the target construct), construct validity (how well the measured construct relates to other well-established measures of the same underlying construct), or criterion-related validity (whether measures correlate with performance on some task). The validity of these IF performance measures needs to be verified in an operational setting where operators are making cognitive decisions.

19.5.4   Cognitive Processing in Dynamic Decision Making

There are many ways to address the users’ roles (reactive, proactive, and preventive) in system design. Likewise, it is also important to develop metrics for the interactions to enhance the users’ abilities in a defined role or task. One way to understand how a user operates is to address the automatic actions performed by the user (which includes eye–arm movements). Additionally, the user associates past experiences to current expectations, such as identifying targets in an image. Intelligent actions include the cognitive functions of reasoning and understanding. Rasmussen³⁷ developed these ideas by mapping behaviors into decision actions including skills, rules, and knowledge through a means-end decomposition.

When tasked with an SA analysis, a user can respond broadly by one of three manners: reactive, proactive, or preventive, as shown in Figure 19.11. In the example that follows, we use a target tracking and ID exemplar to detail the user cognitive actions. In a reactive mode, the user makes a rapid detection to minimize damage or repeat offense. An IFS would gather information from a sensor grid detection of in situ threats and is ready to act. In this model, the system interprets and alerts users to immediate threats. The individual user selects the immediate appropriate response (in seconds) with aid of sensor warnings of nonlethal or lethal threats.

In the proactive mode, the user utilizes sensor data to anticipate, detect, and capture needed information before an event. In this case, a sensor grid provides surveillance based on prior intelligence and predicted target movements. A multi-INT sensor system could detect and interpret anomalous behavior and alert an operator to anticipated threats in minutes. Additionally, the directed sensor mesh tracks individuals back to dwellings and meeting places, where troops respond quickly and capture the insurgents, weapons, or useful intelligence.

The mode that captures the entire force over a period of time (i.e., an hour) is the preventive mode. To prevent potential threats or actions, we would (1) increase insurgent risk (i.e., arrested after being detected), (2) increase effort (i.e., make it difficult to act), and (3) lower payoff of action (i.e., reduce the explosive damage). The preventive mode includes an intelligence database to track events before they reach deployment.

The transition and integration of decision modes to perceptual views is shown in Figure 19.12. The differing displays for the foot soldier or command center would correlate with the DM mode of interest. For the reactive mode, the user would want an actual location of the immediate threats on a physical map. For the anticipated threats, the user would want the predicted locations of the adversary and the range of possible actions. Finally, for the potential threats, the user could utilize behavior analysis displays that piece together aggregated information of group affiliations, equipment stores, and previous events to predict actions over time. These domain representations were postulated by Waltz³⁹ as cognitive, symbolic, and physical views that capture differing perceptual needs for intelligent DM.

Intelligent DM employs many knowledge-based information fusion (KBIF) strategies such as neural networks, fuzzy logic, Bayesian networks, evolutionary computing, and expert systems.⁵⁶ Each KBIF strategy has different processing durations. Furthermore, each strategy differs in the extent to which it is constrained by the facility and its use within the user-fusion system. OODA loop helps to model a DM user’s planned, estimated, or predicted actions. Assessing susceptibilities and vulnerabilities to detect/estimate/predict threat actions, in the context of planned actions, requires a concurrent timeliness assessment. Such assessment is required for adequate DM, yet is not easily attained.

This is similarly posed in the Endsley model of SA: the “projection” level 3 of SA maps to this assessment processing activity. DM is most successful in the presence of high levels of projection SA, such as high accuracy of vulnerability or adversarial action assessments. DM is enhanced by correctly anticipating effects and state changes that will result from potential actions. The nested nature of effect-upon-effect creates difficulty in making estimations within an OODA cycle. For instance, effects of own-force intended COA are affected by the DM cycle time of the threat instigator and the ability to detect and recognize them. Three COA processes⁵⁷ to reduce the IFS search space are as follows: (1) manage similar COA, (2) plan related COA, and (3) select novel COA, shown in Figure 19.13.

Given that proactive and preventive actions have a similar foundational set of requirements for communicating actions, employing new sensor technologies, and managing interactions in complex environments, any performance evaluation must start with the construct of generally applicable metrics. Adding to the general framework, we must also create an evaluation analysis that measures responsiveness to unanticipated states, since these are often the first signals of relevant SA threats. A comparative performance analysis for future applications requires intelligent reasoning, adequate IF knowledge representation, and process control within realistic environments. The human factors literature appropriately discusses the effects of realistic environments as a basis for cognitive work analysis (CWA).

19.5.5   Cognitive Work Analysis/Task Analysis

The key to supporting user reasoning is in designing a display that supports the users task and work roles. A task analysis (TA) is the modeling of user actions in the context of achieving goals. In general, actions are analyzed to move from a current state to a goal state. Different approaches based on actions include⁵⁸

• Sequential TA. It includes organizing activities and describing them in terms of the perceptual, cognitive, and manual user behavior to show the man-machine activities.

• Timeline TA. It is an approach that assesses temporal demands of the tasks or sequences of actions and compared time availability for task execution.

• Hierarchical TA. It represents the relationships between which tasks and subtasks need to be done to meet operating goals.

• Cognitive Task Analysis (CTA). It describes the cognitive skills and abilities needed to perform a task proficiently, rather than the specific physical activities carried out in performing the task. CTA is used to analyze and understand task performance in complex real-world situations, especially those involving change, uncertainty, and time pressure.

Level 5 issues require an understanding of the workload, time, and information availability. Workload and time can be addressed through a TA; however, user performance is a function of the task within the mission. A CWA includes (1) defining what task the user is doing, (2) structuring the order of operations, (3) ensuring that the interface supports SA, (4) defining the decomposition of required tasks, and (5) characterizing what the human actually perceives.⁵⁹ The CWA looks at the tasks the human is performing and the understanding of how the tasks correlate with the information needs for intelligent actions. The five stages of a CWA are (1) work domain, (2) control task, (3) strategy, (4) social-organizational, and (5) worker competency analysis. Vicente⁵⁹ shows that a CWA requires an understanding of the environment and the user capabilities. An evaluation CWA requires starting from the work domain and progressing through the fusion interface to the cognitive domain. Starting from the work domain, the user defines control tasks and strategies (i.e., utilizing the fusion information) to conduct the action. The user strategies, in the context of the situation, require an understanding of the social and cognitive capabilities of the user. Social issues would include no-fly zones for target ID and tracking tasks, whereas cognitive abilities include the reaction time of the user to a new situation. Elements of the CWA process include reading the fusion display, interpreting errors, and assessing sensory information and mental workloads. We are interested in the typical work-analysis concepts of fusion/sensor operators in the detection of events and state changes, and pragmatic ways to display the information.

19.5.6   Display/Interface Design

A display interface is a key to allowing the user to have control over data collection and fusion processing.^60,61 Without designing a display that matches the cognitive perception, it is difficult for the user to reason over a fused result. Although many papers address interface issues (i.e., multimodal interfaces), it is of concern to the fusion community to address cognitive issues such as ontological relations and associations.^62,63 Fusion models are based on processing information for DM. A user-fusion model would highlight the fact that fusion systems are designed for human consumption. In effect, the top-down approach would explore an information need—pull through queries to the IFS (cognitive fusion), whereas a bottom-up approach would push data combinations to the user from the fusion system (display fusion). The main issues for user-fusion interaction include ontology development for initiating queries, constructing metrics for conveying successful fusion opportunities, and developing techniques to reduce uncertainty¹ and dimensionality. In Section 19.6, we highlight the main mathematical techniques for fusion based on the user-fusion model for DM.

An inherent difficulty resides in the fusion system that processes only two forms of data. If one sensor perceives the correct entity and the other does not, there is a conflict. Sensor conflicts increase the cognitive workload and delay action. However, if time, space, and spectral events from different sensors are fused, conflicts can be resolved with an emergent process to allow for better decisions over time. The strategy is to employ multiple sensors from multiple perspectives with UR. Additionally, the HCI can be used to give the user a global and local SA/IA to guide attention, reduce workload, increase trust, and afford action.

Usability evaluation is a user-centered, user-driven method that defines the system as the connection between the application software and hardware. It focuses on whether the system delivers the right information in the appropriate way for users to complete their tasks. Once user requirements have been identified, good interface designs transform these requirements into display elements, and organize them in a way that is compatible with how users perceive and utilize information, and how they are used to support work context and to use conditions.³³ The following items are useful principles for interface design:

• Consistency. Interfaces should be consistent with experiences, conform to work conventions, and facilitate reasoning to minimize user errors, information retrieval, and action execution.

• Visually pleasing composition. Interface organization includes balance, symmetry, regularity, predictability, economy, unity, proportion, simplicity, and groupings.

• Grouping. Gestalt principles provide six general ways to group screen elements spatially: principles of proximity, similarity, common region, connectedness, continuity, and closure.

• Amount of information. Too much information could result in confusion. Using Miller’s principle that people cannot retain more than 7±2 items in short-term memory, chunking data using this heuristic could provide the ability to add more relevant information on the screen as needed.

• Meaningful ordering. The ordering of elements and their organization should have meaning with respect to the task and information processing activities.

• Distinctiveness. Objects should be distinguishable from other objects and the background.

• Focus and emphasis. Salience of objects should reflect the relative importance of focus.

Typically, nine usability areas are considered when evaluating interface designs with users:

• Terminology. Labels, acronyms, and terms used

• Workflow. Sequence of tasks in using the application

• Navigation. Methods used to find application parts

• Symbols. Icons used to convey information and status

• Access. Availability and ease of access of information

• Content. Style and quality of information available.

• Format. Style of information conveyed to the user

• Functionality. Application capabilities

• Organization. Layout of the application screens

The usability criteria typically used for evaluation of these usability areas are noted below:³³

• Visual clarity. This displayed information should be clear, well-organized, unambiguous, and easy-to-read to enable users to find required information, draw the user’s attention to important information, and allow the user to see where information should be entered quickly and easily.

• Consistency. This dimension conveys that the way the system looks and works should be consistent at all times. Consistency reinforces user expectations by maintaining predictability across the interface.

• Compatibility. This dimension corresponds to whether the interface conforms to existing user conventions and expectations. If the interface is familiar to users, it will be easier for them to navigate, understand, and interpret what they are looking at and what the system is doing.

• Informative feedback. Users should be given clear informative feedback on where they are, and what actions were taken, whether successful, and what should be taken.

• Explicitness. The way the system works and is structured should be clear to the user.

  TABLE 19.5
  Interface Design Activities

  	Work Domain Activities
  Info requirements	Cognitive work analysis, SA analysis, contextual inquiry—task analysis, scenario-based design, participatory design
  Interface design	User interface design principles, participatory design
  Evaluation	Situation awareness analysis, usability evaluation—scenario-based design, participatory design, usability evaluation

• Appropriate functionality. The system should meet the user requirements and needs when carrying out tasks.

• Flexibility and control. The interface should be sufficiently flexible in structure—the way information is presented and in terms of what the user can do, to suit the needs and requirements of all users, and to allow them to feel in control.

• Error prevention and correction. A system should be designed to minimize the possibility of user error, with built-in functions to detect when these errors occur. Users should be able to check their inputs and to correct potential error before inputs are processed.

• User guidance, usability, and support. Informative, easy-to-use, and relevant guidance should be provided to help the users understand and use the system.

Work domain issues should be considered in the environment in which processes will take place, such as the natural setting of a command post. Ecological interface design is the methodology concerned with evaluating user designs as related to stress, vigilance, and the physical conditions such as user controls (Table 19.5).