26
Survey of COTS Software for Multisensor Data Fusion*
Sonya A. H. McMullen, Richard R. Sherry, and Shikha Miglani
CONTENTS
26.2 Taxonomy for Multisensor Data Fusion
26.3 Survey of COTS Software and Software Environments
26.3.1 Special Purpose COTS Software
26.3.2 General Purpose Data Fusion Software
26.1 Introduction
In 1993, D. L. Hall and R. J. Linn1 conducted a survey of commercial off the shelf (COTS) software to support development of data fusion systems for applications such as automatic target recognition, identification-friend-foe-neutral processing, and battlefield surveillance. In the survey, they described a number of emerging packages containing basic algorithms for signal processing, image processing, statistical estimation, and prototyping of expert systems. Since the publication of that paper, extensive progress has been made in data fusion for both Department of Defense (DoD) applications as well as non-DoD applications. The basic algorithms and techniques for data fusion have evolved2 and engineering standards are beginning to emerge for system design3 and requirement derivation and analysis.4 Numerous data fusion systems have been developed for DoD applications5,6 and systems are beginning to be developed for non-DoD applications such as the condition-based monitoring of complex mechanical systems.7† In addition, non-DoD applications such as data mining, pattern recognition, and knowledge discovery have spurred the development of commercial software tools8 and general packages such as MATLAB9 and Mathematica.10 Because of these rapid developments along with the emergence of COTS packages for prototyping data fusion applications, it was deemed worthwhile to update the survey conducted by Hall and Linn. S. A. Hall and R. R. Sherry updated the original survey by Hall and Linn and presented that survey at the 2002 MSS National Symposium on Sensor and Data Fusion. This chapter provides an update of that paper and presents a summary of the updated survey, identifies some applicable COTS software, and provides a survey of surveys related to data fusion systems and software.
26.2 Taxonomy for Multisensor Data Fusion
Numerous experts have described the concept of the multisensor data fusion process in-depth. Hall, Hall and Oue.11 have previously provided the following description. Traditionally, data fusion systems have been developed to ingest information from multiple sensors and sources to provide information for automated situation assessment, or to assist a human in development of situation assessments.12, 13 and 14 Extensive research has focused on the transformation from sensor data to target locations, target identification, and (in limited cases) contextual interpretation of the target information. Numerous systems have been developed to perform this processing to achieve or support situation assessment.5,6 The data fusion process is often represented conceptually by the Joint Directors of Laboratories (JDL) data fusion process model illustrated in Figure 26.1. The JDL augmented model shown in Figure 26.1 shows the level 5 process recommended by Hall et al.15 (and introduced at the same conference by Blasch) to explicitly account for functions associated with human–computer interactions (HCI).
26.3 Survey of COTS Software and Software Environments
Three significant advances have shaped the evolution of COTS data fusion applications software since the publication of Hall and Linn’s survey in 1993. First is the rapid development of specialized software algorithm packages used within technical computation environments such as MATLAB,16 Mathematica,17 and Mathcad.18 These environments provide unified platforms for mathematical computation, analysis, visualization, algorithm development, and application. The second development is the evolution of general-purpose data fusion software packages with adaptive capabilities to a diverse set of data fusion applications. The third development (described in Section 26.5) involves the development of tools such as the Adobe Flex 2.0 cross-platform development framework for prototyping web-based applications.
FIGURE 26.1
Revised Joint Directors of Laboratories (JDL) data fusion process model. (From Hall, M.J., Hall, S.A., and Tate, T., Proceedings of the 2000 MSS National Symposium on Sensor and Data Fusion, October 2000.)
26.3.1 Special Purpose COTS Software
Table 26.1 shows a summary of some of the specialized software packages. These are all compatible with at least one of the three mathematical environments previously described. There are literally hundreds of software packages available. Hence, Table 26.1 is by no means a complete listing of the available commercial products. However, it provides a basis for a more in-depth search for such products. Use of these types of packages in conjunction with MATLAB, Mathematica, or Mathcad algorithms, visualization features, and user-interface capabilities can significantly reduce development time for prototype application simulations. Projects initially prototyped and implemented in MATLAB or Mathematica environment can be imported to C and C++ to generate parallel C code from high-level MATLAB programs, or MathCode C++19 for generation of C++ code from Mathematica code.
An example of experience utilizing MATLAB as a prototype environment is described by West et al.20 They indicate that MATLAB-based tracking is feasible and relatively efficient. However, they noted that when the MATLAB compiler translated the MATLAB (*.m) files into C coding, there were many calls to MATLAB.dII functions and the resulting runtimes were similar to standard (*.m) file utilization. West et al.20 observed a dramatic runtime improvement (speed-up) by recoding functions in the C language.
26.3.2 General Purpose Data Fusion Software
The second class of COTS software involves packages specifically designed to perform data fusion. A summary of these is provided in Table 26.2 and a brief summary of example packages is provided below.
The Lockheed Martin data fusion workstation21 is directed at integrated multisensor, multitarget classification based on fusing 14 types of sensor inputs including passive-acoustic, ESM/ELINT, magnetic field, electric field, inverse synthetic aperture radar, and geolocation. The data fusion workstation consists of three principal components: (1) the embedded data fusion system, (2) a graphical user interface, and (3) a data fusion input simulator. The fusion engine is fuzzy logic based on Dempster–Shafer reasoning. The software is objected-oriented C++ code, which runs on Silicon graphics and Sub SPARCstation platforms. The Lockheed Martin data fusion system software is utilized as an embedded function in the Rotorcraft Pilot’s Associate.
Hall and Kasmala22 at the Pennsylvania State University Applied Research Laboratory have developed a visual programming tool kit for level 1 data fusion. The tool kit is modeled after the tool Khoros developed for collaborative development of image-processing algorithms. The top-level interface allows a user to define a level 1 processing flow by a point and click manipulation of graphic symbols representing types of fusion algorithms. The user can define multiple processing streams to analyze single sensor data or multiple processing streams to fuse multisensor data. A library of routines is available and can be selected for processes such as signal conditioning, feature extraction, and pattern recognition or decision-level fusion. The tool kit was originally developed for applications involving condition monitoring of mechanical systems. However, the algorithms are general purpose and can be used on any signal or vector data.
TABLE 26.1
Examples of COTS Software for Use in Mathematical Software Environments
TABLE 26.2
General Purpose Data Fusion Software
Brooks and Iyengar23 provide a suite of C language routines as companion software to their book on multisensor fusion. These routines collectively have been labeled the sensor fusion toolkit and contain implementations of machine learning, learned meta-heuristics, neural networks, simulated annealing, genetic algorithms, tabu search, a Kalman filter, and their own distributed dynamic sensor algorithm.
The Department of Energy (DoE) has funded research directed at developing models for groundwater flow and contaminant transport. This work has resulted in development of the hydrogeologic data fusion computer model.24 This model can be used to combine various types of geophysical, geologic, and hydrologic data from diverse sensor types to estimate geologic and hydrogeologic properties such as ground water flow. A commercial version of this software is available under the name Hydro-FACT from Fusion and Control Technology, Inc.
KnowledgeBoard25 is a software framework for information collection, data fusion, and information visualization. KnowledgeBoard is directed at the fusion of distributed high-level multisource data including XML-based documents, video, web-based content, relational and object databases, flat files, etc. Heterogeneous data sources are displayed within a contextual framework to support user-level fusion of the information and to support improved knowledge generation.
PCI Geomatics has developed the ImageLock Data Fusion software26 that is designed to perform semiautomatic geometric correction, fuse the corrected multisensor imagery, and perform noise and artifact removal. The ImageLock Data Fusion software is a component in PCI’s EASI/PACE remote sensing image processing software package.
eCognition27 by Definiens Imaging is designed to extract features from high-resolution satellite and aerial imagery. eCognition provides for multisource fusion from a large variety of formats, sensors, and platforms at any spatial and spectral resolution.
Boeing has also developed a data fusion workstation for fusing data from nondestructive examination (NDE) testing and from process monitoring.28 NDE and process monitoring is applicable throughout the life cycle (manufacturing, repair, in-service inspection) of numerous components.28 The benefit of the data fusion system is that it aids in cross correlation of data, which results in the reduction in ambiguity from NDE inspections.28 Overall, this tool has served to reduce technical labor requirements as well as evaluation variability particularly for aerospace components.28
Defense R&D Canada—Valcartier has developed a test bed for evaluating multisensor data fusion systems against system requirements.29 This test bed is called CASE ATTI (Concept Analysis and Simulation Environment for Automatic Target Tracking and Identification). One of the main objectives of the development of CASE ATTI was to allow the testing and comparison of different sensor fusion algorithms and techniques without extensive recoding. CASE ATTI allows use of either externally generated sensor data or data generated by a high fidelity simulator that emulates the behavior of targets, sensor systems, and the environment. CASE ATTI runs on UNIX and Windows platforms and can be used with multiple computers across a local area network.
Objectivity/DB31 by objectivity is an object-oriented database management system dealing with large volumes of high dimensional interrelated data. Objectivity/DB is a database engineer with a database library linked to application programs rather than a separate data server process. The developers of Objectivity/DB claim that the package integrates with applications software through standard language interfaces such as C#/.NET, C++, java, SQL, XML, and others. They also cite applications to areas such as bioinformatics, manufacturing, and military situation assessment.
Open Ratings’s DNBi Supply Management Solution32 incorporates machine learning and data mining technologies that identify pattern from large sets of noisy data sets and maintains a supplier database of more than 100 million records. In addition, there is a collaborative workspace for collecting and organizing supplier information when an issue arises—including historical supplier performance intelligence, user notes, internet links, supplier-completed assessments, external files, and contingency plans.
Black Coral LIVE33 and Black Coral MOBILE33 are integrated products that facilitate situational awareness and interoperability allowing multiple organizations in emergency response and military environments to communicate, collaborate, and coordinate for distributed personnel and tactical teams. The system utilizes the ESRI ArcGIS software for displaying and manipulating geographical information (and associated overlay data), and provides functions to support distributed users collaborating via annotation of maps and collective development of an evolving situation display.
26.4 A Survey of Surveys
To assist in identifying additional tools, Table 26.3 identifies sources (papers, reports, and websites) that provide additional surveys of relevant software. In addition to these resources, annual software surveys are available via the ACM computing surveys. The website for the ACM special interest group SIGGRAPH (http://www.siggraph.org/) provides general information on graphics and visualization software, whereas the commercial company Zoom Info surveys and identifies companies who offer specialized services or products. Hence, Zoom Info can be used to find data fusion products and related companies.
26.5 Discussion
This brief survey shows the acceleration of the development and application of data fusion software methods across a growing number of application domains. Since Hall and Linn’s original survey, several new trends can be observed:
There is an explosion of commercial software tools for a wide variety of component techniques such as signal processing, image processing, pattern recognition, data mining, and automated reasoning.
New mathematical environments such as MATLAB, Mathcad, and Mathematica provide extensive tool kits (including methods for signal processing, neural nets, image processing, etc.) that are useful for rapid prototyping and evaluation of data fusion methods.
A limited number of more general purpose tool kits are available. These are quasi-COTS tools (they are available for use or purchase but do not have the general utilization or support of software such as MATLAB or Mathematica).
TABLE 26.3
Software Surveys
The software development environment (including new languages such as Visual Basic, Visual C++, JAVA Script, and XML) provides an improved basis for development of object-oriented, interoperable software. In this breed of new software is Adobe Flex 2.035 cross-platform development framework. The purpose of Flex 2.0 is to create software tools that quickly build and deploy rich internet applications (RIA) within the enterprise or across the web. As a result, Flex 2.0 creates intelligent desktop applications that augment overall user experience and assists users in analyzing data and business processes for productive corporate decisions.
Flex applications are developed with MXML and object-oriented programming language Actionscript using the Flex builder integrated development environment or a standard text editor. The Flex compiler then compiles the source code into bytecode as binary SWF file. This file is executed by flash player at runtime on all major browsers and operating systems. Developers can use Flex’s prebuilt visual, service, and behavior components from the class library of components and containers, and customize them by extending these components or their base classes to achieve the desired look and feel. Flex clients can be used in conjunction with any server environment such as JSP, ASP, ASP.NET, ColdFusion, and PHP. The data request over the Internet can be made via standard HTTP calls or web-services. In addition, Java server based Flex data services enable data transfer, data synchronization and conflict management, and real-time data messaging. Furthermore, to develop data dashboard for interactive analysis, Flex charting provides a library of extensible charting component.35
Recently, Flex has been used in analytical applications as a user-interface technology to analyze trends and monitor applications for better business decisions.34
In general, these developments provide an increasingly efficient environment to develop and use data fusion algorithms. However, there is still a need for the widespread availability of a standard package for multisensor data fusion.
References
1. D. L. Hall and R. J. Linn, “Survey of commercial software for multisensor data fusion,” Proceedings of the SPIE Conference on Sensor Fusion and Aerospace Applications, Orlando, FL, 1993.
2. D. L. Hall and J. Llinas, eds. Handbook of Multisensor Data Fusion, CRC Press Inc., Boca Raton, FL, 2001.
3. C. L. Bowman and A. N. Steinberg, “A systems engineering approach for implementing data fusion systems,” chapter 16 in Handbook of Multisensor Data Fusion, D. L. Hall and J. Llinas, eds., CRC Press Inc., Boca Raton, FL, 2001.
4. E. Waltz and D. L. Hall, “Requirements derivation for data fusion systems,” chapter 15 in Handbook of Multisensor Data Fusion, D. L. Hall and J. Llinas, eds., CRC Press Inc., Boca Raton, FL, 2001.
5. M. L. Nichols, “A survey of multisensor data fusion systems,” chapter 22 in Handbook of Multisensor Data Fusion, D. L. Hall and J. Llinas, eds., CRC Press Inc., Boca Raton, FL, 2001.
6. D. L. Hall, R. J. Linn, and J. Llinas, “A survey of data fusion systems,” Proceedings of the SPIE Conference on Data Structures and Target Classification, pp. 13–36, 1991.
7. C. S. Byington and A. K. Garga, “Data fusion for developing predictive diagnostics for electromechanical systems,” chapter 23 in Multisensor Data Fusion, D. L. Hall and J. Llinas, eds., CRC Press Inc., Boca Raton, FL, 2001.
8. M. Goebel and L. Gruenwald, “A survey of data mining and knowledge discovery software tools,” SIGKDD Explorations, Vol. I, issue 1, pp. 20–33, 1999.
10. http://www.wolfram.com/products/mathematica/index.html.
11. M. J. Hall, S. A. Hall, and C. Oue, “A word (may) be worth a thousand pictures: on the use of language representation to improve situation assessment,” Proceedings of the 2001 MSS National Symposium on Sensor and Data Fusion, Vol. II, June 2001.
12. E. Waltz and J. Llinas, Multisensor Data Fusion, Artech House, Inc., Norwood, MA, 1990.
13. D. L. Hall, Mathematical Techniques for Multisensor Data Fusion, Artech House, Inc., Norwood, MA, 1992.
14. D. L. Hall and J. Llinas, “An introduction to multisensor data fusion,” Proceedings of the IEEE, Vol. 85, No. 1, January 1997.
15. M. J. Hall, S. A. Hall, and T. Tate, “Removing the HCI bottleneck: how the human computer interface (HCI) affects the performance of data fusion systems,” Proceedings of the 2000 MSS National Symposium on Sensor and Data Fusion, Vol. II, San Antonio, TX, pp. 89–104, October 2000.
16. Developers of MATLAB and Simulink for Technical Computing. The MathWorks. Retrieved August 13, 2007, from the World Wide Web: http://www.mathworks.com/.
17. Mathematica. Wolfram Research. Retrieved August 13, 2007, from the World Wide Web: http://www.wolfram.com/products/mathematica/index.html.
18. Mathcad. MathSoft. Retrieved August 13, 2007, from the World Wide Web: http://www.ptc.com/products/mathcad/mathcad14/promo.htm.
19. MathCode C++. MathCore AB. Retrieved August 13, 2007, from the World Wide Web: http://www.mathcore.com/products/mathcode/mathcodec++.php.
20. P. West, D. Blair, N. Jablonski, B. Swanson, and T. Armentrout, “Development and real-time testing of target tracking algorithms with AN/SPY-1 radar using Matlab,” presented at the Fourth ONR/GTRI Workshop on Target Tracking and Sensor Fusion, Monterey, CA, 2001.
21. A. Pawlowski and P. Gerken, “Simulator, workstation and data fusion components for onboard/off-board multi-targeted multisensor data fusion,” IEEE/AIAA Digital Avionics Systems, 1998.
22. D. L. Hall and G. A. Kasmala, “A visual programming tool kit for multisensor data fusion,” Proceedings of the SPIE AeroSense 1996 Symposium, Orlando, FL, Vol. 2764, pp. 181–187, April 1996.
23. R. R. Brooks and S. S. Iyengar, Multi-Sensor Fusion: Fundamentals and Applications with Software, Prentice Hall, Upper Saddle River, NJ, 1998.
24. Hydrogeologic Data Fusion. Innovative Technology, 1999. Retrieved August 13, 2007, from the World Wide Web: http://apps.em.doe.gov/ost/pubs/itsrs/itsr2944.pdf.
25. KnowledgeBoard. SAIC. Retrieved August 13, 2007, from the World Wide Web: http://www.saic.com/products/software/knowledgeboard/.
26. ImageLock Data Fusion Package. PCI Geomatics. Retrieved August 13, 2007, from the World Wide Web: http://www.pcigeomatics.com/cgi-bin/pcihlp/IMAGELOCK.
27. eCognition—object-oriented image analysis. Definiens Imaging. Retrieved August 13, 2007, from the World Wide Web: http://www.definiens.com/article.php?id=6.
28. R. Bossi and J. Nelson. NDE Data Fusion. Boeing. Retrieved August 13, 2007, from the World Wide Web: http://www.ndt.net/abstract/asntf97/053.htm.
29. CASE ATTI: A Test Bed for Sensor Data Fusion. Defence R&D Canada—Valcartier. Retrieved August 13, 2007, from the World Wide Web: www.valcartier.drdc-rddc.gc.ca/poolpdf/e/137_e.pdf.
30. http://www.sonardyne.co.uk/Products/PositioningNavigation/systems/pharos.html.
31. http://www.objectivity.com/data-fusion.shtml.
32. http://www.openratings.com/solutions/solutions_overview.html.
33. http://www.blackcoral.net.
34. A. Lorenz and G. Schoppe, Developing SAP Applications with Adobe Flex, Galileo Press, Bonn, Germany, 2007.
35. Adobe Flex 2.0. Adobe. Retrieved August 13, 2007, from the World Wide Web: http://www.adobe.com/products/flex/whitepapers/pdfs/flex2wp_technicaloverview.pdf.
* This chapter is an update of original paper: Sherry, R.R. and Hall, S.A. (2002). “A survey of COTS software for multi-sensor data fusion; what’s new since Hall and Linn?” Proceedings of the MSS National Symposium on Sensor and Data Fusion, San Diego, CA.
† Reference Chapters 21, 22, 27, and 28 of this Handbook for updated information on the above references.