RSS feed LinkedIn Watch us on Youtube

Activity title

High-level Fusion of Hard and Soft Information for Intelligence

Activity Reference

IST-165 (AI2S)


Information Systems Technology

Security Classification




Activity type


Start date


End date



Context Information, Hard Fusion, Highlevel, Information Fusion, Intelligence, Lowlevel, Metadata, Sensemaking, Sensor Data Fusion, Soft Fusion


With non-traditional, asynchronous warfare on the increase, often in the form of widespread global networks of organized terror and insurgent-based warfare, there is a need to gather, sift, analyse and fuse information which is collected from a wide variety of diverse sources that may be geographically quite distant from one another and which may be collected over days, weeks or even months. These sources provide data and information in many varied formats such video, audio, and written and spoken text, and in such prohibitive volume that hand processing of this tsunami of information is impossible. Computerized (algorithmic) processing is required to deal with the massive inflow of information, at least at the first pass. As of now, this automatic processing has often confined to a single type of source (e.g., array of sensors) or various sensors with a single specific purpose (e.g., coastal protection). The goal of seamlessly combining information from diverse sources including HUMINT, OSINT, and so on exists only in a few narrowly specialized and limited areas. In other words, there is no unified, holistic solution to this problem. In the predecessors to this proposed activity (IST-106 and IST-132) the focus was on fusion of “hard” (device-derived) and “soft” (natural language-based) information within the overall ISTAR chain. The research moved along a spectrum from a focus on a very limited time-scale (several minutes) and geographic (single location) sensor-based scenario in IST-106 to a somewhat longer-scale (hours) and wider geographic (100 km²) scenario in which HUMINT and OSINT filled in gaps in sensor coverage. The next logical step in this research is to move to the next higher level of complexity by extending the lessons learned from the previous Task Groups into an intelligence scenario wherein the geographical area and time frame are considerably greater than previously looked at, integrating not only sensor and human data but also background knowledge and contextual sources to achieve sense making in complex situations. Through the work of these previous Task Groups, hurdles and problems were identified, the role of context information was examined and an approach for standardization of information in context using a Controlled Language was devised for the lower level fusion processes. In this proposed task group, the focus moves up another notch in complexity to focus on larger intelligence issues including topics such as sense-making of patterns of behaviour, global interactions and information quality, integrating sources of data, information and contextual knowledge.


The objective of this task group is to expand upon the results of the preceding two RTGs by expanding the research to cover the fusion of higher-level device-derived information with structured and unstructured human generated information (soft data) including as binding elements complex relevant information context and heterogeneous data structures from various civilian and military organisations and sources. Additionally, there will be a focus on the use of lower-level fusion products as basic “data elements” in higher-level fusion algorithms, in particular, information flow from lower- to higher-level algorithms. The results of the previous task groups will serve as a springboard for these investigations. The main objectives will be to investigate the information needs for higher-level fusion for intelligence purposes, especially the need for contextual information both physical (e.g., geographical, structural) and identify what expansions or extensions are needed to the results which were previously formulated in the earlier work. As with the preceding groups, it is planned to further demonstrate the usefulness of a Controlled Language, in this case, Battle Management Language (a product of the STO Modelling and Simulation Group) as a mechanism to facilitate integrated exploitation of data and information from different type of sources and on different levels of fusion, and to improve threat detection and modelling, including dealing with uncertainty and detection of anomalous behaviour as a result of the holistic data and information processing.


a. Examine and identify information and modelling needs for (longer time span, wider scope, higher level) intelligence purposes as contrasted to (shorter time span, narrower scope, lower level) situation awareness requirements. b. Further explore the potential of BML to be a common framework to uniformly represent results of lower-level fusion products in a way that can be used for higher-level information fusion algorithms. c. Develop structured representations of lower-level situational elements for enhanced integration in higher level information processing using situation context information d. Further develop computationally exploitable information with robust quality measures exploring various mathematical formalisms: e.g. computational linguistics, grey information theory, Bayesian, DS, DSmT, fuzzy logic), in order to support the quality and reliability of the algorithms. e. Formalization and representation of contextual information sources at physical and logical levels

Contact Panel Office